Imaging device with 4-lens time-of-flight pixels and interleaved readout thereof

Ranging devices, systems, and methods are provided. In embodiments, a device includes a casing with four openings and an array with depth pixels. The depth pixels are arranged in four quadrants, so that pixels in each of the quadrants receive light through one of the four openings. The depth pixels may generate samples in response to the received light. For a certain frame, a controller reads out samples from each of the quadrants before completing reading out the samples of any one of the quadrants. In some embodiments, reading out is performed by using interleaved rolling shutter for the rows.

BACKGROUND

Ranging devices can be used for ranging, i.e. determining the distance of the device from a person, object or scene. The distance is also known as range. In some instances imaging devices, which include cameras, are also capable of ranging. This is why descriptions of a certain types of ranging devices sometimes resemble descriptions of cameras.

Modern imaging devices use pixels to capture images. The pixels divide an input image in elements, and capture values for the elements of the image. These values for the image are captured by various techniques, such as numbers of electrons per pixel after a brief exposure time. The output image is typically constructed from the captured values, whether in color or in black and white.

BRIEF SUMMARY

The present description gives instances of devices, systems, and methods, the use of which may help overcome problems and limitations of the prior art.

In embodiments, a device includes a casing with four openings. The device also includes an array with depth pixels. The depth pixels can be arranged in four quadrants, so that pixels in each of the quadrants receive light through one of the four openings. The depth pixels may generate samples in response to the received light. For a certain frame, a controller reads out samples from each of the quadrants before completing reading out the samples of any one of the quadrants. In some embodiments, reading out is performed by using interleaved rolling shutter for the rows.

An advantage over the prior art is that artifacts from motion, which could cause channel misregistration, can be reduced or even eliminated. Moreover, a buffer used in the prior art for half a frame might not be necessary, therefore reducing image processing memory size and thus device cost. Additionally, image processing lag can be reduced.

These and other features and advantages of this description will become more readily apparent from the Detailed Description, which proceeds with reference to the associated drawings in which:

DETAILED DESCRIPTION

As has been mentioned, the present description is about devices, systems, and methods. Embodiments are now described in more detail.

FIG. 1is a block diagram of a device100, which can be implemented according to many different embodiments. Device100could have many embodiments. For example, device100may be a ranging device, configured to determine a distance of object101from device100. That distance is also called the range.

For another example, device100may be an imaging device, configured to capture an image of an object101. In some embodiments, device100is both an imaging device and a ranging device.

Device100can have a casing102that can also be called housing. An opening104is provided in casing102, which will be used for the image. A lens106may be provided optionally at opening104. In embodiments, four additional openings195are provided in casing102. These openings195are openings for obtaining depth data, so as to ultimately construct a depth image. In embodiments, openings195are provided in a 2×2 matrix, with two openings195in a top row, and another 2 in the bottom. They may also have lenses, etc.

Device100also has a pixel array110. Pixel array110is configured to receive light through opening104, and capture it. Accordingly, pixel array110, opening104and lens106define a field of view112. Of course, field of view112and object101are in three dimensions, whileFIG. 1shows them in two dimensions.

Casing102can be aligned, so that object101, or a person or a scene as desired, will be brought within field of view112, so that it presents an input image. A light source114, such as an LED, may be further provided on casing102, so as to assist in the imaging and/or ranging operation of device100. Light source114can be configured to transmit light116towards field of view112, so as to illuminate persons and objects within it. Light116can be reflected by object101and then be received via opening104, in addition to ambient light received by reflection from object101. Accordingly, light source114can assist in imaging by illuminating object101better. Or, light source114can assist in ranging by modulating transmitted light116in a way that is already known to device100. Light source114may be operating in response to a drive signal, and thus it may modulate transmitted light116similarly to how the drive signal is modulated.

As mentioned above, pixel array110can capture light received via opening104. More particularly, in many embodiments, pixel array110has a two-dimensional array of pixels, which are also sometimes known as sensors. The pixels can be arranged in rows and columns, although other arrangements are also possible. When the pixels are exposed to an input image, i.e. receive light from the image, they generate signals in response to the light they receive. Typically these signals are in the form of electric charges. By their magnitude, these signals encode individual sensed values for the light, which is why they are also called samples. Taken together, the samples may render an output image that is a version of the sensed input image. This is also why the entire pixel array110is sometimes called an image sensor.

The pixels mentioned above can also be called image pixels, since they help recreate the input image, for the imaging function. In embodiments, depth pixels may be included for a ranging function, which are additional to Image pixels. These depth pixels may be part of image array110, or of a separate array, and can operate the same way as the image pixels, except that they ultimately help create a depth image.

Device100may additionally include a processor120. Processor120may perform image processing functions upon receiving the signals or samples from pixel array110. Processor120may also perform additional functions, for example adjust imaging parameters of the samples, of the exposure, etc.

Device100may further include a controller130, which can be configured to control the operation of pixel array110and other components of device100. In some embodiments, controller130receives inputs from processor120. Processor120and/or controller130can be implemented with one or more Central Processing Units (CPUs), digital signal processors, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), and so on. Controller130may optionally be formed integrally with pixel array110, processor120, and possibly also with other components of device100, perhaps in a single integrated circuit. Controller130may control and operate pixel array110, by transmitting control signals from output ports, and so on, as will be understood by those skilled in the art.

Device100may further include a memory140. The samples can be stored in memory140, preferably as digital values representing the signals generated by the pixels. The samples may be further processed before and/or after being stored in memory140. In embodiments, memory140is configured to store final samples computed by processor120as the output image.

Device100may moreover include a user interface150, which can be configured to receive inputs from the user. The inputs can be for controlling the operation of device100, such as for adjusting imaging parameters and/or image processing parameters. In some embodiments, interface150is implemented by one or more standalone components, such as actuators, buttons, circular wheels and the like on casing102.

Optionally, device100also includes a display160, which can be considered to be part of user interface150. Display160can include a screen. When provided, display160can display the samples as the rendered image. A user can view this image, and use it to better align casing102, so that object101will be placed within field of view112. Moreover, a user may decide to adjust imaging parameters and/or image processing parameters while receiving feedback from the image displayed in display160. The screen of display160can be a touchscreen, through which inputs can be received by the user.

FIG. 2depicts a controller-based system200for a device made according to embodiments. As will be appreciated, system200can include components of device100ofFIG. 1.

System200includes a pixel array210that is made according to embodiments, and which could be pixel array110ofFIG. 1. As such, system200could be, without limitation, a computer system, an imaging system, a camera system, a ranging system, a scanner system, a machine vision system, a vehicle navigation system, a smart telephone, a video telephone, a personal digital assistant (PDA), a mobile computer, a surveillance system, an auto focus system, a star tracker system, a motion detection system, an image stabilization system, a data compression system for high-definition television or a moving picture (“movie”), and so on.

System200may additionally include a processor220and a controller230, which may be similar to processor120and to controller130respectively. In some embodiments, these components communicate among themselves over a bus235, as shown inFIG. 2.

System200may also include a memory240, which could the previously mentioned memory140. Memory240can be a Random Access Memory (RAM), a Read Only Memory (ROM), a combination, and so on. Memory240may be configured to store instructions that can be read and executed by processor220and/or controller230. Memory240may be configured to store the samples captured by pixel array210, both for the short term and the long term.

System200further optionally includes a user interface250, which can be made as the previously described user interface150. Moreover, since system200does not necessarily have to be implemented with a casing, there can be more and different configurations for user interface250, such as by using a keypad, a keyboard and so on. Memory240may be configured to further store user data that is accessible to a user via user interface250.

System200further optionally includes a display260, which can be considered to be part of user interface1S0. Display260could be display160ofFIG. 1, or a computer screen display, a custom display, a plasma screen, and so on. Display260can show to a user an image captured by pixel array210, whether tentative or final, and so on.

Furthermore, system200may include an external drive270, which can be a compact disk (CD) drive, a thumb drive, and so on. System200can also include a network interface module280. System200may use module280to transmit data to or receive data from a communication network. The transmission can be via wires, for example via cables, or USB interface. Alternately, the communication network can be wireless, and module280can be wireless and include, for example, an antenna, a wireless transceiver and so on. The communication interface protocol can be that of a communication system such as CDMA, GSM, NADC, E-TDMA, WCDMA, CDMA2000, Wi-Fi, Muni Wi-Fi, Bluetooth, DECT, Wireless USB, Flash-OFDM, IEEE 802.20, GPRS, iBurst, WiBro, WiMAX, WiMAX-Advanced, UMTS-TDD, HSPA, EVDO, LTE-Advanced, MMDS, and so on.

The previously mentioned depth images can be used according to time-of-flight (TOF) principles. Now they are described in more detail.

FIG. 3shows an array310, which is made according to embodiments. Array310includes depth pixels that are not shown individually. Array310can be a standalone pixel array, or a portion of pixel array that also includes other pixels such as image pixels.

The depth pixels of array310can be arranged in four quadrants A0, A1, A2and A3. In addition, openings390,391,392and393are shown with reference to their respective quadrants A0, A1, A2and A3. These openings390,391,392and393can be as described for openings195ofFIG. 1. Of course, these openings390,391,392and393are not in the same plane as pixel array310, and that is why they are shown in dashed lines inFIG. 3.

The pixels in quadrants A0, A1, A2and A3can be arranged so as to receive light through a respective one of the four openings390,391,392and393. In other words, light received through opening390could be imaged by pixels in quadrant A0, and so on.

The depth pixels in array310can be configured to generate samples in response to the received light. The light can be received as a reflection of light116from object101, so that ultimately depth can be determined using time-of-flight principles. The samples generated by each depth pixel can be for the quadrant of their respective depth pixel.

It will be recognized that this arrangement can generate four depth images, one in each of quadrants A0, A1, A2and A3. These may have been obtained at a different time instant, i.e. at a different phase of modulated light116, and a set of all four may be needed to ultimately generate a depth image. In other words, for obtaining the final depth information in terms of a depth image, a group of the samples will be read out. The group may include samples from all four quadrants, which are treated as the respective four phases of a time-of-flight (T-O-F) technique.

In embodiments, a controller330, which can be the same as controller130, may control array310. In fact, controller330may be configured to control array310so as to read out of array310a group of the samples in any desired way.

In embodiments, the reading out of the group of the samples from all four quadrants as the four phases of a T-O-F technique is such that at least some samples of each quadrant have been read out, before completing reading out the samples of any one of the quadrants. In some embodiments, therefore, the samples of quadrants A0, A1, A2and A3are read directly into image signal processor320. Then, these samples in the group may be combined to generate a single depth image324according to the time-of-flight technique.

Embodiments are different from prior art, where first quadrants A0and A1are completely read out pixel row by pixel row, and then quadrants A2and A3are read out pixel row by pixel row. In such prior art embodiments, an additional frame buffer340is often used to store the read out samples of quadrants A0and A1as the samples of quadrants A2and A3are read out of the pixel array.

It should be noted that present embodiments do not always need a frame buffer340, which is why it is shown crossed out inFIG. 3. In fact, in present embodiments, after being read out of array310and prior to being combined by image signal processor320, all the samples in the group may have been stored in the same one or more devices. In other words, in embodiments, none of the samples may have been stored in a buffer such as buffer340, or all of them may have. In embodiments, the samples might not be stored in different devices based on which of the quadrants they came from.

More particular ways according to embodiments are now described. In some embodiments, the array can be oriented so that the four quadrants are arranged two in the top and the other two in the bottom. This is also the example inFIG. 3, where the quadrants are arranged two in the top (A0, A1) and the other two in the bottom (A2, A3). The depth pixels in each quadrant can be arranged in pixel rows, and reading out for the group can be performed by reading samples from at least one pixel row of the two quadrants in the top, then samples from at least one pixel row of the two quadrants in the bottom, and then samples from at least one pixel row of the two quadrants in the top.

Another example is shown inFIG. 4, which shows an array410. Array410could be the same as array310. The division in quadrants and the openings are shown without markings, so as to not clutter the drawing.

Array410has depth pixels that are arranged in pixel rows. Pixel row numbers402are shown, ranging from 1 to 2m. Rows 1 through m could be for quadrants A0and A1, while rows m+1 up to 2m could be for quadrants A2and A3.

A read out order412is also shown by row. Specifically, samples are read from pixel row 1 of top quadrants A0, A1, then samples are read from pixel row m+1 of bottom quadrants A2, A3, and then samples are read from additional pixel row 2 of top quadrants A0, A1, etc.

In embodiments, the samples can be read out from the pixel rows in an interleaved fashion. In embodiments, the samples can be read out from the pixel rows according to a rolling shutter scheme. If this is applied to the example ofFIG. 4, the full readout order would be 1>m+1>2>m+2> . . . >m−1>2m−1>m>2m.

FIG. 5shows a flowchart500for describing methods according to embodiments. The methods of flowchart500may also be practiced by embodiments described elsewhere in this document.

According to an operation510, light may be received in each of four quadrants. The light can be received through an opening corresponding to each quadrant.

According to another operation520, samples can be generated in response to the received light. The samples can be generated in the depth pixels, for the quadrant of their respective depth pixels.

According to another operation530, a group of the samples can be read out of the pixel array. The group might include samples of the four quadrants as respective four phases of a time-of-flight technique. The reading out can be such that samples of each quadrant have been read out before completing reading out the samples of any one of the quadrants.

In the methods described above, each operation can be performed as an affirmative step of doing, or causing to happen, what is written that can take place. Such doing or causing to happen can be by the whole system or device, or just one or more components of it. In addition, the order of operations is not constrained to what is shown, and different orders may be possible according to different embodiments. Moreover, in certain embodiments, new operations may be added, or individual operations may be modified or deleted. The added operations can be, for example, from what is mentioned while primarily describing a different system, apparatus, device or method.

Implementation of these embodiments is now described in more detail. To this end,FIG. 6includes two parts and illustrates advantages when capturing a scene with motion.

In the left side ofFIG. 6is a diagram illustrating samples generated by depth pixels in different quadrants680,681,682and683. The pixels in these four quadrants can be considered as four phases of the time-of-flight technique. These pixels are represented by the information they carry, which are images of a man walking his dog. Embodiments of prior art, where first quadrants A0and A1are completely read out pixel row by pixel row, and then quadrants A2and A3are read out pixel row by pixel row, for example using a rolling shutter, would cause images captured in quadrants680and681to be considerably different from images captured in quadrants682and683. Specifically, since the timing of the exposure of the scenes by quadrants682and683is delayed compared to quadrants680and681, images captured by quadrants682and683would show the man and the dog having walked further to the right. For clarity, by exposure timing we mean the time when a quadrant begins its light capture for a particular frame and the time when light capture for that quadrant, for that frame ends.

However, embodiments benefit if the exposure timing of all quadrants is practically identical, for example by implementing the interleaved exposure and readout illustrated inFIG. 5.

InFIG. 6to the right is a time diagram relating to the reading out of the samples ofFIG. 6. Readout621is for rows 1 through m of quadrants680and681, while readout622is for rows m+1 through 2m of quadrants682and683. The Y axis indicates the row number, 1 through 2m. The X axis indicates time when light capture starts and/or ends for each row from 1 to 2m. Left edges of the slanted parallelograms621and622indicate the time when light capture starts. For example, both row 1 and row m start capturing light nearly simultaneously with each other at time zero, where Y axis crosses the X axis. For example, both row 2 and row m+1 start capturing light practically simultaneously with each other, but a short time 2*TROWafter rows 1 and m start their light capture. Such delays of exposure timing from one row to the next row are a hallmark of rolling shutter operation in image sensors.

Right edges of the slanted parallelograms621and622indicate the time when light capture ends and readout of the accumulated sample takes place. For example, both row 1 and row m end capturing light and are read out nearly simultaneously with each other after exposure time TEXP. For another example, both rows 2 and m+1 end capturing light and are read out nearly simultaneously with each other at time TEXP+2*TROW. Similarly, other row pairs, e.g. row 3 and row m+2, row 4 and row m+3 and so on, begin their light capture nearly simultaneously with each other and end their light capture nearly simultaneously with each other.

For clarity of explanation, a case can be considered where a) minor parallax effects can be ignored, b) the object is assumed to be at an infinite distance away, and c) all lenses are perfectly aligned with respect to their quadrants. In this case, exposure of pixels in all quadrants correspondingly imaging a point in space will be nearly simultaneous. Thus images from all quadrants will show the man at an identical position—not some quadrants showing the man when he was on the left while other quadrants show the man after he has moved to the right.

It will be understood that there is “skewness”, which is why the man and the dog appear skewed to the left. This is due to the rolling shutter technique, amplified by the fact that it takes about twice as long to roll through all the pixels in each quadrant.

A person skilled in the art will be able to practice the present invention in view of this description, which is to be taken as a whole. Details have been included to provide a thorough understanding. In other instances, well-known aspects have not been described, in order to not obscure unnecessarily the present invention. Plus, any reference to any prior art in this description is not, and should not be taken as, an acknowledgement or any form of suggestion that this prior art forms parts of the common general knowledge in any country.

This description includes one or more examples, but that does not limit how the invention may be practiced. Indeed, examples or embodiments of the invention may be practiced according to what is described, or yet differently, and also in conjunction with other present or future technologies. Other embodiments include combinations and sub-combinations of features described herein, including for example, embodiments that are equivalent to: providing or applying a feature in a different order than in a described embodiment; extracting an individual feature from one embodiment and inserting such feature into another embodiment; removing one or more features from an embodiment; or both removing a feature from an embodiment and adding a feature extracted from another embodiment, while providing the features incorporated in such combinations and sub-combinations.

In this document, the phrases “constructed to” and/or “configured to” denote one or more actual states of construction and/or configuration that is fundamentally tied to physical characteristics of the element or feature preceding these phrases and, as such, reach well beyond merely describing an intended use. Any such elements or features can be implemented in any number of ways, as will be apparent to a person skilled in the art after reviewing the present disclosure, beyond any examples shown in this document.

The following claims define certain combinations and subcombinations of elements, features and steps or operations, which are regarded as novel and non-obvious. Additional claims for other such combinations and subcombinations may be presented in this or a related document.