Patent Description:
Image sensing devices typically include an image sensor, generally implemented as an array of pixel circuits, as well as signal processing circuitry and any associated control or timing circuitry. Within the image sensor itself, charge is collected in a photoelectric conversion device of the pixel circuit as a result of the impingement of light. There are typically a very large number of individual photoelectric conversion devices (e.g. tens of millions), and many signal processing circuitry components working in parallel. Various components within the signal processing circuitry are shared by a large number of photoelectric conversion devices; for example, a column or multiple columns of photoelectric conversion devices may share a single analog-to-digital converter (ADC) or sample-and-hold (S/H) circuit. <CIT> relates to a solid-state imaging apparatus including a pixel array section in which pixels including photoelectric conversion elements are two-dimensionally arranged in a matrix form, and plural systematic pixel drive lines to transmit drive signals to read out signals from the pixels are arranged for each pixel row, and a row scanning section to simultaneously output the drive signals through the plural systematic pixel drive lines to plural pixel rows for different pixel columns.

Various aspects of the present disclosure relate to an image sensor and distance determination method therein.

The following aspects may include some but not all features as literally defined in the claims and are present for illustration purposes only. In one aspect of the present disclosure, there is provided a time-of-flight sensor, comprising: a pixel array including a plurality of pixel circuits arranged in an array, wherein a first column of the array includes: a first pixel circuit including a first photodiode, a first capacitor coupled to the first photodiode, and a second capacitor coupled to the first photodiode, and a second pixel circuit including a second photodiode, a third capacitor coupled to the second photodiode, and a fourth capacitor coupled to the second photodiode; a first signal line coupled to the first capacitor; a second signal line coupled to the second capacitor; a third signal line coupled to the third capacitor; a fourth signal line coupled to the fourth capacitor; a first switch circuitry; a second switch circuitry; a first comparator coupled to the first signal line and the third signal line through the first switch circuitry; and a second comparator coupled to the second signal line and the fourth signal line through the second switch circuitry.

In another aspect of the present disclosure, there is provided a time-of-flight system, comprising: a light source configured to emit a light; and a sensor comprising: a pixel array including a plurality of pixel circuits arranged in an array, wherein a column of the array includes: a first pixel circuit including a first photodiode, a first capacitor coupled to the first photodiode, and a second capacitor coupled to the first photodiode, and a second pixel circuit including a second photodiode, a third capacitor coupled to the second photodiode, and a fourth capacitor coupled to the second photodiode, a first signal line coupled to the first capacitor, a second signal line coupled to the second capacitor, a third signal line coupled to the third capacitor, a fourth signal line coupled to the fourth capacitor, a first switch circuitry, a second switch circuitry, a first comparator coupled to the first signal line and the third signal line through the first switch circuitry, and a second comparator coupled to the second signal line and the fourth signal line through the second switch circuitry.

In another aspect of the present disclosure, there is provided a system, comprising: a first sensor configured to generate an image data, the first sensor comprising a first pixel array; and a second sensor configured to generate a distance data, the second sensor comprising: a second pixel array including a plurality of pixel circuits arranged in an array, wherein a column of the array includes: a first pixel circuit including a first photodiode, a first capacitor coupled to the first photodiode, and a second capacitor coupled to the first photodiode, and a second pixel circuit including a second photodiode, a third capacitor coupled to the second photodiode, and a fourth capacitor coupled to the second photodiode, a first signal line coupled to the first capacitor, a second signal line coupled to the second capacitor, a third signal line coupled to the third capacitor, a fourth signal line coupled to the fourth capacitor, a first switch circuitry, a second switch circuitry, a first comparator coupled to the first signal line and the third signal line through the first switch circuitry, and a second comparator coupled to the second signal line and the fourth signal line through the second switch circuitry.

As such, various aspects of the present disclosure provide for improvements in at least the technical field of depth sensing, as well as the related technical fields of imaging, image processing, and the like.

This disclosure can be embodied in various forms, including hardware or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, and application programming interfaces; as well as hardware-implemented methods, signal processing circuits, image sensor circuits, application specific integrated circuits, field programmable gate arrays, and the like. The foregoing summary is intended solely to give a general idea of various aspects of the present disclosure, and does not limit the scope of the disclosure in any way.

The quantities I and Q may be obtained in the Nth frame, and the quantities -I and -Q may be obtained in the (N+<NUM>)th frame.

These and other more detailed and specific features of various embodiments are more fully disclosed in the following description, reference being had to the accompanying drawings, in which:.

In the following description, numerous details are set forth, such as flowcharts, data tables, and system configurations.

Moreover, while the present disclosure focuses mainly on examples in which the processing circuits are used in image sensors, it will be understood that this is merely one example of an implementation. It will further be understood that the disclosed systems and methods can be used in any device in which there is a need to detect distance in a wave-based sensor; for example, an audio circuit, phononic sensor, a radar system, and the like.

<FIG> illustrates a first example 100a of a TOF imaging system 101a configured to detect and/or an object <NUM> located a distance d away. The TOF imaging system 101a includes a light generator <NUM> configured to generate an emitted light wave <NUM> toward the object <NUM> and a TOF image sensor <NUM> configured to receive a reflected light wave <NUM> from the object <NUM>. The emitted light wave <NUM> may have a periodic waveform. The TOF image sensor <NUM> may be any device capable of converting incident radiation into signals. For example, the TOF image sensor <NUM> may be implemented by a Complementary Metal-Oxide Semiconductor (CMOS) Image Sensor (CIS), a Charge-Coupled Device (CCD), and the like. The TOF imaging system 101a may further include distance determination circuitry such as a controller <NUM> (e.g., a CPU) and a memory <NUM>, which may operate to perform one or more examples of time-of-flight processing as described further below.

<FIG> illustrates a second example 100b of a TOF imaging system 101b configured to detect and/or image an object <NUM> located a distance d away. The TOF imaging system 101b includes a light generator <NUM> configured to generate an emitted light wave <NUM> toward the object <NUM>, a TOF image sensor <NUM> configured to receive a reflected light wave <NUM> from the object <NUM>, and an RGB image sensor <NUM> configured to capture an RGB image of the object <NUM>. The emitted light wave <NUM> may have a periodic waveform. The TOF image sensor <NUM> may be any device capable of converting incident radiation into signals. For example, the TOF image sensor <NUM> and the RGB sensor <NUM> may each be implemented by a Complementary Metal-Oxide Semiconductor (CMOS) Image Sensor (CIS), a Charge-Coupled Device (CCD), and the like. While the second example 100b is described with reference to an RGB image sensor <NUM>, in practice the image sensor <NUM> may capture a monochromatic image or may include color filters different from RGB. Furthermore, while <FIG> illustrates the TOF image sensor <NUM> and the RGB image sensor <NUM> as separate components, in some aspects of the present disclosure the TOF image sensor <NUM> and the RGB image sensor <NUM> may be integrated as a single chip and/or utilize a single pixel array. The TOF imaging system 101b may further include distance determination and processing circuitry such as a controller <NUM> (e.g., a CPU) and a memory <NUM>, which may operate to perform one or more examples of time-of-flight and image processing as described further below.

The light generator <NUM> may be, for example, a light emitting diode (LED), a laser diode, or any other light generating device or combination of devices, and the light waveform may be controlled by the controller <NUM>. The light generator may operate in the infrared range so as to reduce interference from the visible spectrum of light, although any wavelength range perceivable by the image sensor <NUM> may be utilized. The controller <NUM> may be configured to receive an image from the image sensor and calculate a depth map indicative of the distance d to various points of the object <NUM>.

<FIG> illustrates an exemplary image sensor <NUM> according to various aspects of the present disclosure. The image sensor <NUM> may be an example of the TOF image sensor <NUM> illustrated in <FIG>. As illustrated in <FIG>, the image sensor <NUM> includes an array <NUM> of pixel circuits <NUM>, each of which are located at an intersection where a horizontal signal line <NUM> and a set of vertical signal lines 213a, 213b, 213c, 213d cross each other. The horizontal signal lines <NUM> are operatively connected to vertical scanning circuitry <NUM>, also referred to as a "row scanning circuit" or a "vertical driving circuit," at a point outside of the pixel array <NUM>. The horizontal signal lines <NUM> carry signals from the vertical scanning circuitry <NUM> to a particular row of the pixel circuits <NUM>. While <FIG> illustrates a single horizontal signal line <NUM> for a given row of the pixel circuits <NUM>, in practice a plurality of the horizontal signal lines <NUM> may be provided for each row of the pixel circuits <NUM>.

The pixel circuits <NUM> store a charge corresponding to an amount of incident light alternately in floating diffusions FDa and FDb (for example, as illustrated in <FIG>) and selectively output an analog signal corresponding to an amount of the charge to the vertical signal lines 213a, 213b, 213c, 213d in a manner that will be described in more detail below. While <FIG> illustrates the vertical signal lines 213a and 213c on one side of a given pixel circuit <NUM> and the vertical signal lines 213b and 213d on the other side of the given pixel circuit <NUM>, in practice the vertical signal lines 213a, 213b, 213c, 213d may all be provided on a single side of the given pixel circuit <NUM>; or one of the vertical signal lines 213a, 213b, 213c, 213d may be on one side of the given pixel circuit <NUM> and the other three of the vertical signal lines 213a, 213b, 213c, 213d may be on the other side of the given pixel circuit <NUM>. Furthermore, for illustration purposes, only a subset of the pixel circuits <NUM> in the array <NUM> are actually shown in <FIG>; however, in practice the image sensor <NUM> may have any number of the pixel circuits <NUM>. <FIG> illustrates two vertical signal lines 213a and 213b or 213c and 213d for each of the pixel circuits <NUM> (a "two-tap" system); however, in practice the image sensor <NUM> may incorporate a larger number of the vertical signal lines for each column of the pixel circuits <NUM>.

The pixel circuits <NUM> in some rows of the array <NUM> are connected to the vertical signal lines 213a and 213b, while the pixel circuits <NUM> in other rows of the array <NUM> are connected to the vertical signal lines 213c and 213d. In some aspects, the pixel circuits <NUM> are connected to particular vertical signal lines in groups of four rows; that is, the pixel circuits <NUM> in the first four rows of the array <NUM> are connected to the vertical signal lines 213a and 213b, the pixel circuits <NUM> in the second four rows of the array <NUM> are connected to the vertical signal lines 213c and 213d, the pixel circuits in the third four rows of the array <NUM> are connected to the vertical signal lines 213c and 213d, and so on.

The vertical signal lines 213a, 213b, 213c, 213d conduct the analog signals (A for the vertical signal lines 213a and 213c and B for the vertical signal lines 213b and 213c) for a particular column to a readout circuit <NUM>, which includes a switching circuit <NUM> and includes two comparators <NUM> for each column of the pixel circuits <NUM>. Each comparator <NUM> compares an analog signal to a reference signal output from a reference signal generator <NUM>. The reference signal generator <NUM> may be, for example, a digital-to-analog converter (DAC) and the reference signal may have, for example, a periodic ramp waveform. Each comparator <NUM> outputs a digital signal indicative of a comparison between the input analog signal from the corresponding signal line input and the reference signal.

The output of the readout circuit <NUM> is provided to a signal processing circuit <NUM>. The signal processing circuit <NUM> may include additional components, such as counters, latches, S/H circuits, and the like. The signal processing circuit <NUM> may be capable of performing a method of correlated double sampling (CDS). CDS is capable of overcoming some pixel noise related issues by sampling each pixel circuit <NUM> twice. First, the reset voltage V ¬ reset of a pixel circuit <NUM> is sampled. This may also be referred to as the P-phase value or cds value. Subsequently, the data voltage Vdata of the pixel circuit <NUM> (that is, the voltage after the pixel circuit <NUM> has been exposed to light) is sampled. This may also be referred to as the D-phase value or light-exposed value. The reset value Vreset is then subtracted from the data value Vdata to provide a value which reflects the amount of light falling on the pixel circuit <NUM>. The CDS method may be performed for each tap of the pixel circuit <NUM>.

Various components of the signal processing circuit are controlled by horizontal scanning circuitry <NUM>, also known as a "column scanning circuit" or "horizontal driving circuit. " The horizontal scanning circuitry <NUM> causes the signal processing circuit to output signals via an output circuit <NUM> for further processing, storage, transmission, and the like. The vertical scanning circuitry <NUM>, the switching circuit <NUM>, the reference circuit generator <NUM>, and the horizontal circuitry <NUM> may operate under the control of a driving controller <NUM> and/or communication and timing circuitry <NUM>, which may in turn operate based on a clock circuit <NUM>. The clock circuit <NUM> may be a clock generator, which generates one or more clock signals for various components of the image sensor <NUM>. Additionally or alternatively, the clock circuit <NUM> may be a clock converter, which converts one or more clock signals received from outside the image sensor <NUM> and provides the converted clock signal(s) to various components of the image sensor <NUM>.

<FIG> illustrates a first exemplary pixel circuit 300a having a two-tap configuration. The pixel circuit 300a may be an example of the pixel circuit <NUM> illustrated in the first row or second row of the array <NUM> in <FIG>. As shown in <FIG>, the pixel circuit 300a includes a photoelectric conversion device <NUM> (e.g., a photodiode), a pixel reset transistor <NUM>, a first transfer transistor 303a, a second transfer transistor 303b, a first floating diffusion FDa, a second floating diffusion FDb, a first tap reset transistor 304a, a second tap reset transistor 304b, a first intervening transistor 305a, a second intervening transistor 305b, a first amplifier transistor 306a, a second amplifier transistor 306b, a first selection transistor 307a, and a second selection transistor 307b. The photoelectric conversion device <NUM>, the first transfer transistor 303a, the first tap reset transistor 304a, the first intervening transistor 305a, the first amplifier transistor 306a, and the first selection transistor 307a are controlled to output an analog signal (A) via a first vertical signal line 308a, which may be an example of the vertical signal line 213a illustrated in <FIG>. This set of components may be referred to as "Tap A. " The photoelectric conversion device <NUM>, the second transfer transistor 303b, the second tap reset transistor 304b, the second intervening transistor 305b, the second amplifier transistor 306b, and the second selection transistor 307b are controlled to output an analog signal (B) via a second vertical signal line 308b, which may be an example of the vertical signal line 213b illustrated in <FIG>. This set of components may be referred to as "Tap B. " <FIG> also illustrates a third vertical signal line 308c, which may be an example of the vertical signal line 213c illustrated in <FIG>, and a fourth vertical signal line 308d, which may be an example of the vertical signal line 213d illustrated in <FIG>. As illustrated in <FIG>, however, the pixel circuit 300a is not connected to the third vertical signal line 308c or the fourth vertical signal line 308d.

<FIG> illustrates a second exemplary pixel circuit 300b having a two-tap configuration. The pixel circuit 300b may be an example of the pixel circuit <NUM> illustrated in the last row of the array <NUM> in <FIG>. As shown in <FIG>, the pixel circuit 300b has structural similarities to the pixel circuit 300a of <FIG>, and includes a photoelectric conversion device <NUM> (e.g., a photodiode), a pixel reset transistor <NUM>, a first transfer transistor 303a, a second transfer transistor 303b, a first floating diffusion FDa, a second floating diffusion FDb, a first tap reset transistor 304a, a second tap reset transistor 304b, a first intervening transistor 305a, a second intervening transistor 305b, a first amplifier transistor 306a, a second amplifier transistor 306b, a first selection transistor 307a, and a second selection transistor 307b. The photoelectric conversion device <NUM>, the first transfer transistor 303a, the first tap reset transistor 304a, the first intervening transistor 305a, the first amplifier transistor 306a, and the first selection transistor 307a are controlled to output an analog signal (A) via the third vertical signal line 308c. This set of components may be referred to as "Tap A. " The photoelectric conversion device <NUM>, the second transfer transistor 303b, the second tap reset transistor 304b, the second intervening transistor 305b, the second amplifier transistor 306b, and the second selection transistor 307b are controlled to output an analog signal (B) via the fourth vertical signal line 308d. This set of components may be referred to as "Tap B. " <FIG> also illustrates the first vertical signal line 308a and the second vertical signal line 308b. As illustrated in <FIG>, however, the pixel circuit 300b is not connected to the first vertical signal line 308a or the second vertical signal line 308b.

In either pixel circuit (300a or 300b), the first transfer transistor 303a and the second transfer transistor 303b are controlled by control signals on a first transfer gate line 309a and a second transfer gate line 309b, respectively. The first tap reset transistor 304a and the second tap reset transistor 304b are controlled by a control signal on a tap reset gate line <NUM>. The first intervening transistor 305a and the second intervening transistor 305b are controlled by a control signal on a FD gate line <NUM>. The first selection transistor 307a and the second selection transistor 307b are controlled by a control signal on a selection gate line <NUM>. The first and second transfer gate lines 309a and 309b, the tap reset gate line <NUM>, the FD gate line <NUM>, and the selection gate line <NUM> may be examples of the horizontal signal lines <NUM> illustrated in <FIG>.

In operation, the pixel circuit 300a or the pixel circuit 300b is controlled in a timedivisional manner such that, during one half of a horizontal period, incident light is converted via Tap A to generate the output signal A; and, during the other half of the horizontal period, incident light is converted via Tap B to generate the output signal B. The division of frame among the Tap A portion and the Tap B portion may be referred to as the phase of the tap. For example, where a horizontal period runs from <NUM> to t, the pixel circuit 300a or the pixel circuit 300b may be controlled such that Tap A operates from <NUM> to t/<NUM> (<NUM> phase) and Tap B operates from t/<NUM> to t (<NUM> phase), such that Tap A operates from t/<NUM> to 3t/<NUM> (<NUM> phase) and Tap B operates from <NUM> to t/<NUM> and from 3t/<NUM> to t (<NUM> phase), such that Tap A operates from t/<NUM> to t and Tap B operates from <NUM> to t/<NUM>, or such that Tap A operates from <NUM> to t/<NUM> and from 3t/<NUM> to t and Tap B operates from t/<NUM> to 3t/<NUM>. Under such an operation, the quantities Q and I for the pixel circuit 300a or the pixel circuit 300b may be defined such that Q is given by the <NUM> phase minus the <NUM> phase and I is given by the <NUM> phase minus the <NUM> phase.

While <FIG> illustrate the pixel circuit 300a and the pixel circuit 300b having a plurality of transistors in a particular configuration, the current disclosure is not so limited and may apply to a configuration in which the pixel circuit 300a or the pixel circuit 300b includes fewer or more transistors as well as other elements, such as additional capacitors, resistors, and the like.

An image sensor according to the present disclosure may be capable of a plurality of different readout modes, which will be described initially with reference to <FIG> illustrates a portion of a pixel array, such as the array <NUM> illustrated in <FIG>; as well as a portion of a readout circuit, such as the readout circuit <NUM> illustrated in <FIG>. Specifically, <FIG> illustrates two adjacent columns of pixel circuits <NUM>, which may be the same as or similar to the pixel circuits <NUM> illustrated in <FIG> and/ or the pixel circuits 300a, 300b illustrated in <FIG>; four vertical signal lines 420a, 420b, 420c, 420b for each column, which may be the same as or similar to the vertical signal lines 213a, 213b, 213d illustrated in <FIG> and/or the vertical signal lines 308a, 308b, 308c, 308d illustrated in <FIG>; a switching circuit <NUM>, which may be the same as or similar to the switching circuit <NUM> illustrated in <FIG>; a reference signal generator <NUM>, which may be the same as or similar to the reference signal generator <NUM> illustrated in <FIG>; and a plurality of comparators <NUM>, which may be the same as or similar to the comparators <NUM> illustrated in <FIG>. Each of the pixel circuits <NUM> are illustrated as bisected by a dashed line, thereby to illustrate the two taps of each pixel circuit <NUM>.

The lower four pixel circuits <NUM> are coupled to a first vertical signal line 420a at one tap (for example, at the capacitor forming the first floating diffusion FDa) and coupled to a second vertical signal line 420b at the other tap (for example, at the capacitor forming the second floating diffusion FDb). Thus, the lower four pixel circuits <NUM> may each correspond to the pixel circuit 300a illustrated in <FIG>. The upper four pixel circuits <NUM> are coupled to a third vertical signal line 420c at one tap (for example, at the capacitor forming the first floating diffusion FDa) and coupled to a fourth vertical signal line 420d at the other tap (for example, at the capacitor forming the second floating diffusion FDb). Thus, the lower four pixel circuits <NUM> may each correspond to the pixel circuit 300b illustrated in <FIG>.

For a given column, the switching circuit <NUM> includes a first set of switch circuitry <NUM> and a second set of switch circuitry <NUM>. As illustrated, each set of switch circuitry <NUM>, <NUM> includes three switches, each of which may be individually controllable. The first set of switch circuitry <NUM> includes a first switch connected at a first end to the first vertical signal line 420a of the left column, a second switch connected at a first end to the third vertical signal line 420c of the left column, and a third switch connected at a first end to a first vertical signal line 420a of the right column. A second end of the first, second and third switches is coupled (as illustrated, capacitively coupled) to a first input of a first comparator <NUM> of the left column. The second set of switch circuitry <NUM> includes a first switch connected at a first end to the second vertical signal line 420b of the left column, a second switch connected at a first end to the fourth vertical signal line 420d of the left column, and a third switch connected at a first end to the second vertical signal line 420b of the right column. A second end of the first, second and third switches is coupled (as illustrated, capacitively coupled) to a first input of a second comparator <NUM> of the left column. A second input of the first comparator <NUM> and the second comparator <NUM> are coupled (as illustrated, capacitively coupled) to the reference signal generator <NUM>.

Thus, the first comparator <NUM> is coupled to at least the first vertical signal line 420a and the third vertical signal line 420c through the first switch circuitry <NUM>, and the second comparator <NUM> is coupled to at least the second vertical signal line 420b and the fourth vertical signal line 420d through the second switch circuitry <NUM>. The first switch circuitry <NUM> and the second switch circuitry <NUM> may be controlled by a timing circuit, such as the communication and timing circuitry <NUM> illustrated in <FIG>. The pixel circuits <NUM> in the left column are also coupled to the comparators <NUM> in the right column; for example, the third vertical signal line 420c is connected to a third switch of the first switch circuitry <NUM> in the right column, and the fourth vertical signal line 420d is connected to a third switch of the second switch circuitry <NUM> in the right column.

The various readout modes of the portion of the readout circuit illustrated in <FIG> are described in more detail with regard to <FIG>. The components illustrated in <FIG> correspond to those illustrated in <FIG>, and thus a detailed description of the components is not repeated.

<FIG> illustrate a so-called normal mode for an Nth frame to an (N+<NUM>)th frame, respectively. In <FIG>, the Nth frame is illustrated. As illustrated, the first switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is closed, while the second and third switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are open. The pixel circuits <NUM> in the bottom four rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that, in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

In <FIG>, an (N+<NUM>)th frame is illustrated. As illustrated, the states of the first switch circuitry <NUM> and the second switch circuitry <NUM> are the same as in <FIG>; however, the phases of the pixel circuits <NUM> are modified. Thus, the pixel circuits <NUM> in the bottom four rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that, in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

The outputs for each frame and/or each horizontal period within a frame may be stored in a memory. After the four frames, the quantities Q and I may be calculated as described above. In some aspects of the present disclosure, the quantities Q and I are calculated in signal processing circuitry disposed subsequent to the comparators <NUM>, such as the signal processing circuitry <NUM> illustrated in <FIG>. The signal processing circuitry may include the memory and calculation circuitry such as a processor (e.g., a CPU or a FPGA).

<FIG> illustrates a so-called pixel thinning or pixel skipping mode for an Nth frame. In particular, <FIG> illustrates a "skip <NUM>" mode where one row of pixels is skipped; however, the present disclosure may also be implemented with a "skip <NUM>" readout mode where two rows of pixels are skipped. As illustrated in <FIG>, the first switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is closed, while the second and third switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are open. Every other one of the pixel circuits <NUM> in the bottom four rows are driven in two consecutive horizontal periods <NUM> to <NUM> such that, in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

In <FIG>, the pixel circuits <NUM> in the bottom row and the pixel circuits <NUM> in the third-from-the-bottom row are read out in the frame, while the pixel circuits <NUM> in the second-from-the-bottom row and the pixel circuits <NUM> in the fourth-from-the-bottom row are skipped. Subsequent to the Nth frame illustrated in <FIG>, the phases of the pixel circuits <NUM> may be modified in the manner described above with respect to <FIG> for the (N+<NUM>)th frame to the (N+<NUM>)th frame. Thus, after the four frames, the quantities Q and I may be calculated. In comparison to the normal mode of <FIG>, however, the skip <NUM> mode may be implemented in half the time because only half the horizontal periods are included in each frame.

<FIG> illustrates a so-called pixel binning mode for an Nth frame. In particular, <FIG> illustrates a "<NUM>×<NUM> binning" mode where groups of four pixels are binned; however, the present disclosure may be implemented with a "<NUM>×<NUM> binning mode," a "<NUM>×<NUM> binning mode," a "<NUM>×<NUM> binning mode," and the like. As illustrated in <FIG>, the first and third switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are closed, while the second switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is open.

In a first horizontal period <NUM>, the first, second, fifth, and sixth rows (counting from the bottom) of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase. During the first horizontal period <NUM>, the signals for the first and second rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the fifth and sixth rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column.

In a second horizontal period <NUM>, the third, fourth, seventh, and eighth rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase. During the second horizontal period <NUM>, the signals for the third and fourth rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the seventh and eighth rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column.

Subsequent to the Nth frame illustrated in <FIG>, the phases of the pixel circuits <NUM> may be modified in the manner described above with respect to <FIG> for the (N+<NUM>)th frame to the (N+<NUM>)th frame. Thus, after the four frames, the quantities Q and I may be calculated. In comparison to the normal mode of <FIG>, however, the <NUM>×<NUM> binning mode may be implemented in half the time because only half the horizontal periods are included in each frame.

The binning and skipping modes may be combined into a hybrid mode. <FIG> illustrates such a hybrid mode with <NUM>×<NUM> binning and skip <NUM> implemented. As illustrated in <FIG>, the first and third switches of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the left column are closed, while the second switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the left column is open. All three switches of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the right column are open.

In a first horizontal period <NUM>, the bottom four rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase. During the first horizontal period <NUM>, the signals for the bottom four rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column.

The next four rows of pixel circuits <NUM> are skipped, such that in a second horizontal period <NUM> the bottom four rows of the next set of eight pixel circuits (not illustrated in <FIG>) are driven. Subsequent to the Nth frame illustrated in <FIG>, the phases of the pixel circuits <NUM> may be modified in the manner described above with respect to <FIG> for the (N+<NUM>)th frame to the (N+<NUM>)th frame. Thus, after the four frames, the quantities Q and I may be calculated. In comparison to the normal mode of <FIG>, however, the <NUM>×<NUM> skip <NUM> mode may be implemented in one-quarter of the time because only one-quarter of the horizontal periods are included in each frame.

In each of the above modes, four frames are used to obtain the quantities Q and I because four phases per pixel are utilized. In some modes, however, the quantities Q and I are obtained in only two frames by utilizing two phases per pixel. These modes may be referred to as IQ modes. The IQ modes may be implemented with any of the normal mode, the skipping modes, the binning modes, and the hybrid modes described above. <FIG> illustrates a <NUM>×<NUM> binning IQ mode.

In <FIG>, an Nth frame is illustrated. As illustrated, the first and third switches of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the left column and the second and third switches of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the right column are closed, while the second switch of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the left column and the first switch of the first switch circuitry <NUM> and the second switch circuitry <NUM> in the right column is open. In a first horizontal period <NUM>, the first and third rows (counting from the bottom) of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase, and the fifth and seventh rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

In a second horizontal period <NUM>, the second and fourth rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase, and the fifth and seventh rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

During the first horizontal period <NUM> and the second horizontal period <NUM>, the signals for the bottom four rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the top four rows of the pixel circuits <NUM> in both the left column and the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column.

In <FIG>, an (N+<NUM>)th frame is illustrated. The configuration of each of the switches in the first switch circuitry <NUM> and the second switch circuitry <NUM> remains the same as in <FIG>. In a first horizontal period <NUM>, the first and third rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase, and the fifth and seventh rows of pixel circuits <NUM> are driven such that Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase and Tap B of the corresponding pixel circuit <NUM> operates in the <NUM> phase.

<FIG> illustrates an exemplary mosaic/demosaic (mdm) process which may be implemented with the modes, as will be discussed in more detail below. As illustrated in <FIG>, a first data block <NUM> and a second data block <NUM> are obtained. The first data block <NUM> and the second data block may be obtained, for example, by the processes discussed with regard to <FIG>.

The first data block <NUM> includes pixel data corresponding to the quantities I and Q in alternating columns. The second data block <NUM> includes pixel data corresponding to the quantities -I and -Q in alternating columns. By subtracting the second data block <NUM> from the first data block <NUM>, a third data block <NUM> is obtained. The third data block <NUM> includes data corresponding to the quantities Q' and I', which correspond to the pixel data with ambient error canceled.

Among the sources of ambient error are ambient light, which is generally the same for both Tap A and Tap B of a given pixel circuit <NUM>, and tap gain mismatch, which is not necessarily the same for both Tap A and Tap B of the given pixel circuit <NUM>. The ambient error may be canceled through an IQ demosaic process, in which the third data block <NUM> is converted into a fourth data block <NUM> and a fifth data block <NUM>. The fourth data block <NUM> includes data corresponding to the quantity I' in all columns, and the fifth data block <NUM> includes data corresponding to the quantity Q' in all columns.

The IQ demosaic process may be represented by the following expression (<NUM>):
[Math. <NUM>] <MAT>.

Q, I, Qerror, and Ierror may be given by the following expressions (<NUM>)-(<NUM>), respectively:
[Math. <NUM>] <MAT>
[Math. <NUM>] <MAT>
[Math. <NUM>] <MAT>.

Above, the quantities x, y, m, and n correspond to amounts of active light (for example, light emitted by the light generator <NUM> illustrated in <FIG>); the quantity γ corresponds to an amount of ambient light; and α and β correspond to tap gain mismatch. By utilizing the IQ demosaic process based on the first data block <NUM> and the second data block <NUM>, expression (<NUM>) becomes the following expression (<NUM>):
[Math. <NUM>] <MAT>.

Thus, the ambient error is canceled. The quantities I and Q may be obtained in various modes, as illustrated in <FIG>.

<FIG> illustrate a normal IQ mdm mode for an Nth frame and an (N+<NUM>)th frame, respectively. In <FIG>, the Nth frame is illustrated. As illustrated, the first switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is closed, while the second and third switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are open. The pixel circuits <NUM> in the bottom four rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that, in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

In <FIG>, the (N+<NUM>)th frame is illustrated. As illustrated, the states of the first switch circuitry <NUM> and the second switch circuitry <NUM> are the same as in <FIG>; however, the phases of the pixel circuits <NUM> are modified. Thus, the pixel circuits <NUM> in the bottom four rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that, in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

In the above manner, the quantities I and Q may be obtained in the Nth frame, and the quantities -I and -Q may be obtained in the (N+<NUM>)th frame. The quantities may then be subjected to the IQ mosaic/demosaic process illustrated in <FIG>.

<FIG> illustrate a skip <NUM> IQ mdm mode for an Nth frame and an (N+<NUM>)th frame, respectively. In <FIG>, the Nth frame is illustrated. As illustrated, the first switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is closed, while the second and third switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are open. The pixel circuits <NUM> in the first, third, fifth, and seventh rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase. The pixel circuits <NUM> in the second, fourth, sixth, and eighth rows are skipped.

In <FIG>, the (N+<NUM>)th frame is illustrated. As illustrated, the states of the first switch circuitry <NUM> and the second switch circuitry <NUM> are the same as in <FIG>; however, the phases of the pixel circuits <NUM> are modified. Thus, the pixel circuits <NUM> in the first, third, fifth, and seventh rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase. The pixel circuits <NUM> in the second, fourth, sixth, and eighth rows are again skipped.

<FIG> illustrates a <NUM>×<NUM> pixel binning IQ mdm mode for an Nth frame and an (N+<NUM>)th frame, respectively. In <FIG>, the Nth frame is illustrated. The first and second switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are closed, while the third switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is open. The pixel circuits <NUM> in pairs of rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

In a first horizontal period <NUM>, the bottom two rows of pixel circuits <NUM> are driven such that Tap A and Tap B of the corresponding pixel circuits <NUM> operate in the phases noted above. During the first horizontal period <NUM>, the signals for the first and second rows of the pixel circuits <NUM> in the left column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the first and second rows of the pixel circuits <NUM> in the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column.

In a second horizontal period <NUM>, the third and fourth rows of pixel circuits <NUM> are driven such that Tap A and Tap B of the corresponding pixel circuits <NUM> operate in the phases noted above. During the second horizontal period <NUM>, the signals for the third and fourth rows of the pixel circuits <NUM> in the left column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the third and fourth rows of the pixel circuits <NUM> in the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column. A third horizontal period <NUM> and a fourth horizontal period <NUM> follow similarly.

In <FIG>, the (N+<NUM>)th frame is illustrated. The first and second switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are closed, while the third switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is open. The pixel circuits <NUM> in pairs of rows are driven in four consecutive horizontal periods <NUM> to <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

<FIG> illustrates a <NUM>×<NUM> pixel binning skip <NUM> IQ mdm mode for an Nth frame and an (N+<NUM>)th frame, respectively. In <FIG>, the Nth frame is illustrated. The first and second switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are closed, while the third switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is open. The pixel circuits <NUM> in every other pair of rows are driven in two consecutive horizontal periods <NUM> and <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

In a second horizontal period <NUM>, the fifth and sixth rows of pixel circuits <NUM> are driven such that Tap A and Tap B of the corresponding pixel circuits <NUM> operate in the phases noted above. During the second horizontal period <NUM>, the signals for the fifth and sixth rows of the pixel circuits <NUM> in the left column are provided to the comparators <NUM> for the left column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the left column, while the signals for the third and fourth rows of the pixel circuits <NUM> in the right column are provided to the comparators <NUM> for the right column through the first switch circuitry <NUM> and the second switch circuitry <NUM> for the right column. A third horizontal period <NUM> and a fourth horizontal period <NUM> follow similarly. The third, fourth, seventh, and eighth rows of pixel circuits <NUM> are skipped.

In <FIG>, the (N+<NUM>)th frame is illustrated. The first and second switches of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> are closed, while the third switch of each of the first switch circuitry <NUM> and the second switch circuitry <NUM> is open. The pixel circuits <NUM> in every other pair of rows are driven in two consecutive horizontal periods <NUM> and <NUM> such that in a respective horizontal period Tap A of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap B of the corresponding pixel circuit <NUM> in the left column operates in the <NUM> phase, Tap A of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase, and Tap B of the corresponding pixel circuit <NUM> in the right column operates in the <NUM> phase.

An imaging system, such as the TOF imaging system 101a or the TOF imaging system 101b illustrated in <FIG>, may be operated to implement any of the above readout modes and thereby provide for object detection, depth map generation, face/ gesture recognition, imaging, or combinations of the above.

<FIG> illustrates an exemplary imaging method in accordance with the present disclosure. The imaging method of may be implemented by the TOF imaging system 101a or the TOF imaging system 10b. At <NUM>, a proximity mode selection is made. The selection may be made by a local user, for example by an operation on a button or touch screen of a device implementing the TOF imaging system 101a or the TOF imaging system 101b. The selection may also be made by a controller of a device implementing the TOF imaging system 101a or the TOF imaging system 101b, for example by a remote user request or an automatic or pre-programmed operation. In the proximity mode, at <NUM> a low power mode (LPM) may be selected; again, either by a local user and/or a controller of the device. The low power mode may be any one of the thinning modes, the binning modes, or the hybrid modes described above. At <NUM>, an object detection determination is made. If no object is detected, the exemplary method may reinitialize or restart.

If an object is detected, at <NUM> a depth measurement mode is selected. As above, the selection may be made by a local user and/or a controller of the device. In the depth measurement mode, at <NUM> a readout mode is selected by the local user and/or the controller of the device. The readout mode may be any one of the normal mode, the thinning modes, the binning modes, the IQ mosaic modes, the mdm modes, or the hybrid modes described above. At <NUM>, the device generates a depth map. At <NUM>, the device performs a face recognition operation and/or a gesture recognition operation. In some aspects of the present disclosure, the exemplary imaging method may only generate a depth map (and not perform a recognition operation) or may only perform a recognition operation (and not generate a full depth map).

In this manner, in <NUM> to <NUM> the device determines whether an object is present and, if so, in <NUM> to <NUM>/<NUM> the device may generate a depth map and/or perform a recognition operation.

<FIG> illustrates another exemplary imaging method in accordance with the present disclosure. The imaging method of may be implemented by the the TOF imaging system 10b, which incorporates an RGB sensor in addition to a TOF sensor. At <NUM>, a proximity mode selection is made. The selection may be made by a local user, for example by an operation on a button or touch screen of a device implementing the TOF imaging system 101b. The selection may also be made by a controller of a device implementing the TOF imaging system 101b, for example by a remote user request or an automatic or pre-programmed operation. In the proximity mode, at <NUM> a low power mode (LPM) may be selected; again, either by a local user and/or a controller of the device. The low power mode may be any one of the thinning modes, the binning modes, or the hybrid modes described above. At <NUM>, an object detection determination is made. If no object is detected, the exemplary method may reinitialize or restart.

If an object is detected, at <NUM> a RGB camera, such as the RGB image sensor <NUM>, may be turned on. The power-on operation may be made by a local user and/or may occur automatically by a controller of the device. Once the RGB camera is on, at <NUM> a preliminary face recognition operation is performed using a signal from the RGB camera. Thereafter, at <NUM> a readout mode is selected by the local user and/or the controller of the device. The readout mode may be any one of the normal mode, the thinning modes, the binning modes, the IQ mosaic modes, the mdm modes, or the hybrid modes described above. At <NUM>, the device generates a depth map. At <NUM>, the device performs a face recognition operation. The face recognition operation may utilize inputs from the RGB camera (such as the result of the preliminary face recognition operation) and the TOF camera (such as the depth map). Additionally or alternatively, a gesture recognition may be performed. In some aspects of the present disclosure, the exemplary imaging method may only generate a depth map (and not perform a recognition operation) or may only perform a recognition operation (and not generate a full depth map).

All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as "a," "the," "said," etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

Claim 1:
A time-of-flight sensor (<NUM>), comprising:
a pixel array (<NUM>) including a plurality of pixel circuits (<NUM>, 300a, 300b, <NUM>) arranged in an array, wherein a first column of the array includes:
a first pixel circuit (<NUM>, 300a, <NUM>) including a first photodiode (<NUM>), a first capacitor coupled to the first photodiode (<NUM>), and a second capacitor coupled to the first photodiode (<NUM>), and
a second pixel circuit (<NUM>, 300b, <NUM>) including a second photodiode (<NUM>), a third capacitor coupled to the second photodiode (<NUM>), and a fourth capacitor coupled to the second photodiode (<NUM>);
a first signal line (213a, 308a, 420a) coupled to the first capacitor;
a second signal line (213b, 308b, 420b) coupled to the second capacitor;
a third signal line (213c, 308c, 420c) coupled to the third capacitor;
a fourth signal line (213d, 308d, 420d) coupled to the fourth capacitor;
a first switch circuitry (<NUM>, <NUM>, <NUM>);
a second switch circuitry (<NUM>, <NUM>, <NUM>);
a first comparator (<NUM>, <NUM>) coupled to the first signal line (213a, 308a, 420a) and the third signal line (213c, 308c, 420c) through the first switch circuitry (<NUM>, <NUM>, <NUM>); and
a second comparator (<NUM>, <NUM>) coupled to the second signal line (213b, 308b, 420b) and the fourth signal line (213d, 308d, 420d) through the second switch circuitry (<NUM>, <NUM>, <NUM>).