Imaging sensor with shared pixel readout circuitry

Imaging sensors that detect infrared and visible light are provided herein. In one example, an imaging sensor is presented that includes a semiconductor substrate comprising an array of pixel structures for concurrently sensing infrared light and visible light. Each of the pixel structures include a first pixel element configured to detect the infrared light and a second pixel element configured to detect the visible light. Each of the pixel structures further include a shared output circuit that couples the first pixel element and the second pixel element such that a first output state presents a first signal corresponding to detected infrared light of the first pixel element and a second output state presents a second signal corresponding to detected visible light of the second pixel element.

BACKGROUND

Digital imaging sensors are employed in many devices and systems to capture images, such as in digital cameras. Imaging sensors employ large semiconductor arrays of detection pixels that can comprise charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices, among others. The imaging sensors can be configured to capture a range of the electromagnetic spectrum that spans both visible light and infrared light ranges.

When configured to capture infrared light, the imaging sensors can be employed in time-of-flight (TOF) camera systems. TOF cameras measure a depth of a scene using emission of infrared light that is precisely timed to measurement or detection by an imaging sensor. These TOF cameras can be employed in many applications where identifying relative depths among objects in a scene is useful, such as interactive gaming devices, virtual reality devices, augmented reality devices, industrial controls, medical scanners, or other devices.

Overview

Systems, apparatuses, and methods that employ imaging sensors to detect infrared and visible light are provided herein, such as time-of-flight (TOF) measurement devices and associated imaging sensor arrays. In one example, an imaging sensor is presented that includes a semiconductor substrate comprising an array of interspersed pixel structures for concurrently sensing infrared light and visible light. Each of the pixel structures include at least a first pixel element configured to detect the infrared light and at least a second pixel element configured to detect the visible light. Each of the pixel structures further include a shared output circuit that couples at least the first pixel element and at least the second pixel element such that a first output state presents a first signal corresponding to detected infrared light of the first pixel element and a second output state presents a second signal corresponding to detected visible light of the second pixel element.

TECHNICAL DISCLOSURE

Time-of-flight (TOF) based three-dimensional (3D) cameras have found several applications in industrial automation, medical imaging, automotive driving assistance, virtual reality systems, augmented reality systems, as well as gaming and other consumer areas. TOF sensors can deliver depth information of 3D images by using active illumination for measurement. Many times, the maximum depth that can be detected by a TOF system is limited by infrared filtering components which can reduce sensitivity of the infrared sensor and attenuate detected intensity of the infrared illumination system. Visible imaging sensors can be employed to augment the infrared imaging to overcome some of the limitations on depth of field with infrared imaging sensors. Although a TOF sensor can provide monochromatic two-dimensional (2D) images using common mode outputs, the 2D images can be depth-restricted by associated infrared (IR) filter components when objects in a scene are at a far distance.

In some examples, a separate black-and-white or red/green/blue (RGB) visible-spectrum camera is included, which can lead to bulky imaging equipment as well as increased manufacturing cost and system power consumption. In some instances, two sets of separated imaging sensors are included on the same sensor array, which can increase the architectural complexity of the pixel array. Some systems have combined infrared imaging sensors with visible imaging sensors into a single device or onto a silicon wafer used in a microchip. However, separation distances of the two imaging sensors can lead to parallax issues. Moreover, these devices still encounter problems with high power consumption and sensitivity of the individual pixels. When separate imaging sensors are employed, in can be difficult to ensure accurate timing between capture of visible and infrared images, which can cause problems during image processing and TOF calculations.

In the examples herein, various enhanced imaging sensors and pixel arrangements are discussed which can be employed in TOF camera systems, among other imaging applications. As discussed below, a pixel array architecture and timing method are discussed to overcome problems by using a single pixel arrangement to detect both passive 2D images (RGB or BW) as well as active 3D images (i.e. TOF), which enhances the spatial resolution of an imaging sensor and also reduces system cost and power consumption.

Visible light and infrared (IR) light are discussed herein. Visible light typically comprises wavelengths of light that correspond to the visual range of a human eye, approximately wavelengths 390 nanometers (nm) to 700 nm. IR light comprises wavelengths of light that extend from approximately 700 nanometers to 1 millimeter (mm). Variations of wavelength ranges are possible, but in general visible light and IR light discussed herein refer to the above approximate ranges.

As a first example,FIG. 1is presented.FIG. 1is a system diagram illustrating TOF camera environment100. Environment100includes time of flight camera system110and scene elements101-102. A detailed view of TOF camera system110is shown including infrared emitter120, combined IR/visible sensor121, and TOF processing circuitry122mounted on one or more circuit boards123. TOF camera system110communicates with external systems over communication link125. In some examples, elements of IR emitter120and image processing circuitry are included in sensor121.

In operation, TOF camera system110emits IR light111using IR emitter120to illuminate elements in scene103, such as scene elements101-102. IR light111reflects off objects and elements in scene113and is received as reflected IR light112by sensor121. Sensor121detects reflected IR light112as well as objects and elements in the scene illuminated by ambient light113. Sensor121can detect both IR light and visible light using an array of pixels that are interspersed with each other on a semiconductor substrate of sensor121.

Once the IR and visible light is detected by sensor121, pixel data representative of this detected light is provided to TOF processing circuitry122which processes the pixel data to determine one or more images, with at least one of the images comprising a depth map of the scene resultant from IR illumination and another other the images comprising a passive visible image resultant from ambient light113. Each pixel in sensor121can have an associated filtering element to allow detection of either IR light or selective portions of the visible light, which will be discussed in more detail below.

Referring back to the elements ofFIG. 1, IR emitter120can comprise one or more infrared light emitters, such as light-emitting diodes (LEDs), laser emitters, laser diode emitters, or other components. IR emitter120can also include various driver circuitry configured to provide power to IR emitter120and synchronize emission of IR light with timing signals provided by TOF processing circuitry122.

Sensor121comprises an array of pixels formed on a semiconductor substrate, along with associated driver, power, and output circuitry. The individual pixels can incorporate techniques and semiconductor structures found in CCD pixels or CMOS pixels, among other semiconductor-based light detection techniques and elements. Further examples of sensor121will be discussed inFIGS. 3-13herein.

Link125comprises one or more wired or wireless communication links for communicating with external systems, such as computing devices, microprocessors, servers, network devices, smartphone devices, or other processing systems. Link125can carry imaging data and related data, such determined by TOF camera system110, or can carry commands and instructions transferred by an external control system. Link125can comprise a Universal Serial Bus (USB) interface, Peripheral Component Interconnect Express (PCIe) interface, wireless interface, IEEE 802.15 (Bluetooth) wireless link, IEEE 802.11 (WiFi) wireless link, Direct Media Interface (DMI), Ethernet interface, networking interface, serial interface, parallel data interface, or other communication or data interface, including combinations, variations, and improvements thereof.

To further illustrate the elements ofFIG. 1and provide a detailed view of one example TOF camera system,FIG. 2is presented.FIG. 2is a block diagram illustrating TOF sensing system200, which can be an example of any of the TOF systems discussed herein. Elements of system200can be incorporated into elements of TOF camera system110.FIG. 2includes object of interest201within a scene which is imaged by system200to identify TOF information for at least object201and provide this information to external system250over communication link251. The TOF information, such as a TOF signal, can comprise a signal proportional to a phase shift between infrared light pulses detected and a reference signal. The TOF signal can be used to determine distances to objects in a scene, such as object201, from which infrared light was reflected.

System200includes IR emitter210, radio frequency (RF) modulator211, controller212, optics220, sensor221, and phase module222. RF modulator211comprises a system oscillator that generates RF modulation signal240and is controlled by controller212over link231. RF modulation signal240is provided to IR emitter210over link230for emission as IR light203. Emitted IR light203is modulated according to RF modulation signal240by IR emitter210, and illuminates object201.

Experiencing a time-of-flight time delay, the back scattered reflected IR light203is received by optics220and provided via optical path236onto sensor221. Sensor221includes at least one pixel or one array of pixels. RF modulator211simultaneously transfers a reference signal as RF modulation signal240over link232to phase module222. Phase module222is controlled by controller212over link234. Phase module222shifts the phase of signal240generated by RF modulator211and transmits the phase-shifted signal over link233to infrared pixels used for TOF sensing in sensor array221. This phase-shifted signal can be used in performing demodulation/phase processes discussed in further examples below. Sensor221can simultaneously detect both IR light and visible light. Visible light detection is provided by visible light source202, which in some examples comprises ambient light.

Turning to the elements ofFIG. 2, IR emitter210can comprise a light-emitting diode, diode laser, or other IR light emitter which can be modulated according to RF modulation signal240. RF modulator211comprises various circuitry to generate an RF modulated signal based on control instructions from controller212. RF modulator211can include crystal oscillators, clock generation circuitry, phase-locked loop (PLL) circuitry, or other modulation circuitry. Phase module222comprises a phase comparator circuit which can produce phase shifts between RF modulation signal240and a signal sent over link233from sensor221for use in determining a time-of-flight (TOF) signal. In some examples, RF modulator211and phase module222are combined into a single circuit module. Sensor221comprises an IR/visible light sensor used for determining TOF information of object201. Sensor221includes elements discussed herein for the various pixel arrays and pixel architectures. Optics220can comprise optical interfacing elements that can pass and focus both visible light and IR light. Optics220can include prisms, optical adhesives, lenses, mirrors, diffusers, optical fibers, and the like, to optically couple incident light onto sensor221. Links230-235can each comprise wired or wireless links to interconnect the associated modules ofFIG. 2. When combined onto one or more printed circuit boards, links230-235can comprise printed circuit traces.

Controller212can include communication interfaces, network interfaces, processing systems, computer systems, microprocessors, storage systems, storage media, or some other processing devices or software systems, and can be distributed among multiple devices. Examples of controller212can include software such as an operating system, logs, databases, utilities, drivers, caching software, networking software, and other software stored on non-transitory computer-readable media. A further example of controller212is shown inFIG. 15. External system250can comprise a network device, computing device, gaming platform, virtual reality system, augmented reality system, or other device, including combinations thereof. System200can also include power supply circuitry and equipment, enclosures, chassis elements, or ventilation/cooling systems, among other elements not shown inFIG. 2for clarity.

FIG. 3illustrates a top view of pixel structure300. Pixel structure300illustrates one pixel ‘pitch’ which includes one or more pixels configured to sense visible light nested in a pitch area of a pixel configured to sense infrared light. Pixel structure300can be employed in an array of pixels to form an image sensor, with a plurality of nested pixel structures forming the array. For example, imaging sensor370is shown inFIG. 3which includes pixel array371and pixel control circuitry372. InFIG. 3, a top view is shown of pixel structure300, which represents a single pixel structure area of imaging sensor370.

The pixels in pixel structure300are configured to sense incident light propagating to the pixel structure from the top and into the figure. This example is referred to as front side illumination (FSI). Other configurations are possible with the pixels configured to sense incident light propagating to the pixel structure from the bottom and out from the figure, referred to as back side illumination (BSI). Any associated filtering layers are positioned between light sources and the pixel, namely on the ‘top’ side in FSI examples, and on the back side in BSI examples.

FIG. 3illustrates semiconductor topology for pixel structure300.FIG. 3also indicates a profile ‘cut’ along A-A′ which is used in the side view illustrations in the figures below. A semiconductor substrate is employed onto which various structures are formed using various lithography fabrication processes, such as etching, deposition, masking, diffusion, ion implantations, and the like. A semiconductor wafer is typically used as the substrate, which in this example is a p-type wafer labeled as311inFIG. 3. Although n-type wafers can be employed, the examples herein will focus on p-type wafers for clarity.

Pixel structure300comprises more than one individual pixel nested within a single IR pixel pitch, with at least one of the individual pixels configured to sense IR light and at least another of the pixels configured to sense visible light. The individual pixels are each individual photodetectors, which can comprise active pixel sensor (CMOS) style pixels, photo sensitive diodes, photo gate diodes, or pined photodiodes, among other photodetectors.

Pixel structure300includes at least two demodulation polysilicon (poly) gates, namely gates340and341, which are used in the sensing of IR light by at least creating potential wells for detecting and integrating infrared light-induced charges. Pixel structure300includes two poly gates, namely gates342and343used in the sensing of visible light by at least creating potential wells for detecting and integrating visible light-induced charges. Associated gate oxide region303is included underneath each poly gate, such as for gates340,341,342, and343, which reside on top of gate oxide region303.

Pixel structure300also includes readout floating n+-diffusion on p-type silicon320-321. During operation, charge from the pixel regions will be dumped or transferred to associated ones of floating diffusions320-321for readout by a shared readout circuit, shown as element350inFIG. 3and highlighted in the subsequent figures. To enable both the visible light pixels and infrared light pixels to share ones of floating diffusions320-321, charge transfer gates330,331,332, and333are included in pixel structure300. Transfer of infrared light-generated charges are controlled by gates331and333. Transfer of visible light-generated charges are controlled by gates330and332. Although two charge transfer gates per pixel pair (e.g. gates330-331or gates332-333) are shown inFIG. 3, in other examples one degenerated gate is shared per pixel pair.

Also shown inFIG. 3is IR bandpass filter301which filters light incident into the infrared pixel regions and acts as an IR light bandpass filter (only a corner of the rectangular filter is illustrated in the overview view). IR bandpass filter301can be deposited as a layer on top of pixel structure300during manufacturing. InFIG. 3, IR bandpass filter301is shown as layered on top of back end oxidation310, and back end oxidation310covers the entirety of pixel structure300. IR bandpass filter301can comprise a bandpass filter matched to IR light wavelengths used in active illumination of a scene, such as matched to an emissions spectrum of emitter210inFIG. 2.

In some examples, red/green/blue (R/G/B or RGB) filter302is employed over each of the visible light pixel regions. RGB filter302can be omitted in some examples. When used, RGB filter302filters light incident into the visible pixel regions and acts as a light bandpass filter for selected wavelengths of light, such as red, green, or blue. In an arrayed structure, such as an imaging sensor, the color of visible light filters can be selected to be alternating among the various pixels to provide for pixels with red filtering, pixels with green filtering, and pixels with blue filtering which can be used to produce a full-color image. In examples where specific color filtering is not desired, RGB filter302can be omitted and greyscale images can be produced. RGB filter302can be deposited as a layer on top of pixel structure300during manufacturing. RGB filter302can be applied to individual pixels to spread to cover more than one pixel, such as to have a single filtering layer cover more than one neighboring visible pixel. The IR bandpass filters and RGB filters can each be interlaced within one single 3D/2D detector pixel pitch region.

When included in array371that forms image sensor370, visible pixels are interspersed with IR/TOF pixels onto a semiconductor substrate, such as a silicon wafer substrate. The visible light pixels are typically smaller in size than the IR light pixels, and can be included in the marginal area proximate to each of the IR light pixels, making for a tight packing of interspersed pixels. This interspersed arrangement uses the marginal area inside of a sensor to collect RGB or gray value information, and thus less or no additional silicon real estate is needed. The shared floating diffusion can also reduce real estate for a pixel array, even with both IR and visible light pixels. This interspersed arrangement also enhances the spatial resolution of the 3D/2D pixels.

The materials and geometries of elements of pixel structure300and imaging sensor370can vary. Various semiconductor fabrication techniques and materials are employed for the pixel structures herein. Typically, the various elements of the pixel structures comprise epitaxial layers of silicon, which can be doped or ion implanted to form various regions. Polysilicon gates are employed and can be deposited by chemical vapor deposition or patterned with photolithography and etched, among other processes. Various oxides can be grown, using thermally grown techniques or other oxide formation processes.

As a further example of pixel structure300,FIG. 4is presented which includes a cross-sectional configuration400of pixel structure300, as cut fromFIG. 3along section lines A-A′. Thus,FIG. 4focuses on only a portion of the entire pixel structure that is shown inFIG. 3, and is labeled as pixel structure300for illustrative purposes in the following figures. InFIG. 4, example readout circuitry350is shown in more detail. Specifically, readout circuitry350includes reset metal-oxide semiconductor (MOS) transistor351with the source terminal of transistor351connected to positive voltage VD351. VDcan comprise a logic-level voltage which presents a reset voltage level to floating diffusion321(which is connected to buffer355) when VRESET354is enabled and allows the input to floating diffusion321and buffer355to be pulled ‘up’ to voltage VDand register as a logic ‘1’ by buffer355. The input of buffer355is connected to the drain of MOS transistor353. Capacitance352and buffer355covert the IR light/TOF charges and visible light charges to voltages at buffer output node356. Each pixel capture cycle is reset by the VRESETwhich is clocked by a control circuit212, or other control circuitry.

FIG. 5illustrates configuration500which includes similar features asFIG. 4, and adds detail in for three optional features—split floating diffusions, pixel separation features, and back-side cavities. Pixel structure501ofFIG. 5also includes similar elements as pixel structure300, as well as the optional features discussed below.

In a first feature example, an optional split diffusion arrangement formed from floating diffusions320-321. In this example, floating diffusions320-321are electrically connected, such as by metal interconnect, and thus present a common electrical potential. Floating diffusion321is located proximate to a first set of IR/visible light pixels, such as shown inFIG. 3as being positioned near gate330and331. Floating diffusion320is located proximate to a second set of IR/visible light pixels, such as shown inFIG. 3as being positioned near gate332and333. Each of the individual floating diffusions320-321can receive charge from the nearby pixel potential wells when the associated charge transfer gates330,331,332, and333are enabled to allow dumping of charge onto the associated floating diffusions320-321.

FIG. 5also illustrates a pixel separation feature502to isolate adjacent pixel structures from one another and substantially reduce charge migration to adjacent pixel structures of a pixel array. This substantial reduction in charge migration can include complete prevention of charge migration, or might instead include an inhibition of charge migration or other attenuations in charge migration, where complete charge migration might not be prevented in some cases. The magnitude of charge migration inhibition can vary based on the desired performance, material properties, or induced potential well levels, among other factors. In some examples, feature502comprises a cavity or etched out region which physically separates pixels from neighboring pixels. In other examples, a poly gate arrangement is employed which places a separation gate similar to charge transfer gates330,331,332, and333between each pixel, where the separation gates are configured to provide a potential barrier between neighboring pixels. In some examples, feature502can be omitted, when charge migration to neighboring pixels is mitigated by other features or techniques.

FIG. 5also shows illustrates back-side cavity503. In most of the examples herein, front side illumination (FSI) techniques are employed where light is incident from ‘above’ inFIG. 5. However, back side illumination (BSI) techniques can also be employed where light is incident from ‘below’ inFIG. 5. IR light can penetrate to an acceptable depth through p-type silicon wafer311inFIG. 5. However, visible light attenuates quickly in silicon wafer311and acceptable light levels might not reach the pixel region that captures visible light. Cavity503can be provided which reduces a depth or thickness of silicon wafer311at the visible pixel region and allows penetration of light. In some examples, silicon wafer311is about 7 micrometers thick, and the cavity provides a locally reduced thickness of about 3 micrometers or less. Various semiconductor fabrication techniques can be employed to form cavity503, such as photoresist/etching techniques. It should be noted that IR bandpass filter301and optional RGB filter302would be layered onto the ‘bottom’ side of silicon wafer311in BSI examples instead of the ‘top’ side as shown inFIG. 5.

FIG. 6illustrates configuration600which includes similar features asFIG. 4, and adds detail in for electrical connections of the various elements. Specifically,FIG. 6illustrates details of connecting to electrical sources and the timing clock diagram inFIG. 7. In the examples herein, various high/low voltage levels can be employed, which can correspond to logic-level voltages. However, for the examples inFIGS. 6-13, a low voltage is considered approximately 0.5V and a high voltage is considered approximately 2.0V. The terms ‘voltage’ and ‘potential’ can also be used interchangeably.

Turning first to the pixel elements that comprise the IR or TOF sensing portions of pixel structure300, poly gates340-341(found inFIG. 3) are employed as demodulation gates to produce TOF information based on IR light and associated timing signals provided to gates340-341. InFIG. 6, only gate340is shown for clarity, but gate341is also employed in this demodulation technique. Gate340is coupled to associated phase clock signal720at node613. Gate341has a similar connection node, although not shown inFIG. 6, gate341is coupled to associated phase clock signal730at an associated node. Specifically, gate340is driven by signal Vma720inFIG. 7and gate341is driven by signal Vmb730inFIG. 7. By driving gates340-341in this manner, demodulation occurs which provides charge representative of a TOF signal which can be coupled to an associated floating diffusion, as will be discussed in more detail below. Charge separation/transfer gates331and333are connected to bias voltage VTX1740, such as at node612for gate331inFIG. 6.

The visible sensor portion can comprise a photo diode, e.g., a photo gate diode332, which is connected to a controlling voltage source Vpat node610. Charge separation/transfer gates330and332are connected to bias voltage VTX2750, such as at node611for gate330inFIG. 6. Both the IR light pixel structure and the visible light pixel structure share the same output floating diffusion321and the same readout circuitry350.

As mentioned above,FIG. 7is a timing diagram which illustrates signaling employed to drive pixel structure300, among other pixel structures found herein. To perform the TOF sensing and visible light sensing concurrently, pixel structure300is operated by the timing diagram inFIG. 7. The timing diagram is characterized with five (5) operational stages within one image capture cycle, which are related to pixel structure300operation explained below. Operational stages I-V complete one cycle of visible light and IR light detection/sensing processes all in one pixel structure. In typical examples, an array comprised of the pixels perform 3D/2D measurements with independent responses to active IR illumination and passive visible light at different optical spectra domains.

FIG. 8illustrates a reset process800for pixel structure300. Stage I ofFIG. 7corresponds to reset process800inFIG. 8. Both IR and visible light pixel charges are reset in operational stage I, which inFIG. 7occurs between time T0and T1. The surface potential well configuration underneath all the poly gates is shown inFIG. 8, which sets up a free charge channel path performing the reset process. The polarity of each potential well is indicated with potential gauge820which indicates relative potentials from 0 to +Ψ. It should be noted that the surface potential is denoted by Ψ which is typically not equal to a voltage applied to a corresponding gate due to ‘flat band’ conditions found in typical MOS structures.

When Vresetof the MOS transistor gate is held at a high potential, both visible light generated charges and IR light generated charges will be drained away to the voltage source VD351, thus resetting the floating diffusion321to the voltage VD. During the reset process, the TOF demodulation clock provided by Vma720and Vmb730can be activated or instead be set to the same level. The timing diagram inFIG. 7shows a one clock cycle reset process, although the reset process can occur for any longer amount of time.

To allow visible light generated charges and IR light generated charges to be drained from floating diffusion321to the voltage VD, transfer gates330and331can both be enabled which allows charge in each pixel region to be dumped onto floating diffusion321. In some examples, transfer gates330and331are both enabled, while in other examples, transfer gates330and331are alternatingly enabled. For visible light generated charges,FIG. 8shows transfer gate330as enabled, as noted in VTX2740inFIG. 7, which brings potential underneath gate330to potential ΨTX2831, which is similar in potential to Ψp830established by Vp760underneath gate332. This configuration drains charge from below gate332to floating diffusion321and resets the visible pixel region. To reset the IR pixel region underneath gate340, a similar process is followed but instead transfer gate331is enabled which brings potential ΨTX2831underneath gate331to a similar potential as Ψma834established by Vma720underneath gate340. This configuration drains charge from below gate340to floating diffusion321and resets the infrared pixel region. A similar process can be followed for regions underneath gates341and343which have a similar operation but share a different transfer gate320.

It should be noted that potential Ψma835is shown inFIGS. 8-13to illustrate potential/voltage associated with gate341. Gate341is controlled by signal Vmb730inFIG. 7. When gate340and gate341are driven to opposite potentials as seen inFIG. 7, demodulation of charges generated by modulated IR light is performed.

FIG. 9illustrates configuration900which includes pixel structure300ready to receiving light after a reset process.FIG. 9illustrates the operational stages at time T1inFIG. 7. InFIG. 9, both transfer gates330-331are disabled which inhibit any light generated charges accumulated in potential wells underneath gates332and340from transferring or dumping onto floating diffusion321. Specifically,FIG. 9shows potential ΨTX2831and ΨTX1833as creating potential barriers which inhibits charge propagation from the associated potential wells ΨP830and Ψma834to floating diffusion321.

FIG. 10illustrates configuration1000which shows pixel structure300detecting light by visible light pixel360and IR light pixel361.FIG. 10corresponds to operational stage II inFIG. 7, between times T1and T2. InFIG. 10, both visible light and infrared light (VS+IR)1060are received into both pixels at the same time. IR light is emitted from an emitter device, not shown inFIG. 10, but can be similar to that seen inFIGS. 1 and 2, which illuminates a scene and objects within the scene. Ambient light and active light sources can also provide visible light. RGB filter302is positioned over visible light pixel360and IR bandpass filter301is positioned over IR light pixel361. These filters act as bandpass filters which allow selected wavelengths of light to pass while blocking/absorbing unwanted wavelengths. Thus, IR bandpass filter301allows IR wavelengths to pass, while blocking visible wavelengths. Conversely, RGB filter302(if applied) allows selected red, green, or blue visible wavelengths to pass, while blocking IR wavelengths. If RGB filter302is not applied, the sensor delivers a gray value. This light, after selective filtering, reaches the potential wells created for pixel360and pixel361.

After completion of the reset process for IR charges, the voltage VTX2611is applied to gate330is set to low voltage, as seen inFIG. 7VTX2750. The surface potential ΨTX2831underneath the gate is lower than ΨP830which acts as a ‘valve’ and closes the passage for the visible light-generated charge to reach at floating diffusion321. Therefore visible light-generated charge1042is isolated and resided in the potential well ΨP830underneath the photo gate332by keeping the gate voltage Vp610high. On the other side of the pixel structure, for pixel361, IR generated charges1040and1041are in associated potential wells Ψma834, Ψmb835induced by clock signals Vma, Vmb613(i.e.720and730inFIG. 7) are continuously demodulated and transferred to floating diffusion321via a channel under transfer gate331with potential ΨTX1833under a gate bias condition VTX1612. The pixel structure detects the visible signal and TOF signal and integrates the photo charges concurrently; the visible light generated charges are storage underneath the gate332, while the IR generated charges underneath gate340are ready to be dumped from floating diffusion321.

FIG. 11illustrates the IR dump and IR readout processes. This process is indicated inFIG. 7as operational stage III between times T2and T3. Demodulation clock voltage signals Vma720and Vmb730are held low in operational stage III. The charges in potential wells Ψma834and Ψmb835will be dumped over to floating diffusion321, which is converted to voltage by the readout unit350to a corresponding voltage value at output terminal356. Any active IR light can then be disabled to save power after this stage completes.

FIG. 12illustrates the IR reset process. This process is indicated inFIG. 7as operational stage IV between times T3and T4. After the IR dump and IR readout of the previous stage, IR charges1040and1041(FIG. 10) are reset by switching on MOS transistor353, referred to inFIG. 7for Vreset710. Floating diffusion321is reset to the voltage value VD351, creating the potential well configuration shown inFIG. 12at location832. Meanwhile, the visible light-generated charges1042are still isolated and remain in potential well ΨP830ready to be dumped and read out.

FIG. 13illustrates the VS (visible) dump and VS readout process. This process is indicated inFIG. 7as operational stage V between times T4and T5. Photo gate voltage Vp610(760) is clocked to a lower voltage level, e.g. Vp≤VTX2, and the corresponding potential well ΨP830dumps visible light charges1042to floating diffusion321via channel with potential ΨTX2831induced under transfer gate330by VTX2611. The visible light-generated charge is then converted by the same readout unit350to a corresponding voltage value at output terminal356.

The operations ofFIG. 7can then be repeated for continued detection and readout of visible and IR light. Advantageously, each pixel360-361detects associated light simultaneously and thus any movement in objects in the scene, movement of the imaging sensor, or changes in aspect of the imaging sensor are do not result in temporal image artifacts between 3D depth images and 2D images. The IR light-generated charge is read out as a corresponding voltage while the visible light-generated charge waits in a potential well to be later dumped off to floating diffusion321using transfer gate330as a charge valve. This has the technical effect of reducing the number of components for an imaging sensor and associated pixels. Also, the technical effect of reduced power consumption is achieved.

The examples herein provide for a single combined pixel architecture with both 3D (IR, TOF) and 2D (RGB or BW). This pixel arrangement is formed on a semiconductor substrate, and comprises a first pixel configured to sense infrared light, a second pixel configured to sense visible light, and an output element shared by the first pixel and the second pixel, where a first output state of the output element presents a first signal corresponding to detected infrared light of the first pixel and a second output state presents a second signal corresponding to detected visible light of the second pixel.

When included in an array that forms an image sensor, visible pixels are interspersed with IR/TOF pixels onto semiconductor substrate, such as a silicon wafer substrate. The visible light pixels are typically smaller in size than the IR light pixels, and can be included in the marginal area proximate to each of the IR light pixels, making for a tight packing of interspersed pixels. This interspersed arrangement uses the marginal area inside of a TOF sensor to collect RGB or gray value information, and thus less additional silicon real estate needed. The shared floating diffusions can also reduce real estate for a pixel array, even with both IR and visible light pixels. The IR bandpass filters and RGB filters can each be interlaced within one single 3D/2D detector pixel pitch region. This interspersed arrangement also enhances the spatial resolution of the 3D/2D pixels.

It should be noted that the shared floating diffusion with transfer gates architecture discussed herein can be applied to imaging sensors that employ other types of pixels. For example, if the imaging sensor does not include IR light pixels, then pairs of visible light pixels can share the same readout structure (i.e. a shared floating diffusion with the transfer gates). Likewise, if the imaging sensor does not include visible light pixels, then pairs of IR light pixels can share the same readout structure.

FIG. 14is provided to provide an additional example method1400for operation of a pixel structure or pixel array. The operations of method1400can include similar processes as found in the discussion surroundingFIG. 7, although variations are possible. The operations ofFIG. 14are also discussed in the context of pixel control circuitry372and pixel structure300ofFIG. 3. However, the operations ofFIG. 14can be executed by any control module employed herein for control of other pixel structures or pixel arrays, such as TOF processing circuitry122ofFIG. 1, controller212ofFIG. 2, pixel control circuitry372ofFIG. 3, and computing system1501inFIG. 15.

InFIG. 14, circuitry372resets (1401) pixel areas for measurement of light. InFIG. 8, Vresetis enabled which turns on transistor353and pulls node357to VD. Floating diffusion321is then pulled ‘high’ to VDand any charge remaining on floating diffusion321is drained to VD. Pixel structure300senses (1402) light in pixel areas360-361, keeping charge confined in pixel areas360-361. Transfer gates330-331held to a low potential voltage during the period of light collection to act as barriers to charge movement to shared floating diffusion321, and any accumulated charge is held in associated potential wells Ψp830and Ψma834by created by gates342and340. Pixel structure300first dumps (1402) IR-generated charge to shared floating diffusion321for readout by readout circuit350. Gate340is brought to a low potential, which creates a channel for charge accumulated in pixel361to flow to shared floating diffusion321, which is converted into a voltage by buffer355for readout on node356. Circuitry372resets (1403) shared floating diffusion321after IR light-generated charges are dumped and read out. Vresetis enabled which turns on transistor353and pulls node357to VD. Floating diffusion321is then pulled ‘high’ to VDand any charge remaining on floating diffusion321is drained to VD. Pixel structure300then dumps (1404) visible (VS) light-generated charge to shared floating diffusion321for readout by readout circuit350. Gate342is brought to a low potential, which creates a channel for charge accumulated in pixel360to flow to shared floating diffusion321, which is converted into a voltage by buffer355for readout on node356. The operations in method1400can be repeated as necessary to detect and sense TOF/RGB data in a cyclical fashion, or to provide for further imaging processes.

Thus, IR pixel361is read out first, and VS pixel360is read out second, creating a time-multiplexed readout operation for pixel structure300. It should be noted that although the IR pixel is read out first in the above examples, other examples can have the VS pixel read out first. Once the pixel voltages are presented on output node356, these voltages are transferred to an image processing circuit for conversion into an image and into TOF data, such as a depth map image. Typically, an array of pixels will be employed, and the pixel array will be read out using the time-multiplexed IR/RGB process for each pixel structure. A 3D image can be formed based on the TOF information measured using the IR pixel data and a 2D image can be formed based on the RGB pixel data (or greyscale if applicable).

FIG. 15illustrates controller1500that is representative of any system or collection of systems in which the various time-of-flight detection, pixel control, pixel timing, and image processing operational architectures, scenarios, and processes disclosed herein may be implemented. For example, controller1500can be employed in TOF processing circuitry122ofFIG. 1, controller212ofFIG. 2, or pixel control circuitry372ofFIG. 3. Examples of controller1500can be incorporated into further devices and systems, such as virtual reality devices, augmented reality devices, gaming machines, camera devices, TOF cameras, smart phones, laptop computers, tablet computers, desktop computers, servers, cloud computing platforms, hybrid computers, virtual machines, smart televisions, smart watches and other wearable devices, as well as any variation or combination thereof.

Controller1500may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. For example, controller1500can comprise one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGA), or discrete logic and associated circuitry, including combinations thereof. Although not shown inFIG. 15, controller1500can include communication interfaces, network interfaces, user interfaces, and other elements for communicating with a host system over communication link1520. Computing system1501may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Controller1500can also comprise one or more microcontrollers or microprocessors with software or firmware included on computer-readable storage media devices. If software or firmware is employed, the computer-readable storage media devices may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

Controller1500includes various controller portions to enhance time-of-flight sensing, namely emitter controller1510, pixel array controller1511, and optional image processor1512. Emitter controller1510provides timing of emission of IR light to be synchronized with measurement of IR light by IR pixels, and typically operates in conjunction with pixel array controller1511. In some examples, emitter controller1510provides RF modulator control signaling to indicate an RF modulation frequency and phase to RF modulator circuitry and to pixel array controller1511. Pixel array controller1511provides pixel control signaling to control the pixel structures discussed herein, whether the pixels are individual pixels or included in an array of pixels. Specifically, pixel array controller1511provides for resetting IR pixel areas for measurement of light, controlling transfer gates to transfer charge to shared floating diffusions, and time multiplexing readout of IR/VS pixels, among other operations. Pixel array controller1511provides for receiving pixel readout and providing pixel readout information to optional image processor1512. Image processor1512provides for accumulating pixel data for an array of pixels to create 3D and 2D images and providing associated TOF information or 3D/2D image data to a host system over communication link1520. Image processor1512also processes TOF information generated by IR pixels and RGB/greyscale information generated by VS pixels to form 3D digital images, such as depth map digital images, and form 2D digital images, such as visual light images, among other operations. When image processor1512is omitted, pixel array controller1511can provide pixel readout data to a host system over communication link1520. In some examples, pixel array controller1511controls or includes an analog-to-digital conversion circuit to convert pixel readout signals to digital formats.

Certain inventive aspects may be appreciated from the foregoing disclosure, of which the following are various examples.

An imaging sensor, comprising a semiconductor substrate comprising an array of pixel structures for concurrently sensing infrared light and visible light, each of the pixel structures comprising a first pixel element configured to detect the infrared light and a second pixel element configured to detect the visible light, each of the pixel structures further comprising a shared output circuit that couples the first pixel element and the second pixel element such that a first output state presents a first signal corresponding to detected infrared light of the first pixel element and a second output state presents a second signal corresponding to detected visible light of the second pixel element.

The sensor of Example 1, comprising the shared output circuit of each of the pixel structures comprising a floating diffusion element configured to receive charge from the first pixel element or the second pixel element based at least on a selection state of the shared output circuit.

The sensor of Examples 1-2, comprising based at least on the shared output circuit being in the first output state, a readout circuit communicatively coupled to the floating diffusion element is configured to dump first charge representative of the detected infrared light to the floating diffusion element and convert the first charge to a first voltage. Based at least on the shared output circuit being in the second output state, the readout circuit is configured to dump second charge representative of the detected visible light to the floating diffusion element and convert the second charge to a second voltage.

The sensor of Examples 1-3, comprising based at least on the shared output circuit being in the first output state, the charge representative of the detected visible light is inhibited from dumping to the floating diffusion element by a first potential barrier proximate to the first pixel element that is established by the shared output circuit. Based at least on the shared output circuit being in the second output state, the charge representative of the detected infrared light is inhibited from dumping to the floating diffusion element by a second potential barrier proximate to the second pixel element that is established by the shared output circuit.

The sensor of Examples 1-4, comprising the shared output circuit of each of the pixel structures comprises a first transfer element configured to create a potential barrier proximate to the first pixel element and inhibit dumping of charge to the floating diffusion element from the first pixel element when the shared output circuit is in the second output state. The shared output circuit comprises a second transfer element configured to create a potential barrier proximate to the second pixel element and inhibit dumping of charge to the floating diffusion element from the second pixel element when the shared output circuit is in the first output state.

The sensor of Examples 1-5, comprising for each of the pixel structures, the first pixel element configured to detect the infrared light and hold charge representative of the detected infrared light until the first output state, and the second pixel element configured to detect the visible light concurrent with the first pixel element detecting the infrared light and hold charge representative of the detected visible light until the second output state.

The sensor of Examples 1-6, comprising for each of the pixel structures, the first pixel element having an infrared light bandpass filter, and the second pixel element having at least one of a red, green, or blue light bandpass filter.

The sensor of Examples 1-7, comprising each of the pixel structures comprising the second pixel element positioned in a marginal area proximate to the first pixel element on the semiconductor substrate.

The sensor of Examples 1-8, comprising for each of the pixel structures, the first pixel element configured to detect the infrared light through the semiconductor substrate, and the second pixel element configured to detect the visible light through a cavity in the semiconductor substrate that reduces a depth of the semiconductor substrate through which the visible light travels to reach the second pixel element

A time-of-flight (TOF) sensor apparatus, comprising an emitter configured to emit infrared light onto a scene for detection by an imaging sensor. The imaging sensor comprising an array of pixels for concurrently sensing depth values for the scene and visible light intensity for the scene, with sets of the pixels of the imaging sensor each having a shared floating diffusion element in an output circuit configured to multiplex pixel output among the pixels of the associated set. A processing circuit configured to process at least the pixel output for each of the sets of the pixels to provide image data indicating the depth values of the scene and the visible light intensity of the scene.

The sensor apparatus of Example 10, comprising the output circuit of each of the sets comprising the shared floating diffusion element configured to receive first charge from at least a first pixel representative of the infrared light detected by at least the first pixel and receive second charge from at least a second pixel representative of the visible light detected by at least the second pixel.

The sensor apparatus of Examples 10-11, comprising the output circuit of each of the sets comprising a first transfer gate that when in a first output mode, is configured to allow the first charge to be transferred to the shared floating diffusion element, and when in a second output mode is configured to inhibit the first charge from being transferred to the shared floating diffusion element. The output circuit of each of the sets comprising a second transfer gate that when in the second output mode, is configured to allow the second charge to be transferred to the shared floating diffusion element, and when in the first output mode is configured to inhibit the second charge from being transferred to the shared floating diffusion element.

The sensor apparatus of Examples 10-12, wherein the first output mode comprises the first transfer gate at a first potential level, the second transfer gate at the first potential level, a gate of at least an associated infrared light pixel at the first potential level, and the shared floating diffusion element having been reset to a second potential level higher than the first potential level, and wherein the first output mode comprises the first transfer gate at a first potential level, the second transfer gate at the first potential level, a gate of at least an associated visible light pixel at the first potential level, and the shared floating diffusion element having been reset to the second potential level.

The sensor apparatus of Examples 10-13, comprising ones of the pixels for sensing the depth values comprising infrared light bandpass filter elements, and ones of the pixels for sensing the visible light intensity comprising at least one of red, green, or blue light bandpass filter elements.

The sensor apparatus of Examples 10-14, comprising ones of the pixels for sensing the depth values interspersed with ones of the pixels for sensing the visible light intensity, with the ones of the pixels for sensing the visible light intensity nested within marginal areas of associated ones of the pixels for sensing the depth values.

A method of operating a pixel arrangement formed on a semiconductor substrate, the method comprising receiving first light in a first pixel, and receiving second light in a second pixel concurrent with the first pixel receiving the first light. The method includes in a first output state, transferring first light-generated charge from the first pixel to a shared floating diffusion element for readout as a first voltage level, and in a second output state, transferring second light-generated charge from the second pixel to the shared floating diffusion element for readout as a second voltage level.

The method of Example 16, further comprising in the first output state, inhibiting transfer of the second light-generated charge from the second pixel to the shared floating diffusion element.

The method of Examples 16-17, further comprising resetting the shared floating diffusion element after readout of the first light-generated charge from the first pixel, and resetting the shared floating diffusion element after readout of the second light-generated charge from the second pixel.

The method of Examples 16-18, wherein the shared floating diffusion element comprises a floating diffusion region of the semiconductor substrate configured to receive charges from the first pixel or the second pixel based at least on potentials formed by transfer gates associated with each of the first pixel and the second pixel.

The method of Examples 16-19, further comprising in the first output state, inhibiting transfer of the second light-generated charge from the second pixel to the floating diffusion region using at least a first potential barrier formed by at least a first of the transfer gates associated with the second pixel, and in the second output state, inhibiting transfer of the first light-generated charge from the first pixel to the floating diffusion region using at least a second potential barrier formed by at least a second of the transfer gates associated with the first pixel.