CMOS active pixel sensor with improved dynamic range and method of operation for object motion detection

A CMOS imaging array includes a plurality of individual pixels arranged in rows and columns. Each pixel is constructed the same and includes a photodetector (e.g., photodiode) receiving incident light and generating an output. A first, relatively lower gain, wide dynamic range amplifier circuit is provided responsive to the output of the photodetector. The first circuit is optimized for a linear response to high light level input signals. A second, relatively higher gain, lower dynamic range amplifier circuit is also provided which is responsive to the output of the photodetector. The second circuit is optimized to provide a high signal to noise ratio for low light level input signals. Subtraction of output signals from the amplifier circuits, preferably after gain compensation, can be used to detect moving objects in the scene where the integration times for the two circuits are not the same, e.g., are staggered in time or have a different period.

BACKGROUND OF THE INVENTION

A. Field of the Invention

The present invention relates to solid state electronic image sensors and in particular to methods for detecting a moving object in a scene using exposures of an image sensor. The methods are particularly suitable for application to airborne imaging systems, such as military aerial reconnaissance image sensors.

B. Description of Related Art

Image sensors for capturing electronic representations of images in the visible wavelengths have been around since about 1970, when the silicon charge coupled device (CCD) was invented. Over the years, CCDs have become the standard for visible spectrum electronic imaging, replacing film in most applications. Various architectures have been developed to make CCD operation more effective for the specific application. Such architectures include the full frame, frame transfer, and interline transfer CCD.

The processes related to silicon CCD manufacturing have always been reasonably expensive and complex, with the implementation technology of choice being N type metal oxide (NMOS). The associated costs and operational characteristics of NMOS CCDs, and later P type (PMOS)CCDs, eventually persuaded designers to seek an image sensor solution using the more easily manufactured Complimentary Metal Oxide Semiconductor (CMOS) technology. CMOS had become the technology of choice utilized by most foundries for making computer integrated circuits.

The advantages of using CMOS for image sensors included having a wider selection of foundries, lower manufacturing costs, lower power/voltage operation, and especially the ability to add peripheral CMOS circuitry on the image sensor substrate, the later feature being attractive for developing an image sensor on a single chip. CMOS imagers therefore could have active circuitry added within the picture sensing element structure (pixel). These imagers became known as Active Pixel Sensors (APS). The APS CMOS image sensor (CIS) became feasible approximately ten years ago when CMOS processing technology advanced to the point of enabling minimum feature sizes small enough to meet the required pixel circuit density, while maintaining adequate optical aperture for a small pixel (diffraction limited) visible image sensor.

As CMOS image sensor technology continues to progress, the comparison of performance between the CCD and the CMOS image sensor has become an important topic of discussion by designers and end users. Today, CMOS imager development has yielded a sensor with characteristics close to that of the CCD, but still lacking in some respects. Overall, the CMOS APS has the advantage of lower cost and design flexibility, but has yet to fully match the performance of the CCD.

Currently, foundry-based CMOS image sensor technologies largely focus on manufacturing of low cost visible sensors for use in high volume consumer products, such as cell phones, PC cameras, toys, automotive sensors, camcorders and low cost SLR cameras. Consequently, the CMOS manufacturing technologies are largely the same as those used in manufacturing of high volume computer memory or mixed signal products. Few, if any, high volume commercial grade CMOS imaging products are manufactured using uniquely customized image sensor specific processes or complex pixel design technologies.

Although some CMOS imager foundries have begun to support unique process modifications to create specific process and design features for application specific integrated circuit (ASIC) clients, in nearly every case the custom CMOS image sensor technologies that enable these high performance applications are unique to the ASIC client, and not accessible to other users.

As will be explained in further detail below, improvements are disclosed in an APS CMOS imager which improve the dynamic range of the CMOS image sensor such that it meets or exceeds current customer expectations for imager sensitivity, response and signal to noise ratio such that it can be used for airborne imaging, especially in a military reconnaissance application. Such an imager, when deployed in an aerial reconnaissance camera, can be operated in accordance with the disclosed methods to detect moving objects (e.g., vehicles) in a scene.

The performance requirements for aircraft based Visible and Near Infra-Red (Vis and NIR) wavelength imaging systems mandate that solid state Vis/NIR focal plane technology enable unique capabilities unlike most commercial or consumer applications of imaging technology. Historically, CCD image sensor technology has provided the needed critical performance and configuration demanded by airborne imaging applications. However, as airborne reconnaissance systems begin to incorporate additional functionality and interoperability through reductions in size, weight, power, and cost, it is becoming increasingly difficult for CCD technology to support these objectives, due to limitations in the fundamental fabrication and operational principles of the CCD image sensor itself. Although present CMOS image sensor technology does not support the broad, demanding performance requirements currently provided by the CCD, the successful evolution of airborne reconnaissance imaging systems will rely on the adaptation of CMOS image sensor technology to enable the improvements necessary to advance the art. The CMOS imager design aspects disclosed herein enable this result to be achieved.

In order to achieve both the stringent and contradictory performance requirements of the airborne reconnaissance grade Vis/NIR image sensor in a CMOS technology, customization of both the CMOS process and the pixel design technologies is required. To further understand the problem, it will be noted that optimum sensor performance requirements include seemingly contradictory aspects. On the one hand, each pixel in the focal plane must have a large full well electron capacity for shot noise (thermal) limited high background, large signal performance, as would be expected during the daytime with full scene illumination. However, on the other hand, each pixel must have high sensitivity and low noise for small signal extraction, in order to provide for reconnaissance capability in low light level situations, such as when shadows are present in the field of view. Therefore, aspects of the present disclosure provide for a novel, modified pixel design to satisfy this dynamic range dilemma.

Furthermore, present CMOS imaging technologies are optimized for photoresponse in the visible band from 450 nm to 700 nm, with the desired NIR response filtered or dramatically reduced. Customization of the CMOS process is therefore also needed to further extend the sensor response from 700 nm to 950 nm for airborne reconnaissance applications.

The embodiments disclosed solve the contradictory dynamic range dilemma of the airborne CMOS image sensor by providing a pixel design, which achieves large well fill performance as well as high sensitivity and a high signal to noise ratio (SNR) while preserving linearity of response.

It is known in the art that CMOS active pixel sensors rely on each pixel having a charge to voltage conversion amplifier to create local voltages representative of the signal value illumination recorded at the pixel. A representative prior art CMOS pixel10is shown inFIG. 1A. Incident radiation12impinges on a photodetector14. Collected charge is supplied to an amplifier16. A gate18is clocked to direct a voltage signal on the output of the amplifier to an output multiplexer (not shown) along conductor24. A reset gate20resets a transistor22at the output of the photodetector14as shown.

The typical prior art (FIG. 1A) teaches that a CMOS APS pixel contains one detector14and one amplifier circuit16per pixel. The performance of the pixel is determined by the detection, amplification, and noise performance of the single detector, single amplifier combination. Typically, most imaging applications will either fall in the category of still or video photography, or controlled light source photography. Prior art pixel design has not previously been required to provide both high sensitivity and signal to noise ratio for small signals while simultaneously providing good contrast signal to noise for small signals buried within high background illumination scenes.

FIG. 2is an illustration of a CMOS image sensor30having an array32of CMOS pixel sensors10, one of which is shown enlarged. The array32is arranged in rows and columns of pixels, perhaps 25 or 50 million in all.FIG. 2shows a column amplifier36which amplifies the output voltage from the pixel sensors10. The amplified signals are supplied to a multiplexer34. The multiplexed output signals are amplified by an amplifier38. The amplified signal is supplied to A/D converters and signal processing circuitry which is conventional.

FIG. 3is an illustration of a four transistor prior art CMOS Active Pixel sensor. This design includes a global shutter transistor40, reset transistor22, amplifier transistor16and select transistor18. By controlling the biasing of the shutter transistor40, the accumulated charge on the photodiode detector14is selectively applied to the charge to voltage conversion amplifier transistor16.

FIG. 4is another prior art four transistor CMOS pixel design. It includes a transfer transistor42, a reset transistor44, charge to voltage conversion amplifier16and a select transistor18transferring the voltage signal from the output of the amplifier16to the conductor24. The transfer gate transistor42acts in a similar fashion to the shutter transistor40ofFIG. 3.

Prior art pixel design has addressed the extension of CMOS pixel dynamic range to accommodate large background signals by compressing the response to large signals with non-linear amplification or by subtracting signal values through down stream signal processing. The compressed response to large signals degrades the contrast signal to noise ratio for the airborne reconnaissance imaging application by reducing the contrast between the small signals present in the large background signal and the background signal itself. The associated graph shown inFIG. 1Billustrates the compromised signal handling capacity and the non linearity of the response as the illumination increases and is compressed. For example, in region26, there is very little increase in the output signal on conductor24as illumination increases.

Various other methods have been employed to compress the input dynamic range including the utilization of variable detector voltage values, multiple storage of lines of signal integrated with different integration times then subtracted, logarithmic response amplifiers, anti-blooming structures to limit collected signal charge, pixel level analog to digital conversion (ADC) and gain adjustment, as well as other concepts.

Typically, the response of a visible detector is linear with respect to optical signal input to the point of either pixel well saturation or full signal swing of the detector sense node. Beyond the saturation illumination level, the response curve becomes essentially flat. SeeFIG. 1B. The prior art methods to extend detector dynamic range do not preserve the linear response of the detector while maintaining signal integrity for both high and low scene illumination levels.

The full well capacity of a CCD image sensor pixel is generally limited by the charge handling capacity of a gated potential well that usually has an antiblooming feature to avoid signal charge from mixing with adjacent pixels when a pixel that has reached its well saturation limit. Conversely, the saturation limit of a CMOS image sensor is typically more a complex function of the floating diffusion sense node voltage swing, photodiode storage capacity, and additional capacitances added to the pixel amplifier circuit. The critical relationship for the saturation value of a CMOS sensor pixel is CV=Q, where V=the useful linear voltage swing allowed at the detector sense node (reset node typically), C=the capacitance of the sense node (amplifier16input node) (including all parasitics), and Q=all photogenerated and thermally generated signal electrons collected and incident on the sense node.

The voltage limit at the sense node is determined by the supply voltage and the threshold of the reset transistor22ofFIG. 1A. The largest voltage swing at the sense node for an N-channel transistor is Vdd−Vt. The practical reality is that the sense node is typically coupled to the gate of a source follower circuit, which has a range of useful input voltages for operation. Typically, the maximum useful voltage swing at the sense node of a CMOS image sensor is <60% of Vdd.

In general, the P/N junction diode photodetectors have a much higher limit for charge storage capacity than the voltage swing at the sense node will allow. The photodiode typically begins to forward bias when over-illuminated such that excess charge is swept into the substrate, thus creating a built-in antiblooming circuit.

The relationship that describes the conversion of signal charge to useful voltage signal is determined by the sense node capacitance or qe/Cnode (microvolts per electron). As the node capacitance increases, the charge to voltage gain decreases, but the dynamic range of the allowable input signal increases as Vmax is reached more slowly.

Increasing the node capacitance is the simple method for increasing the saturation charge limit for the CMOS sensor pixel. Increasing the maximum voltage swing in the sense node is another method. Unfortunately, the maximum voltage swing is limited by the maximum allowable Vdd voltage for the CMOS process design rules being used. Smaller dimension design rules dictate smaller voltages. For circuits that require small geometries for packing density and/or low voltages for low power operation, the maximum Vdd supply value scales to ever-lower values.

The effect of increasing the capacitance at the sense node is to decrease the signal to noise ratio by 1/√C. Hence, for small signals, it is desirable to minimize the sense node capacitance. However, in the airborne imaging and reconnaissance application, it is also necessary to have a large charge handling capacity to improve shot noise limited signal to noise ratio (varies as Vsignal/√Vsignal) for scenes with high background signal and low contrast between target features and the background signal. For the case of optimized small signal gain, adding gain (g) to the sense amplifier while decreasing the sense node capacitance results in an increase in the signal to noise ratio proportional to √g/√C (in simplistic terms).

A critical requirement for airborne reconnaissance imaging systems is to have high signal to noise ratios, for both small signal and high background signal imaging conditions. As previously stated, this creates a conflict for conventional pixel design. Small node capacitance favors better small signal SNR, while large node capacitance favors better SNR for large signals. The sensor designs described herein achieve both SNR goals by utilizing novel CMOS image sensor pixel designs.

In aerial reconnaissance applications, it is desirable to detect moving objects in a scene. Such objects could be for example a military vehicle in a battlefield scene. This disclosure provides for methods for identifying a moving object in a scene. The methods are particularly useful in an advanced CMOS imaging sensor which provides for both low small and high background imaging conditions.

SUMMARY OF THE INVENTION

In a first aspect, a method is provided for detecting a moving object in a scene using a imaging array having a plurality of pixels, each pixel having at least a first detector and a second detector. The method comprises the steps of, for each pixel:

exposing the first and second detectors to the scene, wherein the integration time for the first and second detectors is not the same (e.g., staggered in time, or starting at the same time but one integration time ends before the other);

generating a first output signal from the first detector and a second output signal from the second detector;

subtracting the first output signal from the second output signal to produce a difference signal (either with or without gain compensation); and

wherein a moving object present in the scene is identified from the difference signal.

The invention takes advantage of the notion that the timing of the detectors' shutter integration pulse may be configured such that the two detector/amplifier output circuits within each pixel contain signals from the scene with different integration windows. The difference in the integration windows allows for differentiation, i.e., subtraction, of the two separate data sets within the same frame. If nothing in the scene has moved within the differenced integration window, the result of subtracting the two data sets would be a null difference signal (at least with balanced gain compensation). Conversely, if an object in the field has moved within the time period of the differenced field integrations, e.g., after signal collection has ended on one detector but continues on another detector, a bright object (represented by a non-zero difference signal) would be detected from the subtraction of the data sets. The subtraction can occur in the analog domain (subtraction of analog voltage levels from the two detectors), and can be performed by circuitry in the pixel. Alternatively, the subtraction can be performed in the digital domain (subtraction of digitized detector values) and can be performed “off-chip”, e.g., in a data processing unit associated with the imaging array.

In another aspect, a method is provided for detecting a moving object in a scene using an imaging array having a plurality of pixels, each pixel having a single detector connected to a first charge-to-voltage conversion amplifier and a second charge-to-voltage conversion amplifier. The amplifiers are separated by electronic shutter transistors. The method comprises the steps of, for each pixel:

exposing the detector to the scene and integrating charge during the exposure representing scene illumination on the first and second amplifiers, wherein the integration period for the first and second amplifiers is not the same;

comparing image data from the charge integrated on the first detector amplifier with image data from the charge integrated on the second detector amplifier; and

detecting a moving target in the scene as a result of the comparison.

These methods are particularly useful for aerial reconnaissance and surveillance applications. Moreover, the moving object information can be converted to pixel locations, that is, pixel addresses on the photodetector array. Such pixel address data could be used to determine the velocity and direction of moving objects or could be supplied to geolocation algorithms for determining the location of the moving objects on the surface of the earth.

The preferred embodiments may be employed in an array which provides two amplifier circuits within each pixel to achieve both small signal and large signal wide dynamic range performance. In presently preferred embodiments, the imaging array comprises a CMOS image sensor.

DETAILED DESCRIPTION OF REPRESENTATIVE PREFERRED AND ALTERNATIVE EMBODIMENTS

Before describing the methods of detecting moving objects in a scene, a description of presently preferred CMOS imaging arrays which are suitable for practicing the invention will be described first. The inventive methods are applicable to other CMOS imaging array designs and the following descriptions are offered by way of illustration and not limitation.

An improved CMOS imaging array is described below comprising a plurality of individual pixels arranged in an array of rows and columns of individual pixels. The number of pixels can vary, but embodiments of 25 and 50 million pixels are considered exemplary. Each pixel in the plurality of individual pixel sensors is preferably the same. A representative example of the CMOS pixel sensor10is shown inFIG. 5A. In this embodiment, the pixel sensor includes the following elements:

b) A first relatively lower gain, wide dynamic range amplifier circuit100responsive to the output of the photodetector14, the first circuit100optimized for a linear response to high light level input signals. The amplifier circuit100may also function as a charge to voltage conversion circuit.

c) A second relatively higher gain, lower dynamic range amplifier circuit200responsive to the output of the photodetector14, the second circuit200optimized to provide a high signal to noise ratio for low light level input signals. The second circuit may also function as a charge to voltage conversion circuit.

d) A first output select circuit102comprising, in the preferred embodiment a transistor gate for selectively directing the output of the first circuit to a first output multiplexer (not shown inFIG. 5A, see34B inFIG. 6).

e) A second output select circuit202comprising a transistor gate for selectively directing the output of the second circuit to a second output multiplexer (not shown inFIG. 5A, see34A inFIG. 6).

Separate voltage signal outputs of the first and second circuits100,200, respectively are provided for each of the individual pixel sensors of the CMOS imaging array. In particular, the voltage output from the first circuit is optimized for high light level conditions with wide dynamic range, and the voltage output from the second circuit is high gain, optimized for low light level condition, with less dynamic range but with high signal to noise ratio.

The CMOS active pixel sensor is designed with CMOS processing technology for enhanced dynamic range and signal integrity by utilizing two distinct detection processes and two charge to voltage conversion amplifier circuits100,200within the unit cell of each CMOS pixel10. The detector14is combined with two amplifiers101,201to supply two separate voltage outputs on conductors24A,24B from each pixel. In an alternate embodiment (seeFIG. 9), two detectors are combined with the two amplifiers. In either case, the two output signals can be read out simultaneously, line sequentially, or frame sequentially. The use of two outputs from the first and second circuits provides for optimal signal conditioning and performance for both high level and low level optical input signals that are incident on the CMOS image sensor within the same imaging frame and within the same integration or exposure period. One amplifier output voltage (amplifier201) is optimized for best signal to noise ratio for low light level input signals. The second output amplifier voltage (amplifier101) is optimized to provide a linear response to large light level input signals, such that the signal to noise ratio for small signals of interest residing in an area of large background scene illumination will be improved by shot noise limited imaging principles. The disclosed array solves the problem of providing both extended linear dynamic range for enhanced contrast signal to noise and high sensitivity for low light level content in the image.

The graph shown inFIG. 5Billustrates the improvement provided by the disclosed array where the image sensor signal performance is optimized for both large and small signals without degradation to linearity or SNR. Also note the well capacity for the pixel of the present invention is superior to the prior art since the node capacitance can also be optimized. In particular, the output of the first circuit100on conductor24A ofFIG. 5Aproduces a linear response at high illumination levels as shown inFIG. 5B. However, the output of the second circuit200on conductor24B shows a high signal response at low illumination levels. While this response becomes non-linear and flat at illumination level I, this is of no great concern since the signal output from the first circuit can be selected to produce a linear response at this and higher levels of illumination.

The preferred embodiments provide for charge detection, charge transfer, and charge conversion structures within each pixel sensor of the CMOS imaging array. There are several motivations for creating these various embodiments due to pixel area constraints, noise performance optimization, sensor yield, pixel fill factor, as well as other considerations. As an example of an alternate pixel technology, a pinned photodiode or PPD detector technology can be applied to replace the p/n junction photodiode technology of the preferred embodiment.

An alternative embodiment is described in the circuit diagram ofFIG. 6. The circuit consists of a reverse biased photodiode14detecting optical radiation in the visible and near infrared spectra, and two charge to voltage amplifier circuits,100and200. The first amplifier circuit100consists of a transfer gate106, Capacitance Node (Cnode)108, capacitor110, reset transistor112, source follower transistor114and output selection gate116. The second amplifier circuit200consists of a transfer gate206, reset gate208, Cnode2(210), source follower transistor212and output selection gate214. The amplifier circuit100is designed to optimize signal to noise performance for large signals. The amplifier circuit200is optimized for small signals and low light level conditions. The optimization is achieved, in part, by adjusting the capacitances at Cnode1(108), capacitor110, and Cnode2(210). Transfer gates106and206are positioned between the photodiode detector14and the two resettable floating diffusion capacitors Cnode1(108) and Cnode2(210) and serve to isolate each amplifier114and212from the photodiode14. Each of the floating diffusion nodes is reset to the voltage Vdd by a separate reset transistor112,208. Cnode1is reset by transistor reset112and Cnode2is reset by transistor reset208. The total capacitance on each of the floating diffusion nodes Cnode1and Cnode2consists of photodiode to substrate capacitance, gate overlap capacitance, and parasitic capacitances from the process layers used to construct the interconnection layers. In some cases, additional capacitance may be added to the floating node Cnode1through additional circuit elements such as poly to poly capacitors, as indicated by capacitor110in circuit100.

In this embodiment, the capacitances at the floating diffusion sense nodes of the two circuit paths Cnode1(108) and Cnode2(210) are designed to be different values. Cnode1, for example, could be approximately equal to the capacitance of the photodiode detector14and therefore capable of accommodating the signal charge associated with the largest signal integrated at the photodiode detector. Extra capacitor110is optional. The second capacitance Cnode2(210), in contrast, is then made substantially smaller in value than Cnode1or that of the photodiode detector14, and therefore could provide larger charge to voltage conversion gain than that of the circuit100. The larger charge to conversion gain and small capacitance provided by Cnode2(210) provide optimized small signal response and high signal to noise ratio (SNR). The source-follower amplifier transistors114and212are designed with differing physical layouts (W/L ratios) to optimize the signal to noise performance for each circuit's performance objective. Cnode1(108) and Cnode2(210) are shown in dashed lines inFIG. 6to indicate that they may represent inherent capacitance of the sensor substrate, gate overlap capacitance, and parasitic capacitance and need not consist of an additional physical capacitor.

Each of the two amplifier circuits100and200has a separate source follower output transistor114and212and associated gate transistor116and214for selecting the output of either amplifier114or212. Source follower amplifier114is selected by selector gate transistor116which then drives external column amplifier36A for readout by the external output multiplexer34A. Source follower amplifier212is selected by selector gate transistor214which then drives external column amplifier36B for readout by external multiplexer34B. In this embodiment, separate column lines24A and24B are utilized and are separately selected corresponding to the amplifier output signal of choice. Each column output is directed to a separate column amplifier36A,36B then to a separate multiplexing circuit (34A,38A) (34B,38B) as shown inFIG. 6. Each column amplifier36A,36B design is optimized for signal to noise performance for each of the two signals generated. Those skilled in the art may also recognize that the two signals could be combined, using another multiplexing switch, into a single output multiplexer circuit to simplify the design.

The operation of each amplifier circuit100and200in the embodiment shown inFIG. 6as follows. The amplifier sense node voltages are reset through the reset transistors112and208. Integration of signal charge occurs at the photodiode14when optical signal (illumination) is present. The photodiode signal charge is transferred to the sense node (Cnode)1(108) and/or Cnode2(210) through transfer gates106and/or206. Charge is converted to voltage at the source follower transistors114and212. The signal voltage is scanned by random access scan clocks such that a line sequential parallel/serial readout of the image frame is achieved by the operation of the output select circuits (select gates116and214, associated output conductors24A and24B) and the column amplifiers36A and36B and multiplexers34A and34B.

The integration cycles for each of the two circuits100and200may be set for the same duration but staggered by one integration time, or alternatively, set for different integration times staggered by either the shorter or longer integration time. This operation would be dependent on the system clocking and influenced by scene illumination and feedback from the down stream signal processing. If the integration and readout times of the images from the two circuits100and200are separated, it is possible to obtain useful information regarding image motion within the imaged scene. This is discussed in more detail later on. The transfer gates106and206would not be turned on simultaneously, but would be alternating during the composite integration time. The user would have the option of integrating and transferring signal through the output multiplexers in any of three modes: 1) Simultaneously, 2) Line interlace mode, or 3) Alternate frame mode.

The simultaneous readout mode affords the user the ability to extract both large signal and small signal information from the image scene within the shortest time interval between integration periods; image motion effects between the two scene samples would be minimized. Another technique is line sequential, meaning that a line of data from amplifier1is read out through the multiplexer followed by a line of data from amplifier2, and alternating in that way throughout the frame of imagery. Another alternative is frame sequential, meaning that data is read out from amplifier1only during a first frame, and then data is read out from amplifier2during a second frame. Image data could be from the same integration time, or it could be from different integration times. Using either the line sequential or frame sequential methods would require faster readout multiplexers to maintain higher frame rates. In particular, if two multiplexers are running simultaneously at rate F, as in the preferred embodiment, one frame of data can be read out in time X. If the output multiplexers are operated singly at rate F, the frame of data requires 2× time to read out. Alternately, if the multiplexers are clocked at a rate2F, the total time to read out a frame of data from two multiplexers alternately is X, just as in the case of reading the two multiplexers simultaneously.

Combinations of the two output frames could be selected such that both small signal performance in low light level portions of the scene and low contrast high signal performance can be optimized within the same scene, taken in a snapshot or global shutter mode. An additional benefit to combining or comparing two frames integrated at nearly the same time or within a precisely measured time period is that objects moving at a rate that is discernable within relatively short integration times may be accurately measured such that the velocity of the object may be precisely determined. This feature will be described in further detail below.

FIG. 7illustrates an alternative embodiment which is a variation of the embodiment depicted inFIG. 6. The embodiment ofFIG. 7combines an electronic shutter gate transistor42of the prior art shown inFIG. 4into the large signal wide dynamic range amplifier circuit100, while retaining the transfer gate shutter transistor206for the small signal amplifier circuit200as depicted inFIG. 6. The design architecture embodied inFIG. 7enables the photodiode14and sense node capacitance to be combined (Cnode1(108) plus photodiode14capacitance Cpd). The advantage of this configuration is that it reduces the signal lag through the transfer gate42for the case where the photodetector is a p/n junction photodiode as opposed to a pinned photodiode. Extra sense node capacitor110is optional.

FIG. 8shows the pixel10circuit design for a third embodiment. The circuit contains a first charge to voltage amplifier circuit100configured with the same elements as shown inFIG. 7and functioning as a large signal, wide dynamic range amplifier utilizing the combined photodiode14capacitance Cpd and sense node, Cnode1(108) capacitances and optional capacitor110with shutter gate42between the source-follower transistor M1(114) and the detector/sense node, Cnode1, (108) and having output selector gate116. However, the second charge to voltage amplifier circuit200has been reconfigured. Transfer gate1(206) remains connected to the photodiode14on one side and to reset gate208, Cnode2(210) and transistor amplifier212on the other side; However, transistor212has now been configured with transistor INV1to function as an inverting amplifier. Transistor220has been added as a source follower to isolate the inverting amplifier INV1from output transfer gate214and subsequent column amplifier36B. The inverter gain is controlled by the voltage Vs and the W/L layout ratios of INV1and transistor212such that the voltage output signal gain is improved over the typical source-follower gain of 0.7<g<0.85. However, it must be pointed out that the inverter performance characteristics can become non-linear at the extremes of the operating voltage range. Attention to the inverter design is necessary since this non-linearity can have a negative affect on fixed pattern noise correction and other signal processing applications.

For small input signals, the signal to noise ratio of the inverter gain amplifier circuit (212and INV1) is improved by the ratio of √g/√Cnode2compared to a pixel circuit not optimized for small signal gain. Since the capacitance of the small signal amplifier circuit is low, the maximum allowed signal charge for that node is dictated by the CV=Q relationship, and is typically limited to 20% to 30% of the optimized signal capacity of the large signal wide dynamic range amplifier circuit, which can accommodate the full signal charge capacity of the detector photodiode.

Those skilled in the art will also recognize that it is possible to implement the pixel architecture of any of the embodiments shown inFIGS. 6,7, and8using two separate photodiode detectors within the pixel sensor10such that each amplifier (114,212) has its own photodetector. An example of a dual photodiode detector embodiment is shown inFIG. 9. Basically, in this embodiment, a CMOS imaging array is provided with a plurality of individual pixels10. Each pixel in the plurality of individual pixels comprises a first photodetector14A receiving incident light and generating an output and a first relatively lower gain, wide dynamic range charge to voltage amplifier circuit100responsive to the output charge of the first photodetector14A, the first circuit100optimized for a linear response to high light level input signals. Each pixel10further includes a second photodetector14B for receiving incident light and generating an output charge and a second relatively higher gain, lower dynamic range charge to voltage amplifier circuit200responsive to the output of the second photodetector, the second circuit200optimized to provide a high signal to noise ratio for low light level input signals. A first output circuit (116,24A) is provided for directing the output of the first circuit100to a first output multiplexer34A provided for the imaging array. A second output circuit (214,24B) is provided for directing the output of the second circuit200to a second output multiplexer34B provided for the imaging array. Accordingly, separate outputs of the first and second circuits are provided for each of the individual pixels in the imaging array. A mix gate230is shown inFIG. 9which allows charge from photodiode14B to be accumulated with the charge from photodiode14A in the first circuit100. The other elements as shown inFIG. 9are as described previously inFIG. 7. The mix gate230allows the output of the two photodetectors to be combined to act as a single photodetector, with a combined detection area of the two individual photodetectors. The output of the combined photodetectors from the mix gate is supplied to both the first and second amplifier circuits.

FIG. 10is a layout view of a CMOS APS pixel10with drawn geometries that represent transistor gates, drains, sources, electrical interconnects, and photodiodes. The figure is representative of the masking features in a layout of a CMOS APS pixel with two photodiodes and two pixel amplifiers, but is not an actual pixel layout. Shown in the Figure is a possible configuration of two photodiode detectors,14A and14B, separated by a mix gate230. Photodiode14A is larger in area than photodiode14B and the connect gate could be used to isolate the two photodetectors for separated integration and output functions or to combine the integrated signals from both of the photodiodes.

FIG. 11graphically depicts the relationship of the input optical illumination level incident on the photodiode detector and the analog voltage output seen at the output of the pixel charge to voltage conversion amplifier. The two lines labeled100and200represent the output characteristic of the two amplifier circuits100and200described above in the embodiments of the invention. Line100represents the response to input illumination that the amplifier with extended linear dynamic range generates while line200represents the response to input illumination that the high sensitivity amplifier circuit generates. The high sensitivity amplifier generates larger signals for small input illumination, while the wide dynamic range amplifier generates linear response to large signals. The prior art curve represents pixel and imaging system designs that incorporate circuits that extend the input dynamic range by creating a logarithmic or compressed response to large input illumination. In the prior art embodiments, small signals can be detected with relatively high sensitivity, however, large signal response is compressed such that the variation or contrast of signals in scenes of large illumination magnitude is reduced.

A disadvantage of using two photodiodes in a front-side illuminated CMOS pixel is that the pixel fill factor is reduced. The importance of fill factor is discussed below.

As shown inFIGS. 6 and 7, this embodiment requires that the pixel contain as many as eight (8) transistors, while the embodiment shown inFIG. 8requires ten (10) transistors. It is desirable to use the minimum necessary number of transistors in a pixel so that the pixel may be compacted to the smallest pitch allowed by the CMOS process design rules and practical optics limitations. The photodiode detector generally requires a 50% to 60% unobstructed area within the pixel for optical input to be sufficiently absorbed. The efficiency or responsivity of the detector is critically dependent on the open photodiode area and is known in the industry as “fill factor”. To maintain a constant acceptable fill factor for a CMOS image sensor while the gate or transistor count increases, generally requires the process design rule dimensions to decrease and the supply voltages to decrease, causing the process complexity to increase. If the process technology utilized does not support an increased gate count while maintaining an adequate fill factor, the pixel dimensions must increase to accommodate. For example, a 10 micron by 10 micron pixel, with four transistors (4T), designed using a minimum feature size of 0.35 microns will maintain a fill factor of approximately 50%. Using the same technology and pixel pitch but adding four more transistors (8T) would shrink the fill factor to less than 10%, which is unacceptable unless the sensor is constructed with very small design rules or as a backside illuminated image sensor. In the backside illuminated pixel case, the frontside fill factor is not a consideration since the fill factor of the photodiode aperture is nearly 100%. Therefore one further embodiment is to configure the CMOS image sensor with dual amplifier pixel circuitry as a backside illuminated detector array. SeeFIG. 13and the discussion below.

For the case where the eight transistors are incorporated into a frontside illuminated CMOS array, either the CMOS technology design rule must shrink (to 0.18 micron or less) for the 10 micron pixel example, or the pixel size must grow to approximately 14 microns by 14 microns, to assure adequate fill factor. Hence, each embodiment of the present invention either requires advanced CMOS processing technology, backside illumination sensor technology, or pixel enlargement in order to be implemented.

FIG. 12Ais a cross section of a CMOS pixel10showing the fill factor restriction for a front-side illumination indicated at12. The illumination12impinges on the p/n junction photodiode detector14which is embedded in a p-Si substrate. Polysilicon transfer and reset gates, metal bus wires, and other circuit elements in the pixel sensor10are shown by rectangular boxes300. The area bounded by the box302represents the area available for conversion of illumination to charge by the detector14. The fill factor is about 30-40%.

FIG. 12Bis a plan (layout) view of the pixel10ofFIG. 12A. The pixel is frontside illuminated with a high gate count in accordance with a design rule “A”. A larger pixel pitch (total pixel size) is needed to achieve an acceptable fill factor of 30-40%.

FIG. 13is a cross section of a backside illuminated pixel10in accordance with one embodiment. This embodiment has an unrestricted fill factor—it is essentially 100% as indicated by the box302. This design can tolerate a high gate count design with smaller pixel pitch using the design rule “A” ofFIG. 12B.

FIG. 14is a top view of a frontside illuminated pixel10with a high gate count using the design rule ofFIG. 13and a smaller pixel pitch. The smaller box302indicates the available area for scene illumination to impinge upon the photodiode detector14. The fill factor (ratio of area bounded by box302to total area of pixel10) is too small for most reconnaissance applications. Conversely, if back side illumination is incorporated into the design (as shown inFIG. 13), the fill factor is about 100%, as indicated by the larger box304.

It is possible to utilize three or more different amplifier circuits and three or more separate output circuits, one per photodetector. Each photodetector and associated amplifier circuit could be optimized for a different signal level, spectral band, or other criteria. Furthermore, in a three-detector embodiment, each detector could be associated with a separate red, blue or green color filter and thereby provide separate red, blue and green output signals for incident radiation. The array could be backside illuminated with windows adjacent to the array which have color filters applied to the windows. Alternatively, the array could be front side illuminated with separate imaging areas for each of the three colors.

FIGS. 15A-Cdepict an alternative embodiment which could be employed using the invention to extract spectral information within a scene. Using multiple photodetectors, multiple pixel amplifiers, and a filter matrix such as filters400A (e.g., red),400B (e.g., green),400C (e.g., blue), multiple wavelength bands or colors could be separately passed, such as red, green, and blue light (or near infra-red or other band) to the respective photodetectors14A,14B,14C. Each pixel could simultaneously output multi-spectral scene data (such as red, blue and green scene data) that could be optimized for best signal to noise ratio. Because the photoresponse or quantum efficiency of the silicon photodetector is dependent on the wavelength of light (blue being weakest and red being strongest), the three detector/amplifier circuit combinations100,200,300within the pixel could be used to amplify the blue and green response to match that of the red response—thus normalizing the image sensor at the pixel level. Generally, this color normalizing function has to be done using downstream signal processing. Each detector/amplifier circuit combination output is transferred by its own output select circuit to a separate multiplexer, as is the case with the other embodiments. As shown inFIGS. 15B and 15C, the color filters400are mounted on top of a color filter substrate material402that is in close, but not touching, proximity to the silicon CMOS substrate404.FIGS. 15A and 15Cshow a front-side illuminated configuration.FIG. 15Bshows a backside illuminated configuration. Thus, both configurations are possible.

Object (Motion) Detection

As stated above, integration of signal at the detector can be separated for two amplifiers (“amplifier A” and “amplifier B”) within one frame time, whereby it is possible to obtain useful information regarding motion within an imaged scene. In other words, moving objects in the scene can be detected using the combined outputs of the two circuits, where each circuit has an associated integration time that does not completely overlap the other. This section will explain this feature in further detail.

Detecting the motion of objects in the image scene can be achieved for either fast moving or slow moving targets. Slow moving objects would generally be detected by capturing frames of image data over extended periods of time, storing the images, and comparing (subtracting) image data through digital image processing techniques.

The motion of fast moving objects can be detected within a single frame time using the embodiments described above. For the embodiment utilizing a single photodetector connected to two charge-to-voltage conversion amplifiers and separated by electronic shutter transistors (FIG. 5A), image data integrated on the detector with amplifier circuit100(“amplifier A”) can be compared with image data integrated on the detector with amplifier circuit200(“amplifier B”) by staggering the integration times of the amplifiers A and B and performing real time signal differentiation, i.e., subtraction, of the separate outputs of the two amplifiers. Since the data sets representing outputs of amplifiers A and B will typically have different gains, the amplifiers must first be calibrated and adjusted for balance so as to remove the effect of the gain offset difference from the output when the subtraction step is performed. A moving object will then be detected as a region of increased (or non-zero) signal one or more several pixels long or wide. Such objects can be distinguished from the zero signal background that occurs when gain-compensated detector outputs are subtracted from each other.

For the embodiment which uses two diode photodetectors (FIG. 9), each coupled to a separate charge to voltage conversion amplifier, where the first photodetector is connected to amplifier A and the second is connected to amplifier B, the same method of utilizing different integration times for the two amplifiers and performing real time signal differentiation to detect moving objects can be applied.

The detection range of object motion velocities may be affected by adjusting the frame rate and the pixel integration times for signal delivered to amplifier nodes A and B. Longer or shorter frame times would allow longer or shorter relative pixel exposure times and related capture of object motions.

The total frame time comprises the integration time, the time required to read out the imaging array, and other ‘overhead time’. The total frame time (Ft) for a pixel with a single detector is the sum of the integration times for each amplifier (TintA and TintB), the readout time (Tro), and the overhead time (Toh), i.e., Ft=TintA+TintB+Tro+Toh. Referring now toFIG. 18, the exposure (integration) time may not simply be the time between the start and end of the integration period. If the pixel is operating in the non-integrating mode with the reset and shutter switches enabled (pixel off) when the start of frame timing pulse is generated, the detector will not integrate signal, and will not do so until the both the reset and shutter switches are turned off, which turns the pixel back on. Then, when the next shutter pulse occurs, integrated charge is transferred from the detector to the sense node(s). The time between the turning off of the reset and shutter signals and the next shutter pulse is the integration time for that node (A or B). In the case where sense nodes A and B are connected to a single detector through the two separate shutter switches, the integration of signal transferred to node A (or B) occurs before integration of signal transferred to node B (or A). In other words, for the single detector pixel case, the integration times are staggered such that node A or B integration cannot begin until the integration time for the other node is completed, and the effective integration time is sum of the integration times of nodes A and B as shown inFIG. 16A.

For the case where two detectors are contained within the pixel, integration of signals sent to nodes A and B can begin simultaneously and the effective integration time is the longer of the two integration times as shown inFIG. 16B. Referring toFIG. 18again, one component of overhead time may result after the shutters have finished cycling, since there may be a delay before the start of the line1select clock pulse. As each line is selected, the horizontal scan circuit reads out the line1to line N signals through the horizontal shift register. Hence, the true total frame time includes the pixel off time, plus the effective integration time, plus whatever time delay there is between the last shutter switching off and the select pulse start/end (hold time), plus the full frame readout time, plus the whatever time delay between the end of the select N pulse and the start of the next frame start pulse. The pixel off time plus the time delays make up the ‘overhead time’, which can be minimized to zero, but there are often system reasons for having the frame time include a long interval of non-integration time in the pixel.

To summarize, the timing of the shutter integration pulse may be configured such that the photodetector/amplifier output circuits within each pixel contain signals from the scene with different integration windows. The difference in the integration windows allows for differentiation, i.e., subtraction, of the two separate data sets within the same frame. If nothing in the scene has moved within the differenced integration window, the result of subtracting the two data sets would be no signal (at least with balanced gain compensation). If an object in the field has moved within the time period of the differenced sub-frame time (field) integrations, a bright object would be detected from the subtraction of the data sets. Useful operation in the differencing mode would require that the focal plane array or camera platform incorporating the imager designs of this disclosure be stabilized, or fixed on a constant point during the time of integration as in the case of a moving airframe, since camera or ground motion would disturb the intra-field differentiation result. Methods and apparatus for stabilization of aerial reconnaissance and surveillance cameras are well known in the art and thus omitted in order to not obfuscate the present disclosure.

The moving target detection mode will now be illustrated conceptually with reference toFIGS. 16A-16B,17,18and19. As stated earlier,FIG. 16Ashows the functional timing of a dual amplifier pixel with a single detector whileFIG. 16Bshows timing for a dual amplifier pixel with a two detector configuration operating in the target motion detection mode. The timing for the first amplifier shutter is indicated by the upper line500; the timing for the second amplifier shutter is indicated by the lower line502. For the two-detector pixel configuration (FIG. 16B), at time to integration of charge from the scene commences on both detectors. At time t1, the shutter for the second amplifier is closed, i.e., signal collection from the second detector ceases. However after time t1signal collection still occurs in the first detector. At time t2, the shutter for the first amplifier is closed, i.e., signal collection in the first detector ceases. The difference in integration time between the two detectors is t2−t1or TintA−TintB where signal values SA and SB are proportionate to I×t1and I×t2and I is the photon flux at the pixel which is considered constant for this illustration. The subtracted signal is therefore SA−SB=I×(t2−t1). For the case of a single detector being shared by the nodes A and B (FIG. 16A), the start of integration time t1for node A must be delayed by t0′ such that the start of integration for SA is t0+TintB=t0′. Then, for t0=0, the integration time TintB for signal SB must be t1in duration. In this way the difference signal will still be I×(t2−t1) but the total length of time required to complete the integration cycle is increased by t0′. At some later time, a select voltage signal is applied to the pixel amplifiers (bottom line inFIGS. 16A, B) and the voltages representing the accumulated charge from the first and second amplifiers are transferred to the readout circuit. The above process occurs for every pixel in the imaging array.

Line503ofFIG. 16Bshows an alternative timing signal for the second amplifier shutter for the two detector pixel case which is staggered relative to the timing signal on line500. The shutter signal on line503is offset from the shutter signal on line500by the time amount t3−t0. The integration time in this alternative embodiment (TintB′) is the same as the integration time TintA, but is staggered in time. The timing for the shutter signal pulses is either staggered or overlapped such that there would be a difference in the starting time of integration for one detector vs. the second detector and/or a difference in the duration of the integration time. However, for either the two detector or single detector pixel, once signal is transferred to sense nodes A or B, the length of time that useable signal information can be held on the sense node of the associated amplifier after the integration time ends and the signal is read out is dependent on the leakage current of the node. As the signal is held on the node, the signal to noise ratio will decrease due to accumulating dark current shot noise. Therefore, it may be advisable to reduce the period t2−t1to the minimum needed (and this may vary depending on light level and the luminance characteristics of the moving objects that are expected) and have the select signal occur immediately after the longest integration period ends.FIG. 18shows all the timing signals for the embodiment in which the integration periods start at the same time but have different durations, indicated by TintA, TintB, and TintA*.

FIG. 17is a graph of pixel signal (voltage) across one line of pixels in an imaging array using the technique ofFIG. 16Bwith timing lines500and502. The upper line504ofFIG. 17indicates the signal from the first detector and the lower line506indicates the signal from the second detector. The two lines are separated from each other, but with gain compensation they would essentially have the same values, and if graphed, the lines would be superimposed over each other, except for those pixels where an object was moving in the scene during the integration period t2−t1. The pixel(s) imaging the moving object will have different detector outputs for each of the two detectors, and after the subtraction operation, would produce the non-zero difference signal at510. The subtraction of the two pixels (and more preferably taking the absolute value of the signal difference), is indicated in the line508representing a difference signal. This signal has a null or close to null value for most pixels (with gain control), and the different detector outputs for the pixel representing the moving object is indicated at510. The signal510thus indicates the moving object. Obviously, this technique explained in conjunction withFIG. 17is performed for all the pixels in the imaging array.

FIG. 19is a schematic diagram of a pixel with a single detector14, two charge to voltage conversion amplifiers “A” and “B” and select gates Sel1and Sel2which couple the amplifiers A and B to a readout circuit602having column amplifiers36A and36B, the output of which is supplied to a column differential amplifier600in the readout circuit602. The output of the column differential amplifier600is a difference signal (FIG. 17) which is used to detect a moving object in the scene. The output signal from amplifier600is supplied to a multiplexer whereby a subtraction signal from all the pixels in a row of pixels is obtained. X and P are transfer gates for transferring charge from the photodiode detector14to the “A” and “B” amplifiers, respectively.

Signal differentiation (i.e., subtraction of detector output signals) could be achieved on-chip, i.e., in the pixel circuitry itself, by adding a differential amplifier between the two column amplifiers for each pixel detector pair as shown inFIG. 19. Alternatively, signal differentiation could be performed off-chip in a processing unit processing digital pixel values after gain compensation on the detector outputs has been performed.

Once moving objects in an image have been detected using the described signal subtraction procedure, the photodetector pixel addresses for the moving objects can be ascertained and such pixel addresses can be used for a variety of purposes including determination of the velocity and direction of the objects as well as geolocation. For example, using the known time difference between the integration shutter pulses (t2−t1), it is possible to calculate the velocity of the moving object by measuring the length of the trail of the differenced signal510(in terms of pixels) and translate that value into ground distance (by taking advantage of camera parameters such as focal length and the range from the camera to the ground), and then divide by the integration time difference t2−t1. Furthermore, the velocity of the direction the object is moving can also be ascertained. Other applications include but are not limited to scene change detection for locating brief events such as objects appearing/disappearing in the image, and for tracking the travel of high velocity projectiles. Such methods may take advantage of ranging and geolocation from imagery methods which are known in the reconnaissance art and therefore a detailed discussion is omitted for the sake of brevity.

It should be pointed out that the effectiveness of the detection of fast moving objects may be influenced by the range to the object and the associated system parameters including optical configuration and performance, image stabilization, signal to noise ratio, signal processing algorithms, and other related factors.

From the foregoing discussion, it will also be appreciated that a method of detecting motion in a scene with an imaging array having a plurality of pixels has been descried. The array is capable of generating two images in a single exposure of the array to scene radiation (by means of the two amplifier circuit as explained above). The method comprises the steps of:

during the exposure of the array, in each pixel collecting a first image and a second image, wherein the first image and the second image are associated with image integration times which are not the same;

subtracting the first image from the second image to obtain a difference signal; and

detecting motion in the scene from the difference signal.

The method is applicable to the single detector14per pixel embodiments as well as embodiments with two detectors14per pixel.

Presently preferred and alternative embodiments to this invention have been described. However, variation from the specifics of the disclosed embodiments is contemplated without departure from the scope of the invention. For example, while CMOS pixels have been described with one and two individual photodetectors, it is contemplated that additional photodetectors (3, 4, or more in all) could be used. Further, the motion detection features are applicable to a variety of camera configurations, including surveillance cameras, such as those used in ground-based security applications, as well as cameras used in aerial reconnaissance.