WIDE DYNAMIC RANGE IMAGE SENSOR

Abstract: An abstract sensor array (10) includes adaptive sensitivity control circuitry (12) for obtaining multiple outputs at different exposure levels. These are combined to yield an output with enhanced dynamic range. This enhanced dynamic range may be reduced in a manner which preferentially retains edge information in order to maintain high contrast.

Description: WIDE DYNAMIC RANGE IMAGE SENSOR FIELD OP THB INVENTION The present invention relates to video imagery and sore particularly to apparatus and techniques for providing enhanced video images. BACKGROUND OP THE INVENTION Various types of video image enhancement apparatus and techniques have been proposed. AGC techniques have been employed, inter alia, in video signals for reducing the dynamic range of the video signal by subtraction of its DC level. Such a technique is described in U.S. Patent 4,719,350, wherein linear AGC is applied to the video image on a pixel by pixel basis. U.S. Patent 4,601,055 describee an- image processor which provides, inter alia, neighborhood transforms. U.S. Patent ,471,228 describes a solid state image sensor with exposure controller, The exposure control appears to be applied to the entire frame and is not addressed to individual pixels. U.S. Patent ,556,908 describes a solid state image sensor including a photodiode array which is pixel addressable in what appears to be a random-accessible manner. A photometric output froa a selected group of pixels to a light meter is provided, but pixel addressable sensitivity control is not suggested or provided. U.S. Patent 4,472,638 describes a two-dimensional solid state image sensor which is able to directly output image information subjected to two dimensional image processing. U.S. Patent 4,734,776 describes a system for extending the dynamic range of a CID array. The CID array is designed to provide tvσ integration times, one short and one- long, for every pixel in the array, and to read out both of the signals thus produced. The appropriate signal is then selected for each pixel, and, if it is the signal produced from the short integration time, it is amplified to compensate for the difference in sensitivity between the long and short integration times. Pixel- addressable sensitivity is not suggested or provided. U.S. Patent 4,473,836 teaches a sensor or sensor array achieving wide dynamic range by means of a log amplifier integrated with a detector. It does not offer any variation of integration time. U.S. Patent 4,636,865 discloses a detector design which allows discharge of excess photocarriers. This is an improved anti-blooming scheme. U.S. Patent 4,656,361 discloses an nMOS photodiode array with improved sensitivity and dynamic range achieved through the use of an integrated scheme for sensing photo- generated current. U.S. Patent 4,609,825 teaches a line-transfer type CCD array providing variable integration time on a row-by-row basis. The variable integration time is achieved through variation of the time between the reset and the signal transfer pulses. U.S. Patent 4,583,002 discloses a solid-state image sensor which provides sensitivity adjustment in response to the overall image intensity. 4 U.S. Patent 4,669,688 teaches a CID array with an individual amplifier for each row in order to reduce circuit capacitance and to improve the signal/noise ratio. U.S. Patent 4,775,799 improves the dynamic range of an image scanner throug the techniques of digital and analog signal averaging in a manner which adapts to the condition of the input signal. U.S. Patent 4,706,123 discloses an apparatus for changing the exposure times of individual photodβtectαrs in relation to the amount of incident light on only the individual photodβtβctor. The exposure times are limited to a small number of distinct integration times, typically 2, and is subject to restrictions as to which pixels can be controlled. Each line of detector* oust b* aeeeαpanled by at laaat two additional shift registers which is a prohibitive _usount of cicuitry for two- dimensional arrays. S. P. Dβ βerth and C. A. ! Mead, in an article entitled H A Two-Dimensional Visual Tracking Array", presented at the Fifth MIT Conference on Advanced Research in VLSI, March 1988, MIT Cambridge, Masβaehuaβtts, U.S.A. and published ia he proceedings thereof, disclose a semiconductor-photosensitive device which computes, per pixel, the difference between the incident light intensity and the average intensity over some neighborhood. This device LΛ limited by the dynamic range of the individual photodβtector and by the complexity of circuitry required per pixel. E.P. Patent 233.464 appears to show a photodiode array providing selectable feedback amplification for each photodiode signal. SUMMARY OP THE INVENTION The present invention aeeks to provide improved video image enhancement apparatus which overcomes limitations of the prior art apparatus and techniques discussed above. There is thus provided in accordance with a preferred embodiment of the present invention, video imaging apparatus comprising a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation and detector control apparatus for controlling the response of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range which is greater than the operative input radiation dynamic range of any given detector. In accordance with a preferred embodiment of the invention, the apparatus for controlling the response of the photosensitive detectors is addressable simultaneously to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field. In accordance with a preferred embodiment of the invention, the staring array is a generally two-dimensional array. Alternatively, the staring array may comprise a one- dimensional array. Further in accordance with a preferred embodiment of the invention, the detector control apparatus includes means for selectably and variably determining the integration time of each of the multiplicity of detectors. Additionally in accordance with a prefe re _«__Lκ______ent of the invention, the staring array may be embodied on a unitary integrated chip, such as in the form of an array of diffusion diodes. on a MOS chip. In accordance with one embodiment of the invention, the detector control apparatus comprises apparatus for providing two control signals to each of the multiplicity of detectors. The detector is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals. According- to an alternative embodiment of the invention, a control circuit is provided for each detector of the multiplicity of detectors. Additionally in accordance with a preferred embodiment of the invention, the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamic range produced by operation of the detectors with variable integration times. Further in accordance with a preferred embodiment of the present Invention, the detector control apparatus comprises apparatus responsive to radiation incident on the multiplicity of photosensitive detectors for governing the response of the individual ones of the photosensitive detectors. Still further, in accordance with a preferred embodiment of the present invention, the apparatus responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and in response to radiation incident thereon and in- the vicinity thereof. Moreover, in accordance with a preferred embodiment of the present invention, the detector control apparatus includes apparatus for storing a value representing the response of each of the individual ones of the photosensitive detectors and apparatus for updating the stored values. The detector control apparatus is also operative to cause the multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of the input radiation or having a selectable dynamic range. There is provided in accordance with a preferred embodiment of the present invention, video imaging apparatus also comprising image processing apparatus receiving the outputs of the multiplicity of photosensitive detectors and control signals supplied to the multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors such as would be obtained were the array composed of sensors having a uniform sensitivity and an extended dynamic range. The image processing apparatus is operative to cause the multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of the input radiation or having a selectable dynamic range. Further, in accordance with a preferred embodiment of the present invention, the image processing apparatus receives control signals representing integration times of the •* multiplicity of detectors and provides in response thereto modified outputs of the detectors correspondins to a uniforo sensitivity. Additionally, in accordance with a preferred embodiment of the present invention, the multiplicity of photosensitive detectors inciud?? g plurality of detectors, etch aβnsitivR r<Q A different color. The detector control apparatus is operative to control the response of the plurality of detectors to a plurality of colors. Moreove , in accordance with a preferred embodiment of the present invention, the color detector control apparatus includes apparatus responsive to achromatic intensity at a given image location for determining the response and for providing a single control output governing it for the plurality of detectors at that image location. Further, in accordance with a preferred embodiment of the present invention, the video imaging apparatus comprises a multiplicity of detectors, column and row control apparatus together determining the sample timing of the plurality of photosensitive detectors and apparatus for reading out a video signal from the plurality of photosensitive detectors. Additionally, each of the multiplicity of detectors comprises column and row control apparatus together determining the precharge timing of the plurality of photosensitive detectors. Still further, in accordance with a preferred embodiment of the present invention, the column and row control apparatus include apparatus for providing multiple precharges of at least some of the plurality of photosensitive detectors in order prevent accumulation of excessive charge thereat. There is provided in accordance with a preferred embodiment of the present invention, image processing apparatus comprising apparatus for receiving a video signal and for providing- an output video signal having a dynamic range narrower than the input dynamic range, while preserving edge information and removing some relatively spatially unvarying information. Additionally, in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing video signals comprises at least two neighborhood operator apparatl and summation apparatus for combining the outputs of the at least two neighborhood operator apparatl. Further in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing comprises at least one neighborhood operator and at least one look up table which together receive the input video signal and provide a multiplication factor which is multiplied by the input video signal to provide the output video signal. •4 BRIEF DESCRIPTION OF THE . DRAWINGS The present invention will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which: Fig. 1 is a block diagram illustration of a video Imaging system constructed and operative in accordance with a proferred-embodiment of the present invention; Fig. 2Λ is a simplified block diagram of a detector array and associated detector control apparatus in accordance with one embodiment of the invention; Figs * 2B and 2C are simplified circuit diagrams of two alternative embodiments of detector elements useful in the present invention; Fig. 20 is an illustration useful in the understanding of the operation of the detector array and control apparatus of Fig.2A; Piff. _ 3 _ i A HM o t. _ r -»s di mgτ>a~t- . _ _ ti-_.t_r __t. _ne; hm operation -of the apparatus of Fig. 2A; Figs. 3A and 3B are simplified block diagram* of a detector array and associated detector control apparatus in accordance with an alternative embodiment of the invention; Fig. J. is an electrical -diagram of detector control apparatus for a given detector in the embodiment of Figs. 3 and 3BJ Fig. 5 la a pictorial illustration of the relationship between pixels processed in a typical neighborhood transform used in an embodiment of the invention; Λ Fig. 6 is an electrical diagram of circuitry useful in carrying out a neighborhood transform in accordance with an embodiment of the invention; Fig. 7 is a simplified block diagram of control circuitry suitable for controlling the apparatus of Fig. 2A; Figs. δA and 8B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling the apparatus of Fig. 2A; Fig. 8C is a block diagram of circuitry useful in the circuitry of Figs. 8A and 8B; Figs. 9A and 9B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling apparatus similar to that illustrated in Fig. 2A for full color operation; and Figs. IDA, 10B and 100 are:simplified block diagrams of three alternative embodiments of image processing circuitry useful in the present invention. DETAILED DESCRIPTION OF ^REFERRED EMBODIMENTS Reference is now made to Fig. 1. which is a schematic block diagram illustration of a video imaging system constructed and operative in accordance with one preferred embodiment: or tne - present invention. The system comprises α generally two- dimensional solid state detector array 10, which is intcrconnαotαd with adaptive sensitivity control apparatus 12. The output of the detector array 10 is supplied via a suitable bus l* to video readout control circuitry 16, which may comprise conventional raster scan circuitry or alternatively intelligent 10 scan circuitry as described and claimed in Israel Patent Application 83213. Fig. 2A illustrates the interconnection between th adaptive sensitivity control apparatus and the detector array 10 15 in accordance with one preferred embodiment of the invention. It is seen that each detector element 20, corresponding to an Individual pixel, is provided with connections to both a row sample control circuit 22 and a column sample control circuit 24, both of which may βaploy shift register*. 20 Fig. 25 illustrates the.specific! interconnections of each detector element 20. Here it is seen that first and second AND gates 30 and 32 each have β. first input connected to column sample control circuit 24 (Fig. 2A) via respective precharge (P) leads 3^ and sample (S) leads 36 and have a second input 25 connected to row sample control circuit 22 (Fig. 2A) via a lead 38. The number of leads for each detector element 20 may be reduced to two if a local delay circuit for determining the time -_-_ ,__ - a Λ A \ PCT/US89/03346 WO 90/01844 I O 41 between sample and precharge is provided. The photodstector element such as a photodiode 40 corresponding to each pixel is coupled to a precharging voltage V via a P switch 42. governed by the output of AND gate 30. and P 5 is coupled via an S switch 44, governed by the output of AND gats 32, to a buffer 46 and thus to a capacitor 48. Alternatively, it is possible to reverse the location of switch 44 and buffer 46. It is also possible to reverse the lonation of buffer 46 and capacitor 48. It is to be appreciated that each pixel cell may . 0 include additional circuitry not shown in Fig. 2B. The circuitry of fig.23, is however, all that is required for an understanding of the present invention. It may thus be appreciated that the integration time of each photodetector element corresponding to a pixel is governed 15 by the provision of appropriate electrical signals along leads 34, and 38. Initialization of integration is provided when AND gate 30 simultaneously receives inputs along leads 3 and 38, thus precharging the photodiode 4θ. Termination of integration is • provided when AND gate 32 simultaneously receives inputs along 20 leads 36 and 38, thus closing switch 44 and coupling the photodiode via buffer 46 to capacitor 48. An alternative embodiment of the detector element 20 eliminates AND gate 32 and its connαot±on, v.n load 38. to the row control circuit 22. The S lead 36, connected to column 25 control circuit 24, is directly connected to S switcft 44. Thus, all pixel* in a given column are simultaneously sampled and their exposure times are controlled by the precharge signals alone. It will be appreciated that in the embodiment of Figs. 2A and 2B, the selection of the integration time for each pixel is governed externally of the detector array 10 by the row and column sample control circuits 22 and 24. Circuits 22 and 24 are preferably embodied as binary data registers, such as shift registers. It will be ap recia ed thfct the embodiment of Figs. 2A and 2B may be applied to groupφ of pixels as well as to individual pixels. The embodiment of Figs. 1, 2A and 2B may be applied also to modified CCD devices. The overall chip architecture Mna.na ■__ is shewn in Fig. 2λ but t e detector- elaαont 2Θ is as shown schematically in Fig. 2C. In this alternative embodiment, the pbotόdβteetor element is a MOS depletion capacitor 100, commonly used in COD detector arrays. As in the previous embodiment, the P switch 42 releases a precharging voltage V P when AND sa e 30 simultaneously receives inputs along leads 3 and 3δ. Precharging voltage V typically removes any accumulated P photeehmrgeø froa the capacitor 100; Unlike the previous embodiment, the sampling circuit comprises a conventional CCD transfer gate 102 * A complete line (row or column) of such photo-sites is triggered simultaneously to transfer their charges to a parallel line of a CCD shi register 104 * In due course, the CCD shift register 104 transfers the charge out of the photo-site. Thus, each detector element 20 receives its own individual precharge but all detector elements 2Θ along a line are sampled together. This embodiment of the detector element 2Θ is similar 4 to that of conventional CCD arrays with the exception that each detector elem-wit. 20 is independently preαhargod by its own precharging circuit controlled by AND gate 30. It is a feature of the present invention that each pixel la independently precharged or βaβpled through switching of the signals on the column and row control leads 34, 36 and 33. The switching operation is illustrated in Fig. 20. The signals are typically determined by control words externally loaded into registers 110 and 112 of rot. and column control circuits 22 and 2 (Pig. 2k) , respectively. The control words typically are comprised of a sequence of 1-bit values, typically of either 1 or 0. The control word of column register 112 typically comprises a 1 at a column address j to be precharged and 0's at the remaining column addresses. The row control word loaded into register 110 simultaneously with the control word for column j comprises a 1 in each row address i at which a pixel (i.j) of detector array 10 is to be precharged. In Fig. 2D the letter P is placed at those pixels which would be precharged given the column and row control words shown. The column control words are typically designed such that the 1 is shifted sequentially through register 112, thereby- sequentially activating the columns of the detector array 10. The. loaded row control word corresponds to the pixels to be precharged in the activated column-. This cycle of the column precharge signal 1 through the column control register 112, while a new row word is loaded for each column, will be known hereinafter as a "column cycle". 4 ,ϋ In order to produce a plurality of integration times, the column cycle is repeated a plurality of times where each successive column cycle corresponds to a shorter integration or exposure time. Since the pixels of a given column have varied integration times, the row control word loaded for a given column changes with each column cycle * The repetitions end once all of the pixels have completed their reset cycle. At that point, the pixel photocharga values are sampled and their photocharga is transferred out of.the detector array 10. The sampling is performed on a column-by-column basis in order to maintain a constant integration time, for a given precharge cycle, over all the columns. Thus, the column cycle is repeated once more to produce the sampling. A separata column sample register is typically included in column control circuitry 24 for this purpose. In the first embodiment, shown in Fig. 2B, a row word of all i's is loaded into the row control register before sampling commences. For the other embodiments, whose S switch 44 is not connected to row lead 38, a row word generally is not loaded. Fig. 2E illustrates the precharging process described hereinabove and is an example schematic timing diagram for the pixels (i.j) through (i+2, J«-l) of detector array 10. Fig. 2S presents the precharge signals for the two columns, j and j+1, and for the three rows, 1, i+1 and 1+2, as well as the sample signals for the two columns. A pixel is precharged when- its column and row addresses simultaneously receive the falling edge of their respective precharge signals. Thus, integration-of the photon flux for each lβ pixel begins with a precharge signal which is common to both tha row and the column in which the pixel resides. Integration ends on the rising edge of the sample signal for the column in which the pixel resides. Fig. 2E illustrates that each column, receives its column precharge signal once per column cycle and that, for a given choice of integration time, column j is always precharged before column J+l. The timing of the row precharge signals is not as regular as that for the columns and depends on the integration time desired for each pixel. As shown in Fig. 2Ξ, there are thraβ example integration times, 11, T2 and T3 where T3 is the longest and corresponds to the first column • cycle. Although not illustrated in Fig. 2E. it is possible for a pixel to be precharged a multiplicity of times. Each time the pixel la precharged, its accumulated photochmvge is d.βrJiarsβd. Thus, charge ntegration for sampling begins at the final precharge. It will be αpproolated by those skilled .n the art that this mechanism is operative to prevent blooming in pixels subjected tu high photon flux, mrraapondinβ generally to bright areas of the optical image, which would tend to cause saturation at those pixels in conventional sensor arrays. Since video readout is independent of the preehargo/aample proerαinabov*, integration times are typically of arbitrary length. In the preferred embodiment of the present invention, useful for atandaard video applicationa, integration time is equal to or less than the video frame rate of typically 30 fraαe/βec. The integration time resolution of detector array 10 depends on the length of time required for a single column cycle. For example, detector array 10 could be a 51 X 512 pixel detector array operating at a 10 KHz clock rate with a lέ-bit data bus for loading in the control words. The detector elements 20 are the CCD type illustrated in Fig. 2C. The example detector array 10 will take 3.2 uaec to load a single 512-bit row word and 1.6 msec to step through all 12 columns of the array once. To increase the speed of the detector array 10, a faster clock rate and/or wider data busses may be utilized. According to an embodiment of the invention, selection. of the integration time for each pixel is governed Internally of the detector array 10 by adaptive sensitivity control circuitry associate with the photodetector elements corresponding to each pixel * This embodiment is illustrated in Figs. 3A and 3B. Fig. illustrates a detector array 0 wherein each pixel, module 51 communicates via a bus 2 with a video readout control circuit 4, which may be similar to circuit 16 (Fig. 1). Fig, 3B illustrates the structure of each pixel module. It is seen that each module 1 comprises a photodetector element such as a photodiode 6 which is coupled via a F switch 58 to a pre¬ charging voltage source 60 and via an S switch 62 to a pixel buffer 64, which in turn communicates via bus 52 with the video readout control circuitry 4. Tiie output V of pixel buffer 64 is also supplied to i-i a neighborhood transform processor 66 which is operative to compute a signal N , which is the neighborhood transform of V id U and of the output of the pixel buffer 64 for the β neighbors of pixel ij , as illustrated in Fig. 5 * Alternatively another type of n_ i ghborhood transfers, may be employed, such aβ one which includes only the two vertical and two horizontal neighbors only θ- αβ α further alternative, a neighborhood luiηςαv than 3 x 3. In th case of a linear scan, a neighborhood incorporating only the neighbor a along the scan line may be employed. The signal N is supplied to an adaptive sensitivity control circuit 68, which provides control outputs to switches 58 and 62. A preferred embodiment of neighborhood tranaform processor €€ is shown in Fig, 6. The circuitry of Fig, 6 indicates that the light intensity values from the eight pixels which abut on a given pixel are averaged by an averaging circuit 90, such as an 8-input differential amplifier or a resistor network as described in the article entitled "A Two-Oimenβlonal Visual Tracking Array" by S. P. DeWeerth and CA. Mead. The output of averaging circuitry 90 is supplied to a differential amplifier 92, which also receives the video output signal V from the nominal pixel (i,j), via a 1 + epsilon amplifier 94. The output of differential amplifier 92 is the neighborhood αignαl N for pixel (i.j). U According to an alternative embodiment of the invention, any suitable neighborhood transform may be applied to obtain a neighborhood signal N . i_ A preferred embodiment of circuit 68 is illustrated in Fig. 4. As seen in Fig. 4, a differential amplifier 70 receives αignαl N froa neighborhood tranaform processor 66 and a ii 4 ^ reference voltage V from an external source. The output of 1 differential amplifier 70 is proportional to the difference between N ' and V and governs the operation of a transistor 72, ii 1 which in turn governs charging o ' a capacitor 7**. A switch 7β resets capacitor 71 when provided with a P signal. The capacitor 7ή is coupled tn nn .tipnt. n a comparator 80 which also receives a reference voltage V . The r output of comparator 80 is supplied to a first one shot multivibrator 82 which provides a control signal to switch 62 (Fig. 3B). The output of outs ahut 82 is also supplied to a second one shot multivibrator 8*., which provides a control signal to switch 8 (fig. 3B) which is also operative to reset switch 76. The circuitry of Fig..4 defines a first order negative feedback loop having a gain less than unity in order to avoid oscillations. Since the neighborhood signal N is monotonic in V , the circuitry of Fig.4 is operative to compress the dynamic aϋnøe of the image. Thus for a given cycle, the higher the image video signal is, the shorter the integration time for the next cycle will be, producing a lower V in that subsequent cycle. U Higher order feedback lααpx may also ba used for this purpose. Instead of a photodetector which operates cyclically, it is possible to employ a photosensitive transistor, aa deβeribed in the article entitled "A Two-Dimensionai Visual Tracking Array" as referenced hereinabove. Reference is now made to Fig. 7 which illustrates a second embodiment of the control circuitry. In this embodiment of the present invention, the adaptive sensitivity control eirott±try 12 iα partially internal to a sensor chip 11$ and partially external to it. The internal circuitry of sensor chip 119 comprise* _h_. detector array 10 and the elements directly controlling it, inter alia, the row and column control circuitry 22 and 2 , respectively, and video readout control circuitry lδ. Tho adaptive control circuitry 12 external to the chip are those elements which determine the integration time for each pixel on tho detector array 10 and which determine the wide dynamic range video signal which is typically the output of the video imaging system of the present invention. Ilia sensor chip 119 typically opers aβ as described hereinabove for the individual elements. A timing control 120 defines several, typically 3, integration times for the array pixels βuoh that a reasonably wide range of integration times is defined. An example set of integration times of 0.1, 1.6 and 25.6 aaec yields a 256il range within the constraint of a 30 msec frame rate. To achieve these integration times typically requires a 25 MHβ clock and a 64-bit data bus. Row control words corresponding to the set of integration times are defined as f llowβ. A row control memory 122 stores the row control word for each of the H columns of the H x N detector array 10 for each of r repetitions, where r is also the ' number of integration tides. For the example set of integration times, each pixel has associated with it one of three entries, either [0,0,1] for the shortest integration time, [0,1,0] for the intermediate length integration time and [1,0,0] for the longest integration time. Row control memory 122 additionally stores the chosen integration time for use in processing described hereinbelow. Timing control 120 causes the column control 22 to cycle through r column cycles while indicating to row control memory 122 the appropriate row control word R to load nto row J control register 110. It will be appreciated that it may be necessary to have a wait state between column cycles to achieve the requested integration times. Once the repetitions have finished, timing control 120 sends the appropriate sample signal to the column control 2*. to cauae the array to be sampled. It will be appreciated that all of the pixels of a ζ±von column are simultaneously sampled and that a gap between the time a given column is sampled and the time its lefthand neighbor is sampled is maintained in order to maintain uniformity of pixel integration times across -detector array 10. For the example given above, the gap is 3.2 sec. After sampling, the pixex- intensities are read by the video readout control 16. The video output of readout 16 is amplified, in an amplifier 126, and digitized, in A/D converter 128, to produce V for use in an integration time update and extended dynamic range video output calculation operation. The algorithm utilized in the operation is disclosed in Israel Patent Application 87306, commonly owned by the common owners of the present application. Briefly, the algorithm definea the value for each pixel as that which waa received when the pixel detector element 20 waa operating in the optimal, linear range. The algorithm then multiplies (or shift ' s) each value by the inverse of the Integration time which produced the value within the linear ra ee of the pixel detector element 20. The pixel values 01844 are then combined into a single, expanded dynamic rang* picture. Accordingly, a comparator 130 determines whether or not V is within the linear range of the detector element 20. If it id is, the entry for pixel (i,j) stored in row control memory 122 remains unchanged. If V is out of the linear range, T is ϋ « incremented or decremented to the next higher or lower integration time, depending on whether it was respectively below or above the linear range, by a Change T block 132 and the ii entry for pixel (i,J) is changed accordingly. If V is very far ϋ out of the linear range, the change in T may not be sufficient ϋ and ..harn nrfi, T will nβft n hn nήi nr,ptή Again «r. fchft nRx ϋ frame. This will continue until V is within the linear range or i until T has assumed an extreme value and cannot be adjusted any ϋ further. Thus, there may be a lag of r-1 frames in adjusting the sensitivity to sudden temporal changes in scene brightness. V is also input to a multiplier 134 which is ii operative to adjust the video output signal for variations in pixel sensitivities, or integration times, over the detector array 10. Multiplier 134 transforms V into V /T where T is ϋ iJ ϋ ϋ calculated from the appropriate pixel entry in row control memory 122. For integration times which are multiples of 2, as in the example above, multiplier 134 performs a shift operation by the appropriate number of bits . In the current embodiment, multiplier 134 typically produces a 16-bit digital video signal from the 8- --_ bit video signal produced by detector array 10. Standard video display equipment is generally capable of displaying only analog signals of no more than 6-7 bits dynamic range. Therefore, the preferred embodiment of the present invention also Includes a dynamic range compressor 136. Compressor 13$ tøpiςally ςQB risβs a FIFO aβrπantins buffer 1 S typically of size M*(m-1) ♦ m)*16, three mxm convolvers 140, 142 and 144, an adder 146 and a D/A converter 148 for producing the output analog video signal. For typical 3x3 neighborhoods, FIFO buffer 138 typically stores 2*M + p tlu _._ AΛy liώώ ____. i» p *d Lo £** out a given x3 neighborhood around a given pixel until the entire array has passed through and been processed. Each of the 16-bit valuβa of the given neighborhood is 'sliced' into three 8- bit values known as the high byte, the low byte and the middle byte. The middle byte is that which is produced by removing the four highest and four lowest bits. The high bytes of the neighborhood are convolved together in convolver 140, the middle bytes are convolved by convolver 142 and the low bytes are convolved by convolver 144. The kernel for the three convolvers 140, 142, 144 is as follows: -1 -1 -1 "1 8*eps -1 -1 -1 -1 where βps is a small-valued constant stored in block 150. The outputs of the convolvers 140, 142 and 144 are summed and truncated by adder 146 to produce the 8-bit value which is passed to D/A 148 which, in turn, produces the analog value of the 8-bit signal. t will be appreciated by those skilled in the art that the result of the operation described hereinabove is to stretch tha inherent input dynamic range o Uia dececcor elements 20 in detector array 10 from a range αf 7-8 bits to fully 16 hits while maintaining an output dynamic range that is compatible with common display and processing means. In an alternative embodiment, the lέ-bit digital value read out of multiplier 13 is furnished directly to a digital computer or other image processing means capable of processing the extended dynamic range Signal. As mentioned hereinabove, the stretc is accomplishβc. by choosing one of r possible integration times for each pixel. It will further be appreciated that, with the exception of the detector array 10 and its column and row controllers 24 and 22, the necessary components are currently available. Reference is now made to Fig. fiA which illustrates an alternative embodiment of the adaptive sensitivity control 12 of Fig. 7, In this embodiment, the analog video output is directly produced, thus eliminating the need to produce the l6-bit intensity values and to subsequently compress them. The Integration time calculation is modified to increase the number of integration time intervals such that the integration time can be any value which is an integral multiple of the time required for a column cycle. For a minimum column cycle time of 0.16 msec and a frame rate of 30 msec, integration time may be varied over a 200tl range. By restricting sampling time in 2x2 pixel neighborhoods to be identical avar the neighborhood, a range of integration time of 800:1 can be attained. Sensor chip 119 produces analog video output which is aaplified by an amplifier 172 before output to display equipment. O90/01844 - ^ As in the previous embodiment, the integration time processing elements of the adaptive sensitivity control apparatus 12 ara external to sensor chip 119 and are comprised of generally digital circuits. The external integration time processing elements ara typically comprised of on A/D converter 17^ for converting-the analog video output to digital intensities V for id processing, an update integration time block 175 which produces optimal integration times T' given both the output intensity i_ V and the integration time T used to produce that output, and 0 a ιf_rame timing memory 178 of s aize M k N x 8 bits for storing the selected integration times as 8-bit values► The update block 175 comprises a lookup table 176, a multi-cap delay line l8θ and an accumulator 182. Lookup cable 176 * chooses a preliminary new value, T new, for the integration time L5 T . Many di ferent functions o af V and T can be used to 3 .ij u generate the values, stored in lookup table 17o, which determine T new. In the preferred embodiment, the following logic is used: a 1. If V is at the saturation level, which is typically a value of 255, then T new is set to T /2? otherwise, a a 20 2. T new - int( k # V # ej_p(-V ) ) (1) ii 1 ii ii where k is a normalization constant chosen to generate a smooth 1 spread of values of T over the range desired and the function i int produces an integer value froa a-non-integer value. In accordance with an alternative embodiment, T new is ±i 25 defined according to equation 2. T new - k *(i-exρ<lnE /E. tau)) (2) 14 2 i. 1. where k is a normalization constant chosen to produce an Ξ 2 ij which is in a range which is convenient for computations, such as the range be wee θ and 65,335, useful for 16-bit computations to the low valua__r _u_d--cand*nses the high valluβs of T new * _o_n ϋ-r " όSC to locally enHanca edge l s* and to suppress artifacts,: T n* is modified by the average of the new iΛ egi iiαn _____r"in its neighborhood. hus, it is typically stored ' __rf v _ιulti'~fc3p delay line 180 'to wait for the neighborhood values- tύ trβ produced. The neigbfctarhood is then averaged by accumulate* I8≥- to produce T f , the ew value of the integration time fibfr ir._______ pass. V' is stored as an 8-bit number in frame timing memory 178 foT> tø»β i_r * B__}uiring the next image. The appropriate row control wtsrds- which will produce an integration time of T" at pixel fi ' . df . are generated by row control logic 184- as follows and are loaded ' into row control circuitry 22 as described hβrainaSovi-, flow control logic 181 is shown generally in Fig. 8C. it generates ÷-"thv : MM. idrd corresponding to column i at a current integration ' tins- ~ T by reading the: T values or that column n i from memory 178 and 'comparing each or them, in a comparator 3<DQ, to T , A 'c$uήtδt '" B52 provides the value T in units of the basic n n timing- gap, which is 0.16 msec in the example. If T is less id than o fKiual ' td , then address* i in the current row word n receives B. value of 1, indicating that pixel (i,J) will be precharged on the current column cycle. Otherwise, address 1 receives a 0 value, indicating that pixel (i,j) will continue integrating charge until it is sampled. It will be appreciated that counter 302 is typically decremented each column cycle until a sampling cycle occurs; thus, T begins the Integration cycle at n its largest value and ends at its smallest value, decrementing by one column cycle each time. Additionally, it will be appreciated that each pixel may be precharged at every column cycle until T n reaches its integration rime value. Timing control 120 causes the column control circuitry 24 to precharge each column once per column cycle. It will be appreciated that the pixel-by-pixel update of the integration times T occurs independently of the ij- prechauoβ and aam lo operations of tho sonsor chip 119• The off- chip generation of row words is typically very fast; for example, to achieve a O.lό msec minimal Integration time, a row word of 512 bits must be generated every 300 nsec. A 6.--bit bus operating at 25 MHz is capable of producing row words at such a speed. An alternative embodiment of the adaptive sensitivity control 12 which directly produces analog video output is shown In Fig, 8B. and differs from the embodiment illustrated in Fig. 8A in the update block 175 which produces the optimal new integration time value, T*. i_ In accordance with this alternative embodiment, update block 175 comprises a first and a second lookup table. 190 and 192, respectively, and a multiplexer 19 . First lookup table 190 receives the output intensity V and the integration time T and chooses a value Ξ which i_ ij 14 corresponds to the signals it rsceivaa. The values Ξ are id typically generated according to equation 2. As in the previous embodiment, local edge enhancement and artifact suppression is provided by delay line 180 and accumulator 82 which together produce the neighborhood average E avg. i_ Second lookup table 192 receives E avg and chooses a ±i value for T new which corresponds to the input E avg. The U U values stored in second lookup table 192 are typically generated according to equation 4. T new • k *(1 - exp(-lnE avg/E avgtau)) (4) id 4 i ij where k and tau are normalization constants as described with 4 respect to equation 3 hereinabove. If V is at the saturation level, then the value of i. T new computed in equation 4 is invalid and multiplexer 14 sets i_ V to T /2. Otherwise, multiplexer 194 sets T* to T new. i. i. id i_ It will be appreciated that the embodiment of Fig. 8A first determines T new and then performs the local edge i. enhancement and artifact suppression wheras the embodiment of Fig.83 first determines the neighborhood average I avg and then id uses it to compute the integration-time T* . id Reference is now made to Figs. A and 9B which illustrate two embodiments of the adaptive sensitivity control 12 for a system which produces color Images. Three sensor arrays 200, 202 and 204, respectively, separately but synchronously generate the red (R), green (0) and blue (B) components of an image. The R, Q and B components are individually digitized by 4 A/D converters 236, 208 and 210 respectively and the digital output is converted into the hue (H) , saturation (S) end intensity (I) color representation by color converter 212. ∑t will be appreciated that the operations described above can also be performed on output froa a single ROB color detector array. In accordance with a first embodiment of the adaptive sensitivity control 12 for color, the intensity values I are d treated exactly as the video output V is treated in the d monochrome adaptive sensitivity embodiment of Fig, 7. I i» multiplied (or shifted) by a multiplier 214, compress aed by compressor 136 and then recombined, in a combiner 216, with the corresponding hue and saturation values, H and S id ϋ r»_αpβcfc v*ly, to produce the corrected RGB signal for display. Tlv* 4±β_.*__l. αo___*_««ad KCGS olgncl £0 subβαquoatlr coπve__"bo__ -ba on analog signal by D/A converters 218, 220 and 222. The intensity value X is also utilized for id computation of the integration time T , as V was utilized in id i the monochrome embodiment depicted in Fig. 7 and described liβrβiπabσvβ. The operation is performed In update integration trim hloetr 99k which eettpviβaa cetαpavafcez. 130 (ffifif. 7) *»d Chans* T block 132 (Fig. 7} . The new integration time la identical for id all three sensor arrays 200, 202 and 204 and ia stored in row control memory 122. Thus, a single set of row control words is read out of row control memory 122 to be used for controlling the three sensor arrays 200« 202 and 204. Alternatively, three separate control circuits can be employed, for each of the three sensor arrays 200, 202 and 204, each sϋaiiar to one described in Figs * 7 and 8. 844 -~- ■ A βecond eαbodi ant of the adaptive sensitivity control for color, shown in Fig.9B, is similar to that of the first color embodiment in that the intensity I is used to calculate id tho now integration tines. However, update integration time block 224 of Fig. 9 is replaced by either of the update integration time blocks 175 of Figs.8A and Fig.8B and row control memory 122 is replaced by frame timing memory 178. Additionally, there is no 16-bit intensity value calculation nor the attendant compressor 136 since the analog video output is directly produced froa sensor arrays 200, 202 and 204. Reference is now made to Fig. 10 which illustrates three alternative image processing means useful for reducing the dynamic range of a wide range input video signal, V, without losing any details of the image. Fig. 10A comprises a small number, typically 3, of convolvers 310, 312 and 314 each receiving a different consecutive portion of the bits of the original pixel word, and together covering all of the bits of the many bit pixel word. Three selectors 311, 313 and 3 ? each produce one of the consecutive portions of the original many bit word. The convolvers typically use a 3*3 kernel, as described hereinabove in reference to Fig, f . The results of the convolvers are added together in an adder 316 to produce a smaller dynamic range signal, v. An alternative embodiment shown in Fig. 10B, employs a circtiit similar to that used in Fig.8B to compute T new and id comprises an averaging circuit 322 and a lookup table 324. The averaging circuit 322 is typically comprised of a delay line 320 and an accumulator 321 and produces an average neighborhood intensity. Lookup table implements any of the functions described hereinabove. The output of the lookup table 2 is multiplied, in a multiplier 326, with the extended dynamic range input signal V to produce the reduced dynamic range output signal v. A further embodiment of the dynamic range compressor is illustrated in Fig. OC. The circuitry is similar to that shown in Fig. 10B with the exception that the lookup table 24 is Placed before the averaging circuit 322. It will be appreciated by persons skilled in the art that the present invention is not limited to what haβ been particularly shown and described hereinabove. Rather the scope of the present invention is defined only by the claims which follow:

Claims: C I M S - - Video imaging apparatus comprising; a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation; and detector control means for controlling the response of individual ones of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range which is greater than the operative input radiation dynamic range of any given detector. 2. Apparatus according to claim 1 and wherein said means for controlling the response of the photosensitive detectors is addressable simultaneoualy to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field. 3• Apparatus according to either of claims 1 and 2 and whrein-said staring array is a generally two-dimensional array. 4. Apparatus according to any of the preceding claims and wherein said detector control means includes means for selectably and variably determining the integration time of each of the multiplicity of detectors. 5. Apparatus according to any of the preceding claims and wherein said staring array is embodied on a unitary integrated chip. 6. Apparatus according to any of the preceding claims and wherein said detector control means comprises means for providing two control signals to each of the multiplicity of detectors whereby each detactor is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals. 7• Apparatus according to any of the preceding claims 1 - 5 and wherein said detector control means comprises a control circuit for epςh dettctor of the multiplicity nf άntnatxitn . 8. Apparatus according to any of the preceding claims and wherein the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamitl range produced by operation of the detectors with variable integration times. 9» Apparatus according to any of the preceding claims and wherein said detector control means comprises means responsive to radiation incident on said multiplicity of photosensitive detectors for governing the response of said individual onea of the photosensitive detectors. 10. Apparatus according to claim 9 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon. 11. Apparatus according to claim 9 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and ' in the vicinity thereof. 12. Apparatus according to any of the preceding claims and wherein said detector control means includes means for storing a value representing the response of each of said individual ones of said photosensitive detectors and means for updating the stored values. 13. Apparatus according to any of the preceding claims 1 - 12 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of said input radiation. 14. Apparatus according to any of the preceding claims 1 - 12 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a selectable dynamic range. l^. Apparatus according to any of the preceding claims and also Qomprlsing image processing means receiving the outputs of the multiplicity of photosensitive detectors and control signlaa supplied to said multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors corresponding to a uniform sensitivity. 16. Apparatus according to claim 5 and wherein said image processing means is operative to cause said multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of said input radiation. 17 * Apparatus according to claim 15 and wherein said image prβeaaaiac meana In operative _> CXU M HK_I1 mul-_±p_.±__±t_y of detectors -to provide an output having a selectable dynamic range. 18. Apparatus according to any of the preceding cladma 1 - 17 and wherein said image processing means receives control signals representing integration times of said multiplicity of detectors and provides in response thereto modified outputs of the detectors corresponding to a uniform sensitivity. 19• Apparatus according to any of the preceding c__t.π.s and wherein said multiplicity of photosensitive detectors includes a plurality of detectors, each sensitive to a different color and wherein said detector control means is operative to control the response βf said plurality of detectors to a plurality of colors. 20. Apparatus according to claim 19 and wherein said detector control means includes means responsive to achromatic intensity at a given image location for determining the response of said plurality of detectors at said image location. 21. Apparatus according to claim 20 and wherein said •_ detector control means includes means responsive to achromatic UXCa alϊ U.Z __ given Xαiαβ-. loc_. _ _.on for- providing α. βiagla control output governing the response of said plurality of detectors at said image location. 22 * Apparatus according to any of the preceding claims and wherein each of said multiplicity of detectors comprises: a plurality of photosensitive detectors; column and row control means together determining the sample timing of said plurality of photosensitive detectors; and means for reading out a video signal froa said plurality of photosensitive detectors. 23 - Apparatus according to any of the preceding claima and wherein each of said multiplicity of de ectors comprise*t a plurality of photosensitive detectors; column and row control means together determining the precharge Liming of said plurality of photosensitive dotβetonβi and αaαns for reading: out a video aignal froa said plurality*of photosensitive detectors. 24. Apparacua acuu»21ug to claim 23 and horoin said column and row control means include means for providing multiple precharges of at least some of said plurality of photosensitive detectors in order to prevent accumulation of exceaβiva charga thereat. 25. Image processing apparatus comprising! means for receiving a video signal having a first dynamic range and for providing an output video signal having a second dynamic range narrower than said first dynamic range, whila preserving edge information όnd removing acme relatively spatiall -unvarying information. 25. Apparatus according to cliia 25 and wherein said means for receiving and providing compri4eβ at least two neighborhood operator means and summation means or combining the outputs of said at least two neighborhood operator means * 27 * Apparatus according to claim 3 and wherein said means for receiving and providing comprises at least one neighborhood operator and at least one look up tjabla which together receive said input video signal and provide a multiplication factor which is multiplied by the input video siςnal to provide said output video signal. 28. Apparatus substantial!^ as shown and described hereinabove. 29. Apparatus substantially as illustrated in any of the drawings.

Inventors: GINOSAR RAN, HILSENRATH OLIVER, KLIGLER DANIEL J, ZEEVI YEHOSHUA Y, ZINATY OFRA, GINOSAR, RAN, HILSENRATH, OLIVER, KLIGLER, DANIEL, J., ZEEVI, YEHOSHUA, Y., ZINATY, OFRA
Applicants: SOREX CORP, SOREX CORPORATION
Application year: 1989
Main classification: H04N5
IPC codes: H04N3