Abstract:
The subject matter described herein includes a camera sensor with event token based image capture and reconstruction. The sensor includes a photodetector for capturing light from a portion of a scene and for producing a signal indicative of the light. An integrator is coupled to the photodetector for accumulating charge resulting from the signal output by the photodetector and can be reset each time the charge reaches a predetermined level. An in-pixel processor is coupled to the integrator for resetting the integrator and generating an event token each time the predetermined level of charge is accumulated. A communication pipeline communicates the event tokens for downstream processing. A postprocessor is coupled to the pipeline for receiving the event tokens and for determining output intensity for the portion of the scene being reconstructed based on a number of reset events and a

Description:
PRIORITY CLAIM 
       [0001]    This application claims the priority benefit of U.S. Provisional Patent Application Ser. No. 62/011,403, filed Jun. 12, 2014, which is incorporated by reference herein in its entirety. 
     
    
     TECHNICAL FIELD 
       [0002]    The subject matter described herein relates to digital image capture and reconstruction. More particularly, the subject matter described herein relates to a camera sensor with event token based image capture and reconstruction. 
       BACKGROUND 
       [0003]    Still image and video cameras are ubiquitous in today&#39;s society. Almost all mobile telephones include an integrated camera, and consumers often select one mobile telephone over another based on the quality of the integrated camera. In addition, portable point and shoot, video, and multi-function cameras are also common. Single lens reflex (SLR) cameras have become commonplace for users that desire to enhance image capture speed and quality. 
         [0004]    Features that are desirable in any camera include a high dynamic range such that bright regions in captured images do not appear washed out and dark regions do not lose detail. Another desirable camera feature is the ability to select regions of interest in a captured image and to selectively apply higher dynamic range or other image processing enhancement to the region of interest. In video cameras, higher or variable frame rates may be desirable. It is also desirable to be able to dynamically trade off one feature, such as dynamic range, for another feature, such as frame rate. Another camera feature that may be desirable is frameless image capture, where the exposure time for the individual photodetectors (pixels) does not depend on a predefined start and stop time. 
         [0005]    These often competing goals are not achievable by today&#39;s cameras. For example, the integrators that accumulate the charge output by pixels on camera phones are typically small capacitors that are easily saturated. Once saturated, the resultant image appears white or washed out. Increasing the integration rate may yield higher quality in darker images but quicker saturation for lighter images. Using a larger integrator that takes longer to saturate may lead to lower quality in dark regions. 
         [0006]    Additional areas for improvement for existing cameras include in-pixel processing and column readout. In-pixel processing refers to on-chip processing of the output from the integrator and typically involves an analog to digital converter (ADC). Column readout refers to communication signals for a column of pixels from the photodetector array to post-processing circuitry for image reconstruction. It is possible to convert the signal output from the integrator to digital signal prior to column readout. However, analog to digital conversion prior to readout requires an ADC adjacent to each pixel, which reduces the amount of chip area for the photodiodes. Another option is to send analog values to the post processing circuitry. This solution takes up less area in the photodetector array, but analog signals are susceptible to degradation during transmission (because of noise and crosstalk, for example). 
         [0007]    Thus, in light of these difficulties, there exists a need for an improved camera sensor that achieves at least some of the goals described above. 
       SUMMARY 
       [0008]    The subject matter described herein includes a camera sensor with event token based image capture and reconstruction. The sensor includes a photodetector for capturing light from a portion of a scene and for producing a signal indicative of the light. An integrator is coupled to the photodetector for accumulating charge over time from the signal output of the photodetector which can be reset each time the accumulated charge reaches a predetermined level. An in-pixel processor is coupled to the integrator for generating an event token and resetting the integrator in response to the integrator accumulating the predetermined level of charge. A communication pipeline is provided for each column of pixels, through which event tokens are transmitted from the photodetector array to the post-processing circuitry. This circuitry comprises an array of post-processors (PPs), one per pixel, into which the arriving tokens are sorted. Each PP stores the token codes and arrival time of tokens from its corresponding column pixel. These codes and times contain all the necessary information to calculate the total pixel charge accumulated between any two tokens. It is essential that the PPs are physically removed from the pixel array so their necessarily large areas do not compete with the pixel detectors themselves. Indeed, the PPs need not necessarily be on the same chip as the pixel array. 
         [0009]    The subject matter described herein may be implemented in hardware, software, firmware, or any combination thereof. As such, the terms “function” “node” or “module” as used herein refer to hardware, which may also include software and/or firmware components, for implementing the feature being described. In one exemplary implementation, the subject matter described herein may be implemented using a computer readable medium having stored thereon computer executable instructions that when executed by the processor of a computer control the computer to perform steps. Exemplary computer readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media, such as disk memory devices, chip memory devices, programmable logic devices, and application specific integrated circuits. In addition, a computer readable medium that implements the subject matter described herein may be located on a single device or computing platform or may be distributed across multiple devices or computing platforms. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0010]    The subject matter described herein will now be explained with reference to the accompanying drawings of which: 
           [0011]      FIG. 1  is a block diagram illustrating an exemplary camera sensor with event token based image capture and reconstruction according to an embodiment of the subject matter described herein; 
           [0012]      FIG. 2  illustrates the timing relationship of events and postprocessing calculations according to an embodiment of the subject matter described herein; and 
           [0013]      FIG. 3  is a flow chart illustrating an exemplary process for event token based image capture and reconstruction according to an embodiment of the subject matter described herein. 
       
    
    
     DETAILED DESCRIPTION 
       [0014]    The subject matter described herein includes a camera sensor with event token based image capture and reconstruction. Rather than relying on the capacity of the integrator and the resolution of the A-D converter to determine output image intensity, the subject matter described herein includes in-pixel processing and readout, where the intensity of a pixel in an output image is determined by event tokens and the time of receipt of event tokens by post-processing circuitry and where an event token is generated each K times the integrator becomes full, where K is a prescaling factor. Using such a system, the output image granularity depends on the measurement granularity of the system clock, rather than the capacity or sensitivity of the integrator or an A-D converter to different voltage levels. 
         [0015]      FIG. 1  is a block diagram illustrating an exemplary camera sensor with event token based image capture and reconstruction according to an embodiment of the subject matter described herein. For clarity, the description of  FIG. 1  assumes a prescaling factor K=1. The general case of K&gt;1 is discussed later. Referring to  FIG. 1 , an exemplary camera sensor  100  includes a plurality of photodetectors  101  for detecting input light. Each photodetector  101  may be a photodiode that detects light for a single pixel used to reconstruct an image. In the example illustrated in  FIG. 1 , photodetectors and associated circuitry for a single column of pixels is shown. It is understood that the circuitry illustrated in  FIG. 1  would be replicated for plural columns to form a 2-dimensional array of photodiodes, integrators, and in-pixel processors. It is understood that a single in-pixel processor may implement the in-pixel processing for all of the pixels. Alternatively, each pixel may include a separate in-pixel processor. 
         [0016]    The camera sensor further includes an integrator  102  coupled to each photodetector  101 . Each integrator  102  may be a capacitor or other element that accumulates charge in response to the signal output from the photodetector. An in-pixel processor  104  is coupled to each integrator for generating an event token  110  in response to the charge accumulated by the integrator  102  reaching a predetermined level. For example, when an integrator  102  is full or within a predetermined amount of its maximum charge, its associated in-pixel processor  104  may produce an event token  110 . Each event token  110  may include an identifier or location of the photodetector, pixel, or integrator for which the event token is produced. Each integrator  102  is reset, either automatically each time the predetermined level of charge is reached, or in response to an external signal. 
         [0017]    A column readout communication pipeline  106  receives the event tokens for a given column and delivers the event tokens to a post processor  108 . Post processor  108 , rather than using analog intensity values that could be output by integrators  102 , uses the number of event tokens for each pixel and times of receipt of the event tokens to generate output image pixel intensities. Post processor  108  may record the time of receipt of each event token to allow accurate image intensity calculation without depending on a predefined frame start or end time. In an alternate embodiment, each event token may carry a timestamp generated by in-pixel processors  104  at the time of event token generation. 
         [0018]    According to an aspect of the subject matter described herein, in-pixel processors  104  output event tokens  110  asynchronously. Thus, rather than waiting for a rising or falling clock pulse edge, each event token is output when the event occurs. As a result, the system illustrated in  FIG. 1  may be more accurate than synchronous camera sensors where events can only be output on rising or falling clock edges. 
         [0019]    In addition, as stated above, the accuracy is greatly improved because time can be measured with finer granularity than an ADC can sense different voltage levels. For example, if a 1 gigahertz clock is used, event times can be measured to 1 nanosecond granularity. Thus, for a 1 millisecond inter-event exposure, the accuracy is 1 in 10 6  or 20 bits. Stated differently, if the lowest non-zero value of image intensity is 1, the next highest image intensity value is 1.000001. This can be contrasted with the granularity of analog to digital converters, which may only sense  12  or fewer bits of intensity levels. The ability to achieve such accuracy results in a much finer gradations in reproducing images of dark regions. This is an important feature since the human eyes&#39; ability to distinguish between intensities is proportional to ΔI/I, where I is the image intensity. 
         [0020]    An additional advantage of the system illustrated in  FIG. 1  is that there is low area cost at the pixel sensor because the ADC function is accomplished by the integrator&#39;s simple reset comparator. In addition, the transmission of event token  110  is digital, so interference from analog noise is avoided. 
         [0021]    It is crucial, however, that the column readout communication pipeline  106  have low transport jitter because if the events are time stamped by the post processor  108 , the time stamping delay caused by the pipeline should be consistent for different event tokens. It is not crucial, however, for the end-to-end propagation delay of the column pipeline  106  be low because propagation delays of two events used for intensity computation cancel each other out. The dominant cause of transport jitter is traffic congestion in the column pipeline  106 . In order to achieve low transport jitter in the column pipeline  106 , it is advantageous to use the following techniques: (i) Using the technique of frequency decimation, which reduces the rate at which events are inserted into the column pipeline  106 , and/or (ii) Using asynchronous circuits for implementing the column pipeline  106 . 
         [0022]      FIG. 2  illustrates exemplary function of the post processor for calculating pixel intensity values. Graphs  200  represent the processes for one particular pixel of the column. The graphs are aligned with each other in time. The post processor is driven in response to two asynchronous processes, one being the occurrence of token evens  202 , the other being the occurrence of readout events  203 . 
         [0023]    For expositional clarity,  FIG. 2  presumes a fixed prescaling factor K=4 and a constant luminous pixel exposure. Accordingly, the in-pixel processor generates a constant rate of reset events  201  and decimates by K to generate event tokens  202  which are transmitted through the column output pipeline. The tokens contain a pixel index and a decimation count K. Assuming a constant delay through the pipeline, the token is routed to a post process corresponding to the pixel index, arriving at a regular rate  202  which drives the said first process at times  210 ,  212 ,  213 , and  214 . 
         [0024]    The last token arrival time prior to a readout event is stored by the post processor in register T_0 as subsequently explained. The arrival of each token triggers the first process, which reads a time stamp from a global high speed system counter and stores it in register T_r. Meanwhile, it increments token counter N_t as shown in  204  and adds K to reset counter N_e as shown in  205 , which represents the number of times the in-pixel integrator has been reset since T_0, which is in turn a precise representation of the total charge integrated at the instant of occurrence of token  202 . 
         [0025]    Accordingly, after the occurrence of at least one new token at time  212 , a precise value of pixel exposure I=C*N_e/(T_r−T_0) may be calculated, where C is a constant coefficient relating radiometric pixel exposure to I. The same formula is valid after receipt of tokens at times  213  and  214 . As an example,  FIG. 2  illustrates the case where three new event tokens have been received by time  214 . With the arrival of each new token, the relative accuracy is improved as the reciprocal of N_e, because the error in (T_r—T_0) never exceeds the least significant bit of the timestamp. 
         [0026]    Concurrently, the post processor receives asynchronous readout requests which arrive as events  203 . Each readout event triggers the second process. For very dark pixels, N_t may be zero, in which case more elaborate algorithms may be invoked, some of which are discussed below. In the usual case however, N_t&gt;0 indicates at least one token has occurred since the last readout, so I may be validly calculated and returned in response. Meanwhile, the time stored in T_r is transferred to T_0, and both the token counter and the event counter are reset to zero. 
         [0027]    Prior to resetting the token counter, count  205  may be read and a new decimation factor calculated for transmittal to the in-pixel processor through the column readout pipeline. The purpose of this is to minimize the pipeline traffic, as the accuracy of the calculated I is insensitive to count N_t to within a factor of two. 
         [0028]    The accuracy of the subject matter described herein may be compared with the conventional approach of counting the number of events occurring in a frame time. In the conventional method, the formula would be C*N_e/(T_f−T_f0), where T_f0 represents time  211  of the beginning of a frame and T_f represents time  215  of the end of the frame. Since T_f0 and T_f are asynchronous with T_0 and T_r, substantial errors can occur from the tightly precise interval (T_r−T_0). The same holds true for T_f even if the two processes are resynchronized at each readout event, for example by using  203  as a global reset signal. 
         [0029]    According to the subject matter described herein, a further advantage is to increase the dynamic range over conventional cameras. The upper bound of the dynamic range is determined by the maximum rate of event tokens a bright pixel can generate, which is limited not by the capacity of the integrator but by the throughput of communication pipeline  106 . The upper bound is also limited by the maximum rate at which the in-pixel processor  104  can generate reset events. The lower bound of the dynamic range for dark pixels is determined by the minimum number of events generated in a desired readout interval. If the last event token prior to the last readout event is not available, at least two events are needed to measure the time in between events. If the last event token prior to the last readout event is available, then only one new event token is needed before the new readout event in order to measure the time between events. 
         [0030]    According to an aspect of the subject matter described herein, in-pixel processor  104  may associate a scaling factor with each event token. The scaling factor is used to pre-scale or divide the frequency of each pixel&#39;s event train. The scaling factor is also referred to as a decimation factor. The scaling factor can be used to limit the number of event tokens that are put on pipeline  106  by a single in-pixel processor  104  but still to allow accurate image reconstruction by post processor  108 . For example, if communication pipeline  106  becomes congested, post processor  108  may request that in-pixel processor  104  reduce the number of event tokens placed on pipeline  106  by a factor of four. Using such a scaling factor, each in-pixel processor  104  may only output an event token for every four reset events generated by its respective integrator  102 . Each event token  110  may either carry its associated scaling factor or post processor  108  may simply maintain the currently requested scaling factor. As a result, when an event token is received and the scaling factor is four, post processor  108  may use the scaling factor to generate a corresponding intensity that counts the single event token as four reset events. It should also be noted that the scaling factor may be set on a per pixel basis such that brighter pixels receive more decimation. In one embodiment, the scaling factor may be set to a power of 2 to facilitate the logical calculations required to implement the scaling factor. However, any integer scaling factor can be used without departing from the subject matter described herein. The scaling factor can be set such that only two event tokens are generated for each pixel within a desired exposure time, which can be arbitrary and set on a per pixel basis. 
         [0031]    In order to reconstruct or generate an image when scaling factors are used, post processor  108  may utilize the number of event tokens generated for a particular pixel, the scaling or decimation factor for each event token, and the time separation between the first and last events. Thus, if the number of event tokens is N, the time separation between the first and last events is ΔT, and if all of the most recent N−1 events have an identical decimation factor K, then the output intensity is calculated as C*K(N−1)/ΔT, where C is a proportionality coefficient. Further, if decimation factors are chosen to be powers of 2 (i.e., K=2 j ), and if a decimation strategy is used to ensure that N is 2, then the output intensity is rewritten as C*2 j /ΔT. Such an intensity representation lends itself to floating point representation (1/ΔT is the mantissa, j is the exponent), which facilitates computation. Alternatively, the reciprocal of the computed intensity is 1/C*ΔT*2 −j , which has a trivial floating-point representation that requires only a trivial negation: (ΔT, −j). As a generalization, if the number of event tokens is N, each event token could use a different decimation factor, K 1 , K 2 , . . . K N , provided the decimation factor for a pixel is updated only at the time instants when an event token is inserted into the column pipeline  106 . In this scenario, the output intensity is computed as C*(K 2 +K 3 + . . . +K N )/ΔT. This is because there are K 2  reset events between event token 1 and event token 2, there are K3 reset events between event token 2 and event token 3, and so on. 
         [0032]    The decimation factor for a given pixel can be selected by in-pixel processor  104  or by post processor  108 . For example, post processor  108  may select a decimation factor for each pixel and communicate the scaling factors to each in-pixel processor  104  using communication pipeline  106 . In one embodiment, the scaling factors may be fed from the top of pipeline  106  where each in-pixel processor accepts the scaling factor if there is an address match. In another embodiment, the scaling factors may be fed into pipeline  106  from the bottom in a counter flow fashion. 
         [0033]    In an embodiment in which each in-pixel processor  104  independently selects its own scaling factor, the in-pixel processor may utilize reset event frequency and/or event token frequency in making the selection. For example, each in-pixel processor  104  may set its scaling factor to a number greater than 1 if it determines that the number of event tokens that it is generating exceeds a certain frequency. 
         [0034]    In one embodiment, a reference timing signal is sent to all the pixels. This reference timing signal is a series of pulses with a certain period/frequency. The in-pixel processor counts the number of reset tokens and/or event tokens during this reference timing period, and accordingly sets/adjusts its scaling factor so that the number of even tokens produced during each subsequent reference timing period is no greater than a number M. The number M could be predetermined, or it could be determined by the post processor; in the latter case, it could be different for different pixels. For example, M could be 2, which means all pixels will calibrate their scaling factors so as to generate no more than two event tokens. 
         [0035]    In another embodiment, the in-pixel processor updates the scaling factor according to a predetermined sequence. For example, the scaling factor could be 1 for the first event token, 2 for the second event token, 4 for the third event token, doubling with each subsequent token. There may be an upper bound so that the scaling factor is not increased beyond the upper bound. This means that, for the given example, the event tokens will be spaced further apart as time goes on. As a result, the event token frequency of the bright pixels will quickly decrease, thereby reducing the demand on the communication network. The post process can tell the in-pixel processor to restart the scaling according to the predetermined sequence either using a special restart signal sent to all the pixels, or a special token addressed to a particular pixel through the column pipeline. 
         [0036]    According to another aspect of the subject matter described herein, frameless image capture can be achieved. There is no need to synchronize the start and stop time for defining the exposure of different pixels. In addition, the readout process for one pixel need not be synchronized with the readout process for other pixels. Each in-pixel processor  104  generates its associated event token train independently and asynchronously. Post processor  108  may keep track of the two most recent event tokens for a given pixel. Each pixel may be decimated to generate a minimum of two events during a predetermined desired readout interval. It should also be noted that the readout interval can be set on a per pixel basis so that different pixels may have different exposure times. The ability to independently define readout intervals for different pixels can be thought of as frameless image capture, and there is no defined time at which all pixels must be simultaneously read. As a result of this frameless image capture, dark pixels can be reproduced. For example, if a given pixel produces only one event token during a desired readout interval, the previously stored final event token for the readout interval can be used to generate the dark pixel&#39;s intensity value. In another example, if zero events are produced during a readout interval, the previously stored two events from one or more previous readout intervals can be utilized. Each dark region can be reproduced without lengthening the exposure time for brighter pixel. Because image generation doesn&#39;t depend on a synchronized frame start and stop time, camera sensor  100  becomes a continuously sensing camera, i.e., the shutter can remain open without saturating the pixels. In addition, the rate at which the image is read out from the processing logic may be decoupled from the per pixel rates with which the image is sensed. Moreover, different pixels of the image may be read out from the processing logic at different rates. 
         [0037]    According to another aspect of the subject matter described herein, regions of interest can be selected and enhanced independently of other regions in the image. Multiple regions of interest can be composed of pixels located at arbitrary locations in the image, not necessarily rectangular, or even contiguous. Moreover, regions of interest can be modified dynamically, for example to follow a rapidly changing feature while retaining a high quality low frame rate output of the relatively static background. A benefit is that overall energy and/or power consumed by the sensor may be lowered by expending less energy and/or power outside of the regions of interest. 
         [0038]    Some initial simulation results of the subject matter described herein indicate pixel values that are 20 bit numbers, which result in an accuracy of 1 in 2 20  (1 in 10 6 ). The dynamic range is 1:2 20 , and the signal to noise ratio is about 120 dB (decibels). Current commodity cameras typically provide 8 to 12 bit values of pixel intensity, with only some high-end ones providing 14 bit values. Their accuracy is limited by the size of their least significant bit, which is poor at low intensities, 1, 2, 3, etc. Thus, visual “contouring” is an unavoidable artifact of enhancing dark regions by simple scaling. This artifact is negligible for the subject matter described herein. 
         [0039]      FIG. 3  is a flow chart illustrating an exemplary process for event token based image capture and reconstruction. The steps illustrated in  FIG. 3  are for a single photodetector, integrator, and in-pixel processing for a single pixel. However, it is understood that the steps illustrated in  FIG. 3  would be performed by each photodetector, integrator, and in-pixel process to form a complete output image. Referring to  FIG. 3 , in step  300 , light is detected from the scene. The light may be detected by a photodetector  101 . In step  302 , charge produced by the photodetector is integrated in proportion to the received light. For example, one of integrators  102  may integrate the signal output from its respective photodetector. Steps  300  and  302  occur continuously as light is collected from the scene. In step  304 , it is determined whether a predetermined level of charge is reached. The predetermined level of charge may be an engineered value that is based on the capacity of the integrator. If the predetermined level of charge is reached, the integrator is reset and control proceeds to step  306  where the decimation counter is incremented. 
         [0040]    In step  308 , it is determined whether a predetermined decimation count has been reached. The predetermined decimation count may be the decimation count set by the in-pixel processor or the post processor based on a desired event frequency for the pixel. If the predetermined decimation count has been reached, the decimation counter is reset and control proceeds to step  310  where an event token is generated and communicated to the post processor. The event token contains the decimation count, which in this example is the same as the reset event count. In step  312 , the post processor receives and stores the event token, including the decimation count and the event token or pixel identifier. In step  314 , the post processor determines a pixel intensity based on the decimation factors and the total time between at least two event tokens. For example, if there are two consecutive event tokens, E 1  and E 2 , if the decimation factor for token E 1  is 3, the decimation factor for token E 2  is 6, and the time between E 1  and E 2  is 4, then the output pixel intensity will be proportional to (6)/4=1.5. This is because the number of reset events between consecutive event tokens E 1  and E 2  is given by the decimation factor of E 2 . The post processor may use the pixel identifiers for event tokens generated by different in-pixel processors to reconstruct the entire image from the intensities calculated for each pixel. 
         [0041]    The process of data acquisition from a spatially compact, area-critical array of analog continuous time sensors imposes performance limits in addition to those inherent to the sensors themselves. We propose methods and apparati described herein for improving this process to more fully exploit the potential of such sensors. We emphasize this is not about improving the sensors themselves, but improving the means for extracting their data. 
         [0042]    Very generally, the methods described herein can be applied to any collection of spatially distinct continuous-time continuous-level signal sources, the desired outputs of which are to be discretely digitized representations of their respective time integrals between distinct instants. Accordingly, any examples we give of preferred embodiments should not be interpreted as limiting the applicability of these methods to the technologies peculiar to these embodiments. That said, the subject matter described herein focuses on CMOS photo and video imaging technology and details of the methods and the apparati used to implement them. However, the subject matter described herein is not limited to processing signals output from video or other optical image sensors. In an alternate embodiment, the subject matter described herein may be used to process signals output from acoustic sensors, thermal sensors, or any other type of sensor that produces a continuous analog signal. 
         [0043]    It will be understood that various details of the presently disclosed subject matter may be changed without departing from the scope of the presently disclosed subject matter. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation.