Patent Description:
The following background is intended solely to provide information necessary to understand the context of the inventive ideas and concepts disclosed herein. Thus, this background section may contain patentable subject matter and should not be regarded as a disclosure of prior art.

In signal processing, there are many techniques for acquiring and reconstructing a signal. One such technique is known as compressed or compressive sensing. The concept of compressed sensing is based on the principle that a sparse signal can be reconstructed from fewer samples than required by the Nyquist-Shannon sampling theorem. Compressed sensing attempts to reconstruct a signal by finding solutions to an underdetermined system. An underdetermined system is one where there are more variables than equations. Capturing images digitally and attempting to reconstruct the images may be one example of an underdetermined system.

One design for a compressed sensing optical system, by Duarte et al. (<NPL>)), includes a digital micro-mirror device that reflect parts of an incoming light beam toward a single photodiode sensor. For every clock cycle, one part of the image may be imaged by the photodiode, and the data is sent offline to be processed with a compressed sensing algorithm. This approach requires additional processing and does not allow for multiple pixels. Furthermore, this technique may be limited to stationary objects and has a low resolution. Thus, there exists a need for a compressed sensing hardware design to allow for better sensing options with higher resolution. With better resolution, additional applications including high-speed imaging, time-of-flight sensing, and high dynamic-range imaging could be available.

<CIT> discloses: An imaging system includes an array of pixel cells and a plurality of digital memory elements disposed physically separate from and coupled to the array of pixel cells. Each of the pixel cells includes a photodetector, an electrical storage device coupled to the photodetector, and quantization circuitry coupled to the electrical storage device. The photodetector is configured to generate a photo-current in response to light impinging thereon. The electrical storage device is configured to accumulate an electrical charge from the photo-current. The quantization circuitry is configured to convert the electrical charge into an analog quantization event signal. Each of the digital memory elements is in electrical communication with at least one of the pixel cells and is configured to store a digital value in response to receiving the analog quantization event signal from the at least one of the pixel cells.

<CIT> discloses: A Complementary Metal Oxide Semi-conductor (CMOS) TDI detector stage comprising a photo-detector and a pre-amplifier containing an integration capacitor and reset switch that proportionally converts the photo-charge to a voltage. The stage further comprises a summing capacitor that is connected to the output of a prior stage and a correlated double sample (CDS) circuit that stores the integrated signal voltages and passes them onto the next stage. Each CDS circuit comprises a plurality of switches and storage circuits (e.g. capacitors). The CDS signal voltages can be passed from one TDI stage to the next along a column for summing. The CDS signal voltages of the last TDI stages may be read out with a differential amplifier. The CMOS TDI structure can be used for implementing X-ray scanning detector systems requiring large pixel sizes and signal processing circuitry that is physically separated from the photodiode array for X-ray shielding.

<CIT> discloses a focal plane imaging apparatus comprising: a plurality of photodetectors, the plurality of photodetectors comprising a first photodetector to convert a first portion of light that is scattered and/or reflected from a scene into a first analog signal and a second photodetector to convert a second portion of light that is scattered and/or reflected from the scene into a second analog signal; a plurality of analog to digital converters (ADCs), the plurality of ADCs comprising a first ADC electrically coupled to the first photodetector and configured to convert the first analog signal into a first digital signal and a second ADC electrically coupled to the second photodetector and configured to convert the second analog signal into a second digital signal; a plurality of digital registers, the plurality of digital registers comprising a first digital register electrically coupled to the first ADC and configured to store a first digital number representing the first digital signal and a second digital register electrically coupled to the second ADC and configured to store a second digital number representing the second digital signal; and a distributed control pattern generator operably coupled to the plurality of ADCs and/or the plurality of digital registers and configured to modulate, at a rate faster than a readout rate of the plurality of digital registers, storage of the first digital number with a first pseudo-random modulation and to modulate storage of the second digital number with a second pseudo-random modulation so as to control spatial correlation of the first digital number with the second digital number.

The pixel architectures may be used in high-speed imaging to integrate a desired number of time slices at certain speed settings; high-dynamic-range imaging to use a desired ceiling value of photon count to limit the output of each pixel for each time slice and then to integrate the time slices together; time-of-flight sensing to take multiple compressed images and recover time slices of each run, finding peak values of each pixel to determine time-of-flight and therefore distance; coincidence detection, and many other applications.

Unless explicitly indicated as "embodiments according to the claimed invention", any embodiment in the description may include some but not all features as literally defined in the claims and are present for illustration purposes only.

In the following section, aspects of the subject matter disclosed herein will be described with reference to example embodiments illustrated in the figures.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. It will be understood, however, by those skilled in the art that the disclosed aspects may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail not to obscure the subject matter disclosed herein.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "according to one embodiment" (or other phrases having similar import) in various places throughout this specification may not be necessarily all referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration. " Any embodiment described herein as "exemplary" is not to be construed as necessarily preferred or advantageous over other embodiments. Additionally, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Also, depending on the context of discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. Similarly, a hyphenated term (e.g., "two-dimensional," "pre-determined," "pixel-specific," etc.) may be occasionally interchangeably used with a corresponding non-hyphenated version (e.g., "two dimensional," "predetermined," "pixel specific," etc.), and a capitalized entry (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be interchangeably used with a corresponding non-capitalized version (e.g., "counter clock," "row select," "pixout," etc.). Such occasional interchangeable uses shall not be considered inconsistent with each other.

Also, depending on the context of discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. It is further noted that various figures (including component diagrams) shown and discussed herein are for illustrative purpose only, and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purpose only. Further, if considered appropriate, reference numerals have been repeated among the figures to indicate corresponding and/or analogous elements.

The terminology used herein is for the purpose of describing some example embodiments only and is not intended to be limiting of the claimed subject matter. The terms "first," "second," etc., as used herein, are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used across two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. Such usage is, however, for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments or such commonly-referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

It will be understood that when an element or layer is referred to as being on, "connected to" or "coupled to" another element or layer, it can be directly on, connected or coupled to the other element or layer or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly connected to" or "directly coupled to" another element or layer, there are no intervening elements or layers present.

The terms "first," "second," etc., as used herein, are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used across two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. Such usage is, however, for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments or such commonly-referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs.

With compressed sensing, sparsity in a signal in some domains is either assumed to be true or may be introduced to allow for solving of the image reconstruction problem. Sparsity reduces the number of variables to solve for in an underdetermined system. For example, several pixels on a pixel array may be forced to a zero value to create a sparser signal that may then be used for image reconstruction. One such approach to force some pixels to zero is to generate a mask. A mask may be a pseudo-randomly generated array of locations that can be applied to a pixel array used for capturing images. It may apply binary values (<NUM> and <NUM>) or other weighted values, to the pixels impacted by the mask.

An example compressed sensing measurement may be described with y=Ax + e, where x is the signal, such as an image, y are the samples, such as what was captured by a pixel array, A is a measurement matrix or sampling operator in an M by N matrix, and e is measurement noise. The variable x may be a vector N by <NUM>, y may be a vector M by <NUM>, and M may be less than N, typically much smaller than N. In another embodiment, the variable x may be a vector N+b by <NUM>, where b represents a shift number associated with image capture from an image capturing device, which will be described in the discussion of <FIG>. Each measurement may be the inner product, or correlation, between the signal x and a sensing function. For vector x, N may be the value of signal x sampled S times, which may be considered S-sparse. With compressive sensing, vector x is assumed to be S-sparse or has a pseudo-random mask applied to it to allow for sparsity. A compressed sensing algorithm can solve for the matrix A.

If signal x is sparse, it may have a representation in a basis Z or a redundant dictionary D, where y = Zx or y = Dx. Additionally, if A satisfies a restrictive isometry property, then the coefficients for A can be reconstructed by an L1 optimization problem: y = AZx +e, where x̂ = min∥x∥<NUM> such that ∥y - AZx̂∥<NUM> ≤ ε If a signal is x is not sparse, then a pseudo-random mask may be applied to the signal such that the optimization problem can be satisfied. Algorithms to solve for the optimization problem may include, but are not limited to, basic purist, gradient projection for sparse reconstruction, L1 regularized least squares, fixed-point continuation, fast iterative shrinkage-thresholding algorithms, deep learning, and Bayesian methods.

Compressed sensing applications include, but are not limited to, optical systems such as time-of-flight sensing, high-dynamic-range image capturing, high-speed imaging, holography, facial recognition; magnetic resonance imaging; and analog-to-digital conversion. Within optical systems, hardware designs of pixel arrays and data acquisition architecture are needed to enable compressed sensing.

<FIG> is an example pixel <NUM> that may be used in an example embodiment of a pixel array that may be used in some example embodiments of a readout method for compressed sensing. Pixel <NUM> includes a photon sensor <NUM>, a memory <NUM>, and an accumulator <NUM>. Photon sensor <NUM> may be used to collect photon information values and may be a digital or analog element, such as a photodiode, pinned photodiode, single photon avalanche diode (SPAD), avalanche photodiode (APD), quanta image sensor (QIS), contact image sensor (CIS), charge-coupled device (CCD), bolometer, or other sensor type. Photon information values produced by photon sensor <NUM> may be digital, such as a count of photons, or may be analog, such as a charge, depending on the particular photon sensor <NUM> used. Memory element <NUM>, which may be referred to as memory <NUM>, may be a digital or analog memory such as RAM, SRAM, DRAM, or a register, capacitor, or other memory source. Photon sensor <NUM> may be connected to accumulator <NUM>. In some example embodiments, accumulator <NUM> may be a register that may act as a temporary storage to hold intermediate values. Accumulator <NUM> may also add together values. For example, a value already in the accumulator may be added to a value (e.g. photon information) received from photon sensor <NUM>. Memory <NUM> may be connected to accumulator <NUM>. During a photon detection event, photon sensor <NUM> may convert photons into an electrical signal and may send data to accumulator <NUM>. Additionally, memory <NUM> may send data to accumulator <NUM>, which may add data together. A clock signal <NUM> may be used to send a clock signal on a data bus <NUM>. Data bus <NUM> may be connected to memory <NUM> and photon sensor <NUM> (not shown) and may send clock signal <NUM>. Data bus <NUM> may be used for sending clock signal <NUM> to pixel <NUM>, reading out data from pixel <NUM>, or both. In some embodiments, a separate bus may be used to send data and clock signals (not shown). In some embodiments, after one clock cycle, photon sensor <NUM> may be reset to a zero value. Furthermore, accumulator <NUM> may also be reset after data is transmitted to a memory <NUM> of a different pixel <NUM>, as will be described with regard to <FIG>.

In some embodiments, memory <NUM> and accumulator <NUM> may be combined into a single logical unit that performs the same tasks of memory storage and accumulation, respectively. For example, a capacitor may be an accumulator and a memory storage device and may be used in an analog pixel design. An existing charge in a capacitor is accumulated and stored with additional charge added from the output of a photon sensor <NUM>.

In an embodiments according to the claimed invention, a mask memory <NUM> connects to photon sensor <NUM> and provides logic to stop photon sensor <NUM> from sensing. In another embodiment according to the claimed invention, mask memory <NUM> is connected to accumulator <NUM> and provides logic to stop accumulator <NUM> from adding values from photon sensor <NUM>.

A data bus <NUM> may be connected to mask memory <NUM> to provide data transfer capabilities. Mask memory <NUM> and data bus <NUM> may be elements used in alternative embodiments for compressed sensing applications that utilize masks, which will be described with respect to <FIG> and <FIG>.

<FIG> is an example pixel <NUM> that may be used in an example embodiment of a pixel array that may be used in some example embodiments of a readout method for compressed sensing. Pixel <NUM> may have the elements of pixel <NUM> of <FIG>, as described above, except data transfer from accumulator <NUM> may be connected to an output selector <NUM>, which may include a memory used to readout values from accumulator <NUM>. Data bus <NUM> may be connected to output selector <NUM> and may provide commands to read out values from accumulator <NUM>.

<FIG> is an example pixel column (or row) <NUM> that may be used by an example embodiment according to the claimed invention of a pixel array that is used in some example embodiments according to the claimed invention of a readout method for compressed sensing. Pixel column <NUM> depicts n example pixels 100a, 100b, and 100c connected together in series as well as an n-th final pixel, 100n (note that n may be any number greater than <NUM>). For example, pixel 100a is connected to pixel 100b, and pixel 100b is connected to 100c, and so on. Each pixel may have photon sensors 101a to 101n. Accumulator 103a is connected to memory 102b and accumulator 103b is connected to memory 102c, and so on such as for accumulator 103c and onward. Data bus <NUM> may be connected to memories 102a, 102b, 102c, all the way to memory 102n, and may provide a clock signal <NUM> from <FIG> or <FIG> to pixels 100a-100n. Data bus <NUM> may be connected to mask memories 106a, 106b, 106c, all the way to mask memory 106n, and may provide mask data from <FIG> to pixels 100a-100n. Accumulator 103n may be connected to an offline storage <NUM> that may be used to collect and process compressed sensing data collected from pixel column <NUM>.

In one embodiment, pixel column <NUM> may comprise a digital pixel architecture where photons may be counted and transferred as digital packets of information between pixels. In other embodiments, pixel column <NUM> may comprise analog pixel architecture where photons may be converted to a charge and the charge may be transferred between pixels.

In other embodiments, data from pixel 100a may not be sent to the memory of pixel 100b and instead may send it to pixel 100c. In yet other embodiments, data from pixel 100a may be sent to a pixel later in pixel column <NUM> (not shown). In yet other embodiments, there may be additional pixel rows <NUM> and pixels from a first row may be connected to pixels from a second row, as will be described further with respect to <FIG>. In other embodiments, a pseudo-random mask may be applied to one or more pixel column <NUM> to force values of select photon sensors 101a-101n to return a zero or a non-zero value, which will be discussed in more detail with respect to <FIG>. That is, the accumulators from one pixel need not feed into the memories of immediately adjacent pixels. There are many accumulator-memory feed patterns, as will be apparent to one after reading the present disclosure.

<FIG> is an example pixel column (or row) 200b that may be used by an example embodiment of a pixel array that may be used in some example embodiments of a readout method for compressed sensing. Pixel column 200b may have the elements of pixel column <NUM> of <FIG> except that it may use pixel <NUM> of <FIG> instead of pixel <NUM> of <FIG>. In one embodiment, pixel column 200b may have data transfer out of pixels 100a to 100n through a set of output selectors 108a to 108n. For example, for pixel 100a, when a data read request for pixel 100a is sent via data bus <NUM>, output selector 108a may select a value from accumulator 103a which may then send the value to output selector 108a and via data bus <NUM> to offline storage <NUM> or to another pixel 100a-n. This process may be repeated for any pixel 100a to 100n in pixel column 200b.

<FIG> is an example pixel array <NUM> that may be used in some example embodiments of a readout method for compressed sensing. Pixel array <NUM> may have a first column (or row) <NUM> comprising a plurality of pixels. In the present example, eight pixels 100a to <NUM> are shown, which may be pixels 100a to 100n of column <NUM> of <FIG> or column 200b of <FIG>. There may be a second column <NUM> with pixels 302a to <NUM>, a third column <NUM> with pixels 303a to <NUM>, and so on to column <NUM> with pixels 304A-B to <NUM>, which may likewise be similar to column <NUM> of <FIG> or column 200b of <FIG>. In other embodiments, there may be different row and column dimensions of pixel array <NUM>, from <NUM> by <NUM> to any other size. As mentioned previously with respect to <FIG>, pixels in first column <NUM> may be connected to each other (for example, the accumulator of one pixel may be connected to the memory of another pixel), for example, pixel 100a may be connected to pixel 100b, pixel 100b may be connected to pixel 100c, and so on. In other embodiments, pixel 100a from first column <NUM> may be connected to pixel 302b of second column <NUM>, which may then be connected to pixel 303c of third column <NUM>, and so on. In other embodiments, there may be other pixel connection configurations across columns, rows, or a combination of both columns and rows.

<FIG> is an example data flow and timing diagram <NUM> that may be used in an example embodiment of compressed sensing for a single column or row of pixels in pixel array <NUM> of <FIG>. The data flow described below may be applied to all or some pixels of pixel array <NUM> of <FIG>. In one embodiment, timing diagram <NUM> may be an example of a pixel column <NUM> of <FIG> with adding and linear shifting of data, without using a mask. In another embodiment, timing diagram <NUM> may be an example of another series of connected pixels.

<FIG> depicts a series of connected pixels 100a to <NUM>, with their respective memories 102a to <NUM>, of first column <NUM> of <FIG>, and offline storage <NUM> of <FIG> and <FIG>, over the course of an example <NUM> clock cycles, (t=<NUM> to t=<NUM>). Pixels 100a to <NUM> may have accumulators 103a to <NUM> (not shown). There are three clock cycles where a shutter value is set to ON, or open, and four clock cycles with a shutter value set to OFF, or closed. A shutter may be a physical shuttering mechanism or it may be a logical shutter. A logical shutter may allow photon sensors 100a - <NUM> to be activated or deactivated.

A shutter value may be a logical state when imaging is occurring or when it is not occurring. A shutter value of ON or open may represent a state where photon sensors 100a - <NUM> are able to sense photons. A shutter value of OFF or closed may represent a state where photon sensors 100a - <NUM> are not able to sense photons.

In this example, data may be transferred between adjacent pixels, for example, data from pixel 100a's accumulator 103a may be transferred to pixel 100b's memory 102b, and data from pixel 100b's accumulator 103b may be transferred to pixel 100c's memory 102c, and so on. For additional rows where there is no pixel, the data can be considered moved to/amongst and stored within one or more locations within offline storage <NUM> of <FIG>. The logic used to describe the data flow in column <NUM> may be applied to other columns of pixel array <NUM> of <FIG>. In general terms, the value of a pixel 100a at time <NUM> will be denoted 100a,<NUM>, and at time <NUM> will be denoted 100a,<NUM>, and so on. The numbers in <FIG> will use this format.

In the first clock cycle (t=<NUM>), pixels 100a to <NUM> image a scene, not shown. Pixel 100a has <NUM> in corresponding memory 102a. Pixel 100a may receive a first photon information value, which may be a count of photons or an electrical charge, and may store it in accumulator 103a. The value stored in accumulator 103a is added to a second photon information value in memory 102a (which at t=<NUM> may be <NUM>), forming a third photon information value, and the third photon information value is shifted to memory 102b of pixel 100b. In one embodiment, the shifting is performed at the end of the same clock cycle. The photon sensor 101a associated with pixel 100a and the accumulator 103a may be then be set to zero before the next clock cycle, or at the beginning of the next clock cycle before imaging begins. In another embodiment, the shifting may be performed in the beginning of the next clock cycle. That is, pixels capture photon counts, store them in their associated accumulators, add the memory values to the accumulator values, and shift the new value to the next memory (which may be a neighboring pixel, a pixel located elsewhere on the pixel array, or a next memory location in offline storage <NUM>). In one embodiment, the pixels capture charges, store them in associated accumulators that also serve as memory, accumulate the charges in memory and from the pixels together, and transfer the charge to an accumulator associated with a different pixel in the pixel array. The photon sensor of pixel 100a (not shown) may be reset to zero in preparation for sensing in the next clock cycle. After the shifting of data, and before any new photon sensing, the photon sensors and accumulators 103a-h are reset.

In the second clock cycle (t=<NUM>), pixel 100b receives a count of photons and may store it in accumulator 103b. The value stored in accumulator 103b is added to the value in memory 102b, and shifted to memory 103c of pixel 100c. The photon sensor 101b and accumulator 103b of pixel 100b (not shown) may be reset to zero in preparation of sensing in the next clock cycle.

Meanwhile, in the first clock cycle (t=<NUM>) pixel 100b has <NUM> in corresponding memory 102b. Pixel 100b may receive a count of photons and store it in accumulator 103b. The value stored in accumulator 103b is added to memory 102b and shifted to memory 102c. In one embodiment, the shifting is performed at the end of the same clock cycle. In another embodiment, the shifting may be performed in the beginning of the next clock cycle. The photon sensor of pixel 100b (not shown) may be reset to zero in preparation of sensing in the next clock cycle. In the second clock cycle (t=<NUM>), pixel 100c receives a count of photons and may store it in accumulator 103c. The value stored in accumulator 103c is added to the value in memory 102c and shifted to memory 103d. The accumulator 103b element of pixel 100c may be reset to zero in preparation of sensing in the next clock cycle.

The above operations may be repeated for each pixel of pixel array <NUM>. Thus, for each clock cycle, each pixel 100a to <NUM> may add the value of its photon count to the value in its memory 102a to <NUM>, which may be the value provided to it from a previous pixel for all t><NUM>. The value may then be sent to the memory of the next attached pixel.

At time t=<NUM>, the clock shutter status changes from ON to OFF, and pixels 100a to <NUM> no longer detect or count photons. The values in each cell memory will no longer be modified by the accumulator, and will be shifted over time to offline storage <NUM>, which is the recipient of data shifts from the last pixel's memory (in this example, <NUM>).

Data stored in offline storage <NUM> may be used for compressed sensing signal reconstruction. Data may be shifted between memories to offline storage <NUM>. In one embodiment, at time t=<NUM>, data from pixel <NUM> from time t=<NUM> may be shifted to offline storage <NUM>. Each data value may be shifted to another pixel memory location, to offline storage <NUM>, or may remain in the same memory location for each clock cycle.

In other embodiments, data stored in memory 102a may be sent to memory 102c or to other memories in pixel column <NUM> of <FIG>. In other embodiments, memory 102a may send data to memory in other columns of <FIG>. In other embodiments, memory 102a may be sent back and forth between memory of another pixel and itself. In other embodiments, data can be transferred only for a region of interest, that is, a subset of pixels within pixel array <NUM> of <FIG>. That is, as mentioned above, each pixel's accumulator need not be connected to the memory of an immediately adjacent pixel- other interconnect patterns are possible.

In other embodiments, a pseudo-random mask may be applied to pixel array <NUM> of <FIG>. Referring back to <FIG>, for example, first column <NUM> may have a pseudo-random mask applied, which is discussed in more detail with respect to <FIG>. Other pixel types, such as pixels in pixel column 200b may be used. In one embodiment, a pseudo-random mask may be the same for all clock cycles. In yet other embodiments, where memory values may not be shifted and may stay stored in the same memory units, a changing pseudo-random mask may be applied. This pseudo-random mask may change for each clock cycle. In one embodiment, masks may be in a series and that series may be repeated. For example, if there are four mask patterns A, B, C, and D, a mask series ABCD may be used and may be repeated over a number of clock cycles. Mask values may be binary, such as <NUM> or <NUM>, such that the output value of photon sensors 101a-<NUM> under the influence of the mask may be forced to <NUM> or may remain unchanged. The further operation of masks with regard to pixels is described further below with regard to <FIG>.

The technique described above may be used for any type of image sensor. As will be described more with regard to <FIG>, the hardware configurations may vary depending on the type of image sensor.

In some embodiments, the compressed sensing measurement y=Ax+e may have the variable x change from an N by <NUM> matrix to an N+b by <NUM> matrix, where b represents the number of "data shifts" that have occurred during an imaging (i.e. shutter ON). For example, if there are three shifts for three time periods, x may be of size N + <NUM> by <NUM> instead of N + <NUM>.

<FIG> is an example data flow and timing diagram <NUM> that may be used in an example embodiment of compressed sensing for a single column or row of pixels <NUM> in pixel array <NUM> of <FIG>. <FIG> also describes the impact of a mask on the pixel array <NUM> and individual pixels <NUM>. The data flow described below may be applied to all or some pixels of pixel array <NUM> of <FIG>. The use of a mask as described below and the data flow may be applied to the logic of <FIG>. In one embodiment, timing diagram <NUM> may be an example used with a pixel column 200b of <FIG> with accumulators 103a-<NUM>, output selectors 108a-<NUM>, and mask memories 106a-<NUM>. Pixel 100a may be connected to data selector 108a, pixel 100b may be connected to output selector 108b, and so on for pixel <NUM> being connected to output selector <NUM>. Mask memory 106a may be part of pixel 100a, mask memory 106b may be part of pixel 100b, and so on for mask memory <NUM> being connected to pixel <NUM>.

In alternative embodiments, pixel column 200a of <FIG> may be used or another series of pixels in a pixel array may be used. Mask memories 106a to <NUM> may be used with the pixels 100a to <NUM> and data flow operations in <FIG> or any other arrangement of pixel connections.

In general terms, the value of a pixel 100a and mask memory 106a at time <NUM> will be denoted 100a,<NUM> and 106a,<NUM>, respectively, at time <NUM> will be denoted 100a,<NUM> and 106a,<NUM> respectively, and so on. The numbers in <FIG> may use this format.

<FIG> depicts a series of connected pixels 100a-h, with their photon sensors 101a-h , their respective memories 102a to <NUM>, and their respective mask memories 106a-h, of first column <NUM> of <FIG> with four clock cycles (t=<NUM> to t=<NUM>). There are three clock cycles where a shutter value is set to ON and one clock cycle with a shutter value set to OFF. In this example, data may remain in each pixel after each cycle, for example, data from pixel 100a's accumulator may be accumulated in memory 102a, and data from pixel 100b's accumulator may be accumulated in memory 102b, and so on. That is, in the present embodiment, each pixel transfers the results of a cycle to its own memory, rather than to a second pixel in a series.

In the first clock cycle (t=<NUM>), pixels 100a to <NUM> image a scene, not shown. Pixel 100a has <NUM> in corresponding memory 102a. Pixel 100a's photon sensor 101a may receive a first photon information value, which may be a count of photons or an electrical charge, that may be multiplied by a binary value from mask memory 106a and the result may be stored in accumulator 103a. If mask memory 106a is equal to <NUM>, the first photon information value accumulated may be stored. If mask memory 106a is equal to <NUM>, the accumulator may store a <NUM> value. This multiplication may be logical, or it may be the result of a physical operation; for example, the mask may enable or disable the photon sensor 101a, resulting in a normal reading or a <NUM> reading. The value stored in accumulator 103a is added to memory 102a. In one embodiment, pixels capture photon counts, masks transmit binary values that are multiplied by the pixel photon counts, the result is stored in their associated accumulators, and the memory values are added to the accumulator values, and may be stored in the memory once more. The photon sensor 101a and accumulator 103a of pixel 100a may be reset to zero in preparation of sensing in the next clock cycle.

In the second clock cycle (t=<NUM>), pixel 100a's photon sensor 101a may receive a first photon information value, the count may be multiplied by the value stored in mask memory 106a, and the resulting value may be stored in accumulator 103a. The value stored in accumulator 103a is added to a second photon information value in memory 102a, and the result may be stored in the memory once more as a third photon information value. The photon sensor 101a and accumulator 103a of pixel 100a may be reset to zero in preparation of sensing in the next clock cycle.

Meanwhile, in the first clock cycle (t=<NUM>) pixel 100b has <NUM> in corresponding memory 102b. Pixel 100b's photon sensor 101b may receive a count of photons that may be multiplied by a binary value from mask memory 106b and the result may be stored in accumulator 103b. If mask memory 106b is equal to <NUM>, the photon count may be stored. If mask memory 106b is equal to <NUM>, the accumulator may store a <NUM> value. The value stored in accumulator 103b is added to memory 102b and stored in memory 102b. Photon sensor 101b and accumulator 103b may be reset to zero in preparation of sensing in the next clock cycle. In the second clock cycle (t=<NUM>), pixel 100b's photon sensor 101b may receive a count of photons, the count may be multiplied by the value stored in mask memory 106b, and the resulting value may be stored in accumulator 103b. The value stored in accumulator 103b is added to the value in memory 102b, and the result is stored in memory 102b. The photon sensor 101b and accumulator 103b may be reset to zero in preparation of sensing in the next clock cycle. In another embodiment, accumulator 103b and memory 102b may be a single unit that serves the purpose of both accumulation and memory storage and during reset the accumulated values may be saved in the memory 102b that is associated with accumulator 103b.

The above operations may be repeated for each pixel of pixel array <NUM>. Thus, for each clock cycle, each pixel 100a to <NUM> may multiply the value of its photon count by its mask value and add the resulting value to its memory 102a to <NUM>, which may be accumulated for t><NUM> when the shutter is on. The value may then be sent to the memory of the next attached pixel.

At time t=<NUM>, the clock shutter status changes from ON to OFF, and pixels 100a to <NUM> no longer count photons. The values in each cell memory will no longer be modified by the accumulator, and will be read by output selector elements (not shown) such as element <NUM> of <FIG>. This data may be sent to offline storage <NUM>. Data stored in offline storage <NUM> may be used for compressed sensing signal reconstruction. The accumulators/memories of the pixels may be reset.

In some embodiments, the mask values may change for each clock cycle. In other embodiments, the mask values may remain constant. In various embodiments, the number of shutter open and shutter closed cycles may vary.

<FIG> is an example process <NUM> that may be used in an example embodiment of the data flow and timing diagram of <FIG> or <FIG>.

Process <NUM> may begin imaging using pixel array <NUM> of <FIG>. A shutter value may be a logical value that may switch from OFF to ON.

Process <NUM> may begin capturing data in pixels <NUM> of pixel array <NUM>. In some embodiments, all pixels <NUM> of pixel array <NUM> may be used to capture data. In other embodiments, a subset of pixels may be used to capture data.

Process <NUM> is a process in an alternative embodiment to apply a pseudo-random mask to pixels <NUM> of pixel array <NUM>. In some embodiments, the pseudo-random mask may be the same pattern throughout the entire imaging process. In other embodiments, the pseudo-random mask may vary by cycle. In some embodiments, process <NUM> may be skipped if a mask is not used. In an embodiment according to the claimed invention, a mask allows or blocks the photon sensor <NUM> from sensing, or in another embodiment according to the claimed invention, it allows or blocks the value of photon sensor <NUM> from being accumulated. A mask may have a value such as <NUM> or <NUM> multiplied by the value of photon sensor <NUM> or may block the charge from photon sensor <NUM> from being stored.

Process <NUM> may accumulate, or add together, values captured in photon sensor <NUM> and memory <NUM> of pixels <NUM> in pixel array <NUM>.

Process <NUM> may store data accumulated in process <NUM>. Process <NUM> may also include resetting photon sensors <NUM> back to zero.

Process <NUM> is an operation in an alternative embodiment that may shift data stored from process <NUM>. In embodiments according to the claimed invention, data is shifted to memory <NUM> associated with a different pixel <NUM>. In other embodiments not according to the claimed invention, data may remain in the same memory <NUM> and may not ever be shifted. In yet other embodiments not according to the claimed invention, data may transfer back and forth between a first memory (such as of a first pixel <NUM>) and a second memory (such as of a second pixel <NUM>).

Process <NUM> may repeat processes <NUM> to <NUM> until a shutter value of OFF is received.

Process <NUM> may transfer data from pixel array <NUM> to offline storage <NUM> for compressed sensing processing.

<FIG> is an example pseudo-random mask <NUM> that may be used in an example embodiment of compressed sensing. Pseudo-random mask <NUM> may comprise a region of interest <NUM> and a masking region <NUM>. Pseudo-random mask <NUM> may block some pixels in pixel array <NUM> from imaging in region of interest <NUM>. Further, a masking region <NUM> may block all pixels outside of a region of interest <NUM> from imaging and may be used to reduce the imaging area on pixel array <NUM>. Pseudo-random mask <NUM> may be used during the processes of <FIG>/<FIG> as described previously. For each clock cycle, pseudo-random mask <NUM> may remain constant or may be dynamic and change. The values of pseudo-random mask <NUM> may be stored for later use during signal reconstruction.

As will be described further with regard to <FIG>, a pseudo-random mask <NUM> may be used for lossless reconstruction by capturing differential images. A pseudo-random mask <NUM> may also be used when data is too dense and may add sparsity to an image to find a solution for image reconstruction. The type of pseudo-random mask <NUM> used may vary by the type of imaging task. A pseudo-random mask <NUM> may be a Bernoulli, Hadamard, Gaussian, discrete cosine transform (DCT), or other mask type. Pseudo-random mask <NUM> may include binary, greyscale, colored, or other values.

In some embodiments, mask <NUM> may be stored in offline storage <NUM> for use during image reconstruction. Mask <NUM> may be generated by a processor (not shown) and provided to pixels in pixel array <NUM> of <FIG> during imaging. Mask <NUM> may be supplied by offline storage <NUM> or may be generated by a processor (not shown) and sent to pixels via data bus <NUM>. Mask <NUM> may be stored in mask memories 106a to 106n to be used by pixels 100a to 100n of <FIG> and <FIG>. Mask <NUM> may be stored in offline storage <NUM> for use during signal reconstruction.

<FIG> is a schematic diagram of another example embodiment of a pixel architecture <NUM> with a secondary memory used for integration. Integration may be the accumulation of photon information within a single memory. <FIG> shows an example pixel column <NUM> of <FIG> or column 200b of <FIG>. Pixel 801a may be a pixel in pixel column <NUM>. Photon sensor 101a may be a photodiode. There may be a second memory 802a, a second accumulator 803a, and a second data path <NUM> that may also be connected to clock signal <NUM>. During imaging, second memory 802a may accumulate values provided by second accumulator 803a which may collect data from photon sensor 101a. For each clock cycle when imaging occurs, second memory 802a may store values and may not shift values to other memories. When imaging is finished, second data path <NUM> may be used to transfer data from memory 802a to a memory 802b, 802c, and/or to offline storage <NUM> (not shown). In alternative embodiments, second data path <NUM> may be connected to output selector <NUM> (not shown), such as in <FIG> and data may be transferred to offline storage <NUM> by selection instead of shifting data through other pixels to offline storage <NUM>.

<FIG> is a schematic diagram of another example embodiment of a digital pixel architecture <NUM> with a SPAD sensor. <FIG> shows an example pixel column <NUM> of <FIG> or column 200b of <FIG>. Pixel 901a may be a pixel in pixel column <NUM>. Photon sensor 101a may be a single photon avalanche diode or other digital sensor. There may be a counter 902a to 902c and a second data path <NUM> that may also be connected to clock signal <NUM>. Counter <NUM> may count photons detected by photon sensor 101a, accumulate and store this data for each clock cycle, and transfer data. During imaging, counter 902a may accumulate, or integrate, values provided by photon sensor 101a. For each clock cycle when imaging occurs, counter 902a may store values and may not shift values to other counters. When imaging is finished, second data path <NUM> may be used to transfer data from a memory from counter 902a to a counter 902b, 902c, and/or to offline storage <NUM> (not shown). In another embodiment, counter 902a may employ an algorithm to enable high-dynamic-range imaging. If photon sensor 101a is saturated, or has a photon signal stronger than the sensor is capable of sensing, counter 902a may employ an algorithm to avoid saturation.

<FIG> is a schematic diagram of an example embodiment of a time-resolving pixel architecture <NUM>. <FIG> shows an example pixel column <NUM> of <FIG> or column 200b of <FIG>. Pixels 1001a to 1001c may be pixels in pixel column <NUM>. Photon sensor 101a may be combined with memory accumulator unit 1002a to represent an analog output sensor, such as a CCD, CIS, or QIS pixels that may transfer a charge to neighboring pixels. Similarly, photon sensor 101b may be combined with memory accumulator 1002b and photon sensor 101c may be combined with memory accumulator 1002c. Photon sensor 101a may transfer charge to combined memory accumulator unit 1002a. The time resolution of photon sensor 101a may be based on the clock period of clock signal <NUM>. For example, if a clock period is shorter, there may be fewer photons that can be detected (higher resolution). If a clock period is longer, there may be more photons that can be detected (lower resolution). During a sensing event, for a single pixel 1001a, a detected charge may be transferred from photon sensor 101a to unit 1002a, that may be a combined memory and accumulator unit. Unit 1002a may have a memory module that includes a mode to store memory and an accumulator module that includes a mode to accumulate charge. After a clock cycle, charge may be transferred from unit 1002a to a unit 1002b associated with pixel 1001b. After sensing completes, charge stored in unit 1002a may transfer to unit 1002b, which may then transfer to a unit 1002c, which may send charge or a digitized representation of charge to offline storage <NUM> (not shown). In another embodiment, charge stored in unit 1002a may be connected to output selector <NUM> (not shown), such as in <FIG> and data may be transferred to offline storage <NUM> by selection instead of shifting data through pixels.

<FIG> is an example embodiment of a differential pixel architecture <NUM> with a pseudo-random mask <NUM> applied. <FIG> shows an example pixel column <NUM> of <FIG> or column 200b of <FIG>. Pixels 1105a to 1105c may be pixels in pixel column <NUM>. Photon sensors 101a to 101c may be an analog output sensors, such as a CCD, CIS, or QIS sensors. In some embodiments, photon sensor 101a to 101c may be digital output sensors. Photon sensor 101a may transfer charge (in the analog cases) or digital values (in the digital cases) to combined memory accumulator units 1100a and 1100b. The resolution of photon sensor 101a may be the clock period of clock signal <NUM>. During a sensing event, mask <NUM> may be applied to pixels 1105a to 1105c. For a single pixel 1105a, a detected charge/value may be transferred from photon sensor 101a to units 1100a and 1100b, which may be a combined memory and accumulator unit. Units 1100a and 1100b may have a memory module and mode to store memory and an accumulator module and mode to accumulate charge. Unit 1100b may receive a charge/value multiplied together with a mask value from a pixel on mask <NUM>. Unit 1100a may receive a charge/value multiplied together by the inverse or reverse mask value from a pixel on mask <NUM>.

For example, if the value of a pixel of mask <NUM> is <NUM>, the inverse or reverse value of the pixel may be <NUM>. By providing values of mask <NUM> and reverse values of mask <NUM> to pixels in pixel architecture <NUM>, a lossless signal reconstruction may be achieved.

After a clock cycle, charges/values may be transferred from unit 1100a to a unit 1101a and from unit 1100b to a unit 1101b associated with pixel 1105b. After sensing completes, charges/values stored in unit 1100a may transfer to a unit 1101a, and may then transfer to a unit 1102a, which may send the charges/values to offline storage <NUM> (not shown). A similar charges/value transfer may occur for units 1100b to 1102b. This may be used for a lossless image reconstruction. In another embodiment, charge stored in unit 1100a may be connected to output selector <NUM> (not shown), such as in <FIG> and data may be transferred to offline storage <NUM> by selection instead of shifting data through pixels.

Embodiments of the subject matter and the operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of the foregoing. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer-program instructions, encoded on a computer-storage medium for execution by, or to control the operation of, a data-processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. A computer-storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random-or serial-access memory array or device, or a combination thereof. Moreover, while a computer-storage medium is not a propagated signal, a computer-storage medium may be a source or destination of computer-program instructions encoded in an artificially generated propagated signal. The computer-storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Additionally, the operations described in this specification may be implemented as operations performed by a data-processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

While this specification may contain many specific implementation details, the implementation details should not be construed as limitations on the scope of any claimed subject matter, but rather be construed as descriptions of features specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Claim 1:
A method for processing imaging data from a set of pixels comprising a first pixel (100a) and a second pixel (100b), wherein each of the first pixel (100a) and the second pixel (100b) comprises:
a photon sensor (<NUM>);
a memory (102a, 102b)
an accumulator (<NUM>); and
a mask memory (<NUM>) connected to the photon sensor (<NUM>) or the accumulator (<NUM>) providing a logic signal to the photon sensor (<NUM>) or the accumulator (<NUM>),
wherein if the mask memory (<NUM>) is connected to the photon sensor (<NUM>), the method comprises:
receiving a clock signal (<NUM>) at the set of pixels, whereupon the first pixel (100a) performs the actions of:
allowing or stopping the photon sensor (<NUM>) from sensing depending on the logic signal from the mask memory (<NUM>);
collecting a first photon information value from the photon sensor (<NUM>); storing the first photon information value in the accumulator (<NUM>);
adding a second photon information value from the memory (102a) of the first pixel (100a) into the accumulator (<NUM>) to obtain a third photon information value;
storing the third photon information value in the memory (102b) of the second pixel (100b); and
resetting the photon sensor (<NUM>) and the accumulator (<NUM>) to zero,
wherein if the mask memory (<NUM>) is connected to the accumulator (<NUM>), the method comprises:
receiving a clock signal (<NUM>) at the set of pixels, whereupon the first pixel (100a) performs the actions of:
collecting a first photon information value from the photon sensor (<NUM>);
allowing or stopping the accumulator (<NUM>) to accumulate the first photon information value depending on the logic signal from the mask memory (<NUM>);
adding a second photon information value from the memory (102a) of the first pixel (100a) into the accumulator (<NUM>) to obtain a third photon information value;
storing the third photon information value in the memory (102b) of the second pixel (100b); and
resetting the photon sensor (<NUM>) and the accumulator (<NUM>) to zero.