# The Quanta Image Sensor: Every Photon Counts
## 2 Creating Images from Jots  
Readout of jets results in a bit cube of data, with two dimensions representing spatial dimensions of the field-of-view of the sensor, and the third representing time. Each bit-plane slice is a single readout field. For single-bit QIS devices, the jet data cube is binary in nature. For multi-bit QIS, the jet data cube consists of words of bit length corresponding the readout quantization bit depth described in more detail below. In perhaps the simplest form, image pixels can be created from the sum of a small x-y-t \"cubic\" of bits from the jet data cube (see Figure 1). The dimensions of the cubicle determine the spatial and temporal resolution of the output image, and all jet values within the cubicle are weighted equally. The maximum signal-to-noise ratio (SNR) of the image pixel (assuming an ensemble of pixels created from the same illumination and readout conditions) is determined by the size of the cubicle. For example, a cubicle of size 16 \(\times\) 16 \(\times\) 16 of single-bit QIS jets summed together would have a maximum value of 4096 and a maximum SNR of \(\sqrt{4096}=64\). Illustration of image formation from simulated jet data is shown in Figure 2. Note that cubicles do not have to have equal dimensions in x, y and t. Furthermore, all image pixels need not be formed from the same sized cubicles, and their cubicles may overlap. These choices can be made post-capture and on an output frame-by-frame basis to optimize particular imaging aspects such as the trade-off between image SNR and spatial-temporal resolution. An example of this processing was recently demonstrated using a QIS-like device with SPAD-based jets [28] where the pixels near the edges of rotating fan blade were processed differently than either the slowly varying body of the blades or the static background. Using cubicles that have a non-orthogonal trajectory in the time dimension can be used for performing operations analogous to time-delay and integration (TDI) but along an arbitrary track direction [6]. Different objects in the scene with different motion trajectories could be processed with independent tracks to improve SNR. The utility of such processing is enhanced by deep sub-electron read noise in the QIS and quantizer, so that the noise is inherently low and does not accumulate like \(\sqrt{M}\), where \(M\) is the ensemble size of jets summed together. Measuring photon flux with a single-pixel photon-counting photomultiplier tube was reported as early as 1968 [38]. The generation of an image with improved SNR from a series of readout fields goes back to 1985 when photon-counting detectors were used this way for astronomy purposes [39; 40]. The technique was also applied to CCDs with built-in avalanche multiplication [43]. Similar operations with CMOS image sensors (CIS) were envisioned as early as 1998 [52] and proposed as a digital integration sensor (DIS) [7]. The technique was also applied in SPAD arrays [31]. SPAD arrays operating as QIS devices were demonstrated in [30]. Following the introduction of the early QIS concept, the \"gigavision\" binary image sensor was proposed in 2009 [33; 34]. Image formation from bits in the binary sensor was explored mathematically by EPFL [36] and by Harvard [53]. These investigations concerned a cubicle that was essentially \(1\times 1\times N\), that is, one pixel read out \(N\) times. Figure 1: Conceptual illustration of a jet data cube and a \(4\times 4\times 4\) cubicle subset. Figure 2: Simulation of (**a**) raw jet data; (**b**) same at lower magnification; (**c**) after processing cubicles to form grey scale image. From [16]. QIS devices use both spatial and temporal sampling to create one image pixel using sub-diffraction limit pixel sizes [2], although binning multiple pixels to improve SNR at the expense of spatial resolution dates back to at least the early days of CCDs in the 1970's. Uniform weighting and other weight distributions applied to jets of the bit cube to form image pixels have been explored by [PERSON] [16].