Title: bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction
Venue: NeurIPS 2024
Accepted: yes
Keywords: Denoising; Self-supervised learning; Quanta imaging; Photon counting
Abstract: Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

bit2bit : 1-bit quanta video reconstruction by self-supervised photon location prediction

Abstract

Quanta image sensors, such as single-photon avalanche diode (SPAD) arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit , a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

1 Introduction

Quanta image sensor (QIS) (e.g., SPAD arrays [1, 2]) offer unique advantages over other image sensors for capturing fast temporal dynamics in low-light settings, as they can detect individual photons within nanoseconds exposures [3]. This allows for the measurement of ultra-fast phenomena, such as fluorescence lifetime [4], time-of-flight [5], and other time-resolved processes [6]. Quanta image data is different from traditional digital images. The QIS raw data often consists of sparse detection events described in 1-bit arrays (Fig. 1). In each frame, pixels only indicate the presence or absence of a photon without intensity information [7]. This makes the data non-interpretable or usable as traditional images. A common approach to make effective use of the binary data is to bin the image in the time domain[8]. While this can provide clear images (Fig. 1c), it comes at the cost of losing the valuable high-temporal-resolution information offered by QIS. In this paper, we present a novel self-supervised approach that directly denoises binary photon detection events from a QIS, reconstructing a clean video at the original temporal resolution.

b

a

Signal

1-bit array

Predicted signal

Shot noise

Detectors

Photons

~ Poisson

One at

a time

plz!

c

zoom in to view

height

time

width

d

frame 0

Prediction

time

width

height

01000001…

some function f

θ

I am a

detector.

Real quanta video from SPAD

At the original spatiotemporal resolution!

Frame #

Bin 512 frames

1200

Bin 128 Frames

0

d

frames

frames

height

time

width

-o

o

双

Generative Accumulation of Photons (GAP) recently introduced an effective self-supervised method that denoises shot noise corrupted photon counting images by splitting images into paired data training pairs based on Poisson statistics [9]. The self-supervised method is valuable to scientific imaging applications, where very often ground truth data is unavailable and cannot be simulated. However, this method is ineffective for binary images, where it creates significant artifacts due to the non-Poissonian nature of photon statistics in 1-bit data [10, 11]. This inspired us to develop bit2bit , a new method that specialized for binary data. Our key new insight is a masking strategy that hides the complementary dependency within the training pairs, which effectively addresses the problem. We further extend our method to 3D to leverage the information in both space and time, substantially improving the reconstruction quality. In addition to these measures, we also explored different architectures and sampling parameter spaces to reduce the significant issue of overfitting.

Problem statement and claims. The goal of this work is to predict the underlying spatiotemporal signal from a stack of quanta raw data. Each frame in the stack is a 1-bit 2D array representing locations where at least one photon is detected during the exposure period (Fig. 1a). Our method is based on the following 2 assumptions: 1) Each detection event is independent and follows truncated Poisson statistics. 2) The underlying signal exhibits some local and global spatiotemporal structures that can be described by a model.

Given this challenging reconstruction problem, we claim the following main contributions:

Developing a self-supervised method that denoises photon-sparse binary quanta images, which effectively handles the binary nature of the data through a novel masking strategy.

Enhancing the performance of reconstruction by leveraging the temporal information that is readily available in most quanta image data.

Providing insights on stochastic sampling strategies, network designs, and regularization techniques to manage overfitting and improve the reconstruction quality.

Presenting a novel dataset with real and simulated 1-bit SPAD data to support further research and quantitative evaluation of quanta image processing.

H&E slide

Fan + motion

Plasma ball

Sonicator bubbles

USAF 1951 + Drill

zoom in to view

Color-temporal-coded 50 frames

001

frame

050

2 Related Work

Quanta image reconstruction. A common quanta image reconstruction task is finding the optimal transformations of individual 1-bit frames to minimize motion blur prior to applying binning [8, 1214]. Noise is another problem being addressed, especially in SPAD-based 3D reconstruction [15-17]. Some higher-level tasks address the motion deblur of the binned data using specially engineered deep learning methods because the mixing of shot noise renders traditional debluring methods ineffective [18]. Besides direct image reconstruction, there are also efforts to perform traditional computer vision (CV) tasks directly to quanta image data (e.g., image classification) [19-21]. The task described in this work - reconstructing single binary frames at original temporal resolution and addressing both shot noise and motion blur - is challenging and has not been well explored. Only one previous work explored a similar task using the supervised method with ground truth data [22], in contrast to our self-supervised method.

Self-supervised denoising. Convolution neural networks (CNN) can be trained to denoise images by minimizing the difference (e.g., mean squared error (MSE)) between the prediction and a reference target [23, 24]. This is usually referred to as the minimal mean squared error (MMSE) denoising. Noise2Noise (N2N) demonstrated that it is possible to minimize the MSE loss between a pair of noisy images and still get quality denoising results without knowing the noise-free prior [25]. However, N2N still requires real data pairs with the same underlying signal, which are often not obtainable. More recently, self-supervised denoising methods have been introduced to perform denoising from a single noisy image. The methods create input and target training pairs from the original noisy image by assuming the augmentations minimally affect the underlying signal. Noise2Void (N2V)(2) predicts the value of random pixels in a noisy image by hiding these pixels during training and only using the surrounding context to minimize loss only from the hidden pixels [26, 27]. Noise2Self (N2S) presented a similar concept that uses a moving grid mask [28]. Noisier2Noise adds more noise to the input to alter the original noise [29]. Neighbour2Neighbour, Noise2Fast, etc., assume pixel-wise independence of noise and create image pairs from neighboring pixels [30, 31].

However, the binary, sparse nature of quanta image creates a special, challenging case. A recent self-supervised denoising method, GAP [9], effectively addressed shot noise corruption in photon counting images by Poisson-statistic-based resampling. GAP introduced the idea of splitting the data by pixel-wise binomial sampling to create nearly unlimited training pairs, which inspired our work. However, implementing GAP directly on binary quanta images resulted in extremely poor

input

+

p

1 - p

Random split

voxel by voxel

+

…

,

3D network

: learn from space & time

Photon splitting

: create 'unlimited', 'real' data / prevent overfitting

target

t

Masking

: hide complementary dependency in training

Random

3D crop

h

w

Don't learn

from here!

mask

NOT input

One more split, one more unique

training pair

+ more

1-bit

reconstructions. This motivated us to identify the root cause of the problem and develop a specialized self-supervised method for sparse binary image reconstruction.

3 Theories and methods

Brief review of GAP. Here, we summarize the self-supervised denoising approach described in GAP [9]. GAP is based on the assumption that observed image x is a shot noise corrupted version of the clean image s , with each pixel x i drawn from a Poisson distribution parameterized by the clean signal at the pixel s i , where i indicates the pixel index. GAP proposes to split the noisy image x by sampling from a binomial distribution randomly assigning photons to an input image x k inp and a target image x k tar . The images are conditionally independent given the underlying signal s , so can be treated as two independent observations, which can then be utilized in a N2N-style training procedure [25] to train a neural network f ( x ; θ ) with parameters θ . By attempting to predict x tar from x inp , the procedure trains a network to find the MMSE estimate for the clean signal. It uses a cross-entropy loss over pixels

viewing the problem as a classification task, trying to predict the photon locations in the target image. Here, f ( x t inp ; θ ) i corresponds to the logit output for pixel i . The authors show that in the limit, their loss is equivalent to the MSE loss, which is more frequently used in N2N training.

Photon counting. While the assumption of the Poisson shot noise model in GAP seems reasonable for photon counting, it often does not accurately hold for truncated QIS data such as SPAD. We can understand this by modeling photon counting as a Poisson point process and considering each counted photon as an element of a set defining this process. Subsequently, we can model the compound processes and the sub-processes within the framework of point process statistics and differentiate between photon counting and sector activation events (e.g., binary activations in each frame). This broader insight is discussed in the Appendix. This section focuses on the practical implementation of this insight in self-supervised denoising.

Quanta image generation. Instead of truly counting photons, SPAD (and other quanta imaging methods) acquire a series of binary images x t , with each pixel x t i ∈ { 0 , 1 } indicating whether photons have been detected or not. Unfortunately, this means that the number of photons hitting the pixel is not recorded in a unit detection window, and we cannot know if an active pixel x t i = 1 receives single or multiple photons during one exposure. Formally, pixel values x t i are drawn from a Bernoulli distribution with parameter ρ = 1 -e -s i , with e -s i corresponding to the probability of zero photons hitting a pixel according to the underlying Poisson distribution (Appendix A.1). For small s i , individual pixels rarely receive multiple photons during an exposure. Thus, it is convenient to consider ρ i = 1 -e -s i ≈ s i [11].

In order to generate images that are visually pleasing and easier to analyze, the sparse binary data is usually processed by summing multiple frames x t , effectively counting the number of detection

Raw data

Raw data

Network trained w/o mask

Network trained w/ mask

Split

Input

Target

Mask

a

b

LAB

LABL

U

SNACKS

SNACKS

ATreatfor

xunaans

HOH

HOE

events for each pixel [7]. Unfortunately, this aggregation over time leads to an inevitable loss of temporal resolution, as information about the time in which a photon hit is lost. It is important to note that the noise in such data is not Poisson distributed but instead follows a binomial distribution. Unfortunately, this means GAP, which relies on this assumption, cannot be directly applied to SPAD data. In the Appendix (Photon detection), we give an alternative theoretical perspective on this phenomenon by modeling photon counting as a Poisson point process with sub-processes to understand the difference between photon counting and the counting of photon detection events. While it is true that we can produce approximate Poisson statistics by summing a large number of frames, this comes at the cost of significantly reduced time resolution.

Splitting of 1-bit quanta data. This work explores an alternative approach, attempting to directly split the recorded binary images without temporal aggregation. We split the recorded binary image according to GAP [9], randomly assigning each photon detection event in x t to x t inp or x t tar with a probability p (Fig. 3). Assuming low light intensity s i this means that the probability for finding a photon detection event at a pixel is ρ i, inp ≈ ps i and ρ i, tar ≈ (1 -p ) s i . In order to enable the use of temporal information, we apply the same not to 2dimensional images but to 3D space-timevolumes X = ( x 1 , . . . , x T ) , splitting them into input X inp = ( x 1 inp , . . . , x T inp ) and target volumes X tar = ( x 1 tar , . . . , x T tar ) .

However, unlike for true Poisson distributed photon counts, x t inp and x t tar cannot be considered conditionally independent. Since a photon detection event in x t i = 1 can only be assigned to either x t to x t inp or x t tar , an event x t i, inp = 1 in the input image must necessarily mean that corresponding pixel in the target image does not contain an event x t i, tar = 0 . As a consequence, naively using x t inp and x t tar as training pairs for a denoiser, will lead to the artifacts, with the network predicting dark values for any pixel which has a photon detection event in its input.

Masked loss function photon prediction for quanta images. To remedy this effect of correlated input and target images, we propose an adapted loss function. Considering that our data is sparse, that is, only very few pixels contain photon detection events, we avoid the problem by excluding

pixels that contain a photon detection event in the input image using the loss

with f ( X inp ; θ ) referring to the logit output of the denoiser network and f ( X inp ; θ ) t i referring to the value at the pixel/time index i, t of the output. Note that, while the masking is achieved by multiplying (1 -x t i, inp ) the remainder of the function corresponds exactly to the cross entropy loss over pixels from Eq. 1.

Implementation details. We view each binary quanta image dataset as a 3D array with two spatial dimensions and a temporal dimension. We randomly crop each sample x from the volume (Fig. 2). x inp is sampled from a volume x by Bernoulli sampling with dropout probability p defined based on desired strategy. We then calculate the target x tar by subtracting the input x inp from x . The mask x mask is computed from the input by x mask = 1 -max { x inp , 1 } at runtime. In the binary case, the mask is equivalent to the bitwise complement of the input ¬ x inp (Fig. 3). We multiply the mask by the output of the network and the target before the cross entropy loss is calculated, which is equivalent to the loss function discussed in the previous section.

Selecting photon splitting variable p. The Bernoulli sampling probability p is an important new hyperparameter introduced in the sampling process. Its value and selection strategy can drastically impact the performance of production. We will demonstrate this in our experimental section. If we only work at a fixed input signal level (e.g., mean photon count), we can use a fixed p to generate training pairs. However, it is necessary to reduce the photon count to the same signal level for inference. A fixed large or small p can lead to over-fitting and slow training. In extreme cases, the model will recreate the target if p = 1 , and the target is always empty if p = 0 . A single input at the reduced signal level does not contain all the available information. We could resample multiple copies of the input and combine the inference results, but this strategy requires a longer inference time. Alternatively, we can randomly pick p from a desired range, which allows the model to adapt to different signal levels and produce acceptable results from all the inputs within the range. If we use a large p close to 1 as the upper limit (e.g., 1 -10 -6 ), we should be able to use raw data directly as the input, which is more efficient in practice.

4 Experiments

Network architecture. We used a 3D ResUNet [32] with an option to switch the network to 2D at a specific depth. The primary reason for this design choice is to handle the rapid increase of the receptive field as the depth increases, which eventually becomes bigger than the sampled area. The network will learn a lot from the padded areas during training. In the photon sparse binary case, this problem is more prominent, as the 0 padding is not different from the real data. To avoid significant weight from the padded area, a large crop window size is necessary to achieve better results. However, this can easily lead to memory issues. We can prevent the receptive field from growing in the temporal dimension by switching from 3D to 2D at the desired depth. This allows us to use a smaller window size in the temporal dimension and a larger window size in the spatial dimension. A detailed diagram of the network architecture is shown in Fig. S1. For most experiments, we set the network depth at 5, with only the first 2 levels being 3D to control the size of the receptive field. We used 32 initial input features, which provide a balance between performance and memory efficiency. The number of features doubles at each depth. Group normalization of 8 is applied at each convolution [33]. We use GeLU for the activation function and pixel shuffling for up-sampling [34, 35].

Training. Models were trained using the ADAMW optimizer for 150 epochs, with 250 steps per epoch and 4 batches of random crops of 32x256x256 (TXY) per step [36]. The large patch size is desired to prevent performance degradation due to substantial padding weights. However, 3D UNet with a large patch size is not memory-efficient, and data transfer is a bottleneck. Sometimes, only 2 batches of the crop size can be trained at a time while leaving a substantial memory unused with 24GB VRAM. Therefore, we implemented gradient check-pointing [37], which allowed us to double the batch size and utilize the majority of the VRAM. We primarily used Nvidia 3090/4090 for training, and each training takes about 6-10 hours until the loss curve stabilizes.

Inference. We run patch-based inference on a GPU using the original image width and height, maximizing the frame count for each inference. With 24GB VRAM, we can use a 48x512x512 volume for each inference with the typical 5-depth ResUNet mentioned above. The inference speed is above 3 volumes per second (150 fps) on a NVIDIA RTX 4090 GPU. If the training uses p max ≈ 1 , the raw data is used directly as input. Otherwise, we randomly reduce the photon count in the data by a factor of p max through photon splitting. For a 10-shot inference, we Bernoulli sample 10 times from the raw input at a fixed p value, run inference independently using the model trained at the p value, and calculate the mean of all the outputs.

Data. To evaluate our method quantitatively and qualitatively, we use a synthetic video with simulated noise consisting of 3990 frames and a total of 7 real SPAD videos, each containing 100k-130k frames. Additionally, we use a real video with 100k frames published in [8]. Simulated data is essential for quantitative assessment as QIS ground truth is impractical to get. Assuming reference images/videos are ground truth with no shot noise and the pixel values n i are proportional to the Poisson rate λ i , we calculate a variable q based on the selected mean counting rate λ = q ( 1 M ∑ M i =1 n i ) , where 1 M ∑ M i =1 n i is the mean of the reference. We then run pixel-wise Poisson sampling with λ i = n i p and clip between 0 and 1 to simulate the 1-bit quanta data. A ground truth reference video of a person shaking a ThorLab's box was acquired from iPhone 15 slow motion mode at 240 fps (Fig. S2). The simulation was using λ = 0 . 0625 . After the clipping, the resulting simulation has 0.0590 photons per pixel. Real QIS data were acquired using a SPAD512S camera (Pi Imaging Technology, Switzerland)[2]. Each dataset has 100k-130k binary frames. We created scenes to represent a combination of different imaging conditions, including low-signal, high-signal, highcontrast, high-ambient-light, moving camera, moving object, linear movement, random movement, combined movement, ultra-fast events, and stochastic events (Fig.S6, S7, S8, S9, S10, S11, S12). We made the simulated data, real SPAD data, and reference results available to the community * .

Comparisons of different methods. We tested supervised training, N2N, N2V, and the original GAP with simulated data. To ensure fair comparisons, we incorporated the same network architectures, hyperparameters, and training steps into individual baselines. All implementations are in 3D except for GAP. Inference was conducted on the first 512 frames, and the data was normalized by the mean. PSNR and SSIM were calculated frame-wise, and the statistics were derived from the entire output stack. We tested N2N in three different ways: Using two independent simulated samples for training. Sampling two independent image stacks using the proposed splitting method (with p = 0 . 5 , just once before training). Using the same samples but applying the masking strategy during training. We tested N2V using the original masking strategy. The N2V masks were created from 1000 random points in space. A median filter was applied to these points in the input. The minimum distance between the points was smaller than the median filter size, and loss was only computed from these points. We tested the original GAP in 2D with and without masking. Finally, we evaluated our method with and without masking.

5 Results and discussions

Findings from the comparisons. Tab. 1 summarizes key numerical and visual results. The supervised methods perform the best, which reflects the approximate upper limit of the network's performance. In reality, the ground truth is often not available, and high SNR data is only available from binning, which degrades time resolution and blends dynamics. N2N with 2 independent samples created a granular pattern on the images, as well as increasing validation loss since the beginning of the training (Fig. S16). Besides, two independent quanta data with the same underlying signal do not exist. One way the address this with N2N is to use adjacent frames, but this reduces the temporal resolution and introduces motion artifacts (e.g., Fig. S3: Interframe N2N). Without the mask, the image shows a strong pepper noise. After implementing the mask, it generates a smooth image, but the granular pattern persists. We also tested our method, which is similar to GAP in 3D. Masking addresses the image artifact problem. The 3D implementation led to substantially improved PSNR and image quality. N2V also produced poor reconstruction results. The result from the original 2D GAP produced also has the presence of pepper noise, similar to our implementation with large fixed p values. Implementing the mask in the GAP resolved the problem. The images are smooth but lack

fine details. Our method also shows the pepper noise and the fine details are missing when the mask is not applied. Implementing the mask substantially improved the reconstruction quality.

Inference strategies. We achieved a PSNR of 33.93 using a network with p ∈ [0 , 1 -10 -6 ] that takes the original raw data as input and performs inference in one shot. Further, training a network in a less extreme p range [0 . 3 , 0 . 7] to avoid residual overfitting, then running inferences using 10 different samples sampled from the original data with p = 7 , and finally combining the data by simple averaging, improved the PSNR to 34.35 (Table. S8). It is worth noting that it should be assumed that the result is specific to this simulated dataset and model configurations.

1-bit self-resampling. We noticed that different self-supervised resampling strategies become very similar across different methods in 1-bit. When we create a training pair and largely preserve the original information, we could either keep a positive in one image or move it to another image. Then, the sampling method essentially describes the spatial rule of splitting. For example, GAP uses binomial sampling to randomly drop out some photons. Self2Self [38] uses random binary masks to drop out some pixels, which is equivalent to GAP in 1-bit. N2S and N2V both select some pixels

x-axis = group norm

p

∈

[x, 1 - 10

-6

]

PSNR/STD

a

b

c

d

e

x = start filters

x = p

Ⅰ

Ⅰ

Ⅰ

Ⅱ

Ⅲ

Ⅳ

p

∈

[0, x]

34.5

35

32

k

33.6

33.0

34

30

*

32.8

33

28

*

32.0

31.5

32

26

1

2

4

8

16

32

0.0

0.2

0.5

0.6

0.8

0.9

6

.9x4

6

6

9.

*666

.9x4

.9x5

16

24

28

32

40

48

in the input and use them as a target [28, 27]. The pixels are replaced by a new value using various strategies, such as median, random, and random dropout, which usually remove the selected positive pixel and very occasionally create new positive pixels. If a fixed grid is used, the operation becomes spatial-dependent. In the cases of neighboring pixel-based methods [30, 31], image pairs are created by sub-sampling from a 2x2 grid, equivalent to a balanced spatial-dependent selection followed by downsampling.

Group normalization. We noticed that group normalization is critical to for this task. Fig. 5a and Table. S2 shows that PSNR increased from 30.9 to 33.5 by using group normalization of 2. In creating the group number to 8 further increased the PSNR to 33.9. We could not implement batch normalization [39], as 24GB VRAM is only sufficient for 1 batch in some use cases (e.g., 512x512x32 window size).

Over-fitting, stochastic splitting and choices of p. Overfitting is often unavoidable in selfsupervised denoising because the model can learn to fit the noise patterns specific to the finite training data. We noticed granular image artifacts in N2N, N2V, and GAP with large, fixed p-values, which resulted in substantially reduced reconstruction quality (fig. 3). Further analysis indicates that these three examples share a common issue: the input of the model is approximately a fixed binary pattern. In N2N, there is no variation between the input and target data. In N2V, we used a random grid to select the training target, but the grid size was relatively small, and most selected pixels were 0. The input data was very similar to the raw data, with a few photons removed, making the process similar to GAP with a large, fixed p-value.

To analyze the problem, we conducted an experiment using randomly Poisson noise with a small λ (Fig. S4). Ideally, the method should produce a uniform plane image with small pixel value variations. Our results show that if a new random input is used for every training step, the output of the network converges to a uniform image as expected, regardless of whether a fixed output is used. If we fix the input and randomize the output, the result is also uniform. However, if we fix both the input and output or if the output contains few photons (e.g., GAP with fixed large p), granular patterns similar to the N2N results start to appear, also reflected by substantially higher standard deviation. Increasing the photon counts in the input reduces shot noise and mitigates the problem. This suggests that diversity in the training pairs is crucial to mitigate overfitting in photon-sparse quanta image data.

Model size selection. Increasing model size can negatively impact the performance, potentially due to overfitting. We found a smaller 5-depth network with 24 startup filters outperformed a larger network with 48 filters (Fig. 5e), along with non-significant decaying from 24 to 40 filters. In another comparison, a network with depth 4 and 40 startup filters can achieve comparable performance (PSNR=33.83) as our best 5-depth network with 32 startup filters (33.93) (Table. S7). A 6-depth network (32.55) has a similar performance as a 3-depth network (32.42). The model size parameter space should be explored to balance overfitting and under-representation for different tasks.

Comparison to QBP. Quanta burst photography (QBP) is a recognized state-of-the-art quanta image reconstruction method based on spatial-temporal hierarchical alignment and frequency domain combination [8]. We selected a challenging non-static scene from the original work for comparison, where a person is playing the guitar. The results are shown in Fig. 6 and Video S2. It is important to

Raw

(1 out of 130k frames)

Long bin

(2k → 1 frame)

Short bin

(100 → 1 frame)

QBP

(2k → 1 frame / 1E-4 fps)

Ours

(20k → 20k frames / 1 fps)

time (1000 frames)

a

height (30 px)

b

note that QBP reconstructs 1 image from 100-10,000 frames in relatively static scenes, each taking 30 minutes as reported (not optimized) [8]. In contrast, our method can start to produce quality results in less than 1 hour on a 100k-frame dataset. Longer training further improves the results. Our method achieves over 150 fps inference on this dataset, equivalent to processing the entire 100k frames in less than 30 minutes with 50% patch overlap.

6 Conclusions

We demonstrated a versatile method for predicting the underlying signal of binary quanta image data, offering a practical solution for quanta image reconstruction where high SNR data and ground truth are unavailable. However, it is necessary to consider the limitations of the method when applying it to real data. Our method assumes a simple Poisson point process model, where the observation is sampled from a noise-free signal. Thus, it is not intended to restore the signal from other types of noise. From a different perspective, the method removes the Poisson noise component from the data, making it easier to identify and correct other noise components. For example, hot pixels on SPAD can be revealed and corrected with median filtering.

A fundamental limitation of self-supervised denoising is the lack of ground truth data, making it difficult to evaluate performance and accuracy in specific cases. In our case, performance could be evaluated by simulating the Poisson process, though these simulations should be based on reasonable assumptions and approximations. Additionally, the method requires offline training for each dataset. While there is potential for real-time reconstruction, the model should be trained on similar scenes.

Our approach can potentially be effective on any binary higher-dimensional spatial data created from Poisson point processes. For example, any data represented by an n-dimensional scatter plot, where points are independent measurements, can be analyzed with this method. Fig. S14 and S15 show the application and benchmark of the method to real and simulated 3D stacks of confocal microscope data with extremely high shot noise, which also demonstrates the performance advantage in 3D.

We also see the task could be generalized for all spatial event point processes as follows [40]. For an n-dimensional signal in spacetime, a finite amount of measurements can be made from disjoint, uniform, and bounded sub-regions. Their collection can be viewed as a Poisson point process. The measurements are clipped between 0 and 1, and the process becomes a Bernoulli process. Given a finite number of such measurements, the task is to predict the distribution of the underlying signal measured by the Poisson rate. We envision that the core concept of creating data pairs by randomly splitting a point process and then hiding their complementary dependencies can potentially be used in other relevant regression models.

Finally, it is worth noting that the method can be misused in research if inappropriate data or statistical assumptions about the noise are made (e.g., the data points are not independent). In such cases, the prediction results can be misinterpreted, leading to incorrect scientific conclusions and potential negative societal impact.

References

Kazuhiro Morimoto, Andrei Ardelean, Ming-Lo Wu, Arin Can Ulku, Ivan Michel Antolovic, Claudio Bruschini, and Edoardo Charbon. Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications. Optica , 7(4):346-354, April 2020. Publisher: Optica Publishing Group.

Arin Can Ulku, Claudio Bruschini, Ivan Michel Antolovi´ c, Yung Kuo, Rinat Ankri, Shimon Weiss, Xavier Michalet, and Edoardo Charbon. A 512 × 512 spad image sensor with integrated gating for widefield flim. IEEE Journal of Selected Topics in Quantum Electronics , 25(1):1-12, 2019.

Eric R. Fossum, Jiaju Ma, Saleh Masoodian, Leo Anzagira, and Rachel Zizza. The Quanta Image Sensor: Every Photon Counts. Sensors , 16(8):1260, August 2016. Number: 8 Publisher: Multidisciplinary Digital Publishing Institute.

Lucio Pancheri and David Stoppa. A spad-based pixel linear array for high-speed time-gated fluorescence lifetime imaging. In 2009 Proceedings of ESSCIRC , pages 428-431, 2009.

F. Villa, B. Markovic, S. Bellisai, D. Bronzi, A. Tosi, F. Zappa, S. Tisa, D. Durini, S. Weyers, U. Paschen, and W. Brockherde. Spad smart pixel for time-of-flight and time-correlated single-photon counting measurements. IEEE Photonics Journal , 4(3):795-804, 2012.

A. Kufcsák, A. Erdogan, R. Walker, K. Ehrlich, M. Tanner, A. Megia-Fernandez, E. Scholefield, P. Emanuel, K. Dhaliwal, M. Bradley, R. K. Henderson, and N. Krstaji´ c. Time-resolved spectroscopy at 19,000 lines per second using a cmos spad line array enables advanced biophotonics applications. Opt. Express , 25(10):11103-11123, May 2017.

Stanley H Chan. What does a one-bit quanta image sensor offer? IEEE Transactions on Computational Imaging , 8:770-783, 2022.

Sizhuo Ma, Shantanu Gupta, Arin C. Ulku, Claudio Bruschini, Edoardo Charbon, and Mohit Gupta. Quanta burst photography. ACM Transactions on Graphics , 39(4), August 2020.

A. Krull, H. Basevi, B. Salmon, A. Zeug, F. Muller, S. Tonks, L. Muppala, and A. Leonardi. Image denoising and the generative accumulation of photons. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1517-1526, Los Alamitos, CA, USA, jan 2024. IEEE Computer Society.

Feng Yang, Yue M. Lu, Luciano Sbaiz, and Martin Vetterli. Bits from photons: Oversampled image acquisition using binary poisson statistics. IEEE Transactions on Image Processing , 21(4):1421-1436, 2012.

Jeffrey D. Scargle. Studies in Astronomical Time Series Analysis. V. Bayesian Blocks, a New Method to Analyze Structure in Photon Counting Data*. The Astrophysical Journal , 504(1):405, September 1998. Publisher: IOP Publishing.

Stanley H Chan, Omar A Elgendy, and Xiran Wang. Images from bits: Non-iterative image reconstruction for quanta image sensors. Sensors , 16(11):1961, 2016.

Trevor Seets, Atul Ingle, Martin Laurenzis, and Andreas Velten. Motion adaptive deblurring with single-photon cameras. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1945-1954, January 2021.

Kiyotaka Iwabuchi, Yusuke Kameda, and Takayuki Hamamoto. Image quality improvements based on motion-based deblurring for single-photon imaging. IEEE Access , 9:30080-30094, 2021.

Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa, Vivek K. Goyal, Franco N. C. Wong, and Jeffrey H. Shapiro. Photon-efficient imaging with a single-photon camera. Nature Communications , 7(1):12046, June 2016. Publisher: Nature Publishing Group.

David B. Lindell, Matthew O'Toole, and Gordon Wetzstein. Single-photon 3D imaging with deep sensor fusion. ACM Transactions on Graphics , 37(4):1-12, August 2018.

Gongxin Yao, Yiwei Chen, Chen Jiang, Yixin Xuan, Xiaomin Hu, Yong Liu, and Yu Pan. Dynamic single-photon 3D imaging with a sparsity-based neural network. Optics Express , 30(21):37323-37340, October 2022. Publisher: Optica Publishing Group.

Yash Sanghvi, Abhiram Gnanasambandam, and Stanley H. Chan. Photon limited non-blind deblurring using algorithm unrolling. IEEE Transactions on Computational Imaging , 8:851-864, 2022.

Abhiram Gnanasambandam and Stanley H Chan. Image classification in the dark using quanta image sensors. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16 , pages 484-501. Springer, 2020.

Chengxi Li, Xiangyu Qu, Abhiram Gnanasambandam, Omar A. Elgendy, Jiaju Ma, and Stanley H. Chan. Photon-Limited Object Detection using Non-local Feature Matching and Knowledge Distillation. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) , pages 3959-3970, Montreal, BC, Canada, October 2021. IEEE.

Shantanu Gupta and Mohit Gupta. Eulerian single-photon vision. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10431-10442, 2023.

Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, Kolb, and Andreas. A Bit Too Much? High Speed Imaging from Sparse Photon Counts. In 2019 IEEE International Conference on Computational Photography (ICCP) , pages 1-9, May 2019. ISSN: 2472-7636.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7):3142-3155, 2017.

Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, et al. Contentaware image restoration: pushing the limits of fluorescence microscopy. Nature methods , 15(12):1090-1097, 2018.

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference on Machine Learning , pages 2965-2974. PMLR, 2018.

Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-learning denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2129-2137, 2019.

Eva Höck, Tim-Oliver Buchholz, Anselm Brachmann, Florian Jug, and Alexander Freytag. N2v2-fixing noise2void checkerboard artifacts with modified sampling strategies and a tweaked network architecture. In European Conference on Computer Vision , pages 503-518. Springer, 2022.

Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning , pages 524-533. PMLR, 2019.

Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12064-12072, 2020.

Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14776-14785, June 2021. ISSN: 2575-7075.

Jason Lequyer, Reuben Philip, Amit Sharma, Wen-Hsin Hsu, and Laurence Pelletier. A fast blind zero-shot denoiser. Nature Machine Intelligence , 4(11):953-963, 2022.

Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing , 162:94-114, 2020.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. Efficient image super-resolution using pixel attention. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 56-72. Springer, 2020.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1887-1895, June 2020. ISSN: 2575-7075.

Chunwei Tian, Yong Xu, and Wangmeng Zuo. Image denoising using deep cnn with batch renormalization. Neural Networks , 121:461-473, 2020.

Jesper Moller and Rasmus Plenge Waagepetersen. Statistical inference and simulation for spatial point processes . CRC press, 2003.

7 Data and code availability

Simulated and real SPAD data:

https://drive.google.com/drive/folders/1M5bsmsaLBkYmO7nMUjK5_m71RonOp-P9

Python raw data loader and cleaner:

https://github.com/lyehe/spadtools

SPAD data simulator:

https://github.com/lyehe/spadsampler

Source code of the parameterized model with bit2bit:

https://github.com/lyehe/ssunet

8 Acknowledgment

We thank the following grants for their support of this research: NIH R01EB028635, R01HL126747, S10OD024996, U01EY034693 and R44CA268156.

We thank Axiom Optics for providing the demo unit of the SPAD512s camera.

Special thanks to the DL@MBL (now AI@MBL) program for sponsoring training and collaborations that made this work possible. Cheers Team Restorators and members Anirban Ray, Peter (Hyoungjun) Park, Geneva Schlafly, Guillaume Minet, and Mie Thorborg!

9 Competing interests

Yehe Liu and Michael Jenkins are co-founders and owners of OpsiClear LLC.

A Appendix / Supporting documents

A.1 Photon counting statistics

Photon detection. Here, we are looking at the photon counting from the point process perspective. Imagine an underlying signal indicated by a rate parameter λ describing the expected number of photons hitting the detector over a unit of time. Light is quantized, so a hypothetical detector with infinite depth will record k ∼ Poisson ( λ ) photons over each detection window, with p ( k = n ) = λ n e -λ /n ! , where n ∈ N 0 is the number of the detected photons. Real detectors can only activate once in each detection window (e.g., n ∈ { 0 , 1 } ). Therefore, the Poisson point process is reduced to a Bernoulli lattice process with p ( k = 0 | λ ) = e -λ and p ( k = 1 | λ ) = ∑ ∞ n =1 λ n e -λ n ! = 1 -e -λ . With small λ (e.g., < 0 . 01 ), it is convenient to consider p ( k = 0) ≈ 1 -λ and p ( k = 1) ≈ λ in photon-sparse binary data because the chance of missed counts p ( k > 1) is negligibly small.

Photon counting. From the insight above, it is clear that the "photons" are just "activation events" over some finite detection windows in spacetime. In practice, it is often assumed that λ changes minimally across a few windows [11]. The superposition of n ′ such windows is also a Poisson point process with λ ′ = n ′ λ , assuming this homogeneity. If detection is modeled as a more realistic Bernoulli lattice process with p ′ = p ( k = 1) = 1 -e -λ , summing n ′ windows is equivalent to binomial sampling k ′ ∼ Binomial ( n ′ , p ′ ) , which can be approximated by a Poisson distribution with λ ′ = n ′ p ′ = n ′ (1 -e -λ ) . Both λ and λ ′ describe the underlying signal. It is more trivial to determine λ ′ from multiple windows than λ from a single binary image. However, λ i is constantly varying in reality. Through λ ′ = ∑ i λ i or ∑ i (1 -e -λ i ) , we lose the information of individual λ i .

Photon splitting. Since all points in N t and N ′ are independent, we can apply basic point process operation p -thinning to randomly remove some points and create new independent sub-processes. The thinning probability p can be selected either manually or randomly to control the size of the new process N p . For Λ denoting the original expected count rate, the rate of the thinned process is proportionally reduced to p Λ . Similarly, the removed points can form another subprocess N 1 -p with rate (1 -p )Λ . The new processes N p and N 1 -p are independent Poisson processes. Therefore, if we observe a process N over an exposure τ , we can view each split as two real experiments where we observe a first process N p during τp from p Λ and, right after, a second process N 1 -p during τ (1 -p ) from (1 -p )Λ . GAP originally implemented photon splitting by sampling X ′ p ∼ Binomial ( X ′ , p ) in the binned frames. Similarly, we use a special case X t p ∼ Bernoulli ( p · 1 ( X t = 1)) in binary frames.

Independence of the photon splitting pairs. N p and N 1 -p are both independent Poisson processes. However, they exhibit a complementary dependency due to their definition from the original process, specifically:

This implies that for any x ′ i,p , if x ′ i,p = n ′ , then x ′ i, 1 -p = 0 . For small λ i and large n ′ (even for n ′ = 2 ), this is an extremely rare event as p ( x ′ i = n ′ ) = ∏ n ′ t =1 λ t i and p ( x ′ i,p = n ′ | x ′ i = n ′ ) = p n ′ . Therefore, this was ignored by GAP where the data are accumulations of multiple frames. However, in 1-bit data, all the photons in N p meet this criteria, as ∀ X p : X p = 1 -X 1 -p . This also applies to extremely photon-sparse observation when very few photons are detected over a large n ′ because even with frame bins, we rarely encounter a photon count above 1. We will show this is a crucial problem in binary data, which renders GAP ineffective, and our masking strategy is an effective solution.

MMSEdenoising. Different underlying signals can lead to an observed value denoted by N . Our goal is to predict the signal, described in the compound rate Λ = { λ i | i ∈ N k } . We assume a prior distribution p (Λ) for the signal and a likelihood function p ( N | Λ) representing the probability of observing N given the signal Λ . Using Bayes' theorem, we can express the posterior probability and the expected rate of the signals as:

To estimate the signal from the observation in the sense of MMSE denoising, we look for a function f θ ( N ) and optimal parameter θ that maps the observation N to the signal Λ for each individual dataset. Previous works have shown CNN is effective for this regression task [].

Signal prediction. As shown in GAP, we can randomly remove single photons ϕ i from the process N one at a time. The relationship between the removed photons and remaining processes N ′ i = N \ ϕ i can be used to predict the distribution of upcoming photons. Recall that ϕ i is sampled from a Poisson process with expected rate λ i ∈ Λ and p ( ϕ i ∈ N | λ i ) ≈ λ i . We can write:

If we go over all possible locations i and normalize the signal, it becomes clear that predicting the distribution of the next photon is just predicting ˆ Λ in the MMSE sense. We can train model f θ ( N ) to predict this distribution by reducing the cross entropy loss L ( θ ) = -∑ i ln f θ ( N ′ i ) ϕ i . Assuming N ′ i ≈ N in the case of a single photon is negligible in the data, we can apply f θ ( N ) directly.

Training with one photon at a time is time-consuming. And the high similarity between different N ′ can lead to over-fitting. In practice, we could predict multiple photons at a time using data from photon splitting if we normalize the target:

B Appendix / Supplementary tables

C Appendix / Supplementary figures

Group Norm

Conv nx3x3

Group Norm

Previous block

Group Norm

Conv nx3x3

GeLU

Group Norm

Conv nx3x3

GeLU

Next block

Group Norm

Conv nx3x3

GeLU

Previous block

Pixel Shuffle 2

m

Conv nx3x3

Conv nx3x3

Group Norm

GeLU

Conv nx3x3

Next block

Skip

(Concat)

Residual

(add)

Additional layers

Down conv block

Up conv block

Conv 1x3x3

3D blocks

Pixel Shuffle 2x2x2

Pixel Shuffle 2x2

Conv 3x3x3

2D blocks

Residual

(add)

Network Options

(Typical)

Start feature size (32)

Depth (5)

Depth scale factor (2)

3D conv stages (2)

Group norm (8)

THOR

SNACKS

THOR

Hurmarns

SNACKS

Humans

THOR

SNACKS

Humaafs

THOR

THOR

Humanor

Humans

SNACKS

THOR

Humarns

SNAGKS

THOR

SNACKS

THOR

SNACKS

Hurmans

THOR

Hureafor

Huroarns

THOR

SNACKS

THOR

SNACKS

Hurans

THOR

THOR

SNACKS

THOR

Hurans

THOR

WAB

THOR

THOR

SNACKS

Hurmaaror

THOR

THOR

Hureaarns

SNACKS

Aureans

THOR

SNACKS

THOR

SNACKS

Aureans

THOR

LABL

THOR

Hurmans

SNACKS

Aurmarns

THOR

SNACKS

THOR

Hurans

THOR

Hurnt

SNACKS

Hureaans

THOR

SNACKS

THOR

Hureaars

THOR

ST7

Armarar

THOR

THOR

THOR

SNACKS

THOR

Hurarue

THOR

SNACKS

THOR

SNACKS

THOR

Hurearoe

SNACKS

THOR

SNACKS

THOR

THOR

SNACKS

THORI

THOR

SNACKS

Huraror

SNACKS

THOR

THOR

SNACKS

THORI

Hureatfor

THOR

SNACKS

THORI0

SNACKS

THOR

ST

SNACKS

THOR

THOR

SNACKS

THOR

ES

THOR

SNACKS

Hurmaror

THOR

Huroaror

THOR

SNACKSBE

Huresaror

THOR

SNACKS

THOR

Hurenaros

THOR

THOR

THOR

SNACKS

THOR

SNACKS

THOR!

Hurenaros

SNACKS

THORD

Humans

Htrearor

THOR

Hurars

THORD

Ground truth

2D p

∈

[0, 0.999999]

p

∈

[0, 0.999999]*

Simulated data

10-shot p

∈

[0.3, 0.7]**

Supervised

N2V

2D - no mask

3D - no mask

No Group Norm

p=0.5

Inter frame N2N

Start filter = 48

Start filter = 32

Start filter = 24

Start filter = 40

Start filter = 16

depth = 6

Split N2N 3   w/o mask

Split N2N 2  w/ mask

p=0.9999

p

∈

[0.999, 0.9999]

p

∈

[0.5, 0.999]

Higher SNR input

N2N 1  independent inputs

p

∈

[0.99999, 0.99999]

p

∈

[0.9999, 0.9999]

p

∈

[0, 1]

p

∈

[0.99, 0.99]

p

∈

[0.999, 0.999999]

p

∈

[0.9, 0.999999]

p

∈

[0.6, 0.999999]

p

∈

[0.4, 0.999999]

p

∈

[0.2, 0.999999]

depth = 4

Ⅳ

Ⅲ

Ⅰ

Ⅱ

(Zoom in to view)

IAB

IAB

LAB

LARU

SNACKS

SNACKS

SNACKS

ENRCIES

A Treatfor

Humans

ATreat for

Humans

THORU

THORLABS

THORLATES

LAB

LAB

LAB

LAB

LAB

SNAGKS

SNACKS

SNAGKS

SNACKS

SNACKS

AT

ATrea fw

AT lor

ATreatfor

ATreatfor

Humane

Humane

Humans

Humans

HOHL

THORL/RS

THORLANS

ILAB

LABL

LAB!

LAB

SNAGKSE

SNACKS

SNACKS

SNACKS

SNACKS

eroartor

Humans

greax foa

Humans

Arreatfor

ATreatfor

Humans

Hurhans

THOR

THORUA

THOR

LAB

LAB

U

LAB

LAB

LAB

SNAGKS

SNACKSI

LSNACKS

SNACKS

SNACKS

ATrounfo

Arrenfor

Hutipns

ATroatfor

Humans

THOR

TOR

THORU

LAB

LAB

LABI

IAB

SNACKS

SNACKS

SNACKS

SNACKS

SNACKS

ATreatfor

Humans

ATreunfor

Humans

ATroafn

Homans

THORU

THOR

THORIO

THOR

LAB

山

LAB

LAB

LAB

LAB

SNACKS

SNACKS

SNACKS

SNACKS

SNACKS

ATreatter

Humnns

TT

D

ATreatfor

Humans

Humans

ATreatfor

ATreatfor

Humans

ATreatfor

Humans

THOR

THOR

THOR

THOR!

LAB

LAB

LAB

IAB

LAB

GNACKE

SNACKS

SNACKS

SNACKS

SNACKS

HaMans

ATeoutfor

Humans

ATreatfor

Humans

?

ATreatfor

Humans

THOR

THOR

THOR

THORI

Random Target p=0.0001

Both fixed p=1

Both fixed p=0.01

Both random p=0.01

Random Target p=0.01

Random input p=0.01

GAP p=0.01

Both fixed p=10

Std = 0.010

Std = 0.121

Std = 0.031

Std = 0.009

Std = 0.0013

Std = 0.0012

Std = 0.0013

Std = 0.0045

1.047

1.276

0.228

4.005

0.637

1.509

0.942

1.402

N:134217728

Min:1.047

N:134217728

Min:0.228

N:134217728

Min:0.637

N:134217728

Min:0.942

Mean:1.154

Maxc1.276

Mean:0.981

Max4.005

Mean:1.024

Max1.509

Mean:1.102

Maxc1.402

StdDev.0.0104

Mode:1.155(6816128)

StdDev.0.121

Mode:0.965(7354452)

StdDev.0.0311

Mode:1.022(9308406)

StdDev.0.00889

Mode:1.100 （17075891)

Bins:256

Bins:256

Bin Width:0.0148

Bins:256

Bin Width:0.00341

Bins:256

1.153

1.213

1.044

1.104

1.188

1.273

1.171

1.343

N:134217728

Min:1.153

N:134217728

Min:1.044

N:134217728

Min:1.188

N:134217728

Min:1.171

Mean:1.179

Mean:1.072

Mean:1.227

Mean:1.242

StdDev.0.00131

Mode:1.179(12168666)

StdDev.0.00116

Mode:1.072(14255674)

StdDev.0.00127

Mode:1.227(16137779)

StdDev.0.00456

Mode:1.241(12292923)

Bins:256

Bins:256

Bin Width:0.000235

Bins:256

Bins:256

23456

川三

I川

9

Model applied to the data used in training

Model applied to data from a similar scene but NOT used in training

n12x25PWM

High SNR reference

Ours

Input data

GAP 2D

0.00777 photons per pixel

PSNR:

19.9 / 0.485

SSIM:

0.44 / 0.031

Resampled image

0.0599 photons per pixel

PSNR:

24.0 / 0.493

SSIM:

0.61 / 0.018

0.0306 photons per pixel

PSNR:

23.1 / 0.495

SSIM:

0.57 / 0.023

Real confocal image

Cell nuclei in mouse heart

512x512x812 (XYZ)

44.2 photons per pixel

5x Zoom in

Ground truth / Restored image

1.0

0.5

MMM

0.0

-0.5

-1.0

5000

10000

15000

20000

25000

30000

35000

D Appendix / acronyms

QIS quanta image sensor

CV computer vision

CNN convolution neural networks

MSE mean squared error

MMSE minimal mean squared error

GAP Generative Accumulation of Photons

N2N Noise2Noise

N2V Noise2Void

N2S Noise2Self

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction both state the contributions. The assumptions are clarified in the introduction/claims section. The limitations are discussed in the conclusion section. Our theoretical and experimental results support our claims. The specific task and its generalization are discussed in the introduction/problem statements section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Most of the Conclusion section discusses limitations. The more general limitations of self-supervised denoising, particularly in 1-bit data, are covered thoroughly throughout the manuscript. We made it clear that the method only applies to Poisson noise but not other types of noises (e.g., fixed pattern noise), which could be better revealed by the method. Factors that influence the performance of the approach and the specific task are key elements of this manuscript. The computational efficiency of the proposed algorithms is covered in the Experiments section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our assumptions are discussed in the Introduction/Problem statements. Our proofs are covered in the Theories and Methods section.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is included in the supplemental materials. The raw data files are too large to be included. They will be released after the review period. Simulations and public data from QBP can be used for evaluation.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Anonymized code is provided in the supplemental materials. Full code and data will be released after the review period.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All training details and metadata, including all hyperparameters, are provided in the supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We included the mean, standard deviation, and range for PSNR/SSIM.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We discussed computer resources in the manuscript. The method does not require advanced computing resources beyond consumer GPUs. The range of training time is in the order of hours.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics. Our research in the paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work has no direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Licenses are cited in the code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: New assets are not a primary focus of this paper, although data and documentation will be released for public access after the anonymization period.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n1. The paper introduces a novel self-supervised method for reconstructing high-quality image sequences from sparse binary quanta image data.\n\n2. The paper mainly adapt a self supervised denoising algorithm called GAP. Instead of directly adopting the GAP method, the authors extended it to spatiotemporal structures and proposed a novel masked loss to address the correlation issue between input and target images.\n\n3. Experimental results demonstrate that the proposed method substantially outperforms state-of-the-art techniques such as Quanta Burst Photography (QBP) in both reconstruction quality and throughput.\n\n4. Additionally, the paper presents a new dataset and discusses the potential of the method for generalizing to other spatial event point processes beyond the specific application of quanta image sensors.\n\nSTRENGTHS:\n1. The strengths of this approach mainly lie in its ability to effectively utilize spatiotemporal information, its novel masked loss function that addresses the correlation between input and target images, and its demonstrated superiority over existing methods like Quanta Burst Photography in terms of reconstruction quality and throughput efficiency.\n\n2. In addition, as a self supervised algorithm, the training of this scheme does not require the construction of large-scale synthetic datasets, making it more flexible and convenient.\n\nWEAKNESSES:\n1. In Equation 1, there is a confusion between x_{in} and x_{inp}. Moreover, the right side of Equation 1 seems to have overlooked x_{tar}.\n\n2. To illustrate the application of QIS in high-speed and low-light scenarios, it would be beneficial for the authors to provide specific data on the rotation speed of the fan  and the light intensity of the scene.\n\n3. The authors used a 10-shot approach to calculate the average results. However, methods similar to GAP require many iterations during inference, and the combination of these two factors could lead to significant computational time. The authors should provide more detailed explanations, such as the number of iterations and the computational workload involved.\n\nQUESTIONS:\n1. I still have doubts about the role of the masked loss. The authors state that this loss is intended to address the correlation issue between the input and target images. However, after convolution, the information from locations that were originally 1 in the input image has already spread to other pixel locations. Simply masking the original 1 positions may not fully conceal the information from those positions.\n\n2. I am not quite clear about what the network's output is, given that the input is a t*h*w matrix. Is the output also t*h*w? If so, in the supervised loss comparison experiments, are there 32 grayscale images gt? For the N2N comparison experiment, is the supervision using QIS data with a shape of t*h*w? How was the GAP-2D experiment designed—does the network input consist of the first frame of QIS data from t*h*w?\n\n3. Why does N2N lead to a granular pattern? Why can GAP-like method resolve the granular pattern caused by the N2N method?\n\nA good response to the Weaknesses and Questions will improve my initial rating.\n\nLIMITATIONS:\nThere is no discussion of potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a self-supervised method for reconstructing high-quality video from sparse binary quanta image data produced by single-photon avalanche diode (SPAD) arrays. The authors propose a novel masking strategy to handle the binary nature of the data and extend their method to 3D to leverage spatiotemporal information. They evaluate their approach on both simulated and real SPAD data, demonstrating improved reconstruction quality and throughput compared to existing methods like Quanta Burst Photography (QBP). The paper also introduces a new dataset of real SPAD high-speed videos under various challenging imaging conditions.\n\nSTRENGTHS:\n1.\tThe paper addresses an important problem in computational imaging, proposing a novel self-supervised approach for reconstructing high-quality video from sparse binary quanta data.\n2.\tThe authors' masking strategy to handle binary data is innovative and appears effective based on the presented results.\n3.\tThe introduction of a new real SPAD high-speed video dataset is valuable for the research community.\n\nWEAKNESSES:\nThe paper fails to convincingly demonstrate the effectiveness and advantages of the proposed method over existing approaches. The theoretical foundations are not well-developed, and the experimental results are not sufficiently rigorous or comprehensive to support the claims made. The writing lacks clarity in many sections, making it difficult to fully understand the proposed method and its implications.\n\n1.\tThe discussion on selecting the photon splitting variable p is confusing and lacks practical considerations. The paper doesn't adequately address how to ensure correct signal reconstruction when the signal level is unknown before capture.\n2.\tMany crucial implementation details are absent:\n-\tThe composition and size of the training dataset are not specified.\n-\tIt's unclear if the quantitative results for simulated data are computed from only one video (L204).\n-\tThe number of videos in the real test data is not mentioned.\n3.\tThe comparison with existing quanta video reconstruction methods is limited to only QBP (proposed in 2020). There is a lack of quantitative comparisons with other existing methods.\n4.\tThe paper lacks a detailed analysis of the method's runtime and memory requirements compared to existing approaches.\n5.\tMore extensive ablation studies exploring the impact of various components of the method (e.g., network architecture choices, hyperparameters) are needed to provide deeper insights into the method's performance.\n6.\tSeveral technical terms and abbreviations are not properly defined or explained:\n-\tL1: \"SPAD\" is not defined on first use.\n-\tL15: The unit for \"0.06 photons per pixel\" is not specified (e.g., per frame or per second).\n-\tL22: \"QBP\" is not defined on first use.\n-\tL98: The meaning of s_i and i is unclear (intensity or pixel position?).\n-\tL112: The probability of zero photons hitting a pixel being e^{s_i} needs a detailed explanation.\n-\tL197: The term \"10-shot inference\" is not explained.\n7.\tSome experimental details are missing:\n-\tL205: More information on the iPhone 15 slow motion mode (e.g., resolution, exact frame rate) would be helpful.\n\nQUESTIONS:\n1.\tHow do you propose to select the photon splitting variable p in practice when the signal level is unknown before capture? Can you provide guidelines for selecting an optimal value?\n2.\tCan you provide a more comprehensive comparison with other existing quanta video reconstruction methods beyond QBP, including quantitative results?\n3.\tCan you provide a detailed analysis of the computational complexity and how it scales with dataset size?\n4.\tCould you elaborate on the runtime and memory requirements of your method compared to existing approaches?\n5.\tCan you provide more extensive ablation studies on the impact of various components of your method, such as network architecture choices and hyperparameters?\n6.\tCan you provide more details on the composition and size of the training dataset used? How many videos were used in both simulated and real tests?\n7.\tHow does the method perform on extremely low photon count data, and what is the lower limit of photon count where the method remains effective?\n8.\tHow does the method perform on different types of scenes or motion patterns?\n\nLIMITATIONS:\nThe authors have adequately addressed limitations, discussing the scope of applicability to Poisson noise and computational considerations. They have also provided a thorough discussion of assumptions and potential limitations in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper proposes to extend the Generative Accumulation of Photons that was proposed for Poisson noise to 1 bit Quanta image sensors.

STRENGTHS:
The proposed method is novel, and mathematically interesting. The authors have put in a lot of effort to fit GAP for the problem of 1-bit QIS reconstruction.

WEAKNESSES:
The proposed method does not seem to be performing better than a supervised method. It is not clear what is the supervised method that was used in the comparisons. 

It is not clear what the unique advantages of this method are compared to standard techniques like supervised learning or data simulation based training.

QUESTIONS:
Wouldnt randomly assigning each photon event to one of the two bins lead to loss of temporal information to some extent in both the input and target?
The supervised method seems to be performing better than the proposed method on static simulated scenes? Would it achieve similar fps as the proposed method on the video data too?

LIMITATIONS:
Yes. The authors have made the limitations of the proposed method clear.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to reconstruct/denoise high-resolution high-frame-rate videos captured by 1-bit quanta imaging sensors (e.g., SPAD arrays) without heavy spatio-temporal binning. The paper also captures and will release a new SPAD dataset.\n\nThe proposed method is loosely based on Generative Accumulation of Photons [9], which trains a CNN to reconstruct images in a self-supervised fashion by reconstructing the pdf of the photon arrivals. This paper makes several significant modifications to [9] so that it can work with quantum image sensors: First, in order to account for 1-bit sensors, the proposed method models the photon arrival pdf as a Bournoulli (rather than Poisson) distribution to account for the binary nature of the measurements. Second, in order to improve reconstruction accuracy it incorporates temporal information regularization.\n\nThe proposed method is tested on experimentally captured data and produces visually compelling results, including in the presence of non-rigid motion (e.g., guitar string).\n\nSTRENGTHS:\nA SPAD dataset would be extremely valuable to the research community.\n\nProposed method is novel and works well.\n\nPaper is well written.\n\nDenoising quanta image sensor data is an important problem with many scientific imaging applications\n\nWEAKNESSES:\nProposed technique largely follows from GAP. Unclear if Poisson distribution assumption is invalid in the photon starved regime, i.e., would GAP work for dimmer scenes?\n\nFigure/table captions could use more info. E.g., state whether Table 1 results are experimental or simulated.\n\nUnderperforms supervised methods.\n\nQUESTIONS:\nDoes a Poisson signal model work in the low-flux regime (where it would be rare for more than one photon to arrive in a pixel)?\n\nCan one bin a large # of frames in overlapping windows to ensure Poisson statistics while still recovering high temporal resolution?\n\nHow important is self-supervised training? Is the simulated noise model accurate enough that one could train an effective QIS denoiser using only simulated data?\n\nLIMITATIONS:\nWell discussed", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}