bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction
Authors: Yehe Liu
Alexander Krull
Hector Basevi
Ales Leonardis
Michael W. Jenkins
Venue: NeurIPS 2024 poster (2024)
Keywords: Denoising; Self-supervised learning; Quanta imaging; Photon counting
Primary area: machine_vision
TL;DR: We show that high quality video can be created from extremely sparse 1-bit image sequence, the typical raw data of an emerging image sensor, using self-supervised learning.

Abstract: Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

bit2bit: 1-bit quanta video reconstruction by self-supervised photon location prediction
Yehe Liu 1.2
Ales Leonardis Michael Jenkins 122, Casse Western Reserve University "OpsiClear LLC UUiversity of Birmingham
Alexander Krull
Hector Basevi
(yehe, mxj5) edu fa.f.f.krull, b.r.abasevi, allonnrdis ) @ @bham.ac.uk
Joint supervision"
Abstract
Quanta image sensors, such single-photon avalanche diode (SPAD) arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-proce processed using heavy spatiotemporal binming to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, show that the assumption of Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads the proposal of novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real (PAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.
1 Introduction
(e.g., SPAD arrays 2 offer unique advantages over other image sensors for capturing fast temporal dynamics in low-light settings, they can detect individual photons within nanoseconds exposures (3] This allows for the measurement of ultra-fast phenomena, such as fluorescence lifetime [4, time-of- flight (5), and other time-resolved processes [6]. Quanta image data is different from traditional digital images. The QIS raw data often consists of sparse detection events described in 1-bit arrays (Fig D. each frame, pixels only indicate the presence or absence of photon without intensity information 7 This makes the data non-interpretable or usable as traditional images. A common approach to make effective use of the binary data is to bin the image in the time domain(8). While this can provide clear images (Fig. Tt). it comes the cost of losing the valuable high-temporal tar---epooal--rsouutttii resolution information offered by QIS I this paper, we present novel self-supervised approach that directly denoises binary photon detection events from OIS reconstructing a clean video at the original temporal resolution.
Quanta image sensor OIS
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

time
Photons
detecior
One at
CYONA001....
Detectors Bernouli
some Aunalion 6
Signal
1-bit array
Prediction
frames
Real quanta video from SPAD
frames
Predicted signal
width
time
width
time
spsdolemporal
Figure Visualization the reconstruction task. a. signal in spacetime generates discrete photons through Poisson process. Real detectors only count one photon at time. The discrete nature of photons and the discrete counting process introduce shot noise, resulting in a sparse binary map. Our goal is predict the underlying signal from this information sparrs data. b. Real (PAD raw data captured by a detector. The highlighted box indicates a zoomed-in region, revealing sparse photon detection events. To the right is a cross-section of the time-height dimensions, showing similar binary noisy pattern. c. Our method produces the video from the data in at the original spatiotemporal resolution (Video S1). d. Left: effect of accumulating raw data frames directly, showing shot noise and motion artifacts. Right: additional keyframe pairs are provided for reference.
Generative Accumulation of Photons GAP recently introduced an effective self-supervised method that denoises shot noise corrupted photon counting images by splitting images into paired data training pairs based on Poisson statisties 9 The self-supervised method is valuable to scientific imaging applications, where very often ground truth data is unavailable and cannot be simulated. However, this method is ineffective for binary images, where creates significant artifacts due to the non-Poissonian nature of photon statistics in 1-bit data [TO Α. This inspired us to develop bit2bit, new method that specialized for binary data. Our key new insight is masking strategy that hides the complementary dependency within the training pairs, which effectively addresses the problem. We further extend our method to 3D to leverage the information both space and time, substantially improving the reconstruction quality. In addition to these measures, also explored different architectures and sampling parameter spaces to reduce the significant issue of overfitting.
Problem statement and claims. The goal of this work is to predict the underlying spatiotemporal
signal from stack of quanta data. Each frame in the stack is 1-bit 2D array representing locations where at least one photon is detected during the exposure period (Fig. Ta). Our method is based the following 2 assumptions: 1) Each detection event is independent and follows truncated Poisson statistics. 2) The underlying signal exhibits some local and global spatiotemporal structures that can be described by model.
Given this challenging reconstruction problem, we claim the following main contributions:
1. Developing a self-supervised method that denoises photon-sparse binary quanta images, which effectively handles the binary nature of the data through novel masking strategy.
2. Enhancing the performance of reconstruction by leveraging the temporal information that is readily available most quanta image data.
3. Providing insights on stochastic sampling strategies, network designs, and regularization
techniques to manage overfitting and improve the reconstruction quality.
4. Presenting a novel dataset with real and simulated 1-bit SPAD data to support further research and quantitative evaluation of quanta image processing.

---

zoom to VIEW
Fan motion
H&E slide
Sonicator bubbles
USAF 1951 + Drill
Plasma ball
Colar-Aomporal-coddd 50 frames
Figure 2: Example Results from Our Method Using Real SPAD Data The top row displays raw SPAD data. The middle row shows the corresponding reconstructions using our method. CPU Fan + motion:
Imaged under camera motion. Additional paired raw data and reconstruction keyframes shown below. H&E slide: Moving under a microscope. Sonicating bubbles: Humidifier generates bubbles, water droplets, and mist. USAF 1951 drill: Resolution target spinning drill. Plasma ball: Firing plasma. A color-coded accumulation of 50 frames is shown on the right. [More in supp]
2
Related Work
Quanta image reconstruction. A common quanta image reconstruction task is finding the optimal transformations of individual 1-bit frames to minimize motion blur prior to applying binning X
14]. Noise is another problem being addressed, especially in SPAD-based 3D reconstruction [S 17
Some higher-level tasks address the motion deblur of the binned data using specially engineered deep learning methods because the mixing of shot noise renders traditional debluring methods ineffective [18. Besides direct image reconstruction, there are also efforts to perform traditional computer vision CCV tasks directly to quanta image data (e.g., image classification) [19 The task described in this work reconstructing single binary frames at original temporal resolution and addressing both shot noise and motion blur is challenging and has not been well explored. Only one previous work explored a similar task using the supervised method with ground truth data [22). in contrast to our self- supervieed method.
Self-supervised denoising. Convolution neural networks (CNN) can be trained to denoise images by minimizing the difference (e.g., mean squared error (MSE between the prediction and reference target [23] 24 This is usually referred to as the minimal mean squared error (MMSE) denoising,
Noise2Noise N2N demonstrated that is possible to minimize the MSE between pair of noisy
images and still get quality denoising results without knowing the noise-free prior (25 However, N2N requires real data pairs with the same underlying signal, which are often obtainable. More recently, self-supervised denoising methods have been introduced to perform denoising from single noisy image. The methods create input and target training pairs from original noisy image by assuming the augmentations minimally affect the underlying signal. Noise2 (2) predicts the value random pixels noisy image by hiding these pixels during training and only using the surrounding context to minimize loss only from the hidden pixels [26] 27). Noisc2Self N2S) presented a similar concept that uses a moving grid mask (28). Noisicr2Noise adds more noise to the input to alter the original noise [29] Neighbour2Neighhbur. Noise2Fast, etc., assume pixel-wise
independence of noise and create image pairs from neighboring pixels 30 31]
However, the binary, sparse nature of quanta image creates special, challenging case. A recent self-supervised denoising method, GAP 9, effectively addressed shot noise corruption in photon counting images by Daisoon-taaissti--aases data by pixel-wise binomial sampling to create nearly unlimited training pairs, which inspired our work. However, implementing GAP directly binary quanta images resulted in extremely poor
resampling. GAP introduced the idea of splitting the

---

Photon splitting: create "uniimited" "reaf data i prevent overfitting
Don' team from DINW
One more solt, ane more unque haknngg par
Random
3D
1-bit
farget
3D network: from space & time
NOT lppat
Masking:
complementary dependancy training
Overview of the sampling/masking strategy. The raw data processed 3D to use space and time similarly. Data pairs are created by random 3D crop from the raw data, then randomly split the positive values into an input or a target matrix. The split ratio is controlled by parameter p. A mask is created by flipping the bits in the input image, which prevents gradient back- propagation from locations of Is in the input. This process is repeated indefinitely, each time creating a new pair of data equivalent independent observations from the underlying signal.
Figure 3:
reconstructions. This motivated us to identify the root cause of the problem and develop a specialized self-supervised method for sparse binary image reconstruction.
3 Theories and methods
Here, we summarize the self-supervised denoising approach described in GAP 19. GAP is based on the assumption that observed image is shot noise corrupted version of the clean image s, with each pixel xy drawn from a Poisson distribution parameterized by the clean signal at the pixel s,, where indicates the pixel index. GAP proposes to split the noisy image by sampling from a binomial distribution randomly assigning photons to an input image x and a target image _ The images are conditionally independent given the underlying signal so can be treated as two independent observations, which can then utilized in a N2N training procedure 25 to train neural network [xx 0) with parameters 0. By attempting to predict from Xap- the estimate for the clean signal. uses a cross-entropy
Brief review of GAP
procedure trains a network find the eMSSiestmmaee loss over pixels
exp(f(x O):)
L(f [x () x
Σj exp(f(xfmm" (x))
viewing the problem as classification task, trying to predict the photon locations in the target image. Here, fxx f), corresponds to the logit output for pixel i. The authors show that in the limit, their loss is equivalent to the MSE loss, which is more frequently used in N2N training.
Photon counting. While the assumption of the Poisson shot noise model in GAP seems reasonable for photon counting, often does not accurately hold for truncated QIS data such as SPAD We can understand this by modeling photon counting photon as an element of set defining this process. Subsequently, we can model the compound processes and the sub-processes within the framework of point process statistics and differentiate between photon counting and sector activation events (e.g., binary activations in each frame). This broader insight is discussed in the Appendix. This section focuses the practical implementation of this insight in self-supervised denoising.
Poisson point process and considering each counted
Quanta image generation. Instead of truly counting photons, SPAD (and other quanta imaging methods) acquire a series of binary images x', with cach pixel x E {0, 1} indicating whether photons have been detected not. Unfortunately, this means that the number of photons hitting the pixel is not recorded in a unit detection window, and we cannot know if an active pixel af = 1 receives single multiple photons during one exposure. Formally, pixel values x are drawn from Bernoulli distribution with parameter p= e e-si, with e corresponding to the probability of zero photons hitting pixel according to the underlying Poisson distribution (Appendix A.1). For small s; individual pixels rarely receive multiple photons during an exposure. Thus, it is convenient consider A
In order to generate images that are visually pleasing and easier to analyze, the sparse binary data is usually processed by summing multiple frames x", effectively counting the number of detection

---

Raw data
Split
Input
Target
Mask
Raw data
Network trained wio mask
Network trained wi mask
LAB
TAN
HO
Figure 4: Real data examples of photon splitting and the effect of the masked loss a. Example of splitting randomly selected quanta image data frame. The Raw data consists of only binary pixels indicating the location of the photon counting event. The Split indicates the Input (black) and Target (white) of the split. The Mask is caleulated by inverting the Input and is applied the loss. Comparison the training results with unmasked and masked loss. Without the masked loss, the network learns that whenever a pixel location has photon in the input, never has photon in the target. The deterministic relationship leads to the artifacts. The pixel locations where the input is I appear dark the network output. The masked loss effectively addresses the problem.
events for each pixel 77. Unfortunately, this aggregation over time leads inevitable loss of temporal resolution, as information about the time in which photon hit lost. It is important to note that the noise in such data is not Poisson distributed but instead follows binomial distribution. Unfortunately. this means [APP| which relies on this assumption, cannot be directly applied to SPAD data. In the Appendix (Photon detection), we give an alternative theoretical perspective on this phenomenon by modeling photon counting as Poisson point process with sub-processes to understand the difference between photon counting and the counting of photon detection events. While it is true that we can produce approximate Poisson statistics by summing a large number of frames, this comes the cost of significantly reduced time resolution.
Splitting of 1-bit quanta data. This work explores an alternative approach, attempting to directly split the recorded binary images without temporal aggregation. We split the recorded binary image according to GAP [9, randomly assigning each photon detection event in x' to x or x with
probability (Fig. B}. Assuming low light intensity 8; this means that the probability for finding
photon detection event pixel is Pi.in ps, and Pi.rr (1 p)s;. In order to enable the use of temporal information, we apply the same not to 2dimensional images but to 3D space-time.
volumes X
(x) .'.....x"),, splitting them into input X (x ................................ xTpp) and target volumes
X (x
However, unlike for true Poisson distributed photon counts, X and x", cannot be considered conditionally independent. Since photon detection event in ot can only be assigned to either x x_p X an event x in the input image must necessarily mean that corresponding pixel in the target image does contain an event x 0. As consequence, naively using x and x as training pairs for a denoiser, will lead to the artifacts, with the network predicting dark values for any pixel which has photon detection event in its input.
Masked loss function photon prediction for quanta images. To remedy this effect of correlated input and target images, we propose an adapted loss function. Considering that our data is sparse, that is. only very few pixels contain photon detection events, avoid the problem by excluding
5

---

pixels that contain photon detection event in the input image using the loss
n.T > (1 Tip ) Jine In
exp(f (Xmmp 0)1)
L(F(Xup: f). Xmn)
Egjr exp(s (X 0)))
with mXi 0) referring to the logit output of the denoiser network and f f(X 0) referring to the
inp:
value at the pixel/time index i, of the output. Note that, while the masking achieved by multiplying
(1 2. imp the remainder of the function corresponds exactly to the cross entropy loss over pixels
from Eq.
Implementation details. We view each binary quanta image dataset as 3D array with two spatial
dimensions and temporal dimension. We randomly crop each sample x from the volume (Fig. 2 Xip is sampled from a volume x by Bernoulli sampling with dropout probability p defined based on
desired strategy. We then calculate the target Xur by subtracting the input Xmp from x. The mask Xmaa is computed from the input by Xmma 1 m 1) at runtime. In the binary case, the mask is equivalent to the bitwise complement of the input -Xup (Fig. B. We multiply the mask by the output of the network and the target before the cross entropy loss is calculated, which is equivalent to the loss function discussed in the previous section.
Selecting photon splitting variable p. The Bernoulli sampling probability is important new hyperparameter introduced in the sampling process. Its value and selection strategy can drastically impact the performance of production. We will demonstrate this in our experimental section. If we only work at a fixed input signal level (e.g., mean photon count), we can use fixed p to generate training pairs. However, it is necessary to reduce the photon count to the same signal level for inference. A fixed large or small can lead to over-fitting and slow training. In extreme cases, the model will recreate the target if p = 1, and the target is always empty if p 0. A single input at the reduced signal level does not contain all the available information. We could resample multiple copies of the input and combine the inference results, but this strategy requires longer inference time. Alternatively, we can randomly pick p from desired range, which allows the model to adapt to different signal levels and produce acceptable results from all the inputs within the range. If we use a large p close to 1 as the upper limit (e.g., - 10-"), we should be able to use raw data directly as the input, which more efficient in practice.
Experiments
32 with option to switch the network to 2D Network architecture. We used a 3D ResUNet
at a specific depth. The primary reason for this design choice is to handle the rapid increase of the receptive field as the depth increases, which eventually becomes bigger than the sampled area. The network will learn a lot from the padded areas during training. In the photon sparse binary case, this problem is more prominent, the 0 padding is not different from the real data. To avoid significant weight from the padded area, a large crop window size is necessary to achieve better results. However, this can easily lead to memory issues. We can prevent the receptive field from growing in the temporal dimension by switching from 3D to 2D at the desired depth. This allows us to use a smaller window size in the temporal dimension and a larger window size in the spatial dimension. A detailed diagram of the network architecture is shown in Fig. SI For most experiments, we set the network depth at 5, with only the first levels being 3D to control the size of the receptive field. We used 32 initial input features, which provide a balance between performance and memory efficiency. The number of
features doubles at each depth. Group normalization of 8 is applied at each convolution
[3)
We use
GeLU for the activation function and pixel shuffling for up-sampling [34 35).
Training. Models were trained using the ADAMW optimizer for 150 epochs, with 250 steps per epoch and 4 batches of random crops of 32x256x256 (TXY) per step [36]. The large patch size desired to prevent performance degradation due to substantial padding weights. However, 3D memory-efficient, UNet with a large patch size is not memory efficient, and data transfer is bottleneck. Sometimes,
only 2 batches of the crop size can be trained at a time while leaving a substantial memory unused with 24GB VRAM. Therefore, we implemented gradient check-pointing (37], which allowed us to double the batch size and utilize the majority of the VRAM. We primarily used Nvidia 3090/4090 for training. and each training takes about 6-10 hours until the loss curve stabilizes.

---

Inference. We rum patch- bascd inference on a GPU using the original image width and height, maximizing the frame count for each inference. With 24GB VRAM, we can use a 48x512x5 12 volume for each inference with the typical 5-depth Res mentioned above. The inference speed above volumes per second (150 fps) on a NVIDIA RTX 4090 GPU. If the training uses Pmnax R 1, the raw data is used directly as input. Otherwise, we randomly reduce the photon count in the data by a factor of Pmax through photon splitting. For a 10-shot inference, we Bernoulli sample 10 times from the raw input fixed p value, run inference independently using the model trained at the p value, and calculate the mean of all the outputs.
Data. To evaluate our method quantitatively and qualitatively, we use a synthetic video with simulated noise consisting of 3990 frames and total of 7 real SPAD videos, each containing 100k-130k frames. Additionally, we use a real video with 100k frames published in [8) Simulated data is essential for quantitative assessment as QIS ground truth is impractical to get. Assuming reference images/videos are ground truth with no shot noise and the pixel values n, are proportional to the Poisson rate As. we calculate a variable g based on the selected mean counting rate Ā ( Ein ni), where + _i=1 nu; is the mean of the reference. We then run pixel-wise Poisson sampling with X; n.P and clip between 0 and I to simulate the 1-bit quanta data. A ground truth reference video of a person shaking a ThorLab's 's box was acquired from iPhone 15 slow motion mode at 240 fps (Fig. (22). The simulation was using À = 0.0625. After the clipping. the resulting simulation has 0.0590 photons per pixel. Real Q data were acquired using SPAD512S camera (Pi Imaging Technology, Switzerland))2). Each dataset has 100k-130K binary frames. We created scenes to represent a combination of different imaging conditions, including low-signal, high-signal, high- contrast, high-ambient-ligge
M
moving camera, moving object, linear movement, random movement, combined movement, ultra-fast events, and stochastic events (Fig S6 made the simulated data, real SPAD data, and reference results available to the community"
S7
S8
59
S10
S12). We
N2V and the original GAPwwith simulated data. To ensure fair comparisons, we incorporated the same network architec- tures, hyperparameters, and training steps into individual baselines. All implementations are in 3D except for Inference was conducted on the first 512 frames, and the data was normalized by the mean. PSNR and SSIM were calculated frame-wise, and the statistics were derived from the entire output stack. We tested N2N in three different ways: Using two independent simulated samples for training. Sampling two independent image stacks using the proposed splitting method (with p= 0.3. just once before training). Using the same samples but applying the masking strategy during training. We tested N2V using the original masking strategy. The N2V masks were created from 1000 random points in space. A median filter was applied to these points in the input. The minimum distance between the points was smaller than the median filter size, and loss was only computed from these points. We tested the original GAPlin i 2D with and without masking. Finally, evaluated our method with and without masking.
Comparisons of different methods. We tested supervised training, N2N
5 Results and discussions
Findings from the comparisons. Tab. I summarizes key numerical and visual results. The supervised methods perform the best, which reflects the approximate upper limit of the network's performance. In reality, the ground truth is often not available, and high SNR data is only available from binning, which degrades time resolution and blends dynamics. N2N with independent samples created a granular pattern on the images, as well as increasing validation loss since the beginning of the training (Fig. S16). Besides, two independent quanta data with the same underlying signal do not exist. One way the address this with N2N is to use adjacent frames, but this reduces the temporal resolution and introduces motion artifacts (e.g.. Fig. SS Interframe N2N). Without the mask, the image shows strong pepper noise. After implementing the mask, it generates a smooth image, but the granular pattern persists. We also tested our method, which is similar to GAP in 3D. Masking addresses the image artifact problem. The 3D implementation led to substantially improved PSNR and image quality. N2V also produced poor reconstruction results. The result from the original 2D GAP produced also has the presence of pepper noise, similar our implementation with large fixed p values. Implementing the mask in the GAP resolved the problem. The images are smooth but lack
httpss//drive.gen httpg//ewwooooooooo//ooooo////en /ttp://tthtoo/oooooooooo/ooo////
7

---

Table 1: Results from comparison experiments.
Method
Mask
2D33
PSNR
SSIM
Denoised patch
Raw
n/a
n/a
4.81/0.55
0.017/0.006
Humans
Supervised
N
3D
36.51/1.56
0.976/0.006
THOR LABS
Humans
N2N
N
3D
32.15/1.16
0.931/0.009
(2-sample)
THOR THOSLATS LATS
Rumans
N2N
N
3D
23.14/0.73
0.651/0.027
Humans
N2N
Y
3D
28.83/0.58
0.843/0.010
THOR
N2V
N
3D
26.16/0.97
0.804/0.028
GAP
N
2D
20.54/0.64
0.588/0.059
Humma
GAP
Y
2D
29.09/0.57
0.911/0.015
THOR
bit2bit
N
3D
20.89/0.84
0.599/0.062
Humans
bit2bit
Y
3D
33.93/0.99
0.959/0.007
THORLARS
fine details. Our method also shows the pepper noise and the fine details are missing when the mask is not applied. Implementing the mask substantially improved the reconstruction quality.
Inference strategies. We achieved PSNR of 33.93 using network with p E [0, 10 that takes the original raw data as input and performs inference in one shot. Further, training network less extreme p range [0.3, 0.7] to avoid residual overfitting, then running inferences using 10 different samples sampled from the original data with p = 7, and finally combining the data by simple averaging, improved the PSNR to 34.35 (Table. S8). It worth noting that it should assumed that the result is specific to this simulated dataset and model configurations.
1-bit self-resa We noticed that different self-supervised resampling strategies become very similar across different methods in 1-bit. When we create a training pair and largely preserve the original information, we could either keep a positive one image or move to another image. Then, the sampling method essentially describes the spatial rule of splitting. For example, GAP uses binomial sampling to randomly drop out some photons. Self2Self (38 uses random binary masks to drop out some pixels, which is equivalent to GAP in 1-bit. N and N2V both select some pixels

---

x-axis group norm
= [x. 1 10
= [0.
X
start filters
PSNRISTD
34.5
} *
35
32
33.01
x
33.0
34
32.5
23.0
33
30.0
31.5
16 32
0.0
022
05
06
08
0.
24
26
Figure 5: Results of ablation studies. a. Group normalization substantially improved the PSNR. b. The choice of lower and c. upper bound of the thinning probability p affects the reconstruction quality. (.9x6:1 10,, etc.) d. Fixed large led to performance degradation despite the proposed single photon prediction suggested in GAP e. Large model size could negatively impact PSNR. Rome numbers indicate the corresponding images in Fig. S3 Numerical values in Table S2-6.
The pixels are replaced by new value using various strategies, such as median, random, and random dropout, which usually remove the selected positive pixel and very occasionally create new positive pixels. If fixed grid is used, the operation becomes In the cases of neighboring pixel- based methods 30 BT], image pairs are created by sub-sampling from 2x2 grid, equivalent to a balanced spatial-dependent selection followed by downsampling.
in the input and use them target 28 27
Group normalization. We noticed that group normalization is critical to for this task. Fig. 5h and Table. S2 shows that PSNR increased from 30.9 to 33.5 by using group normalization of In creating the group number 8 further increased the PSNR to 33.9. We could not implement batch normalization [39]. as 24GB VRAM is only sufficient for batch in some use cases (e.g.. 512x512x32 window size).
Over-fitting, stochastic splitting and choices of p. Overfitting often unavoidable self- supervised denoising because the model can learn fit the noise patterns specific to the finite training data. We noticed granular image artifacts in N2N N2V and GAP with large, fixed p-values, which resulted in substantially reduced reconstruction quality (fig. 3). Further analysis indicates that these three examples share common issue: the input of the model is approximately fixed binary pattern. In N2N there is no variation between the input and target data. In N2V we used a random grid to select the training target, but the grid size was relatively small, and most selected pixels were The input data was very similar the raw data, with a few photons removed, making the process similar to GAP with a large, fixed p-value.
To analyze the problem, conducted experiment using randomly Poisson noise with small À (Fig. S4). Ideally, the method should produce a uniform plane image with small pixel value variations. Our results show that if new random input is used for every training step, the output of the network converges to a uniform image as expected, regardless of whether a fixed output is used. If we fix the input and randomize the output, the result is also uniform. However, if we fix both the input and output or if the output contains few photons (c.g.. GAP with fixed large p), granular patterns similar the results start to appear, also reflected by substantially higher standard deviation. Increasing the photon counts in the input reduces shot noise and mitigates the problem. This suggests that diversity in the training pairs is crucial mitigate overfitting in photon mhoton--ppars sparse quanta image data.
Model size selection. Increasing model size can negatively impact the performance, potentially due to overfitting. We found smaller 5-depth network with 24 startup filters outperformed larger network with 48 filters (Fig. Fp). along with non-significant decaying from 24 to 40 filters. In another comparison, network with depth 4 and startup filters can achieve comparable performance (PSNR=33.83) as our best 5-depth network with 32 startup filters (33.93) (Table. S7). A 6-depth network (32.55) has a similar performance as 3-depth network (32.42). The model size parameter space should be explored to balance overfitting and under-repreeeentaion for different tasks.
Comparison to QBP Quanta burst photography (QBP) is a recognized state-of-the-art quanta image reconstruction method based on spatial-temporal hierarchical alignment and frequency domain combination [8) We selected a challenging non-static scene from the original work for comparison, where person is playing the guitar. The results shown in Fig. 6 and Video S2. It is important to

---

our 100k frames)
Long bin (2k Home)
Short bin (100 - frame)
QBP -e hrme 1i-4 (p6) Ours (20% 20k frames/ foo)
Hright c10
Figure 6: Comparison of our method QBP with real SPAD data presented in the paper. a) Different rendering of the data. The data is from the original QBP, indicating dynamic scene with person playing guitar. Our result shown on the right. b) Height-time time slicing of the raw data and our our reconstruction, showing the top strings. Bottom: the
reconstruction. Top: raw data. Middle: difference between adjacent frames, indicating sub-pixel movements of the string.
note that QBP reconstructs 1 image from 100-10,000 frames in relatively static scenes, each taking 30 minutes reported (not optimized) [8) In contrast, our method can start produce quality results in less than hour on a 100k-frame dataset. Longer training further improves the results. Our method achieves over 150 fps inference on this dataset, equivalent to processing the entire 100k frames in less than 30 minutes with 50% patch overlap.
6 Conclusions
demonstrated a versatile method for predicting the underlying signal of binary quanta image data, offering practical solution for quanta image reconstruction where high SNR data and ground truth are unavailable. However, it is necessary to consider the limitations of the method when applying to real data. Our method assumes simple Poisson point process model, where the observation is sampled from a noise-free free signal. Thus, is not intended to restore the signal from other types of noise. From different perspective, the method removes the Poisson noise component from the data, making it easier to identify and correct other noise components. For example, hot pixels on SPAD be revealed and corrected with median filtering.
fundamental limitation of self-supervised denoising is the lack of ground truth data, making difficult to evaluate performance and accuracy in specific cases. In our case, performance could be evaluated by simulating the Poisson process, though these simulations should be based on reasonable assumptions and approximations. Additionally, the method requires offline training for each dataset. While there is potential for real-time reconstruction, the model should be trained on similar scenes.
Our approach can potentially be effective on any binary gigher-dimensiona spatial data created from Poisson point processes. For example, any data represented by an n-dimensional scatter plot, where points are independent measurements, can be analyzed with this method. Fig. S14 and $15 show the application and benchmark of the method to real and simulated 3D stacks of confocal microscope data with extremely high shot noise, which also demonstrates the performance advantage in 3D.
We also see the task could be generalized for all spatial event point processes as follows 44) For an n-dimensional signal in spacetime, a finite amount of measurements can be made from disjoint, uniform, and bounded sub-regions. Their collection can be viewed as a Poisson point process. The measurements are clipped between 0 and 1, and the process becomes a Bernoulli process. Given finite number of such measurements, the task measured by the Poisson rate. We envision that the core concept of creating data pairs by randomly splitting point process and then hiding their complementary dependencies can potentially be used in other relevant regression models.
predict the distribution of the underlying signal
Finally, it is worth noting that the method can be misused in research if inappropriate data statistical assumptions about the noise are made (e.g., the data points are not independent). In such cases, the prediction results be misinterpreted, leading to incorrect scientific conclusions and potential negative societal impact.
10

---

References
[1] Kazuhiro Morimoto, Andrei Ardelean, Ming-Lo Wu, Arin Can Ulku, Ivan Michel Antolovic, Claudio Bruschini, and Edoardo Charbon. Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications. Optica, 744):346-354 April 2020. Publisher: Optica Publishing Group.
[2] Arin Can Ulku, Claudio Bruschini, Ivan Michel Antolović, Yung Kuo, Rinat Ankri, Shimon Weiss, Xavier Michalet, and Edoardo Charbon. 512 x 512 spad image sensor with integrated gating for widefield flim. IEEE Journal of Selected Topics in Quantum Electronics, 25(1):1-12, 2019.
[3] Eric R. Fossum, Jiaju Ma, Saleh Masoodian, Leo Anzagira, and Rachel Zizza. The Quanta Image Sensor: Every Photon Counts. Sensors, 16(8):1260, August 2016. Number: Publisher: Multidisciplinary Digital Publishing Institute.
[4] Lucio Pancheri and David Stoppa. spad- based pixel linear array for spee-speed time-gated fluorescence lifetime imaging. In 2009 Proceedings of ESSCIRC, pages 428 431, 2009.
[5] Villa, B. Markovic, S. Bellisai, D. Bronzi, A. Tosi, Zappa, S. Tisa, D. Durini, S. Weyers, U. Paschen, and W. Brockherde. Spad smart pixel for time-of-flight and time-correlated single-photon counting measurements, IEFE Photonics Journal, 4(3):955-004, 2012.
[6] A. Kufcsák, A. Erdogan, R. Walker, K. Ehrlich, M. Tanner, A. Megia-Fernandez, E. Scholefield, Emanuel, K. Dhaliwal, M. Bradley, R. K. Henderson, and N. Krstajić. Time-resolved spec- troscopy at 19,000 lines per second using a cmos spad line array enables advanced biophotonics applications. Opt. Express, 25(10): 103-11 123, May 2017.
[7] Stanley H Chan. What does one-bit quanta image sensor offer? IEEE Transactions Computational Imaging, 8:770-783, 2022.
[8] Sizhuo Ma, Shantanu Gupta, Arin C. Ulku, Claudio Bruschini. Edoardo Charbon, and Mohit Gupta. Quanta burst photography. ACM Transactions on Graphics, 39(4), August 2020.
[9] A. Krull, H. Basevi, B. Salmon, A. Zeug, F. Muller, S. Tonks, L. Muppala, and A. Leonardi. Image denoising and the generative accumulation of photons. In 2024 IEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 1517-1526, Los Alamitos, CA, USA, jan 2024. IEEE Computer Society.
[10] Feng Yang. Yue M. Lu, Luciano Sbaiz, and Martin Vetterli. Bits from photons: Oversampled image acquisition using binary poisson statistics. IEEE Transactions on Image Processing, 21(4):1421-1436, 2012.
[11] Jeffrey D. Scargle. Studies in Astronomical Time Series Analysis. V. Bayesian Blocks, a New Method to Analyze Structure in Photon Counting Data The Astrophysical Journal, 504(1):405, September 1998. Publisher: IOP Publishing.
[12] Stanley H Chan, Omar Elgendy, and Xiran Wang. Images from bits: Non-iterative image reconstruction for quanta image sensors. Sensors, 16(1 2016.
[13] Trevor Seets, Atul Ingle, Martin Laurenzis, and Andreas Velten. Motion adaptive deblurring with single-photon cameras. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1945-1954, January 2021.
[14] Kiyotaka Iwabuchi, Yusuke Kameda, and Takayuki Hamamoto. Image quality improvements based on motion-based deblurring for single-photon imaging. IEEE Access, 9:30080-3009 2021.
[15] Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa, Vivek K. Goyal, Franco N. C. Wong, and Jeffrey H. Shapiro. Photon-efficient imaging with a single-photon camera. Nature Communications, 7(1):12046, June 2016. Publisher: Nature Publishing Group.

---

[16] David B. Lindell, Matthew O'Toole, and Gordon Wetzstein. Single-photon 3D imaging with deep sensor fusion. ACM Transactions on Graphics. 37(4):1-12, August 2018.
[17] Gongxin Yao, Yiwei Chen, Chen Jiang. Yixin Xuan, Xiaomin Hu, Yong Liu, and Yu Pan. Dynamic single-photon 3D imaging with a sparsity-basee neural network. Optics Express, 30(21):3733--33340 October 2022. Publisher: Optica Publishing Group.
[18] Yash Sanghvi, Abhiram Gnanasambandam, and Stanley H. Chan. Photon limited non- non-blind deblurring using algorithm unrolling. IEEE Transactions on Computational Imaging, 8:851-866 2022.
[19] Abhiram Gnanasambandam and Stanley H Chan, Image classification in the dark using quanta image sensors. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 484-501. Springer, 2020.
[20] Chengxi Li, Xiangyu Qu, Abhiram Gnanasambandam, Omar A. Elgendy, Jiaju Ma, and Stan- ley H. Chan. Photon-Limited Object Detection using Non-local Feature Matching and Knowl- edge Distillation. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 3959-3970 Montreal, BC, Canada, October 2021. IEEE
[21] Shantanu Gupta and Mohit Gupta. Eulerian single-photon vision. In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 10431-10442, 2023.
[22] Paramanand Chandramouli, Samuel Burri, Claudio Bruschini, Edoardo Charbon, Kolb, and Andreas. A Bit Too Much? High Speed Imaging from Sparse Photon Counts. In 2019 IEEE International Conference on Computational Photography (ICCP), pages 1-9, May 2019. ISSN: 2472-7636.
[23] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, 26(7):3142 3155, 2017.
[24] Martin Weigert, Uwe Schmidt, Tobias Boothe, Andreas Müller, Alexandr Dibrov, Akanksha Jain, Benjamin Wilhelm, Deborah Schmidt, Coleman Broaddus, Siân Culley, et al. Content- aware image restoration: pushing the limits of fluorescence microscopy. Nature methods, 15(12):0090-1997, 2018.
[25] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference on Machine Learning, pages 2965-2974. PMLR, 2018.
[26] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void-leariing denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2129-2137, 2019.
[27] Eva Höck, Tim-Oliver Buchholz, Anselm Brachmann. Florian Jug, and Alexander Freytag, N2v2-fixing noise2void checkerboard artifacts with modified sampling strategies and tweaked network architecture. In European Conference on Computer Vision, pages 503-518. Springer, 2022.
[28] Joshua Batson and Loic Royer. Noise2self: Blind denoising by self supervisoo. In International Conference on Machine Learning, pages
PMLR, 2019.
[29] Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22064-1207) 2020.
[30] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighhor:: Self-Supervised Denoising from Single Noisy Images. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14776-14785, June 2021. ISSN: 2575-7075.
[31] Jason Lequyer, Reuben Philip, Amit Sharma, Wen-Hsin Hsu, and Laurence Pelletier. fast blind zero-shot denoiser. Nature Machine Intelligence, 4(11):953-963, 2022.
12

---

[32] Foivos Diakogiannis. François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrummetry and Remote Sensing, 162:94-114, 2020.
[33] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the Luropean conference on computer vision (ECCV), pages 3-19, 2018.
[34] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[35] Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. Efficient image super-resolution using pixel attention. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 56-72. Springer, 2020.
[36] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017.
[37] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.
[38] Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image. In 2020 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 1887-1895, June 2020. ISSN: 2575-7075
[39] Chunwei Tian, Yong Xu, and Wangmeng Zuo. Image denoising using deep cnn with batch renormalization. Neural Networks, 121:461-473 2020.
[40] Jesper Moller and Rasmus Plenge Waagepetersen. Statistical inference and simulation for spatial point processes. CRC press, 2003.
13

---

7 Data and code availability
Simulated and real SPAD data:
sttps:///rrive.oogg.. ott.://rwr.ooaaoool////o//ol/aaa
Python raw data loader and cleaner:
SPAD data simulator:
Source code of the parametcrized model with bit2bit:
https///gihbbb..omm/yeeee//////
Acknowledgment
We thank the following grants for their support of this research: NIH RO1EB028635, ROTHL1267477, 1100D024996, UO1EY034693 and R44CA268156.
We thank Axiom Optics for providing the demo unit of the SPAD512s camera.
Special thanks to the @L@ MBL (now AI MBL) program for sponsoring training and collaborations that made this work possible. Cheers Team Restorators and members Anirban Ray, Peter (Hyoungjun) Park, Geneva Schlafly, Guillaume Minet, and Mie Thorborg!
9 Competing interests
Yehe Liu and Michael Jenkins are co-founders and owners of OpsiClear LLC.
14

---

A Appendix/ Supporting documents
A.1 Photon counting statistics
Photon defection. Here, we are looking at the photon counting from the point process perspective. Imagine an underlying signal indicated by a rate parameter À describing the expected number of photons hitting the detector over unit of time. Light is quantized, so a hypothetical detector with infinite depth will record k ~ Poisson(A) photons over each detection window, with p(k n) X "teA/nnd >/nl, where n E No is the number of the detected photons. Real detectors can only activate once each detection window (e.g., n E {0,1]) Therefore, the Poisson point process is reduced to a Bernoulli lattice process with p(k = 0 A) = e and p(k = 11] X) - A- = - e With small X (e.g., 0.01), it is convenient to consider p(k 0) 1 A'd p(k = 1) A in photon-sparse binary data because the chance of missed counts p(k > 1) is negligibly small.
Photon counting. From the insight above, it is clear that the "photons" are just "activation events" over some finite detection windows in spacetime. In practice, it is often assumed that À changes minimally across a few windows [II] The superposition of n' such windows is also Poisson point process with X' = n'' X, assuming this homogeneity. If detection is modeled as a more realistic Bernoulli lattice process with p' p(k = 1) = e-A, X summing n' windows is equivalent to binomial sampling
~ Binomial(n'' p'), which can be approximated by a Poisson distribution with X' n'p' = n1 - e >- Both À and X' describe the underlying signal. more trivial to determine X' from multiple windows than À from a single binary image. However, X is constantly varying in reality. Through X' = E, A or E, - ^>)), we lose the information of individual X;-
Photon splitting. Since points in N and N' are independent, we can apply basic point process operation p-thinning randomly remove some points and create new independent sub- uub-processes. The thinning probability can be selected either manually or randomly to control the size of the new process Np. For A denoting the original expected count rate, the rate of the thinned process is proportionally reduced to PA. Similarly, the removed points can form another subprocess N1-p with rate (1 A)A. The new processes Np and N1-p observe a process over an exposure т, we can view each split as two real experiments where we observe first process N, during Tp from Pd and, right after, second process N Ni-p during T(1 p) from (1 - p)A. A. GAPloriginally implemented photon splitting by sampling X ~ Binomial(X'' (x'm)aal(XX))) pp) in the binned frames. Similarly, we use a special case X ~ Bernoull(p 1(x* 1)) in binary frames.
independent Poisson processes. Therefore, if we
Independence of the photon splitting pairs. Np and Ni-p are both independent Poisson processes. However, they exhibit a complementary dependency due their definition from the original process, specifically:
UN N & No N
This implies that for any xlt,,- if 244,p = n', then -111- = O. For small A and large n' (even for n' = 2), this is an extremely rare event as p(x²- = n') = II= X and p(ai,p = n' x = n') = pm Therefore, this ignored by Ghere the data are accumulations of multiple frames. However, in 1-bit data, all the photons in N₂ meet this criteria, as VX, : X, = Xi-p- This also applies to extremely photon-sparse observation when very few photons are detected over large n' because even with frame bins, we rarely encounter a photon count above 1. We will show this is a crucial problem in binary data, which renders GAP ineffective, and our masking strategy is an effective solution.
MMSE denoising. Different underlying signals can lead to an observed value denoted by N. Our
goal is to predict the signal, described the compound rate = (Xi | N*). We assume a prior distribution p(A) for the signal and a likelihood function p(N (N A) representing the probability of observing N given the signal A. Using Bayes' theorem, we can express the posterior probability and the expected rate of the signals as:
P(N | A)p(A) (A N) J [NN AM)(AA)) dA
A A= 1-INAA - NNAAA
(4)
15

---

To estimate the signal from the observation in the sense of MMSE denoising, we look for a function fo(N) and optimal parameter 0 that maps the observation N the signal A for each individual dataset. Previous works have shown CNN is effective for this regression task [].
Signal prediction. As shown in GAP we can randomly remove single photons d; from the process N one at a time. The relationship between the removed photons and remaining processes N = N', ( can be used to predict the distribution of upcoming photons. Recall that di sampled from a Poisson process with expected rate X; E A and p(d; N X:) A We can write:
p(oe E ENNEN NÇ) - f p(0: E EN A, NY) dA / MA'AN
(5)
If go over all possible locations and normalize the signal, becomes clear that predicting the distribution of the next photon is just predicting Á in the MMSE sense. We can train model fo(N) to predict this distribution by reducing the cross entropy loss L(
Σ, In fo fo (N)),, Assuming N N in the case of a single photon is negligible in the data, we can apply fo directly.
Training with one photon at a time is conecconu And the high similarity between different N' can lead to over-fitting In practice, could predict multiple photons time using data from photon splitting we normalize the target:
/ p(N, I A,N1-))P(A N1-p) dA = ff D(A N1 PA dA
-f
p(N, N1-p)
(6)
16

---

B Appendix Suuppemmentpry tables
Table S1: The baseline hyperparameters used in training. This produces the PSNR 33.93 33.93/0.99
and SSIM 0.959/0.007 with the simulated data. More details on hyperparameters are included in the supplemental materials in the hp. CSV file.
Start features
Depth
Depth scale
3D conv layers
32
5
Model
Group norm
Up mode pixelshuffle
Down mode
Activation
maxpool
gelu
XY size
size
Stack size
Min_p
256
32
1000
Data
Max
p sampling
Flip/Transpose
Rotation
0,999999
uniform
FALSE
Batch size
Epochs 150
Optimizer
Learn Rate
adamw
0.00032
Training
Patience
Loss function
Masked
VRAM
20
Cross entropy
TRUE
24GB
17

---

Table S2: Numerical results correspond Fig. 5a. essential for better PSNR and SSIM.
The result indicates group normalization is
Group size
PSNR
std
min
SSIM
std
min
32
31.83
1.03
29.26
35.66
0.940
0.009
0.917
0.962
16
32.29
1.12
28.24
36.20
0.956
0.007
0.921
0.972
8
33.93
0.99
29.31
36.57
0,959
0.007
0,924
0.973
4
33.50
1.06
28.93
36.51
0,957
0.007
0.923
0.973
2
33,48
1.09
28.96
36.55
0.957
0.008
0.921
0.974
1
30.85
0.93
27.91
33.76
0.917
0.013
0.876
0.947
Table S3: Numerical results correspond to Fig. 5p. Showing the effect of the lower bound of p.
range
PSNR
std
min
SSIM
std
min
[0.0, 1- [E-6]
33.93
0.99
29.31
36.57
0,959
0.007
0.924
0.973
[0.2, 1E-66]
33.69
1.07
29.31
36.78
0.957
0.007
0.924
0.974
1-
33.34
1.02
29.09
36.43
0,955
0.008
0.922
0.972
[0.6, 1E-66
33.11
1.01
28.73
36.04
0.953
0.008
0.915
0.970
[0.8. 1- 1E-6]
32.55
0.97
28.66
35.57
0.946
0.009
0.914
0.966
[0.9. 1E-6]
32.05
0.90
28.54
34.81
0.941
0.009
0.905
0.962
Table S4: Numerical results correspond Fig. 5
Showing the effect of the upper bound of p.
p range
PSNR
std
min
SSIM
std
min
[0-1]
33.50
1.06
28.93
36.33
0.956
0.008
0.920
0.972
[0, 1- 1E-6]
33.93
0.99
29.31
36.57
0.959
0.007
0.924
0.973
[0. 1- [0-4]
33.66
1.07
29.00
36.48
0.958
0.008
0.920
0.974
-
1- IE-2]
32.62
0.99
28.54
35.48
0.938
0.010
0.895
0.960
Table S5: Numerical results correspond Fig...
Showing the effect of the fixed p.
fixed p
PSNR
std
min
max
SSIM
std
min
[1 IE-5, 1 1E-5]
26.12
0.86
25.23
29.52
0.789
0.026
0.748
0.872
IE-4, 1 1E-4]
27.10
0.94
25.93
30.34
0.812
0.031
0.768
0.896
[1 1-
29.95
0.84
28.39
33.19
0.907
0.013
0.865
0.943
[1 1B-2,
30.99
0.86
28.33
34.08
0.926
0.011
0.886
0.954
Table S6: Numerical results correspond to Fig, and SSIM in this case.
5e.. Start filter size of 32 produced better PSNR
Startup filter size
PSNR
std
min
max
SSIM
std
min
max
48
32.17
0.51
30.91
33.47
0.925
0.008
0.902
0.938
40
32.66
0.60
31.07
34.89
0.945
0.007
0.928
0.959
32
32.98
0.58
31.65
35.09
0.942
0.007
0.925
0.956
28
33.04
0.52
31.58
34.64
0.945
0.006
0.929
0.955
24
33.09
0.54
31.69
34.96
0.947
0.006
0.929
0.959
16
32.14
0.71
30.65
34.97
0.940
0.008
0.920
0.961
18

---

Table S7: Results of PSNR and SSIM from different model sizes. d: Model depth. sf: Start
feature size. Zc: Number of z3D) convolutions.
Model size
PSNR
std
min
max
SSIM
min
max
d=3, sf=48, zc=2
32.42
0.90
28.55
35.01
0.943
0.009
0.911
0.963
d=4, sf=32, zc=1
32.85
0.93
29.11
35.29
0.950
0.008
0.921
0.967
d=4, sf=40, zc=2
33.83
1.13
29.08
36.70
0.958
0.008
0.924
0.974
sf=32, zc=2
33.46
1.12
28.90
36.58
0.955
0.009
0,921
0.972
d=4, sf=32, zc=3
33.36
1.11
28.86
36.61
0.955
0.009
0.922
0.972
d=5, sf=32, zc=1
33.11
1.02
29.19
36.54
0.954
0.008
0.925
0.970
d=5, sf=32, zc=2
33.93
0.99
29.31
36.57
0.959
0.007
0.924
0.973
d=6, sf=32, zc=3
32.55
0.96
28.56
35.62
0.947
0,009
0.917
0.967
Table S8: Effect p range on 10-shot inference. The results demonstrate that multi-sher inference improves reconstruction quality. Within limited number of trials, the range within [0.3, 0.7] produced the best results.
p range
Inference type
PSNR
min
max
SSIM
std
min
0.5 x 10
34.00
1.61
28.39
37.61
0.959
0.011
0.904
0.974
[0.5, 0.5]
0.5 x (same split)
34.00
1.62
28.35
37.52
0.959
0.011
0.903
0.974
0.5 x
33.49
1.57
28.31
37.28
0.954
0.011
0.900
0.970
0.7 x 10
34.35
1.72
29.49
38.19
0.961
0.010
0.919
0.975
0.7 10 (median)
34.33
1.72
29.48
38.16
0.961
0.010
0.919
0.974
[0.3, 0.7]
0.7 + 0.3
34.24
1.72
29.49
38.16
0.961
0.010
0.919
0.975
0.7
34.08
1.68
29.22
37.83
0.958
0.010
0.914
0.973
[0.25, 0.75]
0.75x 10
34.34
1.20
29.33
36.96
0.960
0.007
0.927
0.975
[0.2, 0.8]
x
34.09
1.18
29.08
36.71
0.959
0.008
0.921
0.974
19

---

C
Appendix Supplementary figures
Previous block
Next block
Network Options
Conv nx3x3
GeLU
(Typical)
Group Norm
Group Norm
Start feature size (32)
Depth (5)
Conv nx3x3
Depth scale factor (2)
3D conv stages (2)
Group norm (8)
Conv nx3x3
Residual (add)
Group Norm
Down conv block
Up conv block
GeLU
GeLU
Conv 1x3x3
Residual (add)
Group Norm
Max Pool 2x2
Conv nx3x3
Pixel Shuffle
2D blocks
Conv nx3x3
Group Norm
Group Norm
GeLU
Conv nx3x3
Conv 3x3x3
Max Pool 2x2x2
Skip (Concat)
Pixel Shuffle 2x2x2
Max Pool 2m
Conv nx3x3
3D blocks
Pixel Shuffle 2m
Next block
Additional layers
Previous block
Figure S1: The network diagram showing the primary ResUNet components used in this work.
The down convolution block has 3 convolutions, with residual connection between the first two layers. The up-convolution block has similar structure with the up-sampling performed using pixel shuffle layer followed by a convolutional layer.
20

---

LABO
LAB
CAB
BARR
LAB
LAB
LABO
LAB
LAB
CAB
FABS
BABS
BABS
BAB
LAB
LAB
CAB
LAB
LABO
LABE
LAB
LAB
Figure S2: Example key frames. Top: The ground truth. Middle: The simulated 1-bit quanta data. Bottom: our reconstruction.
21

---

Ground truth
Simulated data
Supervised
N2N' independent inputs
N2V
LAV P
LABB
IAB
LAB
THIOE
TENOR
Zaoo view)
20 mask
2D 0.993998)
3D no mask
[0, 0.999999]
10-shot (0.3, 0.7]
LAB
BAB
LABB
LA
LAB
THOO
THOME
THIRE
Split N2N wio mask
Split N2N w/ mask
Inter frame N2N
p=0.5
p=0.9999
LAB
CAB
LAB
LABB
LAB
SNAFR
(0.99, 0.99]
P (0.999, 0.9999)
p [0.9999, 0.9989]
[0.99999, 0.99999]
[0.5, 0.999]
LABE
LAB
LAB
LAB
LAB
ANACKS
KNACK
depth
depth
Higher
No Group
LA
TA Q
LAB
LAV
LAB
SKACKSS
THO
THOPE
THOI
Start filter 16
Start filter 24
Start filter
Start filter
Start filter
LAVB P
SAB
Q
NAAB
CAVE
TAB
THORK
THOR
THOR
THOR
THOR
[0.999, 0,999999]
p=.099 9,09,000 0.999999]
[0.6, 0.999999]
0.999999]
0.999999]
LAB
MAB
CAB
M
THORR
THORI
THORE
THORE
Figure S3: Visualization of selected training experiments under conditions indicated the title. The Rome numbers correspond to experiments in Fig. 5 The hyperparameters and the PSNR/SSIM of most of these images and additional experiments are provided in the HP.CSV the supplemental materials for reference. (Zoom in for detail)
22

---

GAP p-0
Both fixed pm0.01
Both fixed p=1
Both fixed p= 10
Std 0.011
Shd 0. 0.121
Shd 0.031
Sid 0.009
N-134217789
N:134217729
N-134217766
Meen 168
Maxi 1.W
Veas 883
Max 8.35
Deat 228
Nax 1:300
Hean 1-102
Mix 1.40 Will 013199
deDov 02104
Mode: 1-55
Wilh 0.0148
Willi 0.39341
Both random p=0.01
Random input p=0.01
Random Target p=0.01
Random Target p=0.0001
Std - 0.0013
Std - 0.0012
Stdd - 0.0013
Stid - 0.0045
17122277722
Ma:1:
N-1309-7700
315
N: 1212227722
1-09
13420/
1.171
Meat: 128
Mar 26/00th 3 2202377
Mesk 872
1184
Deat 227
1:222
Uamn 242
1.34
Dix 200
S9/W.. 0.002200
Dixe 200
C6: 0.00331
Dix 226
Ce W1-P 0.00071
Figure $4: Results from fixed and randomized training pairs Poisson random noise. ideal training method and model should produce pure blank image with low standard deviation. Top Row: The input is randomly generated noise sampled from white image using a constant Poisson rate p. In GAP, a large fixed thinning parameter (0.99) is used to split only few photons to the target, resulting mostly blank images with few random photons. In all cases on the top row, the granular pattern obvious in the image. Bottom Row: The problem is less prominent when either both input and target are random. Reducing the number photons the target by 100 times increases the standard deviation but is not significant
in the cases above.
23

---

5
4
3
2
12
Example real SPAD data and reconstruction: Resolution target rotating drill bit. This scene shows a USAF 1951 resolution target spinning in dark room. The object is rotating. Top right: the raw data, Middle: 50 frames of raw data, skipping 50 frames. Bottom: the corresponding reconstruction.
Figure 55:
24

---

Figure S6: Example real SPAD data and reconstruction: Image rotating the CPU fan. This scene shown in Fig I showing a sticker of a mandrill image rotating on a CPU fan acquired in dark room. The scene is in low light. The camera is static. Part of the scene is rotating. The speed of the CPU fan is 1500 RPM. The light intensity of the scene is in the order of 1-10 lux. This is scene dark room with the pointing towards the wall and some small LEDs on the motherboard. is worth noting that even at this low light condition, the imaging not Pooton-limited because the hardware is highly sensitive. We had to reduce the aperture of the camera lenãño ensure the sensor was not over-saturated. Top left: The data. Top right: the reconstruction. Middle: 50 frames of data, skipping 50 frames. Bottom:
light turned off, with the only light source the computer monitor
the corresponding reconstruction.

---

Figure S7: Example real SPAD data and reconstruction: Plasma ball. This scene shows a plasma ball, which produces plasma a vacuum sphere. Plasma is triggered by high voltage generated through buck converter circuit, with measured frequency of 28 kHz. The plasma is released in a pulsed fashion at this frequency, following similar paths between adjacent events. The camera is triggered similar frequency Adjacent frames indicate the flow of the plasma, capturing this photo-sparse scene with extremely high-speed events. Top left: The raw data. Top right: the reconstruction. Middle: 50 frames of raw data, skipping 500 frames, Bottom: the corresponging reconstruction.
capture each image, representing a snapshot of the event.

---

Figure S8: Example real SPAD data and reconstruction: Sonicator and bubbles. This scene shows a piezo transducer sending a 3 MHz sound wave through detergent liquid, creating bubbles, mist, and water droplets. It is highly dynamic, complex, and chaotic scene with random high-speed movement. The high background signals from the mist pose challenge. Despite this, our method demonstrated reasonable performance. Top left: The data. Top right: the reconstruction. Middle: 50 frames of raw data, skipping 500 frames. Bottom: the corresponding reconstruction.
27

---

Figure S9: Example real SPAD data and reconstruction: Moving H&F microscope slide. This scene was taken using bright field microscope under 20x. The sample is H&E staining histology slide of mouse lung tissue. The sample is dynamically moved a square pattern on a motorized stage. Most pixels are saturated in this case. Top left: The raw data. Top right: the reconstruction. Middle: 50 frames of raw data, skipping 500 frames. Bottom: the corresponding reconstruction.
28

---

Figure S10: Example real SPAD data and reconstruction: dynamic focusing on cells. This scene was captured using a wide-field fluorescence microscope at 100x magnification, showing the nuclei of cultured cells. The focus of the microscope was dynamically shifted with electronically tunable lens. A fixed pattern noise is present in this dataset, potentially due to hardware issues. While the pattern is not obvious in single binary frame, it becomes clearly visible after reconstruction at single-pixel width. This is a relatively low-light condition with dynamic motion. Top left: The raw data. Top right: the reconstruction. Middle: 50 frames of raw data, skipping 500 frames. Bottom: the corresponding reconstruction.
29

---

PHIA
Figure S11: Example real SPAD data and reconstruction: CPU fan with camera motion. This scene was taken in a dark room, showing a running CPU fan. The exposure was 6 ns for each image. The camera moving up and down. Individual fan blades and the characters resolved after reconstruction. This is a very low-light scene with dynamic motion on both the object and the camera. Top left: The raw data. Top right: the reconstruction. Middle: 50 frames of raw data, skipping 500 frames. Bottom: the corresponding reconstruction.
the fan were both
30

---

NEII12I VIAAA
Model applied to the data used in training
Model applied to data from similar scene but NOT used in training
1811225 PVIT
Figure S12: The result from a model applied to data not used in the training. The model used in SF11 was applied to data taken in the same scene. The data used in inference was not used in the training. No qualitative difference was noticed.
31

---

Figure S13: Selected crops of selected keyframes from our reconstructions in Fig. the consistent quality and the movement the hand.
Showing
32

---

High
Figure $14: Example results from photon-sparse confocal 3D volume. The data acquired using a Leica SP8 confocal microscope in photon counting mode, showing the floofuuureeeecce of mouse brain tissue. Input data, high SNR reference, the result from the original GAP open source code, and our results are shown on the top row for qualitative comparison. The high SNR reference is scanned separately in different imaging session. There are small differences between the images due to repositioning. Thus, no numerical comparison is available. Raw data and our reconstruction of key z-slices through the volume are shown below.
33

---

image
Metadata:
Real confocal
Cell nuclel mouse heart 512x5 12x112 (KY2) photons
Resampled image photons per
PSNR:
24.0/ 24.0/0.499.
PSNR
23.170.495
0.0306 photons per pixel
SSIM:
0.57 /0,023
PSNR 09.9/0.485
0.00777 photons per
0.031
Figure S15: Example results from simulated photon-sparse confocal 3D volume. The data was acquired using Leica SP8 confocal microscope in normal mode, showing the DAPI-stained cell nuclei in mouse heart tissue. The data was then sampled into different levels of photon counts and truncated at to simulate the (SAD data. PSNR and SSIM are computed for different input photon levels.
34

---

1.0
wyphuppphha My Nnw n mom /
-0.5
5000
10000
15000
20000
25000
30000
35000
Figure S16: Example learning curve from N2N training. The validation loss increases since very early stage of the training. The X-axis indicates trainning steps.
35

---

D
Appendix acronyms
SPAD
single-photon avalanche diode
Q
quanta image sensor
CV
computer vision
CNN
convolution neural networks
MSL
mean squared error
MMSE
minimal mean squared error
GAP
Generative Accumulation of Photons
N2N
Noise2Noise
N2V
Voise2Void
N2S
Noise2Self
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction both state the contributions. The assumptions clarified the introduction/Cllimss section. The limitations are discussed in the conclusion section. Our theoretical and experimental results support our claims. The specific task and its generalization are discussed in the introduction/proolen statements section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Most of the Conclusion section discusses limitations. The more general limitations of self-supervised denoising, particularly in 1-1b 1-bit data, are covered thoroughly throughout the manuscript. We made clear that the method only applies to Poisson noise but not other types of noises (e.g., fixed pattern noise), which could be better revealed by the method. Factors that influence the performance of the approach and the specific task are key elements of this manuscript. The computational efficiency of the proposed algorithms is covered in the Experiments section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
37

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our assumptions are discussed in the Introducction/rooblem statements. Our proofs are covered in the Theories and Methods section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
The code is included in the supplemental materials. The raw data files are too large be included. They will be released after the review period. Simulations and public data from QBP can be used for evaluation.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
38

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Anonymized code is provided in the supplemental materials. Full code and data will be released after the review period.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guse wwwec//ggiss//eeeeeeeeee Stto///oueieiioooooooooooooooeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All training details and metadata, including all hyperparameters, are provided in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
39

---

Answer: [Yes]
Justification:
We included the mean, standard deviation, and range for PSNR/SSIM.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We discussed computer resources in the manuscript. The method does not require advanced computing resources beyond consumer GPUs. The range of training time is in the order of hours.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't itdn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpps://neurips..... wwtpeeurrrrssccpculiieeeeee httpa/rpppp.pcpbbbboccccccccacaa
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics. Our research in the paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
40

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work has no direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licenses cited in the code.
Guidelines:
41

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: New assets
primary focus of this paper, although data and documen- tation will be released for public access after the anonymization period.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
42

---

Justification:
No crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
43


Reviews:
[{"review_id": "oRhiZXN3zO", "reviewer": "Reviewer_abBE", "summary": "This paper introduces a method to reconstruct/denoise high-resolution high-frame-rate videos captured by 1-bit quanta imaging sensors (e.g., SPAD arrays) without heavy spatio-temporal binning. The paper also captures and will release a new SPAD dataset.\n\nThe proposed method is loosely based on Generative Accumulation of Photons [9], which trains a CNN to reconstruct images in a self-supervised fashion by reconstructing the pdf of the photon arrivals. This paper makes several significant modifications to [9] so that it can work with quantum image sensors: First, in order to account for 1-bit sensors, the proposed method models the photon arrival pdf as a Bournoulli (rather than Poisson) distribution to account for the binary nature of the measurements. Second, in order to improve reconstruction accuracy it incorporates temporal information regularization.\n\nThe proposed method is tested on experimentally captured data and produces visually compelling results, including in the presence of non-rigid motion (e.g., guitar string).", "questions": "Does a Poisson signal model work in the low-flux regime (where it would be rare for more than one photon to arrive in a pixel)?\n\nCan one bin a large # of frames in overlapping windows to ensure Poisson statistics while still recovering high temporal resolution?\n\nHow important is self-supervised training? Is the simulated noise model accurate enough that one could train an effective QIS denoiser using only simulated data?", "limitations": "Well discussed", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "A SPAD dataset would be extremely valuable to the research community.\n\nProposed method is novel and works well.\n\nPaper is well written.\n\nDenoising quanta image sensor data is an important problem with many scientific imaging applications", "weaknesses": "Proposed technique largely follows from GAP. Unclear if Poisson distribution assumption is invalid in the photon starved regime, i.e., would GAP work for dimmer scenes?\n\nFigure/table captions could use more info. E.g., state whether Table 1 results are experimental or simulated.\n\nUnderperforms supervised methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments and we want to answer the reviewer’s questions one by one. For conciseness, we grouped similar and relevant questions and comments.\n\n1. **Is the Poisson distribution valid in the photon-starved regime (fewer than one photon per pixel on average)? How does GAP behave in this regime? How does the proposed method differ?**\\\nIt depends on the physics of the photon-counting process. If the sensor produces true Poisson photon counts, GAP should be applicable even for very low photon counts. Pixels can have more than one photon even if the Poisson rate is low. However, single photon detectors (like SPAD) are binary, leading to a truncated Poisson distribution / Bernoulli distribution (see L107). In this case, the Poisson distribution assumption becomes invalid at the pixel level leading to the artifacts shown in the main paper Page 7 Table 1 (Ours / No Mask, which is equivalent to a 3D version of GAP), where every pixel that has a photon in the input image leads to a darker result in the corresponding pixel in the output image (also see the figure in the rebuttal PDF). This is a consequence of correlations introduced by the splitting operation. Reducing the amount of light (and photons) will not remedy this problem, and the output images will still exhibit the same artifacts, albeit for fewer pixels, as there are fewer photons in the input. Addressing this issue is a major contribution of this manuscript. Our masking scheme can reliably avoid these artifacts even in low-light conditions.\n\n2. **Can one bin a large # of frames in overlapping windows to ensure Poisson statistics while still recovering high temporal resolution?**\\\nBinning a large number of binary frames will make the data behave more Poisson, but the temporal resolution cannot be recovered. There is a trade-off. As soon as we sum multiple frames, we can no longer distinguish which frame each photon is coming from. The temporal information is lost in the process. In our experiments, each frame has a 6 ns exposure time, but the frame rate is 100k fps (10 us/frame). There is a relatively long period between frames when the camera is not collecting light. If there is fast movement, the movement between frames will be integrated after binning, and binned frames will represent a much longer time scale, causing motion blur. \n\n3. **Comparison to supervised methods and training using simulated data.**\\\nWe acknowledge that the supervised method is the upper limit of the network performance under ideal conditions. However, supervised training requires high-quality ground truth data that covers the distribution of the noisy measurements, leading to limited practical applicability. In many cases, we cannot acquire such a large amount of ground truth data, cannot acquire ground truth data at all, or cannot ensure the distribution of the measurements matches the distribution of the training data. The mismatch between the training data and the measurements can cause erroneous predictions.\\\nIf noise-free ground truth data is available, we can use simulated noise to produce the required training data. We believe the truncated Poisson noise model is sufficiently accurate to generate training data. In many applications, ground truth data is unavailable. In some cases, it may be possible to simulate the ground truth data, given the knowledge of the underlying ground truth signal distribution. This knowledge is often not available in non-natural imaging applications (e.g. scientific imaging).\\\n*The ability to restore noisy data without ground truth data or prior knowledge about the signal distribution is a necessity for many scientific applications. Using a self-supervised method is important in this situation.*\n\n4. **Figure/table captions could use more info. E.g., state whether Table 1 results are experimental or simulated.**\\\nWe will include more information in the Figure/Table captions in the revision. Table 1 results are from simulated data to have a ground truth for evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OR9V8CceYV", "reviewer": "Reviewer_Z4Sm", "summary": "The paper proposes to extend the Generative Accumulation of Photons that was proposed for Poisson noise to 1 bit Quanta image sensors.", "questions": "Wouldnt randomly assigning each photon event to one of the two bins lead to loss of temporal information to some extent in both the input and target?\nThe supervised method seems to be performing better than the proposed method on static simulated scenes? Would it achieve similar fps as the proposed method on the video data too?", "limitations": "Yes. The authors have made the limitations of the proposed method clear.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is novel, and mathematically interesting. The authors have put in a lot of effort to fit GAP for the problem of 1-bit QIS reconstruction.", "weaknesses": "The proposed method does not seem to be performing better than a supervised method. It is not clear what is the supervised method that was used in the comparisons. \n\nIt is not clear what the unique advantages of this method are compared to standard techniques like supervised learning or data simulation based training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments and we want to answer the reviewer’s questions one by one:\n\n1. **The proposed method does not seem to be performing better than a supervised method. It is not clear what is the supervised method that was used in the comparisons.**\\\nWe apologize for the confusion caused by the insufficiently detailed description of the supervised method. The purpose of the experiment was to characterize the theoretical upper limit of the network performance if clean ground truth was available. The supervised method was trained using pairs of clean ground truth data and simulated noisy data. The supervised method used the identical network architecture as our self-supervised method and was trained with cross-entropy loss. It ensured that the selected network architectures and hyperparameters could effectively represent the ground truth distribution. We will include that in the revised manuscript. \n\n2. **It is not clear what the unique advantages of this method are compared to standard techniques like supervised learning or data simulation-based training.**\\\nSupervised training has many limitations in practice, especially in scientific experiments (e.g., microscopy, ultra-high-speed imaging, etc.). Supervised training requires ground truth data that covers the distribution of the noisy measurements. Unlike natural image processing, very often we cannot acquire such a large amount of ground truth data, cannot acquire ground truth data at all, or cannot ensure the distribution of the measurements matches the distribution of the training data. The mismatch between the training data and the measurements can cause erroneous predictions. Our self-supervised method can restore the image using solely the information from the noisy dataset itself, making it appealing for scientific data processing.\\\nRegarding data simulation-based training, if noise-free ground truth data is available, we can use simulated noise to produce the required training data. We believe the truncated Poisson noise model is sufficiently accurate for training the model. It may be possible to simulate the ground truth data given the knowledge of the underlying ground truth signal distribution. However, this knowledge is often unavailable in non-natural imaging applications. \\\n*In summary, the self-supervised method does not require ground truth data or prior knowledge about the signal distribution, making it an effective solution for many scientific applications.*\n\n3. **Wouldn't randomly assigning each photon event to one of the two bins lead to loss of temporal information to some extent in both the input and target?**\\\nEach photon event has its own spatial-temporal index (pixel location and frame number). Assigning a photon event to one of the two bins (input and target) does not affect its frame number and pixel location, so it does not lead to loss of temporal information.\n\n4. **The supervised method seems to be performing better than the proposed method on static simulated scenes? Would it achieve similar fps as the proposed method on the video data too?**\\\nWe apologize for the confusion. The simulated scenes presented in the work are also dynamic video data (see L204, Fig. S2). There are 3990 frames in the ground truth data with motion, recorded at 240 fps. See the response #1 above regarding the comparison of performance. Regarding the inference performance, the supervised method has the same network architecture as our method and outputs the same temporal resolution. Therefore the fps of inference is identical. We are happy to provide further clarification in the discussion phase of the rebuttal process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5detdfc1z1", "reviewer": "Reviewer_iaWM", "summary": "This paper presents a self-supervised method for reconstructing high-quality video from sparse binary quanta image data produced by single-photon avalanche diode (SPAD) arrays. The authors propose a novel masking strategy to handle the binary nature of the data and extend their method to 3D to leverage spatiotemporal information. They evaluate their approach on both simulated and real SPAD data, demonstrating improved reconstruction quality and throughput compared to existing methods like Quanta Burst Photography (QBP). The paper also introduces a new dataset of real SPAD high-speed videos under various challenging imaging conditions.", "questions": "1.\tHow do you propose to select the photon splitting variable p in practice when the signal level is unknown before capture? Can you provide guidelines for selecting an optimal value?\n2.\tCan you provide a more comprehensive comparison with other existing quanta video reconstruction methods beyond QBP, including quantitative results?\n3.\tCan you provide a detailed analysis of the computational complexity and how it scales with dataset size?\n4.\tCould you elaborate on the runtime and memory requirements of your method compared to existing approaches?\n5.\tCan you provide more extensive ablation studies on the impact of various components of your method, such as network architecture choices and hyperparameters?\n6.\tCan you provide more details on the composition and size of the training dataset used? How many videos were used in both simulated and real tests?\n7.\tHow does the method perform on extremely low photon count data, and what is the lower limit of photon count where the method remains effective?\n8.\tHow does the method perform on different types of scenes or motion patterns?", "limitations": "The authors have adequately addressed limitations, discussing the scope of applicability to Poisson noise and computational considerations. They have also provided a thorough discussion of assumptions and potential limitations in the conclusion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper addresses an important problem in computational imaging, proposing a novel self-supervised approach for reconstructing high-quality video from sparse binary quanta data.\n2.\tThe authors' masking strategy to handle binary data is innovative and appears effective based on the presented results.\n3.\tThe introduction of a new real SPAD high-speed video dataset is valuable for the research community.", "weaknesses": "The paper fails to convincingly demonstrate the effectiveness and advantages of the proposed method over existing approaches. The theoretical foundations are not well-developed, and the experimental results are not sufficiently rigorous or comprehensive to support the claims made. The writing lacks clarity in many sections, making it difficult to fully understand the proposed method and its implications.\n\n1.\tThe discussion on selecting the photon splitting variable p is confusing and lacks practical considerations. The paper doesn't adequately address how to ensure correct signal reconstruction when the signal level is unknown before capture.\n2.\tMany crucial implementation details are absent:\n-\tThe composition and size of the training dataset are not specified.\n-\tIt's unclear if the quantitative results for simulated data are computed from only one video (L204).\n-\tThe number of videos in the real test data is not mentioned.\n3.\tThe comparison with existing quanta video reconstruction methods is limited to only QBP (proposed in 2020). There is a lack of quantitative comparisons with other existing methods.\n4.\tThe paper lacks a detailed analysis of the method's runtime and memory requirements compared to existing approaches.\n5.\tMore extensive ablation studies exploring the impact of various components of the method (e.g., network architecture choices, hyperparameters) are needed to provide deeper insights into the method's performance.\n6.\tSeveral technical terms and abbreviations are not properly defined or explained:\n-\tL1: \"SPAD\" is not defined on first use.\n-\tL15: The unit for \"0.06 photons per pixel\" is not specified (e.g., per frame or per second).\n-\tL22: \"QBP\" is not defined on first use.\n-\tL98: The meaning of s_i and i is unclear (intensity or pixel position?).\n-\tL112: The probability of zero photons hitting a pixel being e^{s_i} needs a detailed explanation.\n-\tL197: The term \"10-shot inference\" is not explained.\n7.\tSome experimental details are missing:\n-\tL205: More information on the iPhone 15 slow motion mode (e.g., resolution, exact frame rate) would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the comments. We noticed several factual errors in this review:\n1. **L1: \"SPAD\" is not defined on first use.**\\\nSPAD is defined in L1  on first use.\n2. **L22: \"QBP\" is not defined on first use.**\\\nQBP is defined in L21-22 on first use.\n3. **L205: More information on the iPhone 15 slow motion mode (e.g., resolution, exact frame rate) would be helpful.**\\\nThe frame rate 240 fps is reported in L205. The video is cropped/resampled to 3990x512x512.\n4. **The composition and size of the training dataset are not specified.**\\\nData size is provided in L208. Each dataset is shown and described in Fig. S5-S12. The intended meaning of the term composition is unclear.\n\nSome questions and comments are vague and not specific. We tried our best to address them. We grouped related questions and comments:\n1. **The paper fails to convincingly demonstrate the effectiveness and advantages of the proposed method over existing approaches. The comparison with existing quanta video reconstruction methods is limited to only QBP (proposed in 2020). There is a lack of quantitative comparisons with other existing methods. Can you provide a more comprehensive comparison with other existing quanta video reconstruction methods beyond QBP, including quantitative results?**\\\nTo our knowledge, our self-supervised 3D binary to 3D grayscale task is novel. We are not aware of any existing work that solves this task. We hope the reviewer can specify.\nAlthough similar, QBP is a substantially different task, which constructs a single 2D grayscale image from many binary frames. QBP also did not show any comparable quantitative results. We compared our method to GAP, N2N, and N2V quantitatively. \n2. **The theoretical foundations are not well-developed, and the experimental results are not sufficiently rigorous or comprehensive to support the claims made.**\\\nIt is unclear which claims are not supported. \n3. **The writing lacks clarity in many sections, making it difficult to fully understand the proposed method and its implications.**\\\nIt is unclear which sections and specific parts of the explanation lack clarity. \n4. **The discussion on selecting the photon splitting variable p is confusing and lacks practical considerations. Can you provide guidelines for selecting an optimal value [for p]?**\\\nWe extensively discussed methods for selecting p in L157, L266 and L465, plus Fig. 3, 4, S3, S4, S6, S8, including two best practices. The simple method is using a fixed p (L160). The more robust method is selecting p randomly between 0-1 for each training pair (L166). Both methods are easy to implement in practice. \n5. **Not adequately address how to ensure correct signal reconstruction when the signal level is unknown before capture. How to select the photon splitting variable p in practice when the signal level is unknown before capture?**\\\nOur method is self-supervised. The model is trained on the very data to be denoised at the detected signal level. The selection of p is unrelated to the signal level, as it corresponds to the fraction of photons to split for already-known measurements.\n6. **It's unclear if the quantitative results for simulated data are computed from only one video (L204).**\\\nWe stated the simulated data are computed from L204: a ground truth reference video. \n7. **The number of videos in the real test data is not mentioned. Can you provide more details on the composition and size of the training dataset used? How many videos were used in both simulated and real tests?**\\\nWe reference all 7 real test data in L211. Each test is a video.\n8. **The paper lacks a detailed analysis of the method's runtime and memory requirements compared to existing approaches. Can you provide an analysis of the computational complexity and how it scales with dataset size?**\\\nWe discussed practical computational requirements, optimization, performance, runtime, and VRAM in L185-L198. We compared our runtime to QBP (L294). The computational complexity is more relevant to the architecture rather than the data size. \n9. **More extensive ablation studies exploring the impact of various components of the method (e.g., architecture, hyperparams) are needed to provide deeper insights into the method's performance. Can you provide more extensive ablation studies on the impact of various components of your method?**\\\nIt is unclear which specific ablation studies and hyperparameters are overlooked, as we provided substantial details (Table S1-8, Fig. 4, Fig. S1, 3, supp hp.CSV) relevant to the core concept of the method. Besides, our contribution is not about the model architecture, but the novel task, a theoretical framework for the solution, and its practical implementation.\n10. **L15: The unit for \"0.06 photons per pixel\" is not specified (e.g., per frame or per second).**\\\nWe will clarify this to 0.06 photons per pixel per frame.\n11. **L98: The meaning of s_i and i is unclear (intensity or pixel position?).**\\\nWe will clarify that i indicates the pixel position, and s_i indicates the Poisson rate at i.\n12. **L112: The probability of zero photons hitting a pixel being e^{s_i} needs a detailed explanation.**\\\nWe explained this in A1 L445-463 in detail. We will clarify that in our revision.\n13. **L197: The term \"10-shot inference\" is not explained.**\\\nThe 10-shot inference refers to the inference strategies in L243. This will be clarified in the revision.\n14. **How does the method perform on extremely low photon count data, and what is the lower limit of photon count where the method remains effective?**\\\nWe simulated low photon count data with an average of 0.06 photons/pixel. Some data have even lower photon count (e.g., plasma data Fig. 2, S7, with photon count in the order of 0.001). Unclear what is considered extremely low or effective. \n15. **How does the method perform on different types of scenes or motion patterns?**\\\nResults cover a wide range of scene and motion patterns are provided in Fig. 1, 2, 5, S4-S14.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3pbQ7cEmGS", "reviewer": "Reviewer_VmGM", "summary": "1. The paper introduces a novel self-supervised method for reconstructing high-quality image sequences from sparse binary quanta image data.\n\n2. The paper mainly adapt a self supervised denoising algorithm called GAP. Instead of directly adopting the GAP method, the authors extended it to spatiotemporal structures and proposed a novel masked loss to address the correlation issue between input and target images.\n\n3. Experimental results demonstrate that the proposed method substantially outperforms state-of-the-art techniques such as Quanta Burst Photography (QBP) in both reconstruction quality and throughput.\n\n4. Additionally, the paper presents a new dataset and discusses the potential of the method for generalizing to other spatial event point processes beyond the specific application of quanta image sensors.", "questions": "1. I still have doubts about the role of the masked loss. The authors state that this loss is intended to address the correlation issue between the input and target images. However, after convolution, the information from locations that were originally 1 in the input image has already spread to other pixel locations. Simply masking the original 1 positions may not fully conceal the information from those positions.\n\n2. I am not quite clear about what the network's output is, given that the input is a t*h*w matrix. Is the output also t*h*w? If so, in the supervised loss comparison experiments, are there 32 grayscale images gt? For the N2N comparison experiment, is the supervision using QIS data with a shape of t*h*w? How was the GAP-2D experiment designed—does the network input consist of the first frame of QIS data from t*h*w?\n\n3. Why does N2N lead to a granular pattern? Why can GAP-like method resolve the granular pattern caused by the N2N method?\n\nA good response to the Weaknesses and Questions will improve my initial rating.", "limitations": "There is no discussion of potential negative societal impacts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The strengths of this approach mainly lie in its ability to effectively utilize spatiotemporal information, its novel masked loss function that addresses the correlation between input and target images, and its demonstrated superiority over existing methods like Quanta Burst Photography in terms of reconstruction quality and throughput efficiency.\n\n2. In addition, as a self supervised algorithm, the training of this scheme does not require the construction of large-scale synthetic datasets, making it more flexible and convenient.", "weaknesses": "1. In Equation 1, there is a confusion between x_{in} and x_{inp}. Moreover, the right side of Equation 1 seems to have overlooked x_{tar}.\n\n2. To illustrate the application of QIS in high-speed and low-light scenarios, it would be beneficial for the authors to provide specific data on the rotation speed of the fan  and the light intensity of the scene.\n\n3. The authors used a 10-shot approach to calculate the average results. However, methods similar to GAP require many iterations during inference, and the combination of these two factors could lead to significant computational time. The authors should provide more detailed explanations, such as the number of iterations and the computational workload involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful comments. Here, we answer the reviewer’s questions one by one:\n\n1. **In Equation 1, there is a confusion between x_{in} and x_{inp}. Moreover, the right side of Equation 1 seems to have overlooked x_{tar}.**\\\nThank you for pointing out the mistake. We will correct the typo in our revision. $x_{in}$ should be $x_{inp}$. There should be a $x_{tar}$ before the $\\ln$ in Equation 1.\n\n2. **To illustrate the application of QIS in high-speed and low-light scenarios, it would be beneficial for the authors to provide specific data on the rotation speed of the fan and the light intensity of the scene.**\\\nAgree and we will include the information in the revision. The rotation speed of the fan is 1500 rpm. The light intensity of the scene is in the order of 1-10 lux. This is a scene in a dark room with the room light turned off, with the only light source the computer monitor pointing towards the wall and some small LEDs on the motherboard. It is worth noting that even at this low light condition, the imaging is not photon-limited because the hardware is highly sensitive. We had to reduce the aperture of the camera lens to ensure the sensor was not over-saturated. \n\n3. **The authors used a 10-shot approach to calculate the average results. However, methods similar to GAP require many iterations during inference, and the combination of these two factors could lead to significant computational time. The authors should provide more detailed explanations, such as the number of iterations and the computational workload involved.**\\\nThank you for this comment. We notice the 10-shot approach is not clearly defined in the original manuscript. We will clarify this in L197 when the term is first introduced in the revision.\nTo clarify, the 10-shot approach is unrelated to the iterative sampling in GAP. GAP uses multiple iterations to achieve posterior sampling and function as a generative model. In this work, we only use the network for MMSE denoising, which requires only a single iteration. The 10-shot refers to running the inference 10 times using data Bernoulli-resampled from the raw data (L246). This takes 10 times the inference time but is not a major time-limiting factor. If the data throughput is a concern, we also provide a practical one-shot inference solution (L166, L195, L243).\n\n4. **I still have doubts about the role of the masked loss. The authors state that this loss is intended to address the correlation issue between the input and target images. However, after convolution, the information from locations that were originally 1 in the input image has already spread to other pixel locations. Simply masking the original 1 positions may not fully conceal the information from those positions.**\\\nThe purpose of the masking is not to fully conceal the information from these positions. The only input information to the network is photon positions, so it is necessary for the network to spread this information to other pixels to make predictions about the clean signal. Due to the binary nature of the sensor and the splitting of the truncated Poisson (Bernoulli) distributed data, pixels that have the value 1 in the input will always have the value 0 in the target. Training a network without masking will lead to the network predicting dark pixels at the locations where there is a photon in the input (see figure in the rebuttal PDF). The mask prevents the network from learning this deterministic relationship by zeroing the loss computation for pixel locations that have value 1 in the input. Other (neighboring) locations are not affected by this issue and do not need to be masked. The mask prevents the network from learning from the target pixels that are determined to be 0 because the input is 1.\n\n5. **I am not quite clear about what the network's output is, given that the input is a thw matrix. Is the output also *thw*? If so, in the supervised loss comparison experiments, are there 32 grayscale images gt? For the N2N comparison experiment, is the supervision using QIS data with a shape of *thw*? How was the GAP-2D experiment designed—does the network input consist of the first frame of QIS data from *thw*?**\\\nCorrect. The network's input and output have the same *thw* dimension (3D to 3D). There are 32 grayscale ground truth frames in the supervised loss comparison experiments. The shape of the training data was always the same *thw* for all 3D network experiments, including the N2N comparison experiments. The GAP-2D experiment is designed so that individual 2D frames are used in training and inference. The network input consists of single frames of the QIS data, randomly selected during training (random *t* from *thw*).\n\n6. **Why does N2N lead to a granular pattern? Why can GAP-like method resolve the granular pattern caused by the N2N method?**\\\nWe believe the granular pattern is caused by overfitting due to limited training data. In N2N, the same binary data pairs are repeatedly used for training. In our method, random splitting alleviates this problem by creating a different binary pair each time. We see this as a type of data augmentation that effectively increases the amount of available training data. L266 has some discussion regarding this. L275 and Fig. S4 demonstrate some aspects of the problem with an experiment using random Poisson noise.\n\n7. **There is no discussion of potential negative societal impacts.**\\\nWe will clarify potential negative societal impacts in our revised conclusion. The method indeed can be misused in research (e.g., using inappropriate data, making wrong assumptions about the noise, etc.) and the prediction results can be misinterpreted, leading to incorrect scientific conclusions and potential negative societal impact.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have addressed each reviewer's comments and questions in detail in reviewer-specific rebuttals underneath each review.

A new figure demonstrating the photon splitting process is presented in the attached PDF. The figure also more clearly indicates the artifact resolved by the masked loss. This figure is cited in abBE Response #1 and VmGM Response #4. We plan to add this figure to our revised manuscript.

Meta-review: This paper received four positive leaning reviews --- one 5 (borderline accept), two 6s (weak accepts), and one 8 (strong accept). 

There was broad appreciation for the novelty of the proposed method, quality of presentation, importance of the problem tackled (denoising and reconstruction from quanta sensor data), and the promise of a novel SPAD dataset

There were some concerns raised around the overall effectiveness of the proposed approach wrt supervised methods, and several other technical / exposition points which were reasonably well addressed in the rebuttal. There was considerable discussion post-rebuttal after which an accept consensus was reached. Therefore, an accept decision is recommended. 

The authors are strongly urged to take into account the reviewers' concerns and include all the promised revisions when preparing the final camera-ready version of the paper.
Decision: Accept (poster)