Title: Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Snapshot Compressive Imaging

Deep Image Prior

Inverse Problem
Abstract: Snapshot compressive imaging (SCI) recovers high-dimensional (3D) data cubes from a single 2D measurement, enabling diverse applications like video and hyperspectral imaging to go beyond standard techniques in terms of acquisition speed and efficiency. In this paper, we focus on SCI recovery algorithms that employ untrained neural networks (UNNs), such as deep image prior (DIP), to model source structure. Such UNN-based methods are appealing as they have the potential of avoiding the computationally intensive retraining required for different source models and different measurement scenarios. We first develop a theoretical framework for characterizing the performance of such UNN-based methods. The theoretical framework, on the one hand, enables us to optimize the parameters of data-modulating masks, and on the other hand, provides a fundamental connection between the number of data frames that can be recovered from a single measurement to the parameters of the untrained NN. We also employ the recently proposed bagged-deep-image-prior (bagged-DIP) idea to develop SCI Bagged Deep Video Prior (SCI-BDVP) algorithms that address the common challenges faced by standard UNN solutions. Our experimental results show that in video SCI our proposed solution achieves state-of-the-art among UNN methods, and in the case of noisy measurements, it even outperforms supervised solutions. Code is publicly available at [https://github.com/Computational-Imaging-RU/SCI-BDVP](https://github.com/Computational-Imaging-RU/SCI-BDVP).

Untrained Neural Nets for Snapshot Compressive Imaging: Theory and Algorithms

Abstract

Snapshot compressive imaging (SCI) recovers high-dimensional (3D) data cubes from a single 2D measurement, enabling diverse applications like video and hyperspectral imaging to go beyond standard techniques in terms of acquisition speed and efficiency. In this paper, we focus on SCI recovery algorithms that employ untrained neural networks (UNNs), such as deep image prior (DIP), to model source structure. Such UNN-based methods are appealing as they have the potential of avoiding the computationally intensive retraining required for different source models and different measurement scenarios. We first develop a theoretical framework for characterizing the performance of such UNN-based methods. The theoretical framework, on the one hand, enables us to optimize the parameters of data-modulating masks, and on the other hand, provides a fundamental connection between the number of data frames that can be recovered from a single measurement to the parameters of the untrained NN. We also employ the recently proposed bagged-deep-image-prior (bagged-DIP) idea to develop SCI Bagged Deep Video Prior (SCI-BDVP) algorithms that address the common challenges faced by standard UNN solutions. Our experimental results show that in video SCI our proposed solution achieves state-of-the-art among UNN methods, and in the case of noisy measurements, it even outperforms supervised solutions. Code is publicly available at https://github.com/Computational-Imaging-RU/SCI-BDVP .

1 Introduction

Snapshot Compressive Imaging (SCI) refers to imaging systems that optically encode a threedimensional (3D) data cube into a two-dimensional (2D) image and computationally recover the 3D data cube from the 2D projection. As a novel approach in computational imaging, SCI has attracted significant attention in recent years. Initially proposed for spectral imaging [1], its application has since expanded to various fields, including video recording [2], depth imaging [3], and coherence tomography [4] (Refer to [5] for a comprehensive review).

The key advantage of SCI systems lies in significantly accelerating the data acquisition process. Traditional hyperspectral imaging methods, for example, often encounter bottlenecks due to their reliance on spatial or wavelength scanning, leading to time-consuming operations. In contrast, hyperspectral SCI systems capture measurements across multiple pixels and wavelengths in a single snapshot, effectively bypassing this limitation [6].

The optical encoding process in SCI systems can be mathematically modeled as a linear measurement system, characterized by a sparse and structured sensing matrix, commonly referred to as a 'mask'. Consequently, SCI recovery algorithms aim to reconstruct high-dimensional (HD) 3D data from a

highly underdetermined system of linear equations. A wide range of SCI recovery methods has been proposed in the literature, which can broadly be categorized into:

Classic approaches: These methods model source structure using convex regularization functions and employ convex optimization techniques (e.g., [7, 8, 9, 10]). While robust to measurement and source distribution non-idealities, they are typically limited to simpler structures and challenging to extend to 3D HD data cubes central to SCI applications. DNN-based methods: These approaches use deep neural networks (DNNs) to capture complex source structures, learning from training data. They can be further categorized as: i) End-to-end solutions (e.g., [11, 12, 13, 14, 15, 16]); ii) Iterative plug-and-play solutions (e.g., [17, 18]); iii) Unrolled methods (e.g., [19, 20, 21, 22, 23, 24]). While these methods extend beyond simple structures to model intricate source patterns, they require extensive training data, often struggle with generalization, and are computationally intensive.

An alternative approach to SCI recovery involves using UNNs, such as deep image prior (DIP) [25] or deep decoder [6], to model the source structure. These methods capture complex source structures without requiring any training data. Existing UNN-based SCI solutions either recover the image end-to-end in one shot [26] or employ iterative methods akin to projected gradient descent (PGD) [27]. Despite their advantages, these approaches often exhibit lower performance compared to pre-trained methods and may require additional data processing steps for enhancement.

In this work, we focus on leveraging UNNs to address the SCI problem. We begin by establishing a theoretical framework for analyzing UNN-based methods, providing insights into optimizing the adjustable SCI masks, under both noise-free and noisy measurements. We then explore DIP-based algorithms and introduce SCI-BDVP solutions. Our results demonstrate the robustness of these solutions to measurement noise and their competitive performance across diverse datasets, using a consistent set of parameters.

1.1 Contributions of this Work

Theoretical: Wetheoretically characterize the performance of DIP-based SCI recovery optimization for both noise-free and noisy measurements. Using our theoretical results, we establish an upper bound on the number of frames that can be recovered from a single 2D measurement, as a function of the dimensions of the DIP. Furthermore, we show how the developed theoretical results enable us to optimize the parameters of the masks for both noisy and noise-free cases, enhancing the performance of the recovery process.

Algorithmic: Inspired by the newly proposed bagged-DIP algorithm for the problem of coherent imaging [28], developed to address common shortcomings of DIP-based solutions for inverse problems, we explore the application of bagged-DIP for SCI recovery. We conduct extensive experimental evaluations, demonstrating the following: i) Confirmation of our theoretical results on the optimized masks for both noise-free and noisy measurements. ii) The proposed SCI-BDVP solution robustly achieves state-of-the-art performance among UNN-based solutions in the case of noise-free measurements. iii) In scenarios with noisy measurements, our proposed method achieves state-of-the-art performance among both end-to-end supervised and untrained methods.

1.2 Notations

Vectors are represented by bold characters like x and y . ∥ x ∥ 2 denotes the ℓ 2 norm of x . For X ∈ R n 1 × n 2 , Vec( X ) ∈ R n denotes the vectorized version of X , where n = n 1 n 2 . This vector is created by concatenating the columns of X . Given A , B ∈ R n 1 × n 2 , Y = A ⊙ B denotes the Hadamard product of A and B , such that Y ij = A ij B ij , for all i, j . Sets are represented by Calligraphic letters, like A , B . For a finite set A , |A| denotes the number of elements in A . Throughout the paper, log refers to the logarithm in base 2, while ln denotes the natural logarithm.

2 Related Work

UNNs for SCI. While the majority of SCI recovery algorithms developed for various applications fall under classic optimization-based methods (e.g., [8, 9, 10]) or supervised DNN-based methods [24], in recent years, there has been increasing interest in leveraging UNNs in solving inverse problems. In SCI recovery, this trend has been motivated by the diversity of applications and datasets encountered

in various SCI applications, necessitating the availability of pre-trained denoising networks tailored to different resolutions and noise levels for various datasets. Another challenge with these traditional solutions is their robustness to various problem settings, such as measurement noise. These challenges have spurred a notable interest in developing solutions that harness the ability of DNNs to capture complex source models while not relying on training data.

While deep image priors (DIPs) have been applied to various inverse problems [29, 30, 31], their application to SCI recovery has been limited. The authors in [27] developed an iterative DIP-based solution for hyperspectral SCI. To enhance the performance and address the challenges faced by DIP-based methods in terms of falling into local minimas, they initialize the algorithm by the solutions obtained by GAP-TV [9]. In [26], the authors propose Factorized Deep Video Prior (DVP), which is a DIP-based SCI recovery algorithm for videos, which is based on separating the video into foreground and background and treating them separately. [32] develops a DIP-based solution for compressed ultrafast photography (CUP), where in addition to the normal SCI 2D measurement and additional side information consisting of the integral of all the frames (referred to as the time-unsheared view in [32]) is also collected. The video is reconstructed using an end-to-end approach using the DIP to enforce the source model. In [33], the authors leverage the concept of video snapshot compressive imaging (SCI) reconstruction to develop an algorithm for snapshot temporal compressive microscopy. They propose an iterative algorithm that utilizes UNNs to incorporate the source structure.

In the context of image recovery from underdetermined measurements corrupted by speckle noise, the authors in [28] recently proposed the idea of bagged-DIP, which is based on independently training multiple DIPs operating at different frame sizes and averaging the results. In this paper, we extend the idea to videos and construct a bagged-DVP, which as we show in our experimental results robustly achieves state-of-the-art performance among all UNN-based SCI video recovery methods.

Mask optimization. In various SCI applications, one can design the masks, which are typically binary-valued, and used for modulating the input 3D data cube. This naturally raises the question of optimizing the masks to improve the performance. To address this problem, several empirical works have designed solutions that simultaneously solve the SCI recovery problem and optimize the masks. In [34], the authors design an end-to-end autoencoder network to train the reconstruction and mask simultaneously for video data and find the trained mask has some distribution such as non-zero probability around 0.4 and varies smooth spatially and temporally. Similarly, in [35], deep unfolding style networks are trained to simultaneously reconstruct 3D images and also optimize the binary masks. They show that for the empirically jointly optimized masks have a non-zero probability of around 0 . 4 . The authors in [16] design an end-to-end VIT-based SCI video recovery solution that simultaneously learns the reconstruction signal and the mask. They consider a special type of mask that constrained by their hardware design.

Due to the highly non-convex nature of the described joint optimization problem, empirically-jointlyoptimized solutions are likely to converge to suboptimal results. Furthermore, the optimized solution, inherently dependent on the training data, lacks theoretical guarantees. To address these limitations, [36] employed a compression-based framework to theoretically optimize the binary-valued masks in the case of noiseless measurements and showed that in that case the optimized probability of non-zero entries is always smaller than 0 . 5 . Here, we theoretically characterize the performance of UNNbased SCI recovery methods and show a consistent result in the case of noise-free measurements. Interestingly, as shown in our experiments, for noisy measurements, the optimized probability can be larger than 0.5. We derive novel theoretical results explaining this phenomenon.

3 DIP for SCI inverse problem

3.1 SCI inverse problem

The objective of a SCI system is to reconstruct a three-dimensional (3D) data cube from its twodimensional (2D) compressed measurement. Specifically, let X ∈ R n 1 × n 2 × B represent the target 3D data cube. In an SCI system, X is mapped to a singular measurement frame Y ∈ R n 1 × n 2 . This mapping, particularly as implemented in hyperspectral SCI and video SCI [2], can be modeled as a linear system as follows [2, 37]: Y = ∑ B i =1 C i ⊙ X i + Z . Here, C ∈ R n 1 × n 2 × B represents the sensing kernel (or mask), and Z ∈ R n 1 × n 2 denotes the additive noise. The terms C i = C (: , : , i ) and

X i = X (: , : , i ) ∈ R n 1 × n 2 correspond to the b -th sensing kernel (mask) and the associated signal frame, respectively.

To simplify the mathematical representation of the system, we vectorize each frame as x i = Vec( X i ) ∈ R n with n = n 1 n 2 . Then, we vectorize the data cube X by concatenating the B vectorized frames into a column vector x ∈ R nB as x = [ x ⊤ 1 , . . . , x ⊤ B ] ⊤ . Similarly, we define y = Vec ( Y ) ∈ R n and z = Vec ( u ) ∈ R n . Using these definitions, the measurement process can also be expressed as

The sensing matrix H ∈ R n × nB , is a highly sparse matrix that is formed by the concatenation of B diagonal matrices as

where, for i = 1 , . . . B , D i = diag ( Vec ( C i )) ∈ R n × n . Using this notation, the measurement vector can be written as y = ∑ B i =1 D i x i The goal of a SCI recovery algorithm is to recover the data cube x from undersampled measurements y , while having access to the sensing matrix (or mask) H .

3.2 Theoretical analysis of DIP-based SCI recovery

The Deep Image Prior (DIP) [25] hypothesis provides a framework for understanding the potential of UNNs in capturing the essence of complex source structures without requiring training data. Define Q ⊆ R n as the class of signals of interest (e.g., class of video signals consisting of B frames.). Also, let g θ : R p → R n represent a UNN parameterized by θ ∈ R k . Informally, DIP hypothesis states that any signal in Q can be presented as the output of the DIP parameterized by parameters θ ∈ R k . This can be represented more formally as follows.

DIP hypothesis: Assume that u ∈ R p is sampled i.i.d. from a uniform distribution U (0 , 1) . For any x ∈ Q , the DIP hypothesis states that for any x ∈ R k , there exists θ ∈ [0 , 1] k , such that ∥ g θ ( u ) -x ∥ 2 ≤ δ , almost surely.

This hypothesis underscores the capability of UNNs to function as powerful priors, capturing intricate data structures inherent in natural images and other complex datasets, thereby bridging the gap between classical analytic methods and modern machine learning techniques.

Given SCI measurements y = Hx + z , as described in (1) with H defined in (2), a DIP represented by g θ : R p → R n can be used to recover x from measurements y as follows: Step 1) Randomly sample u (independent of y and H ), as required by the DIP. Step 2) Solve the DIP-SCI optimization:

Before describing our proposed approach to solving DIP-SCI optimization in Section 4, we theoretically characterize the performance of (3), under noise-free and noisy measurements and use our theoretical results to i) bound the number of frames that can be recovered from a single 2D measurement, and ii) optimize the parameters of the mask H that is used for modulating the data.

3.2.1 Noise-free measurements

The following theorem characterizes the performance of (3) in case where the measurements are noise-free and connects its performance ( ∥ x -ˆ x ∥ 2 ) to the ambient dimension n , number of frames B , number of parameters of the DIP k , the distortion δ and the Lipschitz coefficient L .

Theorem 3.1. Let x ∈ Q , where Q denotes a compact subset of R n , such that ∥ x ∥ ∞ ≤ ρ 2 , for all x ∈ Q . Assume that g θ ( u ) : [0 , 1] N → R nB is L -Lipschitz as a function of θ . Let y = Hx , where H = [ D 1 , . . . , D B ] , where D i = diag( D i, 1 , . . . , D i,n ) , i = 1 , . . . , B , are independently generated with D i, 1 , . . . , D i,n i.i.d. Bern( p ) . Given randomly generated u , let ˆ x denote the solution of (3) . Then, if min c : c = g θ ( u ) ,θ ∈ [0 , 1] k 1 nB ∥ x -c ∥ 2 ≤ δ , we have

with a probability larger than 1 -2 -0 . 5 k log log n +1 .

The bound in (4) consists of multiple terms. The first term, i.e., √ 1 + Bp 1 -p δ , accounts for the effect of the DIP representation error. For instance if x is directly selected from the output space of DIP, then δ = 0 . The goal of the following two corollaries to shed light on the interplay of the three terms in (4) and highlight their implications on the performance of DIP-SCI optimization. First, Corollary 3.2 characterizes an upper bound on the number of frames B that are to be recovered from a single 2D measurement.

Corollary 3.2. Consider the same setup as in Theorem 3.1. If

then 1 √ n ∥ x -ˆ x ∥ 2 ≤ √ 1 + Bp 1 -p δ + c n √ p (1 -p ) , where c n = O (1 / (log n ) 1 4 ) does not depend on p .

Next, Corollary 3.3 states that in the case where the measurements are not corrupted by noise, the value of p , the probability of a mask entry being non-zero, that minimizes the upper bound in (4) is always less than 0 . 5 . This is consistent with the results established i) empirically in the literature [16] and ii) theoretically in [36] using a compression-based framework.

Corollary 3.3. Consider the same setup as in Theorem 3.1. The upper bound in (4) is minimized at p ∗ ∈ (0 , 0 . 5) .

3.2.2 Noisy measurements

In many practical SCI applications, the measurements are corrupted by additive noise. This raises the following natural question: How does the inclusion of noise in the model affects the optimized mask parameters? To address this question, we develop two theoretical results: Theorem 3.4 characterizing the reconstruction error ∥ x -ˆ x ∥ 2 and Theorem 3.5 bounding the error in estimating the mean of the input frames ¯ x = 1 B ∑ B i =1 x i . As we explain later, the combination of these two results provide a theoretical understanding on the performance of SCI recovery methods in the presence of noise and the corresponding optimized masks.

Theorem 3.4. Consider the same setup as in Theorem 3.1. For x ∈ Q , let y = ∑ B i =1 D i x i + z , where z ∈ R n denotes the additive noise and z ∼ N ( 0 , σ 2 z I n ) , for some σ z ≥ 0 . Let ˆ x denote the solution of DIP-SCI optimization (3) . If B satisfies the bound in (5) , then

with a probability larger than 1 -(2 -0 . 5 k log log n +3 + e -0 . 3 n ) . Here, α n = O ( 1 √ log n ) , β n = o ( 1 (log n ) 1 4 ) and γ n = o ( 1 log n ) do not depend on σ z and p .

Theorem 3.5. Consider the same setup as in Theorem 3.4. Assuming that B satisfies the bound in (5) , then with probability larger than 1 -(2 -0 . 5 k log log n +3 +e -0 . 3 n ) ,

where υ n = O ((log n ) -1 8 ) and does not depend on p .

To shed light on the implications of these two theorems, the following corollary characterizes the value of p optimizing each bound.

Corollary 3.6. Consider the same setting as Theorem 3.4. The upper bound in Theorem 3.4 is always optimized at p ∗ < 0 . 5 . On the other hand, the upper bound in Theorem 3.5 is a decreasing function of p and is minimized at p ∗ = 1 .

Let ¯ x B = [¯ x ⊤ , . . . , ¯ x ⊤ ] ⊤ ∈ R nB , i.e., the reconstruction signal derived by repeating the average frame ¯ x = 1 B ∑ B i =1 x i . Then, using the triangle inequality, we have

0.2

0.4

0.6

0.8

25

30

35

40

0.2

0.4

0.6

0.8

25

30

35

0.2

0.4

0.6

0.8

20.0

22.5

25.0

27.5

30.0

32.5

Aerial

0.2

0.4

0.6

0.8

20

22

24

26

28

30

Crash

0.2

0.4

0.6

0.8

20

25

30

Kobe

0.2

0.4

0.6

0.8

17.5

20.0

22.5

25.0

27.5

Traffic

Figure 1 shows ∥ x -ˆ x ∥ 2 , ∥ x -¯ x B ∥ 2 , and ∥ ¯ x B -ˆ x ∥ 2 , for different video test samples. Here are our key observations: 1) ∥ ¯ x B -ˆ x ∥ 2 is an increasing function of p , which is consistent with Corollary 3.6. 2) The optimal value of p ∗ that minimizes ∥ x -ˆ x ∥ 2 , is an increasing function of σ z , for all test videos. 2) In cases where the difference between ¯ x B and x is relatively large, e.g. Traffic, the optimized p ∗ stays smaller than 0 . 5 , even for large values of σ z , as predicted by Theorem 3.4. 3) On the other hand, in cases where ¯ x B provides a high-fidelity representation of x and ∥ ¯ x B -x ∥ 2 is relatively small (e.g., Drop), for large values of noise power, the optimal value of p ∗ can move beyond 0 . 5 , as predicted by Theorem 3.5. In other words, in such cases, the algorithm moves toward estimating the mean of the frames, which is a good representation of the actual data frame.

4 SCI-BDVP: Bagged-DVP for video SCI

Recall the DIP-SCI optimization described in (3), i.e., ˆ x = arg min c ∥ y -Hc ∥ 2 , where c = g θ ( u ) , θ ∈ [0 , 1] k and u generated independently and randomly according to a pre-specified distribution. To solve this optimization, one straightforward approach is to solve min θ f ( θ ) , with f ( θ ) = ∥ y -H g θ ( u ) ∥ 2 2 , by directly applying gradient descent to the differentiable function f ( θ ) . However, given the highly non-linearity and non-convexity of f ( θ ) , this approach is prone to readily getting trapped into local a minima and achieving considerably sub-optimal performance. Generally, a better approach to is to write the DIP-SCI optimization as ˆ x = arg min c ∈C ( u ) ∥ y -Hc ∥ 2 2 , where C ( u ) ≜ { c = g θ ( u ) : θ ∈ [0 , 1] k } . This alternative presentation of the problems leads to minimizing a convex cost function over a non-convex set. A classic approach to solve this optimization is projected gradient descent (PGD), which while in general is not guaranteed to converge to the global minima is more apt to recover a solution in the vicinity of the desired signal.

Remark 4.1. Theoretical feasibility of SCI recovery was first established in [38] using a compressionbased framework for modeling source structure. There, the authors considered ˆ x = arg min c ∈C ∥ y -Hc ∥ 2 2 , where C denotes a discrete set of the codewords of a compression code. They theoretically proved that in that case, despite the non-convexity of the problem, PGD is able to converge to the vicinity of the desired signal.

The PGD applied to ˆ x = arg min c ∈C ( u ) ∥ y -Hc ∥ 2 2 proceeds as follows: Start form an initialization point x 0 . For t = 1 , 2 , . . . , T , perform the following two steps i) Gradient descent: x G t +1 = x t + µ H ⊤ ( y -Hx t ) , and ii) Projection: x t +1 = arg min c ∈C ( u ) ∥ c -s t +1 ∥ 2 , or

To solve the non-convex optimization required at the projection step, one can again employ gradient descent. However, in addition to the non-convexity of the cost function, another common known issue with projection into the domain of a DIP is overfitting [25, 6, 39, 40]. Moreover, in PGD, ideally one needs to set the resolution of the projection step adaptively, such that during the initial steps the DIP has a coarser resolution and as it proceeds it becomes finer and finer. This poses the following question: Which DIP structure should one use to optimize the final performance?

To address this question, the authors in [28] have proposed, bagged-DIP, which consists of employing multiple DIP with different structures in parallel, for the DIP projection step and averaging the outputs. They show that this approach provides a robust projection module which consistently outperforms the performance achievable by each individual DIP network, and also provides, at least partially, the flexibility and adaptability required by PGD.

k

SCI-BDVP

…

g

1,

i

θ

(

u

1,

i

)

u

1

∼

U

(0,1)

DIP

Block

Output

Block

Estimate 1

x

G

t

…

u

k

∼

U

(0,1)

DD

Block

HW

/

h

k

wk

Estimate k

…

u

K

∼

U

(0,1)

Estimate K

.

.

.

.

.

.

MSE

θ

←argmin

θ

∥

x

G

t

,

i

-

,

i

θ

(

u

k

,

i

)∥

2

2

+

ω

∥

y

k

,

i

-

H

k

,

i

,

i

θ

(

u

k

,

i

)∥

2

2

x

P

t

x

P

t

,1

x

P

t

,

k

x

P

t

,

K

DIP

Block

y

3D data cube

2D measurement

DIP

Block

DD

Block

HW

/

h

k

wk

DIP

Block

DD

Block

HW

/

h

k

wk

Output

Block

DD

Block

HW

/

h

DIP

Block

DD

Block

HW

/

h

DIP

Block

DD

Block

HW

/

h

DIP

Block

x

t

3D data cube

y

K

,

i

,

H

K

,

i

y

k

,

i

,

H

k

,

i

,

i

θ

(

u

k

,

i

)

g K

,

i

θ

(

u

K

,

i

)

h

1

=

H

,

w

1

=

W

h

1

=

H

,

w

1

=

W

h

1

=

H

,

w

1

=

W

3D binary mask

H

,

i

θ

(

u

k

,

i

)

y

1,

i

,

H

1,

i

g K

,

i

θ

(

u

K

,

i

)

x

P

t

=

1

K

K

∑

x

P

t

,

k

Average

x

t

=

α

x

G

t

+(1 -

α

)

x

P

t

Fusion

…

3D patches in estimate k

…

3D patches in estimate K

=1

Bagged-DIP, essentially employs bagging idea to mitigate overfitting. As the DIP projection iterations proceeds (within each step of PGD), overfitting tends to occur after a certain threshold. However, due to the variance reduction facilitated by bagging, the bagged estimate can demonstrate less overfitting. In other words, the bagged estimate is less sensitive to the stopping time of the DIP training. In essence, each DIP is not required to produce the best estimate at every iteration of PGD.

Inspired by the bagged-DIP solution, here we propose the bagged-DVP for SCI (SCI-BDVP), as shown in Figure 3. SCI-BDVP, in addition to the standard gradient descent (GD) step, defined as x G t +1 = x t + µ H T ( y -Hx t ) , consists of two main additional components: i) The bagged-DVP module that simultaneously projects the output of the GD step onto the domain of multiple DVP networks operating at varying patch sizes (refer to Figure 2 and then averages their outputs, and ii) a skip connection that computes a weighted average of the output of the GD step and the bagged-DVP step. Next, we briefly explain the detailed construction of each component.

SCI-BDVP. Figure 2 schematically shows the structure of a bagged-DVP consisting of K individual DVPs, each operating at a different scale and trained separately. More specifically, for each k , k = 1 , · · · , K , the 3 D video is partitioned into non-overlapping video cubes of dimensions ( h k , w k ) . For each video cube of dimension ( h k , w k , B ) , we train a separate DVP. In other words, at scale k , we need to train N k = H/h k × W/w k separate DVPs. (The total aggregate number of DVPs that are trained is going to be ∑ K k =1 N k .) At each scale, the separately projected video cubes are concatenated to form x P t +1 ,k , a video frame of the same dimensions as the desired video. At scale k , let g k θ ( · ) denote a DVP that generates an output video frame of dimensions h k × w k × B . To cover the whole video frame at scale k , we need to train N k separate DVPs g k,i θ ( · ) , each having an independently drawn input, u k,i . i denotes the index of partitioned video cube. To train each of these N k DVPs, we first extract the corresponding parts from x G t +1 , y and H and denote them as x G t +1 ,i , y i and H i , respectively. 2 Then, to train the corresponding DVP to form reconstruction x P t +1 ,k,i , we minimize

̂

̂

̂

̂

Gradient Descent

x

G

t

+1

=

x

t

+

μ

H

T

(

y

-

Hx

t

)

x

G

t

+1

x

G

t

+1

=

clip

(

x

G

t

+1

, 0, 1)

Projection

Bagged-DVP

g

θ

(

u

)

θ

←argmin

∥

g

(

u

) -

x

G

∥ 2

x

t

+1

=

α

x

G

t

+1

+(1 -

α

)

x

P

t

+1

x

P

t

+1 ←

g

θ

(

u

)

PGD Bagged-DVP

u

∼

U

(0,1)

x

G

t

+1

̂

θ

θ

t

+1

2

∥ x G t +1 ,i -g k,i θ ( u k,i ) ∥ 2 2 + ω ∥ y i -H i g k,i θ ( u k,i ) ∥ 2 2 , where ω > 0 denotes the regularization parameter. Unlike classic DIP cost function, here we use the measurements y as an additional regularizer. After recovering x P t +1 ,k,i , i = 1 , . . . , N k , we concatenate them based on their locations to form x P t +1 ,k . We repeat the same process, for each k = 1 , . . . , K to find x P t +1 , 1 , . . . , x P t +1 ,K . Finally, we use the idea of bagging and define x P t +1 = 1 K ∑ K k =1 x P t +1 ,k .

Skip connection. After obtaining x P t +1 and x G t +1 , we define x t +1 as their weighted average: x t +1 = α x G t +1 +(1 -α ) x P t +1 , where α ∈ (0 , 1) . (See Figure 3.) In the experiments in Appendix C.3, we show how the addition of this skip connection consistently improves the achievable performance.

5 Experiments

We evaluate the performance of SCI-BDVP and compare it with existing SCI methods, for σ z = 0 and σ z > 0 . Our experimental results are consistent with our theoretical results on mask optimization. To evaluate the performance we use peak-signal-to-noise-ratio (PSNR) and structured similarity index metrics (SSIM) [41]. All the tests are performed on a single NVIDIA RTX 4090 GPU.

Datasets and baselines. We compare our method against the baselines on 6 gray-scaled benchmark videos including Kobe, Runner, Drop, Traffic, Aerial, Vehicle [18], where the spatial resolution is 256 × 256 , and B = 8 . We choose 5 representative baseline methods i) GAP-TV [9] : the Plug-and-play (PnP) method that employs a total-variation denoiser; ii) PnP-FFDnet [17] and PnP-FastDVDnet [18] : PnP methods that employ pre-trained deep denoisers, iii) PnP-DIP [27]: DIPbased iterative method; iv) Factorized-DVP [26]: Untrained End-to-End (E2E) network. Baseline setups follows that exactly stated in the respective papers. The details of proposed SCI-BDVP can be found in Appendix B.

Masks for noiseless and noisy measurements. For the case of SCI without noise, we obtain the measurements from equation (1), where we randomly sample mask values from Bern( p ) with p = 0 . 2 , 0 . 3 , . . . , 0 . 8 . For the noisy setup, zero-mean Gaussian noise with variance ( σ 2 ), σ = 10 , σ = 25 and σ = 50 , is added to the measurements. For the results reported in Tables 1 and 2, the masks are randomly and independently generated as as Bern(0 . 5) .

5.1 Reconstruction results for video SCI

Noiseless measurement. In Table 1 we compare the performance of SCI-BDVP against baselines. To highlight the effectiveness of the bagged DVP idea, we also implemented two versions of our proposed method: i) SCI-BDVP (E2E), an end-to-end BDVP-based solution and 2) SCI-BDVP (GAP): an iterative algorithm that employs generalized alternative projection (GAP) update rule and BDVP projection (Refer to Appendix B.1 for a description of GAP and GD and our rationale for the choice of each method.). It can be observed that both SCI-BDVP (E2E) and SCI-BDVP (GAP) outperform existing untrained methods. Specifically, SCI-BDVP (GAP) achieves state-of-the-art performance and on average improves about 1 dB in PSNR compared to other methods.

Noisy measurement. Table 2 compares the performance of SCI-BDVP (E2E) and SCI-BDVP (GD) with baseline methods. As explained in Appendix B.1, unlike noise-free measurements, in the case of noisy measurements, especially when noise variance grows, GAP update rule is no longer a reasonable

choice. Therefore, for noisy data, we replace the GAP update rule with GD. For completeness, for PnP-FastDVDnet [18], we report both GAP-based version (as implemented in [18]) and GD-based version (newly implemented here). We observe SCI-BDVP (GD) considerably outperforms PnP-DIP [27] 3 . Additionally, SCI-BDVP (GD) in most cases outperforms pre-trained method [18], across noise levels, while showing a robust performance on different datasets and noise levels.

Time/computational complexity. The proposed SCI-BDVP method relies on the bagging of multiple DIP projections. These DIP projections, which vary depending on the patch size, involve different levels of computational complexity. Table 3 shows the average time required to perform each DIP projection for each patch size. As observed, the time increases considerably as the patch size decreases. This is expected because the number of networks that need to be trained grows significantly. (Refer to Figure 2 for a pictorial representation.) Additional computational complexity analysis of our proposed method and its comparisons with other methods is included in Appendix B.3.

5.2 Mask optimization

We consider masks that are generated independent of the data as i.i.d. ∼ Bern( p ) . The question is what value of p optimizes the reconstruction performance? Figures 4 and 5 show the achieved reconstruction PSNR as a function of p , for the cases of noiseless and noisy measurements, respectively. For noiseless measurements, the results are shown both for SCI-BDVP (GAP) and PnP-FastDVDnet (GAP). It can be that for both methods, the optimized value of p is smaller than 0 . 5 (around 0.4) and

consistent with empirical observations reported in [35, 34]. For the noisy measurements, we see that p ∗ is an increasing function of σ z , consistent with our theoretical results discussed earlier in Section 3. (Refer to Appendix C.1 for further results.)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

20

25

30

35

40

45

PSNR

p

0.4

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.75

0.80

0.85

0.90

0.95

1.00

SSIM

p

0.4

0.2

0.3

0.4

0.5

0.6

0.7

0.8

20

25

30

35

40

45

PSNR

p

0.4

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.75

0.80

0.85

0.90

0.95

1.00

SSIM

p

0.4

Crash

Traffic

Kobe

Aerial

Average

0.2

0.4

0.6

0.8

26

28

30

32

34

=10

=25

=50

0.2

0.4

0.6

0.8

22

24

26

28

30

0.2

0.4

0.6

0.8

21

22

23

24

25

Aerial

0.2

0.4

0.6

0.8

22

23

24

25

Crash

0.2

0.4

0.6

0.8

22

23

24

25

26

Kobe

0.2

0.4

0.6

0.8

19

20

21

22

Traffic

6 Conclusion

We have studied application of UNNs SCI recovery. We propose an iterative solution with bagged DVP (multiple, separately trained DVPs with averaged outputs), achieving state-of-the-art performance among unsupervised solutions for noise-free measurements and robustly outperforming both supervised and UNN methods for noisy measurements. Additionally, we provide a theoretical framework analyzing the performance of UNN-based methods, characterizing achievable performance and guiding hardware parameter optimization. Simulations validate our theoretical findings.

An important application of SCI is hyperspectral snapshot imaging (HSI). Our results in this paper provide a theoretical foundation to understand HSI systems and optimize their hardware. Additionally, the developed theoretical framework can be used to explore aspects specific to HSI, such as masks being shifted versions of each other. We also expect our algorithm to effectively address overfitting in HSI tasks, enhancing reconstruction performance. We plan to explore these aspects further in our future research.

Several other aspects remain for future work. Theoretically, we only considered i.i.d. Bernoulli masks, while practical SCI systems typically are more constrained. Additionally, deriving informationtheoretic lower bounds on SCI recovery is an open problem. Experimentally, we focused on classic baseline videos; exploring a richer set of samples and studying noise models beyond additive Gaussian noise are interesting directions for future research. We also plan to enhance the algorithm's efficiency by parallelizing the projections required by the bagged solution.

References

M. E Gehm, R. John, D. J Brady, R. M Willett, and T. J Schulz. Single-shot compressive spectral imaging with a dual-disperser architecture. Opt. Exp. , 15(21):14013-14027, 2007.

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady. Coded aperture compressive temporal imaging. Opt. Exp. , 21(9):10526-10545, 2013.

P. Llull, X. Yuan, L. Carin, and D. Brady. Image translation for single-shot focal tomography. Optica , 2(9):822-825, 2015.

M. Qiao, Y. Sun, X. Liu, X. Yuan, and P. Wilford. Snapshot optical coherence tomography. In Dig. Hol. and Three-Dim. Ima. , pages W4B-3. Optica Publishing Group, 2019.

X. Yuan, D. J. Brady, and A. K. Katsaggelos. Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Sig. Proc. Mag. , 38(2):65-88, 2021.

R. Heckel and P. Hand. Deep decoder: Concise image representations from untrained nonconvolutional networks. In International Conference on Learning Representations (ICLR) , 2019.

D. Kittle, K. Choi, A. Wagadarikar, and D. J Brady. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied optics , 49(36):6824-6833, 2010.

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin. Video compressive sensing using gaussian mixture models. IEEE Trans. on Image Proc. , 23(11):4863-4878, 2014.

X. Yuan. Generalized alternating projection based total variation minimization for compressive sensing. In IEEE Int. Conf. on Image Proc. (ICIP) , pages 2539-2543. IEEE, 2016.

Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. , 2019.

Z. Cheng, R. Lu, Z. Wang, H. Zhang, B. Chen, Z. Meng, and X. Yuan. BIRNAT: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In Proceedings of the European Conference on Computer Vision (ECCV) , August 2020.

Z. Cheng, B. Chen, G. Liu, H. Zhang, R. Lu, Z. Wang, and X. Yuan. Memory-efficient network for large-scale video compressive sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16246-16255, 2021.

Z. Wang, H. Zhang, Z. Cheng, B. Chen, and X. Yuan. Metasci: Scalable and adaptive reconstruction for video compressive sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2083-2092, June 2021.

L. Wang, M. Cao, and X. Yuan. Efficientsci: Densely connected network with space-time factorization for large-scale video snapshot compressive imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 18477-18486, 2023.

L. Wang, M. Cao, Y. Zhong, and X. Yuan. Spatial-temporal transformer for video snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. , 2022.

P. Wang, L. Wang, and X. Yuan. Deep optics for video snapshot compressive imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10646-10656, October 2023.

X. Yuan, Y. Liu, J. Suo, and Q. Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

X. Yuan, Y. Liu, J. Suo, F. Durand, and Q. Dai. Plug-and-play algorithms for video snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell. , 44(10):7093-7111, 2021.

J. Ma, X. Liu, Z. Shou, and X. Yuan. Deep tensor admm-net for snapshot compressive imaging. In Proceedings of the IEEE/CVF Internatinal Conference on Computer Vision (ICCV) , 2019.

P. Peng, S. Jalali, and X. Yuan. Solving inverse problems via auto-encoders. IEEE Journal on Selected Areas in Information Theory , 1(1):312-323, 2020.

Z. Meng, S. Jalali, and X. Yuan. Gap-net for snapshot compressive imaging. arXiv preprint arXiv:2012.08364 , 2020.

Z. Wu, J. Zhang, and C. Mou. Dense deep unfolding network with 3d-cnn prior for snapshot compressive imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4892-4901, 2021.

C. Yang, S. Zhang, and X. Yuan. Ensemble learning priors unfolding for scalable snapshot compressive sensing. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Z. Meng, X. Yuan, and S. Jalali. Deep unfolding for snapshot compressive imaging. International Journal of Computer Vision (IJCV) , pages 1-26, 2023.

D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 9446-9454, 2018.

Y. C. Miao, X. L. Zhao, J. L. Wang, X. Fu, and Y. Wang. Snapshot compressive imaging using domain-factorized deep video prior. IEEE Trans. on Computational Imaging , 2024.

Z. Meng, Z. Yu, K. Xu, and X. Yuan. Self-supervised neural networks for spectral snapshot compressive imaging. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV) , pages 2622-2631, 2021.

X. Chen, Z. Hou, C. A Metzler, A. Maleki, and S. Jalali. Bagged deep image prior for recovering images in the presence of speckle noise. arXiv preprint arXiv:2402.15635 , 2024.

G. Jagatap and C. Hegde. Algorithmic guarantees for inverse imaging with untrained network priors. Advances in neural information processing systems (NeurIPS) , 32, 2019.

H. Zhang, L. Mai, N. Xu, Z. Wang, J. Collomosse, and H. Jin. An internal learning approach to video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2720-2729, 2019.

G. Mataev, P. Milanfar, and M. Elad. Deepred: Deep image prior powered by red. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0-0, 2019.

H. Zhou, Y. Song, Z. Yao, D. Hei, Y. Li, B. Duan, Y. Liu, and L. Sheng. Unsupervised reconstruction with a registered time-unsheared image constraint for compressed ultrafast photography. Opt. Exp. , 32:16333-16350, 04 2024.

M. Qiao, X. Liu, and X. Yuan. Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks. Opt. Lett. , 46(8):1888-1891, Apr 2021.

M. Iliadis, L. Spinoulas, and A. K. Katsaggelos. Deepbinarymask: Learning a binary mask for video compressive sensing. Dig. Sig. Proc. , 96:102591, 2020.

X. Zhang, Y. Zhang, R. Xiong, Q. Sun, and J. Zhang. Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 17532-17541, June 2022.

M. Zhao and S. Jalali. Theoretical analysis of binary masks in snapshot compressive imaging systems. In 2023 59th Annual Allerton Conf. on Comm., Cont., and Comp. (Allerton) , pages 1-8. IEEE, 2023.

A. Wagadarikar, N. Pitsianis, X. Sun, and D. Brady. Video rate spectral imaging using a coded aperture snapshot spectral imager. Opt. Exp. , 17(8):6368-6388, Apr. 2009.

S. Jalali and X. Yuan. Snapshot compressed sensing: Performance bounds and algorithms. IEEE Trans. Inform. Theory , 65(12):8005-8024, 2019.

R. Heckel and M. Soltanolkotabi. Denoising and regularization via exploiting the structural bias of convolutional generators. In International Conference on Learning Representations (ICLR) , 2020.

H. Wang, T. Li, Z. Zhuang, T. Chen, H. Liang, and J. Sun. Early stopping for deep image prior. Trans. on Machine Learning Research , 2023.

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc , 13(4):600-612, 2004.

S. Jalali, A. Maleki, and R. G. Baraniuk. Minimum complexity pursuit for universal compressed sensing. IEEE Trans. Inform. Theory , 60(4):2253-2268, Apr. 2014.

X. Liao, H. Li, and L. Carin. Generalized alternating projection for weighted-2,1 minimization with applications to model-based compressive sensing. SIAM Journal on Imaging Sciences , 7(2):797-823, 2014.

A Proofs

A.1 Preliminary results and definitions

Lemma A.1 (Concentration of χ 2 [42]) . If Z 1 , Z 2 , . . . , Z n are i.i.d. N (0 , 1) random variables, then for any t > 0 ,

Definition A.1. f : R k → R is called an L -Lipschitz function (or L -Lipschitz continuous) if there exists a constant L > 0 such that for all x 1 , x 2 ∈ R k :

The constant L is called the Lipschitz constant of f.

A.2 Proof of Theorem 3.1

Let ˜ x = arg min c = g θ ( u ): θ ∈ R k ∥ x -c ∥ 2 . By assumption 1 √ nB ∥ x -˜ x ∥ 2 ≤ δ . On the other hand, since ˆ x = arg min c = g θ ( u ): θ ∈ R k ∥ y -Hc ∥ 2 2 , ∥ y -Hˆ x ∥ 2 ≤ ∥ y -H ˜ x ∥ 2 , where y = Hx . Therefore,

Let ˆ x q = g [ ˆ θ ] q ( u ) , i.e., the reconstruction corresponding to the q -bit quantized version of parameters θ . By the triangle inequality,

Combining (8) with 9, it follows that

Given our assumption about the L -Lipschitz continuity of g θ as a function of θ , it follows that

For u ∈ R nB , using Cauchy-Schwartz inequality, ∥ Hu ∥ 2 2 = ∑ n j =1 ( ∑ B i =1 D ij u ij ) 2 ≤ ∑ n j =1 ( ∑ B i =1 D 2 ij ∑ B i =1 u 2 ij ) ≤ B (max i,j D 2 ij ) ∥ u ∥ 2 2 ≤ B ∥ u ∥ 2 2 , which follows since D i,j ∈ { 0 , 1 } . Therefore,

For a fixed random initialization z ∈ R p , define the set of reconstructions derived from q -bit quantized parameters as C q ( u ) , i.e.,

Note that |C q ( u ) | ≤ 2 qk . Given ϵ 1 > 0 , ϵ 2 > 0 , and x , ˜ x ∈ R nB , define events E 1 and E 2 as

respectively. Then, conditioned on E 1 ∩ E 2 and noting that i) ∥ ∑ B i =1 ( x i -˜ x i ) ∥ 2 2 ≤ B ∥ x -˜ x ∥ 2 2 and ii) 1 √ nB ∥ x -˜ x ∥ 2 ≤ δ and iii) for any a, b ≥ 0 , √ a + b ≤ √ a + √ b , from (10)- (12), we have

or

On the other hand, by the triangle inequality, ∥ x -ˆ x ∥ 2 ≤ ∥ x -ˆ x q ∥ 2 + ∥ ˆ x q -ˆ x ∥ 2 . Therefore, combining (11) and (16), it follows that

To finish the proof, we need to set the parameters ( q, ϵ 1 , ϵ 2 ) and bound P ( E c 1 ∪ E c 2 ) . For a fixed u ∈ R nB , ∥ Hu ∥ 2 2 = ∑ n j =1 U j , where U j = ( ∑ B i =1 D ij u ij ) 2 . Note that

Moreover, by the Cauchy-Schwartz inequality, U j ≤ ( ∑ B i =1 D 2 ij ) ∑ B i =1 u 2 ij ≤ B ∑ B i =1 u 2 ij ≤ B 2 ( ∥ u ∥ ∞ ) 2 .

Therefore, since by assumption for all x ∈ Q , ∥ x ∥ ∞ ≤ ρ 2 , using the Hoeffding's inequality, we have

Similarly, combining the Hoeffding's inequality and the union bound, we have

Finally, given free parameters η ∈ (0 , 1) , setting the parameters as

we have

For the selected parameters, setting η = 0 . 5 and using ln 2 2 < 1 and ( η 1 4 +(1 + η ) 1 4 ) ≤ 2 , from (17), it follows that

A.3 Proof of Corollary 3.3

Let f ( p ) denote the upper bound in Theorem 3.1. That is,

where ϵ 1 and ϵ 2 are defined as

and do not depend on p . On the other hand, f (0) = f (1) = ∞ . Let p ∗ denote the value of p ∈ (0 , 1) that minimizes f ( p ) , note that

Note that on one hand lim p → 0 f ′ ( p ) = -∞ and on the other hand f ′ ( 1 2 ) > 0 , which implies that p ∗ where f ′ ( p ∗ ) = 0 belongs to (0 , 1 2 ) .

A.4 Proof of Theorem 3.4

Let ˜ x = arg min c = g θ ( u ): θ ∈ R k ∥ x -c ∥ 2 and

That is, ˆ x q denotes the reconstruction corresponding to the q -bit quantized version of ˆ θ . Following the same argument as the one used in the proof of Theorem 3.1, since y = Hx + z , it follows that 1 √ n ∥ x -˜ x ∥ 2 ≤ δ and ∥ H ( x -ˆ x )+ z ∥ 2 ≤ ∥ H ( x -˜ x )+ z ∥ 2 . On the other hand, ∥ H ( x -ˆ x )+ z ∥ 2 2 = ∥ H ( x -ˆ x ) ∥ 2 +2 ⟨ z , H ( x -ˆ x ) ⟩ + ∥ z ∥ 2 , and ∥ H ( x -˜ x )+ z ∥ 2 2 = ∥ H ( x -˜ x ) ∥ 2 +2 ⟨ z , H ( x -˜ x ) ⟩ + ∥ z ∥ 2 . Therefore,

Moreover, using the triangle inequality,

|⟨ z , H ( x -ˆ x ) ⟩| ≤ |⟨ z , H ( x -ˆ x q ) ⟩| + |⟨ z , H ( ˆ x q -ˆ x ) ⟩| ( a ) ≤ |⟨ z , H ( x -ˆ x q ) ⟩| + ∥ z ∥ 2 ∥ H ( ˆ x q -ˆ x ) ∥ 2 , where ( a ) follows from Cauchy-Schwartz inequality. Therefore,

Note that, by the triangle inequality, ∥ H ( x -ˆ x ) ∥ 2 ≥ ∥ H ( x -ˆ x q ) ∥ 2 - ∥ H ( ˆ x q -ˆ x ) ∥ 2 , which implies that ∥ H ( x -ˆ x ) ∥ 2 2 ≥ ∥ H ( x -ˆ x q ) ∥ 2 2 -2 H ( x -ˆ x q ) ∥ 2 ∥ H ( ˆ x q -ˆ x ) ∥ 2 + ∥ H ( ˆ x q -ˆ x ) ∥ 2 2 ≥ ∥ H ( x -ˆ x q ) ∥ 2 2 -2 ∥ H ( x -ˆ x q ) ∥ 2 ∥ H ( ˆ x q -ˆ x ) ∥ 2 . Therefore, combining this inequality with (24), it follows that

Similar to the proof of Theorem 3.1, for a fixed random initialization u ∈ R p , define C q ( u ) = { g [ θ ] q ( u ) : θ ∈ [0 , 1] k } . Also, given ϵ 1 , ϵ 2 , ϵ 3 > 0 , and x , ˜ x ∈ R nB , define events E 1 and E 2 as (13) and (14), respectively. Moreover, define event E 3 as

Compared to the proof of Theorem 3.1, (24) involves three terms that involve Gaussian noise z . For c ∈ C q ( u ) , define ϕ ( c ) as

Conditioned on the mask D , ϕ ( c ) is a zero-mean Gaussian random variable with

Let t ( c ) ≜ σ z ∥ ∑ D i ( x i -c i ) ∥ 2 . Then, for any given ϵ z > 0 ,

where ( a ) follows because for any Gaussian random variable G ∼ N (0 , σ 2 ) , applying the Chernoff bound, we have P( | G | > t ) ≤ 2e -t 2 / 2 σ 2 . Given ϵ z 1 , ϵ z 2 , ϵ z 3 > 0 , define events E z 1 , E z 2 and E z 3 as

and

respectively. From (27), noting that |C q ( u ) | ≤ 2 qk , it follows that

Define event E as E = E 1 ∩ E 2 ∩ E 3 ∩ E z 1 ∩ E z 2 and

Conditioned on E , since by assumption 1 √ nB ∥ x -˜ x ∥ 2 ≤ δ , we have

or

where the last line follows because for any a, b > 0 , √ a + b ≤ √ a + √ b . Define

Then, similar to (33), conditioned on E , since ˆ x q ∈ C q ( u ) , it follows that

Conditioned on E , combining (25), (33), and (34) it follows that

where we have used ∥ H ( ˆ x q -ˆ x ) ∥ 2 ≤ BL 2 -q √ k derived in (12). Note that since ∆ q ≤ ρ and h ( p ) ≤ B , the last term in (35) that corresponds to the quantization error can be bounded as

where

and

and does not depend on ∆ q . Rearranging the terms in (11) and letting

it follows that

Therefore,

This implies that

Therefore, since i) ∥ x -ˆ x ∥ 2 ≤ ∥ x -ˆ x q ∥ 2 + ∥ ˆ x q -ˆ x ∥ 2 and ii) from (11), ∥ ˆ x q -ˆ x ∥ 2 ≤ L 2 -q k , we have

Using √ 1 + α ≤ 1 + 2 α , for α > 0 , and noting that h ( p ) p (1 -p ) = 1 + Bp/ (1 -p ) , we have

Next we set the parameters by analyzing the probability event E . Applying the union bound,

where we have used Lemma A.1 to bound P( E c z 3 ) . Similar to the proof of Theorem 3.1, given free parameters η 1 , η 2 ∈ (0 , 1) , let

Also, set

Following a similar argument as the one used in the proof of Theorem 3.1, it follows that

Setting η 1 = η 2 = 0 . 5 , it is straight forward to see that

where (a) follows from (5). Also, for the selected parameters and noting that B ≥ 1 ,

Therefore, c n = O ( 1 log n ) . Moreover,

Therefore, from (39), and using i) √∑ i a i ≤ ∑ i √ a i , for a i ≥ 0 , ii) h ( p ) ≤ B for all p ∈ [0 , 1] and iii) B ≥ 1 (which implies from our assumption that k (log n )(log log n ) < n ), it follows that

or, rearranging the term in (1), we have

which yields the desired result.

A.5 Proof of Theorem 3.5

Following the steps of the proof of Theorem 3.4, we can derive (25), i.e., ∥ H ( x -ˆ x q ) ∥ 2 ≤ ∥ H ( x -˜ x ) ∥ 2 +2 |⟨ z , H ( x -ˆ x q ) ⟩| +2 |⟨ z , H ( x -˜ x ) ⟩| +( ∥ z ∥ 2 +2 ∥ H ( x -ˆ x q ) ∥ 2 ) ∥ H ( ˆ x q -ˆ x ) ∥ 2 . We defined events E 1 , E 2 , E 3 , E z 1 , E z 2 and E z 3 as before. Note that conditioned on E 2 ,

In the proof of Theorems 3.1 and 3.4, to apply this inequality and bound the error, we ignored the positive term p 2 n ∥ ∑ B i =1 ( x i -ˆ x i ) ∥ 2 2 and used 1 n ∥ H ( x -ˆ x q ) ∥ 2 2 ≥ p -p 2 n ∥ x -ˆ x ∥ 2 2 -Bρ 2 ϵ 2 . Here, instead we ignore p -p 2 n ∥ x -ˆ x ∥ 2 2 and conditioned on E 2 , we use

Define E = E 1 ∩ E 2 ∩ E 3 ∩ E z 1 ∩ E z 2 similar to the proof of Theorem 3.4. Following the steps used in deriving (36) in the proof of Theorem 3.4, and applying i) the lower bound in (45) and ii) ∆ q ≤ ρ , it follows that, conditioned on E ,

Let and

Using these definitions, (46) can be written as

Noting that 1 p 2 B h ( p ) = B -1 B + 1 pB ≤ 1 + 1 pB ,

Using Cauchy-Schwarz inequality, ∥ ∑ B i =1 (ˆ x i -ˆ x q,i ) ∥ 2 2 ≤ B ∥ ˆ x -ˆ x q ∥ 2 2 and combining it with (11), it follows that

Therefore, using the triangle inequality as ∥ ¯ x -ˆ ¯ x ∥ 2 ≤ ∥ ¯ x -ˆ ¯ x q ∥ 2 + ∥ ˆ ¯ x q -ˆ ¯ x ∥ 2 , it follows from (48) and (50) that

where the last line follows from δ ≤ ρ , and √ a + b ≤ √ a + √ b , for a, b ≥ 0 . We set the parameters as in the proof of Theorem 3.5 as

Also, set

Then, using the bounds in (42) and (43), it follows from (51)

where υ n = O ( 1 (log n ) 1 8 ) and does not depend on p .

B Setup of SCI-BDVP

B.1 SCI-BDVP: Descent step

As explained in the paper, to solve the optimization described in (3), we employ the PGD algorithm, with an additional skip connection. The details of the projections step using bagged-DVP and also the skip connection are described in Section 4. Here, we review the descent step, as we employ two different operators depending on whether the measurements are noisy or noiseless.

Descent step:

For noise-free measurements, we use GAP update rule [43]:

or

For nosiy measurement, we use gradient descent (GD):

In both cases µ denotes the step size.

Compared to the GD, if µ = 1 , GAP, at each iteration, projects the current estimate x ( t ) onto the y = Hx hyperplane. Note that due to the special structure of the sensing matrix H , HH ⊤ is a diagonal matrix and therefore it is straightforward to compute its inverse, as required by GAP.

In our experiments, we found that in the case of noise-free measurements, the GAP update rule consistently showed better convergence compared to GD. Therefore, we adopted GAP update rule for the case of noise-free measurements. However, for noisy measurements, even the true signal does not lie on y = Hx hyperplane, and therefore, application of GAP is no longer theoretically founded. Hence, for all experiments done for noisy measurements, we use the classic GD update rule.

In summary, Algorithm 1 below shows the steps of SCI-BDVP.

Algorithm 1 SCI-BDVP

Require: measurement y , mask H

1: Initial x 0 = H ⊤ y .

2: for t = 1 , . . . , T do

3: Descent step

4: Update x G t with Eq. (53) or (54).

5: Projection step

6: Generate x P t as the output of bagged-DVP (refer to Fig. 2)

7: Update x t with x t = α x G t +(1 -α ) x P t

8: end for

9: Output: Reconstructed signal ˆ x = x T .

DVP Block-1

2D

Conv

kernel 3

ReLU

Up-

Sample

scale 2

DVP Block-2

2D

Conv

kernel 3

ReLU

Up-

Sample

scale 2

DVP Block-3

2D

Conv

kernel 3

ReLU

Up-

Sample

scale 2

Video Output Block

2D

Conv

kernel 3

Sigmoid

Channel:

In:128 out:128

Channel:

In:128 out:128

Channel:

In:128 out:128

Channel:

In:128 out:frame size

B.2 Implementation details

In the projection step, we use the same structure design for bagged-DVP, for both noiseless and noisy measurements. Inspired by deep decoder structure [6], we design the neural nets, using three DVP blocks and one video output block shown in Figure 6. Each DVP block is composed of Upsample , ReLU and Conv blocks. The output block only contains the Conv block. Here, we use Conv 3 × 3 and the number of channels are fixed to 128 . Lastly, the input u of each DVP (described in DVP function g θ ( u ) ) is generated independently using a Uniform distribution, U (0 , 1) .

Since the input video consists of B 256 × 256 frames, we choose three DVP structures, one with 16 64 × 64 × B patches, one with 4 128 × 128 × B patches, and one with a single 256 × 256 × B frame. For each size of the patches, we perform mirror padding to augment the each patch with the size of h/ 8 , since it is square patch, where h represent the height of the padded patch. And for each patch of each estimate, we train the separate DVP module.

The hyperparameters are set as follows: the learning rate of the DVPs is set to 0.01; weight ω = 0 . 1 for measurement loss term ω ∥ y i -H i g k,i θ ( u k,i ) ∥ 2 2 in Figure 2. For noise free case, GAP, we set the step size µ = 1 . 0 , and µ = 0 . 1 for the noisy simple GD case.

B.3 Time/computational complexity

Implementing SCI-BDVP involves outer loop iterations (described in Algorithm 1) and also inner loop iterations for training DIPs. Table 4 presents average number of inner loop iterations used for different patch sizes ( 64 , 128 , 256 ) of various videos, and the number of outer loop iterations. Detailed time consumption for each patch level computation is recorded in Table 3. A comparison across different UNN-based methods is provided in Table 5. All comparisons are performed on a single NVIDIA RTX 4090. It is important to note that training a bagged DIP requires training multiple separate DIPs. This process can be readily parallelized , which is expected to significantly speed up the algorithm. We plan to explore this direction to optimize the algorithm's efficiency in future work. Lastly, making a direct comparison among all methods is challenging because, for supervised methods, the main time is spent in training, whereas, for unsupervised methods, the main time is spent on training the UNNs. This is an expected trade-off for requiring no training data and achieving a robust solution.

C Additional studies

C.1 Mask optimization

In this paper, we explored binary masks that are generated i.i.d. Bern( p ) . Figures 4 and 5 show the effect of probability p on the performance of SCI-BDVP (GAP) (noiseless measurements) and SCI-BDVP (GD) (noisy measurements), respectively. In this section, we perform similar investigation of the effect of p on the performance of other SCI methods, namely, PnP-FastDVD [18] and PnP-DIP [27]. Figure 7 shows the results corresponding to PnP-FastDVD. It can be observed that the trends are consistent with the performance of our proposed SCI-BDVP (GAP): i) for noiseless measurements, the optimal performance is achieved at p ∗ < 0 . 5 , ii) for noisy measurements, p ∗ is an increasing function of σ z .

On the other hand, Figure 8 shows the performance achieved by PnP-DIP proposed in [27]. It can be observed that the reconstruction performance is not longer a smooth function of p , unlike the performance of SCI-BDVP and PnP-FastDVD. We believe that the reason lies in the the limitations of UNNs (or DIP) explained before, such as overfitting and instability. In the next section, we further explore this issue and explain how bagging can address the problem.

Also, we include the detailed PSNR and SSIM results using SCI-BDVP on different measurement noise level in Table 6. We can find when σ = 50 , the effect of the mask optimization will improve the overall result around 1 dB in PSNR, 0 . 1 in SSIM. Here, all the algorithm settings are kept intact, and the only variation is in the mask non-zero probability p varies between 0 . 5 to 0 . 7 . This further highlights the stability of the proposed SCI-BDVP solution.

0.2

0.4

0.6

0.8

Probability

15

20

25

30

35

40

PSNR

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

14

16

18

20

22

24

26

PSNR

Crash

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

14

16

18

20

22

24

26

28

PSNR

Traffic

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

15

20

25

30

35

PSNR

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

15

20

25

30

PSNR

Kobe

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

14

16

18

20

22

24

26

28

PSNR

Aerial

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

20

22

24

26

28

30

32

34

PSNR

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

18

19

20

21

22

23

PSNR

Crash

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

19

20

21

22

23

24

PSNR

Traffic

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

20

22

24

26

28

30

PSNR

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

19

20

21

22

23

24

25

PSNR

Kobe

=0

=10

=25

=50

0.2

0.4

0.6

0.8

Probability

17

18

19

20

PSNR

Aerial

=0

=10

=25

=50

C.2 Effect of bagging

We discussed how bagging can help address the DIP (or DVP) overfitting issue and provide a robust projection module, which can robustly capture the source structure, without any training data. Figure 9 shows the impact of bagging on the performance of SCI-BDVP in Section 4, by comparing it with the performances of different SCI-DVP solutions, where instead of a bagged version, we used a simple DVP for projection.

Figure 10 shows the qualitative reconstruction results of our proposed SCI-BDVP in comparision with the non-bagged version, SCI-DVP. An expected, the results show that using bagging improves the reconstruction quality, in both noise-free and noisy cases.

0

5

10

15

20

25

30

35

Iteration Number

8

10

12

14

16

18

20

22

24

PSNR Value

Traffic

SCI-DVP-64

SCI-DVP-128

SCI-DVP-256

SCI-BDVP

0

10

20

30

40

50

60

70

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Kobe

SCI-DVP-64

SCI-DVP-128

SCI-DVP-256

SCI-BDVP

0

10

20

30

40

50

60

70

Iteration Number

18

20

22

24

26

28

30

32

34

PSNR Value

SCI-DVP-64

SCI-DVP-128

SCI-DVP-256

SCI-BDVP

0

5

10

15

20

25

30

35

Iteration Number

16

18

20

22

24

26

PSNR Value

Aerial

SCI-DVP-64

SCI-DVP-128

SCI-DVP-256

SCI-BDVP

Finally, to further highlight the impact of the bagging operation, here we explore the performance of the proposed bagged DVP solution for the classic inverse problem of denoising from additive Gaussian noise. Figure 12 shows the denoising performance of the proposed bagged-DVP solution (refer to Figure 2) in denoising x from measurements y = x + z , where z is generated i.i.d. N (0 , σ 2 z ) . We compare the performance of bagged-DVP with the three DVP structures that are used as the building components of our bagged-DVP. As explained earlier, each of these DVPs operates at a different patch size. It can be observed that BDVP consistently outperforms the individual DVPs and shows a much more smooth convergence behaviour. Given that BDVP only averages the outputs of the three individual DVPs, the observed gain suggests the independence of the estimates (at least partially), which leads to the observed gain.

C.3 Effect of averaging coefficient

We explore the effect of coefficient α used in combining the outputs from gradient descent and projection steps at time t ,

In Figure 11 left panel, it can be observed that without the skip connection, the performance drops by around 4 dB. However, for α ∈ [0 . 1 , 0 . 7] the performance is stable and does not vary considerably as α changes. On the other extreme case when α = 1 , when there is no projection, as expected, the performance severely degrades.

Drop #6

Traffic #36

Kobe #9

Kobe #18

𝜎

!

= 0

𝜎

!

= 10

𝜎

!

= 25

𝜎

!

= 50

SCI-BDVP

SCI-DVP

SCI-BDVP

SCI-DVP

SCI-BDVP

SCI-DVP

SCI-BDVP

SCI-DVP

C.4 Effect of reconstruction loss coefficient

We explore the effect of coefficient in reconstruction measurement loss, the second term of the loss function in Figure 2

In the middle and right panel of Figure 11, we can find that for noise-free case the reconstruction results is not sensitive to the change of ω . However, in the noisy case, when we increase the ω the reconstruction will drop and if not including the measurement loss term, ω = 0 , the results will also

Runner #5

Aerial #22

Crash #27

𝜎

!

= 0

𝜎

!

= 10

𝜎

!

= 25

𝜎

!

= 50

SCI-BDVP

SCI-DVP

SCI-BDVP

SCI-DVP

SCI-BDVP

SCI-DVP

drop. This is due to in noisy case, the measurement y is no longer the actual measurement, but we still need some information from corrupted y to boost the reconstruction results.

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

alpha

5

10

15

20

25

30

35

40

PSNR (dB)

skip connection coefficient - noiseless

Kobe

Traffic

Crash

Aerial

Average

0.0

0.1

0.3

0.5

0.7

0.9

omega

22.5

25.0

27.5

30.0

32.5

35.0

37.5

40.0

PSNR (dB)

reconstruction loss coefficient - noiseless

Kobe

Traffic

Crash

Aerial

Average

0.0

0.1

0.3

0.5

0.7

0.9

omega

20

22

24

26

28

30

32

34

PSNR (dB)

reconstruction loss coefficient - noisy (sigma=25)

Kobe

Traffic

Crash

Aerial

Average

0

1000

2000

3000

4000

5000

Iteration Number

15

20

25

30

35

PSNR Value

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Crash{1}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Crash{2}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

18

20

22

24

26

28

PSNR Value

Crash{3}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Crash{4}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

18

20

22

24

26

28

PSNR Value

Traffic{1}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Traffic{2}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Traffic{3}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Traffic{4}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Traffic{5}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Traffic{6}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Kobe{1}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Kobe{2}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Kobe{3}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Kobe{4}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Aerial{1}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Aerial{2}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

PSNR Value

Aerial{3}

bagged

64

128

256

0

1000

2000

3000

4000

5000

Iteration Number

16

18

20

22

24

26

28

PSNR Value

Aerial{4}

bagged

64

128

256

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the following we list the claims made in the abstract and introduction and references on how they are accurately addressed in the paper. (Section 1.1 lists the contributions of the paper)

(theoretic) Theoretical analysis of untrained-neural-net-based (UNN-based) SCI recovery methods: Sections 3.2 employs DIP hypothesis and theoretically analyzes such DIP-based SCI methods, for both noisy and noiseless measurements. (Refer to Theorems 3.1, 3.4 and 3.5 for the theoretical characterization of UNN-based SCI recovery.)

(theoretic) Theoretical statements on the optimization of binary-valued masks under both noisy and noiseless measurements: Corollary 3.3 and Corollary 3.6 provide the implications of our theoretical results on mask optimization.

(theoretic) Theoretic upper bound on the number of frames that can be recovered as a single encoded 2D measurement frame as a function of the parameters on the UNN: Corollary 3.2 presents this result.

(algorithmic) Introduction of a novel unsupervised SCI recovery algorithm: The proposed method, SCI-BDVP, is introduced in Section 4.

(experimental) We have implemented the proposed method (SCI-BDVP) and compared its performance with both unsupervised and supervised SCI solutions, under both noisy and noiseless measurements. (Refer to Section 5.)

(experimental) We have claimed that the proposed method achieves state-of-the-art performance among existing unsupervised methods. This is confirmed in the reported results in Table 1 and Table 2.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The second paragraph in Section 6 reviews the theoretical and experimental limitations of our results.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our theorems are self-contained, including all necessary assumptions and settings. In the following section, we list the theoretical contributions of our paper and provide references for their proofs.

Theorem 3.1: Proved in Section A.2.

Corollary 3.2: Straightforward result following Theorem 3.1.

Corollary 3.3: Proved in Section A.3.

Theorem 3.4: Proved in Section A.4.

Theorem 3.5: Proved in Section A.5.

Unfortunately we didn't have enough space in the paper to provide highlights of the proofs in the main body of the paper.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

We have released our codes.

Our algorithm is clearly explained in the paper. We have also given a detailed description of the UNNs used in the our proposed bagged-DVP solution.

We have clearly stated our choice of hyperparameters in the paper.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided our code which includes all the parameter settings and implementations. For data, we have used benchmark videos including Kobe, Runner, Drop, Traffic, Aerial, Vehicle [18].

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Our proposed SCI-BDVP algorithm and the structure its main component, i.e., bagged DVP, are presented in Section 4

The descent step used by our proposed SCI-BDVP algorithm is explained in Section B.1 of the supplementary materials.

The hyperparameter settings are explained in Section B.2 supplementary materials.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have not provided error bars due to computational complexity of solving the required NN training. We want to highlight two points: 1) Since our method is unsupervised, its performance on each data cube is independent of any training data. 2) None of the cited prior works on SCI recovery include error bars.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The details of our computation resources are presented in Section B.2, Section B.3 and 5.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our proposed solution is an unsupervised method for SCI recovery, which can enable recovery without any training data. Moreover, we provide theoretical understanding of the problem, which enable hardware optimization, without requiring extensive computationally-intensive empirical optimizations. All the positive aspects are discussed in the paper. The work does not have any negative societal impact.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the source of the benchmark videos and also the codes of different algorithms used in our implementations.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:

We have submitted an anonymized version of our code that implements our proposed SCI-BDVP algorithm.

We will later upload the code to GitHub and make it public.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
comments: SUMMARY:
Please see Strengths and Weaknesses.

STRENGTHS:
1. Rigorous theoretical analysis of the proposed formulation
2. Detailed empirical evaluation.

WEAKNESSES:
1. Formatting in Figure 2 can be improved. The text near smaller cubes is not readable.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work gives some theoretical analysis for mask optimization and DIP-based SCI recovery methods. The work claims that the proposed SCI-BDVP achieves SOTA performance among UNN methods.\n\nSTRENGTHS:\n1. This work provides some theoretical analysis, compared with the conventional work is rare.\n2. Bagged DIP is introduced to develop SCI iterative methods and the proposed method claims to achieve SOTA performance among UNN methods.\n\nWEAKNESSES:\n1. While this work does some theoretical analysis by introducing DIP's theory, it cannot be directly generalized to all untrained networks. On the one hand, it seems impossible that all untrained networks satisfy the DIP hypothesis in the paper, and on the other hand, how can the UNNs used in this work be guaranteed to satisfy the DIP hypothesis and Lipschitz's condition? As far as I know, networks that satisfy Lipschitz's condition require a specific design, and this work does not seem to give some related introduction.\n2. The core of this work is to prove theoretical results for untrained networks, but the actual contribution is less than stated. When the properties of certain untrained networks are demonstrated by DIP, the work does not propose any new hypotheses in snapshot compressive imaging but simply adopts the original one, which decreases the contribution of the paper. \n3. When the DIP hypothesis is used to simplify the UNNs as a minimization operator, the work is merely proving a boundary based on the minimization operator. Similar theoretical results based on the minimization operator can also be found in the theoretical analysis section of GAP-Net[1].\n4. The DIP hypothesis demonstrated in this paper seems to ignore some conditions in the original DIP paper and doesn't capture the unique characteristics of untrained networks, what if the same is true of pre-trained networks? \n5. The comparison algorithm needs to be perfected. The current comparison algorithms seem to be some traditional optimization algorithms and your methods, and we even can't distinguish which of them is previous UNN works. So what does it mean that your method achieves SOTA performance in UNNs？\n6. Considering this work as a deep network-based approach, such comparisons are unfair. Why not add some recent self-supervised methods to demonstrate the effectiveness of the algorithm? In addition, despite claiming to be compared with the supervised algorithms in noisy scenarios, some recent algorithms, such as SCI3D[2] and EfficientSCI[3] have not been included.\n\n[1]Deep Unfolding for Snapshot Compressive Imaging, in IJCV2023.\n\n[2]Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot CompressiveImaging, in ICCV2021.\n\n[3]EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging, in CVPR2023.\n\nQUESTIONS:\nHow can the UNNs used in this work be guaranteed to satisfy the DIP hypothesis and Lipschitz's condition?\n\nLIMITATIONS:\nThe authors provide some limitations, which is their future research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe focus of this paper is on developing recovery algorithms of snapshot compressive imaging (SCI) using untrained neural networks (UNNs). Besides, the paper introduces the concept of bagged-deep-image-prior (bagged-DIP) to create SCI Bagged Deep Video Prior (SCI-BDVP) algorithms, which are designed to address common challenges faced by standard UNN solutions in SCI recovery. Extensive experiments demonstrate the effectiveness of the proposed method in video SCI recovery. In scenarios with noisy measurements, this untrained network even outperforms supervised methods.\n\nSTRENGTHS:\n- This paper is well structured and has clear logic.\n\n- The experimental results on the performance are convincing.\n\n- The proposed method is based on untrained neural network, making it more flexible to be applied in various scenes.\n\nWEAKNESSES:\n- For untrained methods, runtime may be a crucial metric [1,2]. But this paper doesn't provide analysis in comparison to existing methods.\n\n- To improve clarity, consider adding clear notations for the symbols used. For example, what do different colored arrows represent in Figure 2, and the color of \"Fusion\" and \"MSE\" are too similar to be distinguished.\n\n[1] Rui, Xiangyu, et al. \"Unsupervised hyperspectral pansharpening via low-rank diffusion model.\" Information Fusion 107 (2024): 102325.  \n[2] Pang, Li, et al. \"HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n- SCI is also widely used in hyperspectral image, can your method be used in hyperspectral reconstruction task? If so, what's the prominent advantage of your method?\n\n- How does the method handle different noise models beyond additive Gaussian noise?\n\nLIMITATIONS:\n- Exploring the application on additional datasets from different domains or varing conditions in the future may increase persuasiveness, e.g. hyperspectral data. And exploring the performance of the proposed method under different noise models would provide a more comprehensive evaluation of its robustness and general applicability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper leverages untrained neural networks UNN (deep image priors DIP or deep decoder DD) as a prior to solve Snapshot Compressive Imaging (SCI), a technique used in ($n_1$ x $n_2$ x B)-dimensional 3-D imaging where the captured measurements lie in a 2-D plane ($n_1$ x $n_2$). The application of UNNs in the context of SCI itself is not novel; this paper's main contributions are:\n1) Theoretical recovery guarantees for SCI (i.e. existence of a minimizer to the reconstruction problem) denoting number of 2-D frames B that can be recovered using 1 2-D measurement as a function of UNN model complexity (assuming original signal is close to range of UNN by $\\delta$) under noise-free and additive gaussian noise settings.\n2) Use bagged-DIP as algorithm for signal recovery, called SCI Bagged Deep Video Prior (BDVP).\n3) Optimize binary-valued masks used in the measurement process and show empirical analysis.\n\nSTRENGTHS:\nThe main contribution of this paper are theoretical reconstruction guarantees for SCI using UNNs under both noisy and noiseless measurements. They derive a bound on reconstruction error in terms of signal parameter B, Bernoulli sampling pattern mask p, measurement and signal parameter n, and $\\sigma_z$ noise. \n\nAuthors further validate their reconstruction error bound and its dependence on sampling parameter p, on empirical datasets on 6 videos: Drop, Runner, Aerial, Crash, Kobe, Traffic.\n\nThey propose SCI Bagged Deep Video Prior method as an algorithmic framework for solving SCI. On the empirical data, they show improved SNR and SSIM against baseline untrained methods.\n\nWEAKNESSES:\nMain comments:\nComputational complexity: Deep Video Prior/Untrained Network Prior setups have high computational complexity; on top of this a bagging approach requires solving K such problems simultaneously. This requires a discussion on how computational complexity compares to baselines.\n\nTheoretical claims are summarized in main paper; however I was not able to find longer version of paper in supplementary zip folder to validate the proofs.\n\nTheorem 3.1: Are parameters $p$ in $u [0,1]^p$ and Bern($p$) for $D_{i,j}$ the same? \n\nMinor comments:\nline 47-48: \"Existing UNN-based SCI solutions either recover the image 48 end-to-end in one shot or employ iterative methods akin to projected gradient descent (PGD).\" (relevant citations missing)\n\nline 88-97: Literature review of DIP + UNN: Add relevant citation to \"Qiao, M., Liu, X., & Yuan, X. (2021). Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks. Optics Letters, 46(8), 1888-1891.\"\n\nTheorem 3.1. reconstruction error bound - $\\rho$ not defined.\n\nQUESTIONS:\nTheoretical claims are summarized in main paper; however I was not able to find longer version of paper in supplementary zip folder to validate the proofs - this would help consolidate the contents.\n\nLIMITATIONS:\nLimitations have been discussed; they should additionally discuss time/computational complexity of the proposed BDVP algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-23"}