MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging
Authors: Zhenghao Pan
Haijin Zeng
Jiezhang Cao
Yongyong Chen
Kai Zhang
Yong Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Snapshot Compressive Imaging; Compressive Imaging; Deep Learning; Attention; Coded Aperture Compressive Temporal Imaging (CACTI); State Space Models
Primary area: machine_vision

Abstract: Color video snapshot compressive imaging (SCI) employs computational imaging techniques to capture multiple sequential video frames in a single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient reconstruction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba-Blocks, which residually connect the Spatial-Temporal Mamba (STMamba), Edge-Detail-Reconstruction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporal dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA is used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses state-of-the-art methods with lower computational and memory costs. PyTorch style pseudo-code for the core modules is provided in the supplementary materials. Code is at https://github.com/PAN083/MambaSCI.

MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging
Zhenghao Pan Haijin Zeng"
Jiezhang Cao Yongyong Chen !-t, Kai Zhang', Yong Xu'
Harbin Institute of Technology (Shenzhen), Ghent University, Harvard University, Nanjing University
Abstract
Color video snapshot compressive imaging (SCI) employs computational imaging techniques capture multiple sequential video frames in single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color vidco designed based on the traditional Bayer pattern.
SCI reconstruction algorithms When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient recon- struction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba Blockks which residually connect the Mamba (STMamba), Edgee-etaill Recaall-Reoonttuuction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporea dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses ttate-of-the-art methods with lower computational and memory costs. Py PyTorc style pseudo-code for the modules is provided in the supplementary materials. Code is at wtlhttps:///a h/tt:::///tthubb
Introduction
In recent years, there has been significant progress in en- hancing imaging quality in smartphone image sensors. One notable trend is the adoption of the quad quad-Bayer Color Filter Array (CFA) pattern [ Smartphones such as the iPhone Pro/Max, vivo X90 Pro+, Xiaomi 13S Ultra, and OPPO Find X6 Pro utilize quad-Bayer array to enhance image quality low-light conditions and offer higher resolution for mobile photography 2 3. Unlike the traditional RGGB Bayer CFA pattern, the quad-Bayer pattern expands each pixel into four sub-pixels and arranges them periodically 5. as depicted in Fig. a).. This arrangement allows for larger pixels by
17
35
Mambasci-8
PGAR (00)
MambasGi-
EffiiennSCi-8
20
STFomee-5
EMfclencSCi-S
25
5020
10000 FLOPS
(4)
(b)
Figure 1: (a) Bayer CFA vs. Quad-Bayer CFA. (b) PSNR and FLOPS on color sim- ulation videos (larger size means more parameters).
Equual contribution. Corresponding Author: Yongyong Chen YongyongChen.cn@@@mall.cco
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Scene
Quad-Bayer
Quad Sayer CFA
Measurement
Proposed Quad- Bosed Method
Reconstruction
Algorithm
Bayer
Bayer Filter Camera
Recovered Color
Sayer CRA
Previous Bayer Based Method
(b)
Figure 2: (a) Schematic diagram of the comparison between color video SCI based on the proposed quad- Bayer-based method and the previous Bayer-based method. (b) Photo taken by quad-Bayer CFA pattern (Sony IMX689) (top) Bayer CFA pattern (bottom). One can see that the upper image is sharper with less noise.
combining neighboring pixels of the same color, resulting in more light intake compared the Bayer pattern Consequently, quad-Bayed sensors offer enhanced sensitivity and resolution for imaging tasks 6 g As illustrated Fig. b). quad-Bar BaadrBayer technology effectively mitigates resolution loss and enables the capture of low-noise photos in low-light environments. In summary, quad-Bayer sensors provide HDR capability O and improved color accuracy [T0] while effectively mitigating resolution loss and capturing low-noise photos in low-light environments.
Color videos captured by traditional high-speed cameras incur high transmission and storage costs. To solve this issue, color video snapshot compressive imaging (SCI) [IT), which comprises both hardware encoder and software decodes components, has been proposed. During the encoding phase, multiple raw video frames undergo modulation and compression using various masks to generate 2D measurements [2]. Subsequently, in the decoding stage, the desired high-speed color video is reconstructed from the acquired measurements and predefined masks.
So far, all color video SCI encoding and reconstruction algorithms have been designed based on the Bayer pattern [33 14 1S] 16 [7]. As smartphone cameras continue to improve in pixel count and performance, the majority of videos patterns. However, our observation reveals that existing methods struggle to effectively reconstruct videos based quad- quad-Bayer patterns, often resulting in artifacts, color distortions, and incomplete demosaicing. This ineffectiveness poses a challenge for processing videos captured by smartphone cameras. Thus, in this paper, aim break through these challenges and introduce the first color video SCI reconstruction method, as illustrated in Fig. aa).
captured using smartphones equipped with quad-Bayer
When incorporating quad-Bayce in color video SCI reconstruction tasks, two challenges arise. One how to efficiently manage quad-Baye color video processing with reduced computational complexity. Current color video SCI reconstruction methods encompass model-based 13
18 19),
21.. However, model-based and 23 24]. which have not been extensively explored for Baud-Bryrr fields regarding performance and model size trade-offs, leading to suboptimal reconstruction outcomes and inefficiencies. On the other hand, existing E2E methods, primarily transformer-bace require significant parameters and computational resources, making it difficult process long video sequences effectively. The high computational demands transformers and the absence of a global attention mechanism in CNNs hinder their scalability modern, lightweight architectures. Leverag- ing the advancements in State Space Models (SSMs) [25] 26 27), modern SSMs like Mamba [28] effectively capture long-range dependencies while maintaining linear complexity relative to the input size. Furthermore, numerous experiments have illustrated that image-based based Mamba 229 30] achieves promising results and can match the performance of existing transformer 31 32 33 and CNN 35 36) models with smaller parameters. Therefore, we focus processing. This approach aims to create a lightweight design suitable for deployment mobile devices. By leveraging the Mamba-UNo framework (37 38
iteration-basee based 14 15), and End-to-End (E2E) approaches 220
iterative methods necessitate corresponding demosaicing models 6
22
(17) and hybrid CNN-transformer [16] approaches, typically
have demonstrated the ability
on exploring multi-scale reconstruction capabilities of Mamba-UNet for
we can reduce parameters and FLOPS while
still achieving state-of-the-- performance.

---

Projection Layer
Connection
Bementwtte
X mddiglcation
exctivation
Recorstruction
DWConv
RaConv
STiMarbba
Quad Bayer Measurement
Initaaization Block
Mamba5CI
Encoder
Decoder
Laverhorm
M gatonaz
Xuut ER R
RSTMaMbb
Bock
Ball
Upfomple
EDR
&
Projection
Manks
(oQuuedBlayee Petterned Color Video SCI Reconatrurtion Prorerr
G0
10
00
(Camiid
3SM
DoenGample
DiKCam
LayerNorm
Encoder
Decoder
Chainell Attention
Rerpe
KSTManbiu
Block V
57Mamba
(0) MamibaSC
AS7Mamba
Figure 3: The proposed MambaSCI network architecture and overall process color video reconstruction. (a) Quad-Bayer patterned color video SCI reconstruction process. feeds quad-Baye pattern measurement Y and masks M into the initialization block to get X,n and inputs it into MambaSCI network to get the reconstructed RGB color video Xout. (b) The overall network architecture of the proposed MambaSCI network. (c) Structure of Residual-Mamba-Bloek (RSTMamba) with ST EDR, and CA modules connected via residuals. The detailed design of EDR and CA is shown in Fig. + (d) STMamba. It captures spatial-temporal consistency via structured SSMs that enable parallel scanning in the spatial forward- warkwadd and temporal dimensions.
The second challenge is how to eliminate motion artifacts to ensure clarity in dynamic video. Motion artifacts may arise from handheld camera instability and pixel merging in quad quad-Bayer patterns. To pre- serve scene clarity and edge details, we have customized the Residual-Mambat Block, which integrates three key components: the -emporll Mamba (STMamba), the Edge-Detale Detail-Recoosttructin module (EDR), and the Channel Attention module (CA) [41 42). The Residual- Maaall-mmb-BBiie enhances long-range spatiotemporal dependence by leveraging residual connections and learning scales. STMamba replaces the self-attention module by performing parallel scanning of spatial and temporal dimensions, ensuring spatiotemporal consistency and enabling visual reconstruction of videos in a more lightweight manner. The EDR module enhances boundary sharpness perception and restores edge details lost due to compression and motion artifacts. By combining linear transformation with DWConv features, it integrates both local and global information. This approach strengthens the global features extracted by STMamba, fusing them with local details more effectively capture complex edge structures. As result, the module enables high-quality video reconstruction with improved edge clarity. The CA module compensates for the overlooked channel interactions in Mamba by weighting features based on channel importance, reducing the impact of artifact noise on reconstruction and thus improving the clarity of reconstruction results.
further
Building upon the foundation modules discussed earlier, we introduce a novel model called Mam- baSCI, which serves the reconstruction algorithm for our proposed muad-Bayer-basee SCI method as illustrated in Fig. (la). MambaSCI adopts a non-symmetric U-shaped encoder-decode architecture with skip connections to enhance model efficiency. To incorporate spatial-temporal consistency at multiple scales, we introduce employ residual convolution in the decoding stage reduce both parameters and computational complexity. The effectiveness of the MambaSCI model is demonstrated in Fig. [bb), where it achieves superior performance compared to the SOTA methods while requiring fewer parameters and FLOPS. Pseudo-code detailing the core modules is provided in the supplementary materials.
at each encoding stage. Furthermore, we
In short, our contributions can be summarized as follows:
(i) We are the first to introduce Quad-Bayer CFA pattern into color video SCI to accommodate that most videos captured by mobile photographers using quad-Bayer patterned smartphone cameras.
(ii) We are the first to use Mamba model for video SCI reconstruction. By integrating Mamba with non-symmetric UNet, we employ hierarchical encoder to capture spatial-temporal correlations at various scales, thus accelerating the model and enhancing reconstruction quality.
(iii) We customize the Residual-Mamber Block, integrating STMamba, EDR, and CA modules with residual connections and learnable scales enhance reconstruction quality and edge details.
(iv) MambaSCI outperforms other SOTA methods, requiring fewer parameters and FLOPS, and delivers superior visual results on 6 simulation color videos and large-scale simulation color videos.
3

---

2 Related Work
2.1 Mathematical Model for Color Video SCI
For color video SCI systems, the original T T-faame input video E R x3xI is given, along with the mask M E R#xWxT where H, W and T denote color rideo's height, width and number of frames, respectively. As in previous Bayer-based approaches, since each pixel captures only the red (R), green (G) or blue (B) channel of the raw data in a spatial layout, both X and M are divided into four parts obtain four sub-measurements {Y", Y Y Y ER
="=X O M
Y
X M
Y X M
Y'-X'M M.. (1)
Some methods 14 [5] process the four uub-meesurements individually, restore them to RGB color by an off-the-shelf demosaicing algorithm and finally combine them to get the final reconstructed video, which is inefficient and cannot make good use of channel correlation. Thus others [I6] 17 input all uub-measurements into their proposed reconstruction network simultaneously, and finally obtain the final RGB color video by convolutional network. Similarly, we first obtain four sub- mebbueessseeents based on the spatial layout of the quad Badd-Bayer array shown in Fig. aa), and then feed them into the network simultancously.
Following [ 43), we denote the vectorized measurement y E RWW, Then given vectorized color video pWTT and mask & HR WWWT. the degradation model can be formulated
2x + n.
(2)
where E R RWW represents noise measurement. SCI reconstruction is obtain from captured However, for quad-Bayer color videos, demosaicing algorithms bring artifacts, and Bayer pattern-based convolutional reconstruction causes color distortion, significantly affecting quality.
y and the pre-set o using reconstruction algorithm [4
13
16
2.2 State Space Models (SSMs)
systems that map 1D sequence ar(t IR to (t) E ER through hidden state h(t) E RV the process can be expressed as follows:
SSMs common treated as linear
(()) Ah(() Br((), v() = Ch(t),
(3)
where A E NXN BERNII C IRINN It is common to use the zero-order hold (ZOH) method discretize the continuous system, thus Eq. can be discretized following:
h4 Ah + Br
A AA
(4)
yk Chi
B= ae IBA
which uses timescale parameter A to convert continuous and to discrete À and B.
2.3 SSMs for Visual Applications
The 1D $4 model 455 is extented to handle multidimensional data, while TranS4mer model [46 optimizes movie scene detection by combining S4 with self-attention. Vision Mamba and Mam- balR (30 47 introduce SSMs into the vision domain generic backbones. U-Mamba [48] addresses long-range dependencies in biomedical images. Efficient medical image segmentation is achieved with lightM-UNet and UltraLight VM-UNet 40 ViVim 49 is proposed for effective and efficient medical video object segmentation.
These methods focus on (i) image restoration (spatial information), (ii) video understanding (global features), and (iii) medical image or video segmentation (small resolution and frame count). However, they do not apply SSMs to video SCI, a task requiring spatial-temporal consistency and detailed feature reconstruction for high-resolution, long-frame videos. Therefore, there is an urgent need to explore SSMs performance and efficiency in long-sequence problems such as video SCI.

---

3 The Proposed Method
In this section, we introduce our proposed MambaSCI network framework, and detail our customized Maetuaa--Mmaa-BBBae which is capable of capturing long-range spatial-temporal consistency along with edge detail sharpness reconstruction and channel information interaction, thus being able to outperform SOTA results with fewer parameters and FLOPS. Fig. 3 illlsstaaees the reconstruction process, network framework and details of core modules.
3.1 Architecture Overview
Given the 2D measurement Y E BB and mask M RB through the general initialization module of SCI, the general initialization module of SCI provides an initial raw quad- reconstruc- tion video i E HR xxxT This serves as the input to the MambaSCI reconstruction network, which outputs the final color reconstruction video Xout E "X""xxxx"" MambaSCI comprises five main components: (i) shallow feature extraction block, (ii) encoder layer, (iii) bottleneck layer, (iv) decoder layer, (v) color video reconstruction block. When Xin is feed into MambaSCI, it first goes through shallow feature extraction block, which includes depthwise separable convolution (DWConv), producing the shallow feature E Hx xCxT, Next sequentially passes through three encoder layers, each composed of Residual-Mamba-Bie
and Max-Pooling operation, re-
sulting the feature F R111/22x((/2)xxxxx22>>>> where E {1, 2,33. After these encoding layers, the deep feature F E R @xxWxxxx is obtained, with H H W = W bottleneck layer, composed Residual-Mamba- Mamba-Blocks, keeps the feature's 's shape unchanged. Then, through each decoding layer, which includes residual convolutions and upsampling operations, the feature transforms into F di x2"))22))xx x=))x(((2))))x Eventually, feature F d3 is fed into the color video reconstruction block to obtain the reconstructed color video Xout E R HgHxWxx x3xT" See Fig. b) for overall view.
8 and Č = 8 x C. The
3.2 Residual-Mammba-Blecd
Within each encoder layer, we incorporate N DesiddullMamma--Blockk,, specifically designed to capture and enhance temporal-spatial coherence across multiple scales, resulting in more accurate and comprehensive feature representations. As depicted in Fig. 3(c), each Desidual-MammaaBBlos consists of three key components: (i) the STMamba module, (ii) the EDR module, and (iii) the CA module. These components are detailed below, and the overall process can be mathematically described as follows:
F STMamba(LN((")) + F 81,
F Projection (EDR((N(()))())) + F -2))
(5)
F CA(LN(F(2) F2 883
where LN represents LayerNorm and / E []. F' is the le block's input feature and Flut can be treated as F(+1 block's input, $1, $2, 83 represent the learnable scales in the residual connection.
(i) STMamba module. Previous video SCI reconstruction algorithms typically compute attention separately for the temporal and spatial dimensions, then fuse them using a residual network (50] However, this approach may lack temporal-spatial consistency. To address this issue, we employ the STMamba model [49], which integrates Specifically, As illustrated in Fig. d(), the input F E 12 xX XCXT is processed in two parallel branches. In the first branch, F is expanded to Ć = S C channels via linear layer, resulting in X E xxXXX where S is expansion scale. Then, X unfolded along frames T to form X. E (T(H))xC, Forward and backward scanned features, Xsʃ and Xab, are obtained by scanning X, in both directions, efficiently capturing spatial dependencies. Simultaneously, sequence X, (HHYTX is generated to explore temporal dependencies by forward scanning each pixel across the T frames.
information through structured SSMs.
STMamba utilizes parallel SSMs to capture intra-frame and inter-frame correlations, enforcing time-space consistency constraints. This approach enables STMamba to capture both the spatial features within frames and the temporal dependencies between frames, accurately modeling dynamic changes in video data, facilitating efficient spatial-temporal feature extraction and preservation of
5

---

Table 1: Reconstruction quality and computational complexity for different versions MambaSCI.
Method
#Channel
Block
PSNR (dB)
SSIM
Params (M)
FLOPS (G)
MambaSCI-T
[2,4,4,6]
32.13
0.919
1.61
165.47
MambaSCI-S
10
[2,4.4.6]
34.53
0.950
247.53
MambaSCT-B
[2,4,4.6]
35.70
0.959
6.11
556.89
consistency. As shown in Fig. 3 d), the process can be formulated as:
X = SiLUULinear (F));
Z SiLU (F)),
X. X; = Unfolding(X))
X. LN [Farward-SSM([(CCccd((3]]
(6)
X,6 LN (BaccBaard-SSSM. (MncwaardSSSM(Conld(XXX..
X; LN (Forward-SSM(Coomm
F sem = Linear [x,
@ Z x @ Z + X @ Z),
where C represents Hadamard product, LN represents LayerNorm and SiLU is activation function.
(ii) EDR and CA module.
To achieve the fine reconstruction of edge details and to compensate for the missing inter-channel interaction capability the Mamba model, we introduce the EDR and CA modules. The EDR module consists of linear layers and depthwise separable convolution (DW- Conv). By combining linear transformation with DWConv features, the EDR module gains the ability to perform multi-scale feature fusion, allowing the network extract global information from local features and process both global and local information simultane- ously. This enhances the model's capacity to understand complex edge structures. Additionally, through adaptive weight initialization, the model can capture finer details more effectively in the early stages of training, enabling efficient edge detail reconstruction in lightweight manner while enhancing image edge information.
AugPooling
Linear
Conväd
Conuild
DWCCom
GELU
ReiUU
GELU
Corwid
Convöd
Linear
(0.1)
sigmod
(a) EDR
(6) CA
Figure 4: Detailed design of EDR and CA module.
On the other hand, the CA module compresses the spatial dimension of the feature map to 1 x 1 through an average pooling operation, which turn maps the compressed features to the interval (0, 1) through a convolutional layer with an activation function to form channel attention weights. These weights are subsequently multiplied Shannel-by-chamde implement the channel attention mechanism. In addition, the CA module utilizes the convolution operation to further facilitate the fusion and exchange information between channels, thereby enhancing the interaction effect between channels.
with the original feature map to
3.3 Bottleneck and Decoder Layer.
Like transformer, Mamba encounters severe optimization and convergence challenges as network depth increases 44)) In the bottleneck layer, multiple Basidua--Mamba-Bloos are concatenated, keeping the same number feature channels and resolution. This maintains feature richness and clarity, enhances the model's spatial-temporal dependency capture, and improves MambaSCI's performance without significantly increasing computational burden.
Decoding layer decodes features and recover image resolution. It receives two inputs: F. R xw xCxT from the skip connection, which retains original spatial information, and F, pxxwxxxx from the previous decoding layer, containing higher-level spatial-temporal information. The decoding layer fuses features using clement-wise addition to enhance expressiveness. applies DWConv with residual concatenation, an activation function and an upsampling operation. This process produces output features with richer semantics and higher spatial resolution.
3.4 Color Video Reconstruction Block.
The color video reconstruction block reconstructs the desired video Xout E XgxxWx3xX x3xT Instead of the computationally intensive remosaicing and demosaicing of raw quad-Bayer images, we use

---

Table Comparisons between MambaSCI and SOTA methods on simulation videos. PSNR (upper entry in each cell), and SSIM (lower entry in each cell) are reported. The best and second-best results are highlighted in bold and underlined, respectively.
Method
Params (M) FLOPS (G)
Beauty
Bosphours
Ruuner
ShakeNDry
Traffic
Jockey
Avg
33.38
29.53
29.61
29.70
19.64
29.32
28.53
GAP-TY 13
0.965
0.904
0.872
0.884
0.625
0.885
0.856
32.47
27.45
28.66
26.93
20.56
31007
27.86
PaP-FFDeatt-rayy
0.958
0.883
0.864
0.832
0.686
0.9.6
0.855
34.29
33.07
34.18
30.11
23.74
32.70
31.35
*nP-FastDVDDggm
15
0.947
0,928
0.883
0.811
0.921
0.909
19:47
26.88
34.26
24.13
25.98
30.41
26.86
RfficientSCI-S
16
2.21
1434.18
0.402
0.642
0.906
0.639
0.761
0.788
0.690
36.40
24.52
36.34
34.73
26.63
35.52
32.35
EfficientSCI-B
16
8.83
5701.50
0.980
0.497
0.919
0.955
0.774
0.945
0.845
23.15
23.75
34.36
24.78
26.17
30.13
27.06
STFormer-S
1.23
769.23
0.679
0.435
0,885
0.659
0.771
0.785
0.703
36.69
23.84
37.13
34,83
26.62
35.80
32.48
STFomer-B
19.49
12155.47
0,981
0,446
0.927
0.955
0.791
0.952
0.842
33.45
35.07
35.03
31.81
25.31
32.09
32.13
MamhaSCI-T
1.61
165.47
0.965
0.963
0.926
0,912
0.843
0.909
0.919
36.12
37.33
38.35
33.72
26.70
34,98
34,53
MambaSCI-S
2.47
247.53
0.978
0,976
0.968
0.943
0.886
0.951
0.950
36.95
38.62
40.02
34.55
27.52
36.54
35.70
MambuSCI-B
6.11
556.89
0.979
0.982
0.977
0.950
0.904
0.960
0.959
three-layer convolution (kernel sizes xx,, output and obtain the final RGB color video.
3
3x3x3, and 1x1x1 1) to process the decoding layer's
3.5 Network Variants and Computational Complexity
To balance size and performance. we pro- pose three versions of the MambaSCI model: MambaSCI-T (tiny), MambaSCI-S (small), and MambaSCI-B (base). Tab. shows the network hyperparameters, model parameters, and com- putational complexity (FL.PPS). By varying the number of channels from the initial DWCony, our method achieves significantly lower complexity than EfficientSCI [16] and STForme [17]
Table 3: Computational complexity of several SOTAs.
Method
Computational Complexity
STFommr
6HWTC* + G.. HWTC + HWT"C HWTK + HWTC* + HWT SHWTCN + 22WTTN2
EfficientSCT
MambaSCI
We also calculate the computational complexity the attention module n MambaSCI compared to other SOTA methods, shown in Tab. 3] where C' is the number of input features. K represents the kernel size, G₁, and G are the spatial size of local window in Swin tranftranfformr [51]. N is a fixed parameter in Mamba set to 16. In ManbaSCI, input features are unfolded into sequence E RCXT As seen in Tab. 3 while STFormer and EfficientSCI scale linearly with the spatial size (HW), their complexity grows quadratically with video frames T and C, which is typically 64 or larger, resulting in high computational costs. MambaSCI scales linearly with the entire video sequence (HWT)) and C, which capped 64 in MambaSCI-B, enabling efficient reconstruction of longer video sequences. Inference time comparisons are shown under various methods in Tab, 5
4 Experiment
In this section, we evaluate MambaSCI against SOTA video reconstruction methods multiple simulation datasets using PSNR, SSIM metrics, and visual comparisons.
4.1 Experimental Setup
Following STFormer and EfficientSCI, we use DAVIS2017 52 with resolution 480x894 (480p) as the model training dataset. To verify model performance, we test MambaSCI several
7

---

Ground Truth
GAP-TV
PnP-Fastie Dnet
STFormer-B
EfficientSCI-B
MambaSCI-B
28.53 da
24.52
30.02
0.504
29.01 de
34.10
37.13
36.34 dB
0.372
0.929
-..927
-0.919
99977
0
0.791
0.004
0-921
Figure 5: Visual reconstruction results of different algorithms on middle-scale simulation color video dataset (Bosphrous #10, Runner #1 Traffic #32 and Jockey #24 in order from top to bottom). PSNR/SSIM is shown in the upper left corner of each picture.
simulated datasets, six benchmark mid-scale color datasets [5] (Beauty, Bosphorus, Jockey. Runner, ShakeNDry and Traffic of size 512x512x3x32), and four benchmark large-scale color datasets [15] (Messi, Hummingbird, Swinger and Football). Since there currently no real color video SCI dataset based on quad-Bayer pattern, our method is not tested on real datasets.
4.2 Implementation Details
We use PyTorch framework training on NVIDIA RTX4090 GPUs and use random flipping, scaling, and cropping on DAVIS2017 for data augmentation. We use randomly generated masks as training input to enhance model robustness and optimize the model using the Adam (53) optimizer. Since MambaSCI is flexible in input size, first train for 100 epochs with spatial size of 128 x 128. Then, we train for 50 epochs at learning rate of 0.0001, followed by fine-tuning on x56x255 data learning rate of 0.00001 for additional 50 epochs.
learning rate of 0.0005 data
4.3 Results on Middle-scale Simulation Color Video
To test the performance of method for color video reconstruction, perform experiments 32-frame simulation color RGB video dataset with size of 512 x 512 x 3 x 32. We compress the color video with compression rate of B 8 and capture quad-Bayer pattern measurements using camera with Bayd- CFA pattern.
Since all current color video SCI reconstruction algorithms are designed based Bayer pattern, which cannot be directly applied to quad-Bayer pattern. Meanwhile, re-training an E2E model requires much training time and memory. Thus we only re-train two of the latest SOTA E2E models (S (STorrmer [17]. EfficientSCI [6] with quad-Bayer pattern. We compare with iterative optimization algorithm (GAP-TV [33,, PnP algorithms (PnP-FFDnet [14] and PnP-FastDVD [5] and E2E algorithms (STFormer IV and EfficientSCI [66).. The number of parameters, FLOPS, and
reconstruction results shown in Tab. 2
Notably, there no readily available quad-Baye demosaicing package. Therefore, for model-based and PnP algorithms, we first upsample the reconstructed raw mayd-Bayer video X x xIxT

---

Ground Truth
PnP-FastDVDnet
EfficientSCI-S
MambaSCI-s
ND
PSNR SSIM
34.76 dB / 0.959
27.10 d8 0.754
36.31 dB / 0.976
PSNR/ / SSIM
28.60 dB 0.887
20.90 dB 0.588
29.78 dB 0.920
PSNR/ SSIM
32.11 dB/ 0.867
30.73 dB / 0.815 30.15 dB/ 0.811
Figure 6: Visual reconstruction results of different algorithms on large-scale simulated color video dataset (Footbal #11, Swinger #1, and Hummingbird #40 from top to bottom). PSNR/SSIM is under each image.
according to the quad- Baader CFA pattern to obtain a thrce-channel video E [2lxxx3xxx We then demosaic it using the BJDD (54] algorithm to get the final RGB color video. See supplementary materials for details. Visual comparisons of the reconstruction results are shown in Fig.
We summarize our observations: (i) Our MambaSCI model significantly outperforms SOTA methods with lower computational and memory resources. For instance, MambaSCI-B surpasses STFormer-B B by 3.22 dB, using only 31 % (6.11/ / 19.49) of the Params and 4.5% %.5%% (556.89) / 12155 47) of the FLOPS. also outperforms EfficientSCI-B by 3.35 dB with just 69% Params and 9.8 % FLOPS. Additionally, both MambaSCI-S and MambaSCI-T achieve better results than STFormer-S and EfficientSCI-S with fewer Params and FL.PS.. Fig. [b) shows MambaSCI's superior PSNR-FL -LOSS performance with fewer resources.
(ii) In visual comparisons, GAP-TV and PaP-based methods exhibit artifacts, while STFormer and EfficientSCI suffer from color distortions likely because their reconstruction modules being designed for Bayer pattern and not compatible with Quad- pattern. None of these methods achieve high-quality reconstruction. MambaSCI, however, eliminates artifacts and achieves high-quality reconstruction with accurate color fidelity.
In addition, our proposed model demonstrates SOTA re- construction performance even at higher frame rates and compression ratios. We tested it on beauty data across various compression ratios (B = 8, 16, 32), with results detailed in the Tab. 4 Notably, our method requires only 9.8% of the FLOPS needed by EfficientSCI, while also significantly outperforming
Table 4: Performance analysis at B=16 and cases.
Params
PuP-FFDnet
21.85
0.767
16
STFamner
19.49
24311.76
EfficienSCC
883
11406.23
25.35
0.656
MambaSCI
6111
1113.78
25.39
0.817
the SSIM.
PnP-FFDnet
1.82
0.496
32
STFanmer
19.49
00M
EfficienSCC
8,83
22825.34 34
23.24
0,63
4.4 Results on Large-scale Simulation Color Video
MambaSC1
Similar to previous studies, we conduct experiments on a large-scale color video dataset. Due to the significant time and memory required to retrain existing E2E models for Bayer patterns, we only compare with SOTA model-based methods (GAP-TV, PnP-FFDnct, PoP-FastDVDnet) and retrain STFormer-S and EfficientSCI-S for the quad-Bayer pattern. Tab. 5 shows the comparisons on PSNR and SSIM, and Fig. provides visual comparisons.
We summarize the observations: (i) MambaSCI-S outperforms other methods in PSNR and SSIM on football and swinger, achieving over 1.5 dB higher PSNR on football. (ii) In visual comparisons, GAP-TV and PnP algorithms exhibit artifacts, while STForme and EfficientSCI suffer from color distortions. MambaSCI excels in reconstructing detailed information, resulting in superior reconstruction quality. (iii) The poorer performance on Messi and Hummingbird may be due to faster, more detailed motions that strained by limited parameters and FLOOS.. However, the visual results remain superior to other SOTA methods.

---

Table 5: Comparisons between MambaSCI and SOTA methods on large-scale simulation videos. PSNR (upper), and SSIM (lower) are reported. The total time (minutes) taken to reconstruct 4 videos is under each method. The best and second-best results are highlighted in bold and underlined.
GAP-TV
IPP-FFDNET
PaP-FanDVDnet
STrormer-
EfficienSCCSS
MambuSCI-S
Dataset
Pixel resolution
(17.03)
(17.97)
(50.03)
(2.12)
(5.47)
(4.9%
25.00
28.62
29.17
17.77
18:45
26.36
Messi
1080 x 1920 x x 48
Hummingbind 1080 1920 40
29.33
29.72
32,11
31,96
30.15
30.73
0.886
0.924
0.867
0.886
0.811
0.815
24.92
26.72
28.60
20.10
20,90
29.78
Swinger
1080 x 1920 x 20
0.833
0.883
0.887
0.556
0.589
0.920
Foothall
1080 x 1920 x x 48
31.19
0.939
0,963
0,959
0.815
0.754
0.976
4.5
Ablation Study
We conduct ablation experiments to evaluate the effectiveness of each module in MambaSCI, Tab. presents the results, comparing reconstruction quality, Params, and FLOPP across different models. All experiments are performed on six color benchmark datasets.
STMamaba Block: We verify the impact of STMamba blocks on reconstruction qual- ity. As indicated in Tab. 6 replacing vanilla Mamba with STMamba boosts PSNR by 6dB, while Params and FLOPS remain unchanged. STMamba's ' linear scanning enables sat- isfying spatial-temporal consistency without notable increase in computational complexity.
EDR Block: Tab. 6 demonstrates that the EDR module can improve PSNR by approximately 4.3dB, dramatically im- proving the quality of the reconstruction. However, it will result in the increase of Params and FLOPS.
Table Ablation study on each major module.
Baseline STMamba EDR CA PSNR SSIM Params(M) FLOPS(())
24.18
0.811
0.28
35.36
30.31
0.897
0.28
35.36
34,66 0,953
2.53
235.47
35,70 0.959
6.11
556.89 89
CA block compensates for the lack of channel information interaction in Mamba model, achieves the mod- elling of channel importance and improves the reconstruction quality through channel attention mechanism. Tab. 6 shows CA block can significantly improve reconstruc- tion quality. However, the CA module's multiple Conv3d operations result in a no- table increase in parameters and FLOPS, which is aspect to optimize in future work.
CA Block:
Number of Channels: Tab. iilustrates that the only distinction among differ- ent MambaSCI versions is the varying number of channels. Through experi- mentation, we observed that Params and FLOPS are significantly influenced by the number of channels, rather than the number of of Residual-Mamble Mamba-Bllocss prolongs both training and inference times, highlighting the need for trade-off in the current setup.
Table 7: Ablation study Residual- Mamba-Block
Models
PSNR (dB) SSIM Parass (M) FLOPS (G)
w/o learaalee scale
35.33
0.955
6.11
556.89
w/o residual connections
34.71
0.953
6.11
556.89
Residuul-STMMama-BBllc
35.70
0.959
6.11
556.89
Moreover, an excess
Residual Connection and Learnable Scales: The Residual-ST Block is the core customised module of our MambaSCI. Tab. is experimentally demonstrated that the residual connections and the learnable scales are effective in improving the reconstruction quality enhancement.
5 Conclusion
In this paper, we introduced quad-Bayer pattern into video SCI for the first time, enabling SCI to align with the fact that most current videos are captured by mobile phones with quad-Bayer cameras, thus avoiding artifacts and color distortions caused by existing algorithms. Specifically, we integrate Mamba model with an asymmetric UNet in video SCI, leveraging Mamba's s linear complexity and the speed improvements from the non-symmetric architecture for efficient SCI reconstruction. Moreover, we customized Boatttaa---aaa----ee residual connections, ensuring efficient spatial-temporal consistency and detailed reconstruction. Experimental results on simulated color video datasets highlighted that MambaSCI outperformed SOTA methods with fewer parameters, lower computational complexity and better visual effects.
to connect STMamba, EDR, and CA modules through
10

---

Acknowledgments
This work was supported in part by the Shenzhen Science and Technology Innovation Committee under Grant JSGG2022083 104402004 and by Guangdong Major Project of Basic and Applied Basic Research under Grant 2023B0303000000.
References
[1] Irina Kim, Seongwook Song, Soonkeun Chang, Sukhwan Lim, and Kai Guo. Deep image demosaicing for submicron image sensors. Electronic Imaging, 32:1-12, 2019.
[2] Haijin Zeng, Kai Feng, Jiezhang Cao, Shaoguang Huang, Yongqiang Zhao, Hiep Luong, Jan Aelterman, and Wilfried Philips. Inheriting bayer's legacy-joint remosaicing and denoising for quad bayer image sensor. IJCV, 2024.
[3] Jun Jia, Hanchi Sun, Xiaohong Liu, Longan Xiao, Qihang Xu, and Guangtao Zhai, Learning rich information for quad bayer remosaicing and denoising. In ECCV, pages 175-191. Springer, 2022.
[4] Assaf Lahav and David Cohen. Color pattern and pixel level binning for aps image sensor using 2x 2 photodiode sharing scheme, August 10 2010. US Patent 7,773.138.
[5] Trina Kim, Seongwook Song, Soonkeun Chang, Sukhwan Lim, and Kai Guo. Deep image demosaicing for submicron image sensors. Electronic Imaging, 32:1-12, 2019.
[6] Jungwoo Kim and Min Kim. Joint demosaicing and deghosting of time-varying exposures for single-shot hdr imaging. In ICCV, pages 12292-12301, 2023.
[7] Bolun Zheng, Haoran Li, Quan Chen, Tingyu Wang. Xiaofei Zhou, Zhenghui Hu, and Cheng- gang Yan. Quad bayer joint demosaicing and denoising based on dual encoder network with joint residual learning. In AAAI, volume 38, pages 7552-7561, 2024.
[8] Yitong Jiang, Inchang Choi, Jun Jiang, and Jinwei Gu. Hdr video reconstruction with tri- exposure quad-bayer sensors. arXiv preprint 2rXiv:2103.10988 2021.
[9] Jungwoo Kim and Min H Kim. Joint demosaicing and deghosting of time-varying exposures for single-shot hdr imaging. In ICCV, pages 12292-12301, 2023.
[10] Haechang Lee, Dongwon Park, Wongi Jeong, Kijeong Kim, Hyunwoo Je, Dongil Ryu, and Se Young Chun. Efficient unified demosaicing for bayer and non-bayer patterned image sensors. In ICCV, pages 12750-12759, 2023.
[11] Xin Yuan, David J Brady, and Aggelos Katsaggelos. Snapshot compressive imaging: Theory, algorithms, and applications. IEEE SP, 38(2):65-83) 2021.
[12] Patrick Llull, Xuejun Liao, Xin Yuan, Jianbo Yang, David Kittle, Lawrence Carin, Guillermo Sapiro, and David J Brady. Coded aperture compressive temporal imaging. Optics express, 21(9):10526-1053 10545, 2013.
[13] Xin Yuan. Generalized alternating projection based total variation minimization for compressive sensing. In 2016 IEEE ICIP, pages 2539-2544 IEEE, 2016.
[14] Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai. Mlug-and-pl algorithms for large-scale snapshot compressive imaging. In CVPR, pages 1447-1457, 2020.
[15] X Yuan, Y Liu, J Suo, F Durand, and Q Dai. Plug-and-plas algorithms for video snapshot compressive imaging. IEEE TPAMI, 2021.
[16] Lishun Wang, Miao Cao, and Xin Yuan. EfficientSCI: Densely connected network with space-time factorization for large-scale video snapshot compressive imaging. In CVPR, pages 18477-18486, 2023.
[17] Lishun Wang, Miao Cao, Yong Zhong, and Xin Yuan. Spatial-temporal transformer for video snapshot compressive imaging. IEEE TPAMI, 45(7):9072-9089, 2023,

---

[18] Yang Liu, Xin Yuan, Jinli Suo, David J Brady, and Qionghai Dai. Rank minimization for snapshot compressive imaging. IEEE TPAMI, 41(11):2990-3006, 2018.
[19] Jianbo Yang. Xuejun Liao, Xin Yuan, Patrick Llull, David J Brady, Guillermo Sapiro, and Lawrence Carin. Compressive sensing by learning gaussian mixture model from measurements. IEEE TIP, 24(1):106- 119, 2014.
[20] Ziheng Cheng, Ruiying Lu, Zhengjue Wang, Hao Zhang, Bo Chen, Ziyi Meng, and Xin Yuan. BIRNAT: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In ECCV, pages 258-275. Springer, 2020.
[21] Chengshuai Yang, Shiyu Zhang, and Xin Yuan. Ensemble learning priors driven deep unfolding for scalable video snapshot compressive imaging. In ECCV, pages 600-618. Springer, 2022.
[22] Yanchen Dong. Ruiqin Xiong, Jing Zhao, Jian Zhang, Xiaopeng Fan, Shuyuan Zhu, and Tiejun Huang. Joint demosaicing and denoising for spike camera. In AAAI, volume 38, pages 1582-1590, 2024.
[23] Haechang Lee, Dongwon Park, Wongi Jeong, Kijeong Kim, Hyunwoo Je, Dongil Ryu, and Se Young Chun. Efficient unified demosaicing for Bayer and Non-Bayer patterned image sensors. In ICCV, pages 12750-12759 2023.
[24] SMA Sharif, Rizwan Ali Naqvi, and Mithun Biswas. Sagan: adversarial spatial-asymmetric attention for noisy nona-bayer reconstruction. arXiv preprint arXiv:21 2021.
[25] Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
[26] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur, Long range language modeling via gated state spaces. In ICLR, 2023.
[27] Jue Wang, Wentao Zhu, Pichao Wang. Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, pages 6387-6397, 2023.
[28] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00752, 2023.
[29] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu, Vmamba: Visual state space model. arXiv preprint 4rX1v:2401.10166, 2024.
[30] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09411 2024.
[31] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE TPAMI, 45(1):87-110, 2022.
[32] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pages 5728-5739, 2022.
[33] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. CVPR, pages 3202-3211, 2022.
[34] Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In ICCV, pages 9066-9075, 2019.
[35] Jianping Luo, Shaofci Huang. and Yuan Yuan. Vidco super-resolution using multi-scale pyramid 3d convolutional networks. In MM, pages 1882-1890. 2020.
[36] Ya-Liang Chang, Zhe Yu Liu, Xuan-Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In ICCV, pages 9066-9075, 2019.
12

---

[37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and cmmputer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 231-241. Springer, 2015.
[38] Zhaohu Xing, Tian Ye. Yijun Yang, Guang Liu, and Lei Zhu, Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint (rXiv:2401.1356) 2024.
[39] Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmen- tation. arXiv preprint (ixix:2442.024999 2024.
[40] Weibin Liao, Yinghao Zhu, Xinyuan Wang. Cehngwei Pan, Yasha Wang, and Liantao Ma. Lightm-unet: Mamba assists in lightweight unet for medical image segmentation. arXiv preprint 20X1v:2403.05246, 2024.
[41] Renkai Wu, Yinghao Liu, Pengchen Liang, and Qing Chang. Ulralight vm-unet: Parallel vision mamba significantly reduces parameters for skin lesion segmentation. arXiv preprint arXiv:2403.20035, 2024.
[42] J Ruan, S Xiang, M Xie, T Liu, and MALUNet Fu. A multi-attention and light- unet for skin lesion segmentation. In BIBM, pages 6-8, 2022.
[43] Xin Yuan, Patrick Llull, Xuejun Liao, Jianbo Yang, David J Brady, Guillermo Sapiro, and Lawrence Carin. Low-cost compressive sensing for color video and depth. In CVPR, pages 3318-3325, 2014.
[44] Zhenghao Pan, Haijin Zeng, Jiezhang Cao, Kai Zhang, and Yongyong Chen. Diffsci: Zero-shot snapshot compressive imaging via iterative spectral diffusion model. In CVPR, 2024.
[45] Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV, pages 87-104. Springer, 2022.
[46] Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. Efficient movie scene detection using state-space transformers. In CVPR, pages 18749-18758 2023.
[47] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arX1::2402.15648, 2024.
[48] Jun Ma, Feifei Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv.2401.0472. 2024.
[49] Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: video vision mamba for medical video object segmentation. arXiv preprint vxXiv22401.14666 2024.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[51] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[52] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Corkine-Horuune and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint urXiv:1704.00 00675, 2017.
[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[54] SMA Sharif, Rizwan Ali Naqvi, and Mithun Biswas. Beyond joint demosaicking and denoising: An image processing pipeline for pixel-bin image sensor. In CVPR, pages 233 233-242, 2021.
[55] Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, and Lue Gool. Degradation-awar unfolding half-shuffle transformer for spectral compressive imaging. In NeurIPS, pages 37749-37761. 2022.
13

---

A Appendix
In the supplementary material, we provide more details that are not in out main paper:
(a) Mathematical model of color video CACTI in Sec. A.I
(b) The pseudo-code of Residual-Manba-Blode in Sec. A.2
(c) We apply demosaicing techniques to quad-Bay images reconstructed by GAP-TV, PnP- FFDNet, and MnP-FastDDDnet converting them into RGB images for visual comparison. Meanwhile, more visual comparison against the current state-of-art (SOTA) method both middle-scale and large-scale simulation color videos in Sec. A.3
(d) Limitation of our work in Sec. A.4
(e) Broader impacts in Sec. A.5
A.1 Mathematical model of color video CACTI
Coded aperture compressive temporal imaging (CACTT) is one of the famous video SCI systems. Specifically, for color video SCI systems utilizing quad-Bayer arrays, the raw data is spatially structured that each pixel captures only red (R), green (G), blue (B), creating a format similar to 'RGGB' where each color occupies adjacent 2x2 pixels in succession. Thus, the initial color video X E # x3x? is multiplied pixel by pixel with the filter and superimposed in the channel dimension, ultimately producing the raw data X E RR HxHxxxx WI'", Given the mask M HWWW modulation is:
x (=:()=xx
@ M(;;;A),
(7)
where X' denotes the modulated video data and represents element-wise multiplication. 2D compressed measurement Y can be captured across time dimention, which can be expressed as:
T Y
Σ *X, t=1
(8)
where E Hx means the measurement noise. Vectorization, Given:
= Yec E HW
(9)
vec(N) E IR HW
(10)
[x] .... xT HWT
(11)
4 [D,, Dr HWW.HWT
(12)
where vec(-) means vectorization operation, X, vec (((;;;;;)))))))))) f)) represents the vectorization of /-th
frame of X and D, = Dia f))) is a diagonal matrix and its diagonal elements is filled by vec(M[[; ()). The vectorization forward process can be expressed as:
Øx
(13)
And the reconstruction process is to investigate algorithms that can reconstruct given y and Φ.
A.2 Pseudo-code of Residual-Mamba-Bieee Block
As mentioned in the paper. Residual-Mantee -anbba-Blocks can complete high quality reconstruction by capturing spatial-temporal coherence while completing edge detail reconstruction and compensating for channel information interactions not available in the Mamba model. In Algorithm we show the PyTorch style pseudo-code on how construct a Residual-Mammb--Bloss
A.3 Additional Visual Results
In this section, we will demonstrate how we employ certain techniques to apply existing demo- saicing algorithms to the raw quad-Bayer images reconstructed by GAP-TV, PnP-FFDNet, and
14

---

Algorithm Pseudo-code of Residnal-Mamba-Block
class RSTMamba
def __.init__ input__ding output_ nframes, d_state = 16, d_conv expand 2, alp_ratio = act_layer Gn.UE)
(nperNorm(inputddee
## STMamba
self. = Mamba(
d_model input_dimg dim, ## Model dimension d_model d_state d_state,
SSM d_conv d_conv, Local convolution width expand expand, ## Block expansion factor nframes = nframes # Number of frames of video
expansion factor
self.proj Linear (input_diag output_dim)
self.scalei nn.Parameter (torch. ones (1))
self.scale2 an.ammmete (torch. ones (1))
self.scales Parameter (torch. ones (1))
self.norm2 LayerNorm CigeeNormiinpu_ddae
serm. LayerNerm httpllormoouuuuttdtdlt
## Channel Attention
self. ca CA (output_dim))
mlp hap_hidden_diil #* Edge Detail Reconstruction
immt (input_dim mlp_ratio)
self.edr Mppinnut__dim mlp_hidden_din drop)
def forward(self, x):
## x in shape [B, C, T, H, W]
B, C, nf, H, W shape n_tokens = shape [2:] .nmmll()
dime = shape [2:] x_flat r.r.peapeeeee (B, C, n n_tokens) transpose -2)
#s x_flat in shape [B, n_ n_tokens, C]
## (1) STNamba and residual connection
x m_mamba x_flat
sef wft.ggaaal.mmmammall.....c.
wam.a normi (x_ (x_flat)))
(2)
residual connection
x x_mamba x_mamba self
self www.ppppssel... wewreeeerrreeeeeeeeeeeeeeeeeeeee (self norm2 manaba),,,, nf, H, W))
s_mambna - self proj (x_mamba)
out out.uutpee (0,
## (3) and residual connection
out out self.skip_edd calf.ca(sele norm3 permute (0,4,1,2,3)) permute (0,2,3,4,1)
out out permute 40,4,1,2,3)
return out
DnD-FastDVDnet. This process converts the raw format images into RGB color images, facilitating visual comparison. We provide moving images in GIF format for the various methods in Tab. 2 Please refer to the folder 'gif" for further observation.
Demosaicing Process. As mentioned in the paper, previous model-based and PnP reconstruction algorithms need be paired with corresponding demosaicing algorithm get RGB color videos. However there are no available packages to use for to present the reconstructed color images also addressed. As shown Fig. a). it's video frame in raw format obtained after reconstruction, we use Algorithm. to transform it into the form of Fig. b)) and then use the off-the-shelf demosaicing algorithm get the final color video frame shown in Fig. c).
demosaicing, so the problem of how
15

---

Algorithm 2 Code of Transformation
mask CFA_qe int) (NpArray,,,,, ...I:
channels = {channel: : np. zerzero (shape, dtype="bool") for channel in range
## Quad (red)
channels [0] [:4]
channels [0] [1:14,1::4
channels [0] [::4,1::4]=1
channels [0] [1::4,::4] -1
## Quad G1 (grenn)
channels [::4,2::4] =1
channels [1] [::4]3::4] =1
channels [1] [1:14,2::4] -1
channels [1:14,3:14]
## Quad G2 (grenn)
channels [2::4,::4] =1
channels [2::4,1::4] =1
channels [1] [3:14,::4]
channels [1] [3:14,1::4] =1
## Quad (blue)
channels [2] [2: [4;::4]
channels [2] [3: :4;::;2::4
channels [2] [2:
channels [2] [3:14,3::4]
return tuple valiessslllvluueeeeeeeeeeaeaeeee values
def CFA_quad (CFA):
CFA uss_flatt_array(Cn (CFA)
R_m, G_m, B_m masks_CFA_Bayer (CFA. shape)
## obtain red channel
R =
CFA R_m
## obtain green channel
CFA G_m
## obtain blue channel
B CFA B_m
RGB tstack [B, G, R))
return RGB
(a) Reconstruction raw data
(b) Quad- Quad-Bayer pattern data
(c) Demosaicing RGB color data
Figure Process of reconstruction from Raw color RGB image.
Visual comparison. We conduct additional comparative experiments with current with current SOTA
methods and provide more visual comparison. As shown in Fig. Fig and Fig. 10 proposed
16

---

Ground Truth
GAP-TV
PnP-FFDNet
PnP-FastDVDnet
STFormer-B
EfficientSCI-B
MambaSCI-S
MambaSCI-B
Figure 8: Visual reconstruction results of different algorithms middle-scale simulation color video Traffic
#16.
Ground Truth
GAP-TV
PnP-FFDNet
PnP-FastDVDnet
STFormer-B
EfficientSCI-B
MambaSCI-S
MambaSCI-B
Figure 9; Visual reconstruction results of different algorithms on middlc-scale simulation color vidco Runner #7.
MambaSCI method achieves superior color fidelity and detailed reconstruction on middle-scale benchmark simulation color video dataset, offering significant visual advantage over previous methods. Meanwhile, as shown Fig. TT] even though our proposed MambaSCI method may not surpass EnP-FastDVDFnet in terms of PSNR and SSIM metrics, it achieve superior reconstruction visual quality. In comparison to the artifacts introduced by GAP-TV and PnP--aatDVDDFne as well as the color distortions in STFormer-S and EfficientSCI-S, MambaSCI demonstrate significantly better visual results. As shown in Fig. 12 and Fig. our proposed method achieve pleasant visual results.
17

---

Ground Truth
GAP-TV
PnP-FFDNet
DnP-FastDVDnet
STFormer-B
EfficientSCI-B
MambaSCI-S
MambaSCI-B
Visual reconstruction results different algorithms on middle-scale simulation color video Bosphrous #16.
Figure 10:
Ground Truth
GAP-TV
DnP-aatDDVDnet
1
23
23
23
ATAR
NALT
ATAR
WALT
ATAR ARRAAY
WAIT
AIRWAYS
Cempo
STFormer-S
EfficientSCI-S
MambaSCI-S
23
ATAR AIKWAYS
ATAR IRWAYY
ATAR IRWAYY
WAIT
Figure 11: Visual reconstruction results of different algorithms on large-scale simulation color video Messi #8.
A.4 Limitation.
The limitations of our approach are in two respects:
(i) One limitation is the trade-off between computational complexity and performance. As shown in Table to achieve better performance, we incorporate EDR and CA modules. However, the use of Conv3d in the CA module significantly increases the number of parameters and computational complexity.
(ii) The second limitation is that measurements based on quad-Bayer in real datasets are currently unavailable, making it impossible to evaluate the performance of our proposed model in real-world scenarios.
Given these limitations, we to investigate ways to simplify the internal modules while maintaining performance, thereby reducing computational complexity and accelerating inference speed. Addi- tionally, we will focus on collecting Badd-Bayr--basee SCI measurements in real- sal-world scenarios to verify the reliability of our method under real scene conditions.
18

---

Ground Truth
GAP-TV
PnP-FastDVDmet
STFormer-S
EfficientSCI-S
MambasCI-S
Figure 12: Visual reconstruction results of different algorithms on large-scale simulation color video Swinger
#30.
Ground Truth
GAP-TV
PnP-FastDDe FastttDDDDDet
NND
U
NND
U
NND
BI
STFormer-S
EfficientSC-SS
MambaSCI-S
07
LND
AND
NND
Figure 13: Visual reconstruction results different algorithms on large-scale simulation color video Football #38.
A.5 Broader Impacts
Video reconstruction a fundamental task in snapshot compressive imaging (SCI), an area with research history spanning several decades. As artificial intelligence continues evolve, the handling of high-quality, high-dimensional data has emerged as a significant challenge for large-scale deep learning models. Video SCI systems, which utilize low-speed cameras capture high-speed video, present several advantages including low memory requirements, low transmission bandwidth, low cost, and low power consumption [17 (55). Our proposed MambaSCI algorithm enables more efficient high-quality reconstruction of videos captured with quad-Bayer pattern, significantly broadening the application scenarios of video SCI.
To date, video reconstruction techniques have not demonstrated any negative social impact. Similarly, proposed MambaSCI algorithm does not present any foreseeable negative societal consequences.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We reflect the contribution and scope of the paper in detail in the Abstract and Introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation of work in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
20

---

Justification: We conduct experiments based on existing SCI theory and do not give complete mathematical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the detailed process of the network, and detailed parameter settings for the training part for reproduction in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We give the pytorch type pseudo-code of the core module in the supplementary material Sec. [.2] and provide the corresponding gif format dynamic image. The code will be publicly available soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips- cc/
mublicc/guudess/CdeeSttmmmss http://odddSbbbissssoooooooooooo osllclllicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//apps.cc/pulli//uiiiee
swssssssiiiiyyyys for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe all the training and testing details needed to understand the results in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The train/test split, initialisation in our approach does affect the overall run for a given experimental condition, while certain parameters are not randomly plotted.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the experimental platform GPU devices and memory, the number of model parameters, the amount of computation and the corresponding inference time in the paper Sec. 3.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the thesis complies all respects with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are in Sec. A.5
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our papers pose no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The creators original owners of assets used in the paper properly credited, and the licences and terms of use are are are clearly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not deal with crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not deal with crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Ql9ykkuKtA", "reviewer": "Reviewer_M9Ao", "summary": "This paper presents a method for compressive video image using Mamba-Unet for quad bayer sensors.", "questions": "What is the primary motivation to address quad bayer sensors when they are not that prevalent?\n\nWhat is the need of using Mamba-Unet for this work?", "limitations": "Limitations are not addressed well.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The work seems to be in a less explored area of research.", "weaknesses": "- The work is not making significant contribution in terms of method. Directly applying Mamba-Unet to this problem looks unnatural. Why not use a transformer or a CNN? I dont think there will be a much of a difference.\n\n- The writing needs improvement and section 3 is not that clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Response to Reviewer M9Ao**\nThank for your valuable comments.\n\n**Q1. Clarification of why Mamba and not transformers or CNNs.**\n\n**A:** We have added further discussion of this topic in lines 59-61 of main manuscript, with a more detailed analysis and experimental validation as follows:\n\nTransformer and CNN-based Video Snapshot Compressive Imaging methods have been proposed such as **transformer-based method STFormer** and the **hybrid CNN- and transformer-based method EfficientSCI**. However, existing methods still face two issues:\n\n1. **Technique perspective:** \n   - The high computational complexity of Transformers and the lack of a global attention mechanism in CNNs hinder their extension into modern, lightweight, and efficient network architectures. Therefore, we focus on exploring multi-scale reconstruction of Mamba-Unet into the Quad-Bayer pattern to achieve lightweight design and enable deployment on mobile devices.\n   - As shown in Table 2 of main manuscript, MambaSCI-B outperforms STFormer and EfficientSCI, using only **31%** of STFormer’s parameters and **4.5%** of its FLOPS, and **69%** of EfficientSCI's parameters and **9.8%** of its FLOPS.\n2. **Pattern perspective:** All existing video SCI methods are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these methods often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. Thus, we aim to solve these issues with two key contributions: **new task and new method**. That is, efficient and qualitative recovery of quad-Bayer arrays through the asymmetric Mamba-Unet framework. **To the best of our knowledge, we are the first one to develop the Quad-Bayer Patterned Video Snapshot Compressive Imaging task and the first algorithm to elaborate designs of Mamba-UNet to this task.**\n\n**Q2.  Clarification of the necessity and advantages for quad-Bayer .**\n\n**A:** Quad-Bayer arrays are prevalent and almost all current flagship smartphones cameras, such as the iPhone 14 Pro/Max, vivo X90 Pro+, Xiaomi 13S Ultra, and OPPO Find X6 Pro, utilize quad-Bayer arrays. We have added a description of the main applications where quad-Bayer is used and its advantages over Bayer in lines 35-38 of the main manuscript. Meanwhile, integrating quad-Bayer array into video SCI not only is necessary but also offers significant advantages over Bayer arrays:\n\n1. **Necessity:**\n   - **Widespread Use on Smartphones:** Quad-Bayer arrays are common in smartphone cameras [1], which are frequently used for video recording. Existing methods face color distortion and artifacts, which our work aims to provide a customized solution for Quad-Bayer color video SCI tasks to overcome these issues while offering space-saving, high-quality reconstruction through compression and lightweight construction on smartphones.\n   - **New Industrial Demands and Research Trends:** Bayer arrays have been widely used in industry and well-studied in academia due to their long history, performing adequately in bright scenes. However, the imaging capabilities in low-light conditions are limited. Therefore, exploring quad-Bayer arrays is essential to address these deficiencies.\n2. **Advantage:** Quad-Bayer arrays has the following advantages over Bayer arrays:\n   - **Higher Resolution:** Quad-Bayer provides higher resolution than Bayer arrays by realizing higher pixel density on the sensor for better detail reproduction compared to conventional Bayer arrays [2].\n   - **Low-Light Performance Enhancemente:** Quad-Bayer collects more light in low-light environments, improving the sensor's signal-to-noise ratio and reducing noise interference.\n   - **HDR Capability:** By adjusting quad-Bayer sub-pixels and setting different exposure values on different sub-pixels, high dynamic ranging (HDR) images or videos can be captured to improve the detail performance of highlights and shadows [3-4].\n   - **Color Accuracy:** Quad-Bayer pattern captures richer color information by sampling red, green and blue colors in four directions. This fine sampling reduces the likelihood of color distortion and improves the color accuracy and realism of the image [5].\n\n**Therefore, we have introduced quad-Bayer arrays to video SCI for the first time, providing a solution for reconstructing higher-quality videos even HDR videos in the future.**\n\n**Q3. Section 3 lacks clarity.**\n\n**A:** To address this, we have made the following changes to the paper:\n\n- **Top-Down Structure:** We use a top-down structure to introduce our framework, providing an overview followed by detailed component breakdowns. This approach ensures a clear understanding of the overall algorithm and the specific functions of each individual part.\n- **Clear Delineation:** We clearly delineate Section 3.2 to provide a detailed analysis of each module, including a thorough examination of the internal structure and a clear explanation of each module's function.\n- **Figure-Text Interaction:** Figures 3 and 4 in the main manuscript have been revised to align with the content and more effectively present the network framework and internal modules details.\n- **Contextual Analysis:** We ensured the model description is consistent with Section 4’s experimental setup and interacts with the Supplementary Material’s pseudo-code to aid reproduction.\n\n[1] Madhusudana P C, et al. \"Mobile Aware Denoiser Network (MADNet) for Quad Bayer Images\" *CVPRW* 2024.\n\n[2] Zheng B, et al. \"Quad Bayer Joint Demosaicing and Denoising Based on Dual Encoder Network with Joint Residual Learning\" *AAAI* 2024.\n\n[3] Kim J, Kim M H. \"Joint demosaicing and deghosting of time-varying exposures for single-shot hdr imaging.\" *ICCV* 2023.\n\n[4] Wu T, et al. \"High Dynamic Range Imaging with Multi-Exposure Binning on Quad Bayer Color Filter Array. \" *ICIP* 2023.\n\n[5] Lee H, et al. \"Efficient unified demosaicing for bayer and non-bayer patterned image sensors.\" *ICCV* 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "asODoDJzSd", "reviewer": "Reviewer_zshs", "summary": "This manuscript introduces the Mamba model and Quad-Bayer CFA pattern into color video snapshot compressive imaging (SCI) for the first time. Specifically, the proposed MambaSCI adopts a non-symmetric U-shaped encoder-decoder architecture, which includes DWConv, Residual-Mamba-Blocks, and ReConv. The Residual-Mamba-Blocks integrate several modules designed to enhance reconstruction quality and edge details. Experiments demonstrate that MambaSCI outperforms comparison methods in both quantitative and qualitative results, with fewer parameters and FLOPs.", "questions": "Please refer to weakness.", "limitations": "Please refer to weakness.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1.\tThis work introduces a method for quad-Bayer patterned color video SCI for the first time. Quad-Bayer sensors offer better hardware performance than Bayer pattern sensors and represent a promising direction worth exploring in SCI.\n2.\tThe Mamba model, as a popular module for extracting data causality, aligns well with SCI tasks. Its incorporation into SCI is well-justified.\n3.\tThe manuscript is generally well-structured and well-written.", "weaknesses": "+ The proposed MambaSCI takes X_in as input, which is obtained from initialization Block. So the framework can be seen as a simple video enhancement task. Furthermore, MambaSCI does not consider the characteristics of Quad-Bayer pattern like previous work (STFormer, EfficientSCI), but instead directly adopts or integrates popular modules.The in-depth analysis appears to be lacking in certain modules, specifically within the passages from lines 185-187 and 193-197.\n\t\n+The lack of an explanation for the initialization block raises questions about whether special processing was done to address issues introduced by the quad-Bayer pattern. It would be better to add further visualization of the X_in results for comparison.\n\n+In Figure 6, the results of proposed method show artifacts and color distortion in the lower left corner of ``Swinger”. More analysis should be included.\n\n+The manuscript lacks real experiments on videos captured by quad-Bayer patterns. All the results are obtained on  simulated data. Results on real data are expected to evaluate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer zshs\nThanks for your valuable comments.\n\n**Q1: Differences between the proposed method and the video enhancement-based ones.**\n\n**A:** **MambaSCI significantly differs from video enhancement technology in the nature of the task, input differences, and use of prior knowledge**, as detailed below:\n\n- **Nature of the Task:** Video enhancement tasks typically improve RGB video quality by addressing noise or missing frames. In contrast, the MambaSCI network addresses the more complex SCI task, which involves not only overcoming compression degradation but also handling the additional effects of the quad-Bayer array mask. Thus, the MambaSCI network has a dual mission: **accurate reconstruction of compressed data** and **conversion and demosaicing from the raw domain to standard RGB format**, requiring highly flexible and capable of managing the unique complexities of SCI technology.\n- **Huge Gap Between the Input of Two Tasks:** Same to previous models like STFormer and EfficientSCI, our MambaSCI reconstruction network also requires an initialization block for up-sampling, which is a crucial component of the SCI reconstruction task. However, the $\\mathbf{x_{in}}$ obtained after initialization differs significantly from traditional video enhancement inputs. We present $\\mathbf{x_{in}}$ after initialization in **Figure 1** in supplemental PDF, where it is evident that the degradation is much more severe than what common video enhancement networks can handle.\n- **Utilization of Different Prior Knowledge:** In the reconstruction process, MambaSCI leverages the unique priori knowledge of compression and quad-Bayer arrays, contrasting sharply with traditional video enhancement’s traditional focus on smoothness and edge preservation.\n\n\n**Q2: Clarification of the quad-Bayer pattern is not characterized in the same way as in previous work.**\n\n**A:** Both STFormer and EfficientSCI were designed as Bayer array-based video compression reconstruction methods, only considered Bayer pattern features in their initialization blocks. **In contrast, we have developed a specialized initialization block to address the unique characteristics of quad-Bayer physical arrays, and applied it specifically to GAP-TV, PnP-FFDnet, and PnP-FastDVDnet.** Details are as follows:\n\n- GAP-TV, PnP-FFDnet, and PnP-FastDVDnet use iterative optimization by dividing the raw domain into R, G1, G2, and B blocks for independent reconstruction. Similarly, we divided the quad-Bayer array into R, G1, G2, and B sub-blocks as shown in lines 103-107 in the main manuscript. However, due to the quad-Bayer pattern's 2x2 grid representation for each color, processing is significantly more complex than standard Bayer pattern, resulting in an increased processing time from 0.0015s to 7.3312s at 512x512 resolution compared to the SCI generic initialization method.\n- Leveraging the powerful modeling capabilities of the E2E model and considering the need for a lightweight design due to the limited computing power of future handsets, we selected the SCI generic initialization block for STFormer, EfficientSCI, and MambaSCI. Our experiments confirmed nearly no performance loss with this choice.\n\n**Figure 1** of our PDF submission provides a visual comparison of $\\mathbf{x_{in}}$ obtained using the SCI generic initialization block versus an initialization block designed specifically for the physical characteristics of a quad-Bayer array.\n\n**Q3: Enhancing in-depth analysis of certain modules.**\n\n**A:** We have added a deeper analysis of the EDR module, bottleneck layer, and the decoding layer in the following areas:\n\n- **More Detailed Analysis:** We have thoroughly analyzed the EDR module, detailing its inner workings and its critical role in our framework. Additionally, we have improved the presentation and analysis of the bottleneck and decoding layers.\n- **Figure-Text Interaction:** We have redrawn Figures 3 and 4 in the main manuscript and improved their integration with the text to facilitate better understanding and clarity for the reader.\n- **Corresponding to the Ablation Experiment:** The module description is linked to the associated ablation experiments, allowing the reader to intuitively grasp the module's important role in the network.\n\n**Q4:  Lack of real experiments on videos captured by quad-Bayer patterns.**\n\n**A:** The reasons for the lack of real experiments are as follows:\n\n- Since we are exploring a new task and a new discovery that is not currently being investigated, there are no publicly available datasets for comparison, as stated by reviewer **yWFS (there is no real SCI dataset based on quad-Bayer pattern)**.\n- To obtain the real-world datasets, a corresponding optical coding system must be assembled. We are working on building this system, but it will take six months to one year to complete, even with experienced personnel. Additionally, we need to upgrade the existing system by replacing the current modules with four-layer modules.\n- To demonstrate the advantages of quad-Bayer over Bayer, we plan to acquire more datasets from extreme scenes and collect HDR data. This will significantly expand the current research field of SCI.\n\n**Q5: Analysis of artifacts and color distortion in the lower left corner of \"Swinger\".**\n\n**A:** Thanks to your careful observation, we have added the visual comparison in **Figure 2** of the submitted PDF, showing different methods to locally zoom in on the lower left corner of the \"swinger.\" The intricate ropes in this area, as depicted in the GT, present a significant challenge due to their dense packing and rapid movement between frames, leading to ghosting artifacts in the reconstruction.\n\nTo address this limitation, we plan to refine our model further. In future work, we will focus on improving the model's ability to handle fast motion between frames, minimizing artifacts and achieving more accurate reconstructions in complex and dynamic scenes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LxDJuwGly", "reviewer": "Reviewer_xnVL", "summary": "This paper investigate video snapshot compression imaging reconstruction task by Quad-Bayer CFA pattern into color video SCI. They design a Residual-Mamba-Block consisting of ST-Mamba, Edge-Detail-Reconstruction module and Channel-wise attention module to enhance reconstruction quality and edge details. Experimental results on many datasets validate its effectiveness.", "questions": "Why we need Mamba instead of self-attention in this task?\n\nWhy quad-Bayer pattern is better than a single Bayer-patterned measurement?", "limitations": "The author is suggested to compare the efficiency of the proposed method and existing methods on long videos to validate the effectiveness on long sequence.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is interesting to introduce Mamba into a new task.\n\nThey investigate long sequence modeling to further validate the effectiveness of ST-Mamba.\n\nThey proposed a novel framework for video SCI reconstruction task, which outperform existing methods not only in effectiveness but also in efficiency.", "weaknesses": "The contribution (ii) and (iii) have limited novelty. It seems that the author simply adapt ST-Mamba to video SCI reconstruction task without elaborated designs. The proposed Residual-Mamba-Block is an incremental combination of ST-Mamba from Vivim, DWConv and Channel-wise Attention. The author should further clarify the novelty.\n\nMore description of EDR module should be given. Why EDR module can provide more edge details?\n\nObserved from Table 4, the improvements on large-scale simulation color video are not consistent across datasets. More analysis on such phenomenon should be given. While the author claim the results on Messi and Hummingbird is strained by parameters and FLOPs, specific quantitative comparison should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer xnVL\nThanks for your valuable comments.\n\n**Q1: Clarification of the novelty.**\n\n**A:** We have elaborated on the novelty in lines 160-167 and 289-296 of the manuscript. **Rather than merely adapting ST-Mamba, we introduced the following innovations in video SCI reconstruction:**\n\n- **High Performance Lightweight Network Design:** Although STMamba is an effective lightweight global attention mechanism approach, it still struggles with the high demands of SCI video reconstruction compared to Vivim's medical image segmentation. Our framework enables joint global-local reconstruction, addressing these challenges.\n- **Combined Global-Local Features.** DWConv efficiently extracts local features, avoiding the parameter and complexity issues of common 3D convolutions. **Meanwhile, it is not a straightforward incremental combination with STMamba.** By integrating linear transformations with DWConv features, the network integrates local and global information, enhancing the global features extracted by STMamba. This fusion is crucial for accurately capturing complex edge structures.\n- **Enhanced Channel Interaction:** Many Mamba-based frameworks overlook channel interaction limitations, leading to channel isolation and suboptimal information fusion, which adversely affects reconstruction quality. To address this, we introduce a lightweight channel attention mechanism to improve multi-channel perception and feature representation, refining reconstruction quality.\n\n**Q2: In-depth description of EDR module.**\n\n**A:** We have added a detailed analysis of the EDR module's design principles in lines 184-192, explaining how it enhances video reconstruction by improving the representation of depth features. To better illustrate the EDR module's operation, we redraw the relevant section of Figure 4 in the manuscript. Additionally, further analysis of edge detail reconstruction attributed to the EDR module has been added to the ablation experiment section, specifically in lines 297-299. We additionally provide a visual comparison of the edge detail effect with and without the EDR module on the final reconstruction result (**Figure 3** on pdf).\n\nSpecifically, key improvements with EDR include:\n\n- **DWConv :** This technique performs spatial convolution independently on each channel and then applies pointwise (1x1) convolution across channels. It effectively captures localized spatial features, enhancing edge detail perception.\n- **GELU :** Compared to ReLU, GELU handles negative inputs more naturally, improving the model's ability to represent fine details.\n- **Adaptive Weight Initialization:** Truncated normal initialization helps the model capture details more effectively during the early training stages.\n- **Multi-scale Feature Fusion:** Combining linear transformations with DWConv features allows the network to extract global information from local features. This simultaneous processing of global and local information is crucial for understanding complex edge structures.\n\n**Q3: Clarification the reasons for inconsistent improvement effects on large-scale datasets.**\n\n**A:** The inconsistencies may be attributed to:\n\n1. **Evaluation in Raw Domain:** As described in lines 494-500 of the Supplementary Material, we use additional demosaicing model to show RGB images reconstructed by GAP-TV, PnP-FFDnet, and PnP-FastDvDnet, which **ultimately reconstruct videos in the raw domain**. In contrast, three E2E methods reconstruct RGB video directly. For a fair comparison and excluding the influence of demosaicing, we assess performance in the raw domain. Thus, the RGB video from E2E methods is converted back to the raw domain using a quad-Bayer mask. Figure 6 in manuscript shows that while iterative methods perform better in metrics, they exhibit more visual artifacts.\n2. **Training Resolution:** The training resolution is 128x128 and fine-tuned at 256x256, which is a gap with large-scale datasets.\n3. **High-Speed Frame Reconstruction:** The effect of high-speed moving frame reconstruction is to be strengthened under the limitation of the number of parameters and computational complexity. The number of parameters and flops of the model are as follows. Even with the limited FLOPs, we still achieve better performance.\n\n|Method|Params|FLOPs|PSNR|SSIM|\n|-|-|-|-|-|\n|STFormer|**1.23M**|6084.61G|25.11|0.724\n|EfficientSCI|2.21M|11344.367G|24.15|0.710\n|MambaSCI|2.47M|**1957.96G**|**30.80**|**0.896**\n\n**Q4: Clarification of why Mamba and not self-attention.**\n\n**A:** As stated in our response to **Reviewer M9Ao's Q1**, from both the technical perspective, including performance metrics and comparisons of Params and FLOPs as shown in Table 2 of main manuscript, and the perspective of pattern processing, where previous methods based on Transformers and CNNs have struggled with artifacts and color distortions when dealing with quad-Bayer arrays. Mamba has proven to be a more suitable choice for addressing the new challenges we face.\n\n**Q5: Clarification of why quad-Bayer pattern is better than a single Bayer-patterned measurement?**\n\n**A:** As noted in response to **Reviewer M9Ao's Q2**,exploring quad-Bayer-based SCI tasks is both necessary and advantageous. Due to the quad-Bayer's physical structure, where each color is represented by a 2x2 grid, it offers higher resolution, better light intake in low-light conditions, and supports HDR video through different exposure settings.  \n\n**Q6: Comparison on long videos.**\n\n**A:** We have conducted new experiments to further validate our method's effectiveness with long sequence videos and also provided corresponding visual comparison images (**Figure 4** on PDF) and a performance comparison table (**Table 1** in global rebuttal). The default $T$ is set to 8, meaning reconstruction is performed after compressing 8 frames of video into 2D observations. To test our method's performance with longer sequences, we also evaluated it with $T = 16$ and $T = 32$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4vPL8z1p7F", "reviewer": "Reviewer_yWFS", "summary": "The paper proposes a method called MambaSCI for efficient reconstruction of quad-Bayer patterned color video snapshot compressive imaging.This method surpasses state-of-the-art algorithms with lower computational and memory costs, providing improved color accuracy and demosaicing.", "questions": "1) The paper describes the computational benefits of the proposed approach over SOTA in terms of fewer parameters and FLOPS, however no concrete numbers have been provided around the performance. Is it possible to add some information around the time it would take to reconstruct each frame?\n2)  There is no mention on how the masks were generated. Or the conditioning of the masking matrix. This very important information should be included.\n3) What is the relationship between T (number of frames used during reconstruction) and the amount of motion between consecutive frames? For instance I would expect you could use more frames(T) if the object was moving slowly within the scene, compared to when it is moving fast. This might help in further improving the computational cost. \n4) There is no tiling done during reconstruction. I wonder why this is so? Especially if you want to deal with higher resolution images?", "limitations": "Limitations have been adequately addressed in the appendix, so nothing to comment upon here.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The contributions of this paper are quite novel.  The usage of Mamba for video SCI reconstruction has been used for the first time. Moreover, the paper also introduces Quad-Bayer CFA patterned color video SCI reconstruction which is also being done for the first time.\n2) The quality of the results produced is quite high. The average PSNR and SSIM scores produced with the method proposed in this paper are both comfortably beating the state of the art results. \n3) The paper is well written and the presentation is very clear. The citations, and comparisons with SOTA are sufficient to build confidence in the proposed method.\n4) The contributions due to each individual module has been presented which clarifies the necessity for each.", "weaknesses": "1) The reconstruction seems to be performed on fairly low resolution images(512 x 512). I would have liked to see the simulations being done on higher resolution.\n2) The proposed method is tested only on simulated datasets (since there is no real SCI dataset based on quad-Bayer pattern). To me, it is unclear how the simulated datasets have been created. Was noise added while creating the simulated datasets? My biggest concern is the way the noise has been modeled while creating the simulated dataset might not match the actual data(If there was a way to do it). Therefore, currently the efficacy of the provided solution is somewhat hypothetical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer yWFS\nThanks for your valuable comments.\n\n**Q1: Clarification of high resolution images reconstruction.**\n\n**A:** As stated in the main manuscript (lines 233-234), we also evaluated our method on a large-scale simulated dataset. The metrics comparison and visual comparison between our method and the comparison methods for reconstructing 1920x1080 resolution video are presented in Table 4 and Figure 6 in the main manuscript.\n\n**Q2: Noise modeling analysis of simulated datasets**.\n\n**A2**: We have explained how the simulated dataset was created and provided an analysis of our method's effectiveness on a real dataset:\n\n- The middle-scale simulated dataset is created by extracting and cropping frames from high-quality open-source videos available online to obtain the groundtruth (GT), while the large-scale dataset uses videos selected from YouTube as the GT. **Both datasets include noise of real-world scenes.**\n- Given that we are tackling a completely new task, our focus is primarily on addressing the degradation caused by SCI compression and developing effective demosaicing techniques to achieve color fidelity and eliminate artifacts for high-quality reconstruction. The aspect of noise modeling has not yet been fully considered.\n- Through our studies on various Bayer-array-based methods, their performance on simulated and real datasets is positively correlated, thus I believe our method can still achieve significant advantages on real data.\n\n**Q3: Time required to reconstruct each frame.**\n\n**A:** In the table below, we present a comparison of the time required to reconstruct each frame, along with the PSNR and SSIM metrics for the proposed method and other competing methods. Even though our method takes longer than EfficientSCI to reconstruct each frame, it delivers significantly superior performance.\n\n|Method|Time(s)|PSNR(dB)|SSIM\n|-|-|-|-|\n|PnP-FFDnet|0.28|27.86|0.855\n|STFormer|0.20|32.48|0.842\n|EfficientSCI|**0.07**|32.35|0.854\n|MambaSCI|0.16|**35.70**|**0.959**\n\n**Q4: Clarification of how the masks are generated.**\n\n**A:** In the SCI domain, the **mask typically has a specific physical significance**. The basic principle involves modulating different frames within the video cube with varying weights and then integrating the light into the sensor to create a 2D measurement [1-3]. Specifically the masks are the 0-1 matrices that are different from each other in the T-frame. Also this mask is customizable subject to its physical significance and has no effect on the reconstruction results. During training, we simulate the masks of different SCI systems by randomly generating 0-1 matrices to improve the robustness of the system.\n\n**Q5: Clarification of the relationship between T and the amount of motion between consecutive frames.** \n\n**A:** In SCI tasks, $T$ typically represents the compression ratio ($Cr$) and is not directly related to the motion between consecutive frames. A higher $T$ indicates a higher $Cr$ and increased reconstruction difficulty. As highlighted in **Table 1 in the global rebuttal**, both reconstruction difficulty and the required FLOPS escalate with increasing $T$. Nonetheless, our method delivers high-quality reconstruction with superior computational efficiency.\n\nWe greatly appreciate the suggestion to use smaller compression ratios for fast motion scenes and larger ones for slow motion scenes to improve efficiency. This insight has inspired us to consider dynamically adjusting the compression ratio based on object movement speed. Thank you for this suggestion, which opens a new avenue for exploration.\n\n**Q6: Clarification of why there is no tiling done during reconstruction.**\n\n**A:**  In the reconstruction process, we first address the SCI compression degradation in the raw domain and then perform demosaicing to convert from the raw domain to RGB. In the raw domain, two common tiling methods involves divideing into non-overlapping patches and splitting the image into four sub-parts: R, G1, G2, and B. We specify why we did not employ either of these methods:\n\n- **Dividing into Non-overlapping Patches.** Dividing an image into non-overlapping patches will lead to localized detail loss. To address this, we employ convolution for shallow feature extraction, which enables us to train effectively on low-resolution images (using a resolution of 128x128 for training and 256x256 for fine-tuning) and process on high resolution images.\n- **Dividing into Four Sub-parts.** This method is efficient for Bayer arrays due to their physical arrangement. However, the quad-Bayer's unique layout poses challenges. Splitting the image into R, G1, G2, and B components requires a more intricate process because each color forms a 2x2 square, which must be considered as a whole rather than isolated pixel by pixel. We attempted this approach, but found it significantly increased both training and inference times.\n  \nThank you for your question, and we will continue exploring efficient tiling methods to achieve lightweight reconstruction for higher-resolution videos.\n\n[1] Yuan X, Liu Y, Suo J, et al. \"Plug-and-play algorithms for large-scale snapshot compressive imaging.\" *CVPR* 2020.\n\n[2] Yuan X, Liu Y, Suo J, et al. \"Plug-and-play algorithms for video snapshot compressive imaging.\" *IEEE TPAMI* 2021.\n\n[3] Yuan X, Brady D J, Katsaggelos A K. \"Snapshot compressive imaging: Theory, algorithms, and applications.\" *IEEE SP* 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and insightful comments which have helped improve our paper. We are pleased that the reviewers found our introduction of Mamba into the SCI task to be very novel and well justified, and that our proposed quad-Bayer patterned SCI task is a direction worth exploring. Some of the main criticisms are summarized in the next section, after which we will address the comments of the delicious reviewers separately.

In response to the reviewers' constructive criticism, we added some new experiments including differential analysis of whether considering quad-Bayer feature in the initialization block, analysis of Swinger data results, evaluation of the EDR module for edge detail reconstruction, and a comparison of reconstruction in higher frame rate scenes. The new analyses are shown in **Figures 1 to Figure 4** in the associated PDF file.

**New comparative experiments and analysis.** To verify the validity and reasonableness of our method, we have conducted additional experiments and analysis accordingly. These experiments include:

- **Comparison of Initialization Methods with Quad-Bayer Characteristics and SCI Generic Blocks.** We have customized a Quad-Bayer-specific initialization block by considering the characteristics of the Quad-Bayer pattern (each color consists of 2x2 pixels) and have applied it to GAP-TV, PnP-FFDnet, and PnP-FastDVDnet. **Figure 1** shows a visual comparison of the $\mathbf{x_{in}}$ obtained considering Quad-Bayer characteristics in the initialization block and the $\mathbf{x_{in}}$ obtained using SCI generic initialization block, demonstrating the use of the SCI generic initialization block can reduce reconstruction time while maintaining performance. For a more detailed analysis, see **Reviewer zshs's Q2**.
- **Zoomed-In Patch Comparison of "Swinger" with Ghosting Artifact Analysis.** **Figure 2** shows a localized zoomed-in patch comparison image of the lower left corner of the "swinger", along with an analysis of the ghosting artifacts that appear.
- **Impact of EDR Module on Edge Detail Reconstruction.**  **Figure 3** shows the effect of w/ or w/o EDR module on the impact of edge detail reconstruction, we can see the EDR module can significantly improve the reconstruction of edge details.
- **Reconstruction Effects at Higher Frames and Compression Ratios.** We conducted new experiments in longer frame rates and larger compression rate scenarios. **Figure 4** shows a comparison of the reconstruction effects of our method and the comparative methods at higher frames T (compression ratios). The related performance comparison is shown in **Table 1**:

*Tabel 1: Performance analysis at $T$=16 and 32 cases. Our MambaSCI outperforms in PSNR and SSIM while requiring less than **10%** of the FLOPS of the comparison method.* 
| T  | Methods     | Params (M) | FLOPS (G)  | PSNR (dB) | SSIM   | Time (s) |
|----|---------------|-----------|-------|---------|-----------|----------|
| 16 | PnP-FFDnet    | -       | -        | 24.85     | 0.767   | 4.52     |
|    | STFormer      | 19.49   | 24311.76 | 25.21    | 0.685    | 3.13     |
|    | EfficientSCI  | 8.83    | 11406.23| 25.35    | 0.656    | **1.37**     |
|    | MambaSCI      | **6.11**    | **1113.78**  | **25.39**    | **0.817**   | 2.67     |
|
| 32 | PnP-FFDnet    | -       | -         | 1.82     | 0.496 | 9.85     |
|    | STFormer      | 19.49     | OOM      | -     | -  | -    |
|    | EfficientSCI  | 8.83     | 22825.34      | **23.24**    | 0.653  | **2.21**    |
|    | MambaSCI      | **6.11**     | **2227.57**      | 22.44    | **0.785**  | 5.20     |

Meta-review: This paper received 4 mixed reviews with the scores spread across a spectrum - one 3 (Reject), one 5 (Borderline Accept), one 6 (Weak Accept), and one 8 (Strong Accept). 

There was a general appreciation for the addressed problem (video compressive imaging with quad Bayer sensors, which is timely given the increasing popularity of quad-Bayer sensors), quality of the exposition, the principled nature of the proposed solution which is well-motivated, and the overall quality of the results. 

There were several concerns raised, in particular about lack of real experiments with a quad Bayer sensor. The authors submitted a response and there was considerable discussion post-rebuttal. While the concern around lack of real experiments remain (due to the relative difficulty of obtaining large scale quad-Bayer datasets), most other concerns were reasonably addressed. Therefore, on balance, an accept decision is recommended. 

The authors are strongly urged to incorporate the reviewers' feedback and include the additional results / evaluations promised in the rebuttal when preparing the final camera-ready version.
Decision: Accept (poster)