MambaSCI: Efficient Mamba-UNet for Quad-Bayer Patterned Video Snapshot Compressive Imaging

Authors: zhenghao  pan
haijin  zeng
jiezhang  cao
yongyong  chen
kai  zhang
yong  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:58:14

Abstract: Color video snapshot compressive imaging (SCI) employs computational imaging techniques to capture multiple sequential video frames in a single Bayer-patterned measurement. With the increasing popularity of quad-Bayer pattern in mainstream smartphone cameras for capturing high-resolution videos, mobile photography has become more accessible to a wider audience. However, existing color video SCI reconstruction algorithms are designed based on the traditional Bayer pattern. When applied to videos captured by quad-Bayer cameras, these algorithms often result in color distortion and ineffective demosaicing, rendering them impractical for primary equipment. To address this challenge, we propose the MambaSCI method, which leverages the Mamba and UNet architectures for efficient reconstruction of quad-Bayer patterned color video SCI. To the best of our knowledge, our work presents the first algorithm for quad-Bayer patterned SCI reconstruction, and also the initial application of the Mamba model to this task. Specifically, we customize Residual-Mamba-Blocks, which residually connect the Spatial-Temporal Mamba (STMamba), Edge-Detail-Reconstruction (EDR) module, and Channel Attention (CA) module. Respectively, STMamba is used to model long-range spatial-temporal dependencies with linear complexity, EDR is for better edge-detail reconstruction, and CA is used to compensate for the missing channel information interaction in Mamba model. Experiments demonstrate that MambaSCI surpasses state-of-the-art methods with lower computational and memory costs. PyTorch style pseudo-code for the core modules is provided in the supplementary materials. Code is at https://github.com/PAN083/MambaSCI.

Main paper: 
 introduction: 
 In recent years, there has been significant progress in enhancing imaging quality in smartphone image sensors. One notable trend is the adoption of the quad-Bayer Color Filter Array (CFA) pattern [1] . Smartphones such as the iPhone 14 Pro/Max, vivo X90 Pro+, Xiaomi 13S Ultra, and OPPO Find X6 Pro utilize quad-Bayer array to enhance image quality in low-light conditions and offer higher resolution for mobile photography [2, 3] . Unlike the traditional RGGB Bayer CFA pattern, the quad-Bayer pattern expands each pixel into four sub-pixels and arranges them periodically [4, 5] , as depicted in Fig. 1(a) . This arrangement allows for larger pixels by combining neighboring pixels of the same color, resulting in more light intake compared to the Bayer pattern. Consequently, quad-Bayer sensors offer enhanced sensitivity and resolution for imaging tasks [6, 7, 8] . As illustrated in Fig. 2 (b), quad-Bayer technology effectively mitigates resolution loss and enables the capture of low-noise photos in low-light environments. In summary, quad-Bayer sensors provide HDR capability [9] and improved color accuracy [10] while effectively mitigating resolution loss and capturing low-noise photos in low-light environments.Color videos captured by traditional high-speed cameras incur high transmission and storage costs.To solve this issue, color video snapshot compressive imaging (SCI) [11] , which comprises both hardware encoder and software decodes components, has been proposed. During the encoding phase, multiple raw video frames undergo modulation and compression using various masks to generate 2D measurements [12] . Subsequently, in the decoding stage, the desired high-speed color video is reconstructed from the acquired measurements and predefined masks.So far, all color video SCI encoding and reconstruction algorithms have been designed based on the Bayer pattern [13, 14, 15, 16, 17] . As smartphone cameras continue to improve in pixel count and performance, the majority of videos are now captured using smartphones equipped with quad-Bayer patterns. However, our observation reveals that existing methods struggle to effectively reconstruct videos based on quad-Bayer patterns, often resulting in artifacts, color distortions, and incomplete demosaicing. This ineffectiveness poses a challenge for processing videos captured by smartphone cameras. Thus, in this paper, we aim to break through these challenges and introduce the first quad-Bayer-based color video SCI reconstruction method, as illustrated in Fig. 2(a) .When incorporating quad-Bayer in color video SCI reconstruction tasks, two challenges arise. One is how to efficiently manage quad-Bayer color video processing with reduced computational complexity. Current color video SCI reconstruction methods encompass model-based [13, 18, 19] , iteration-based [14, 15] , and End-to-End (E2E) approaches [20, 21] . However, model-based and iterative methods necessitate corresponding demosaicing models [6, 22, 23, 24] , which have not been extensively explored for quad-Bayer fields regarding performance and model size trade-offs, leading to suboptimal reconstruction outcomes and inefficiencies. On the other hand, existing E2E methods, primarily transformer-based [17] and hybrid CNN-transformer [16] approaches, typically require significant parameters and computational resources, making it difficult to process long video sequences effectively. The high computational demands of transformers and the absence of a global attention mechanism in CNNs hinder their scalability to modern, lightweight architectures. Leveraging the advancements in State Space Models (SSMs) [25, 26, 27] , modern SSMs like Mamba [28] have demonstrated the ability to effectively capture long-range dependencies while maintaining linear complexity relative to the input size. Furthermore, numerous experiments have illustrated that image-based Mamba [29, 30] achieves promising results and can match the performance of existing transformer [31, 32, 33] and CNN [34, 35, 36] models with smaller parameters. Therefore, we focus on exploring the multi-scale reconstruction capabilities of Mamba-UNet for quad-Bayer processing. This approach aims to create a lightweight design suitable for deployment on mobile devices. By leveraging the Mamba-UNet framework [37, 38, 39, 40] , we can reduce parameters and FLOPS while still achieving state-of-the-art performance. The second challenge is how to eliminate motion artifacts to ensure clarity in dynamic video. Motion artifacts may arise from handheld camera instability and pixel merging in quad-Bayer patterns. To preserve scene clarity and edge details, we have customized the Residual-Mamba-Block, which integrates three key components: the Spatial-Temporal Mamba (STMamba), the Edge-Detail-Reconstruction module (EDR), and the Channel Attention module (CA) [41, 42] . The Residual-Mamba-Block further enhances long-range spatiotemporal dependence by leveraging residual connections and learning scales. STMamba replaces the self-attention module by performing parallel scanning of spatial and temporal dimensions, ensuring spatiotemporal consistency and enabling visual reconstruction of videos in a more lightweight manner. The EDR module enhances boundary sharpness perception and restores edge details lost due to compression and motion artifacts. By combining linear transformation with DWConv features, it integrates both local and global information. This approach strengthens the global features extracted by STMamba, fusing them with local details to more effectively capture complex edge structures. As a result, the module enables high-quality video reconstruction with improved edge clarity. The CA module compensates for the overlooked channel interactions in Mamba by weighting features based on channel importance, reducing the impact of artifact noise on reconstruction and thus improving the clarity of reconstruction results.Building upon the foundation modules discussed earlier, we introduce a novel model called Mam-baSCI, which serves as the reconstruction algorithm for our proposed quad-Bayer-based SCI method as illustrated in Fig. 2 (a). MambaSCI adopts a non-symmetric U-shaped encoder-decoder architecture with skip connections to enhance model efficiency. To incorporate spatial-temporal consistency at multiple scales, we introduce Residual-Mamba-Blocks at each encoding stage. Furthermore, we employ residual convolution in the decoding stage to reduce both parameters and computational complexity. The effectiveness of the MambaSCI model is demonstrated in Fig. 1(b) , where it achieves superior performance compared to the SOTA methods while requiring fewer parameters and FLOPS. Pseudo-code detailing the core modules is provided in the supplementary materials.In short, our contributions can be summarized as follows:(i) We are the first to introduce Quad-Bayer CFA pattern into color video SCI to accommodate that most videos are captured by mobile photographers using quad-Bayer patterned smartphone cameras.(ii) We are the first to use Mamba model for video SCI reconstruction. By integrating Mamba with a non-symmetric UNet, we employ a hierarchical encoder to capture spatial-temporal correlations at various scales, thus accelerating the model and enhancing reconstruction quality.(iii) We customize the Residual-Mamba-Block, integrating STMamba, EDR, and CA modules with residual connections and learnable scales to enhance reconstruction quality and edge details.(iv) MambaSCI outperforms other SOTA methods, requiring fewer parameters and FLOPS, and delivers superior visual results on 6 simulation color videos and 4 large-scale simulation color videos.2 Related Work 
mathematical model for color video sci: 
 For color video SCI systems, the original T -frame input video X ∈ R H×W ×3×T is given, along with the mask M ∈ R H×W ×T , where H, W and T denote color video's height, width and number of frames, respectively. As in previous Bayer-based approaches, since each pixel captures only the red (R), green (G) or blue (B) channel of the raw data in a spatial layout, both X and M are divided into four parts to obtain four sub-measurements{Y r , Y g1 , Y g2 , Y b } ∈ R H 2 × W 2 . Y r = X r ⊙ M r , Y g1 = X g1 ⊙ M g1 , Y g2 = X g2 ⊙ M g2 , Y b = X b ⊙ M b . (1)Some methods [14, 15] process the four sub-measurements individually, restore them to RGB color by an off-the-shelf demosaicing algorithm and finally combine them to get the final reconstructed video, which is inefficient and cannot make good use of channel correlation. Thus others [16, 17] input all sub-measurements into their proposed reconstruction network simultaneously, and finally obtain the final RGB color video by convolutional network. Similarly, we first obtain four sub-measurements based on the spatial layout of the quad-Bayer array shown in Fig. 1 (a), and then feed them into the network simultaneously.Following [11, 43] , we denote the vectorized measurement y ∈ R HW . Then given vectorized color video x ∈ R HW T and mask Φ ∈ R HW ×HW T , the degradation model can be formulated as:EQUATIONwhere n ∈ R HW represents noise on measurement. SCI reconstruction is to obtain x from captured y and the pre-set Φ using a reconstruction algorithm [14, 15, 16, 44] . However, for quad-Bayer color videos, demosaicing algorithms bring artifacts, and Bayer pattern-based convolutional reconstruction causes color distortion, significantly affecting quality.SSMs are common treated as linear time-invariant systems that map a 1D sequence x(t) ∈ R to y(t) ∈ R through a hidden state h(t) ∈ R N , the process can be expressed as follows:EQUATIONwhereA ∈ R N ×N , B ∈ R N ×1 , C ∈ R 1×N .It is common to use the zero-order hold (ZOH) method to discretize the continuous system, thus Eq. ( 3) can be discretized as following:EQUATIONwhich uses timescale parameter △ to convert continuous A and B to discrete Ā and B.The 1D S4 model [45] is extented to handle multidimensional data, while TranS4mer model [46] optimizes movie scene detection by combining S4 with self-attention. Vision Mamba and Mam-baIR [30, 47] introduce SSMs into the vision domain as generic backbones. U-Mamba [48] addresses long-range dependencies in biomedical images. Efficient medical image segmentation is achieved with lightM-UNet and UltraLight VM-UNet [40, 41] . ViVim [49] is proposed for effective and efficient medical video object segmentation.These methods focus on (i) image restoration (spatial information), (ii) video understanding (global features), and (iii) medical image or video segmentation (small resolution and frame count). However, they do not apply SSMs to video SCI, a task requiring spatial-temporal consistency and detailed feature reconstruction for high-resolution, long-frame videos. Therefore, there is an urgent need to explore SSMs' performance and efficiency in long-sequence problems such as video SCI. 
the proposed method: 
 In this section, we introduce our proposed MambaSCI network framework, and detail our customized Residual-Mamba-Block, which is capable of capturing long-range spatial-temporal consistency along with edge detail sharpness reconstruction and channel information interaction, thus being able to outperform SOTA results with fewer parameters and FLOPS. Fig. 3 illustrates the reconstruction process, network framework and details of core modules.Given the 2D measurement Y ∈ R H×W and mask M ∈ R H×W ×T , through the general initialization module of SCI, the general initialization module of SCI provides an initial raw quad-Bayer reconstruction video X in ∈ R H×W ×1×T . This serves as the input to the MambaSCI reconstruction network, which outputs the final color reconstruction video X out ∈ R H×W ×3×T . MambaSCI comprises five main components: (i) shallow feature extraction block, (ii) encoder layer, (iii) bottleneck layer, (iv) decoder layer, (v) color video reconstruction block. When X in is feed into MambaSCI, it first goes through shallow feature extraction block, which includes a depthwise separable convolution (DWConv), producing the shallow feature F ∈ R H×W ×C×T . Next F sequentially passes through three encoder layers, each composed of a Residual-Mamba-Block and Max-Pooling operation, resulting in the featureF ei ∈ R (H/2 i )×(W/2 i )×(C×2 i )×T, where i ∈ {1, 2, 3}. After these encoding layers, the deep featureF ∈ R Ĥ× Ŵ × Ĉ×T is obtained, with Ĥ = H 8 , Ŵ = W 8 and Ĉ = 8 × C.The bottleneck layer, composed of Residual-Mamba-Blocks, keeps the feature's shape unchanged. Then, through each decoding layer, which includes residual convolutions and upsampling operations, the feature transforms intoF di ∈ R ( Ĥ×2 i )×( Ŵ ×2 i )×( Ĉ/2 i )×T .Eventually, feature F d3 is fed into the color video reconstruction block to obtain the reconstructed color video X out ∈ R H×W ×3×T . See Fig. 3 (b) for an overall view.Within each encoder layer, we incorporate N Residual-Mamba-Blocks, specifically designed to capture and enhance temporal-spatial coherence across multiple scales, resulting in more accurate and comprehensive feature representations. As depicted in Fig. 3(c) , each Residual-Mamba-Block consists of three key components: (i) the STMamba module, (ii) the EDR module, and (iii) the CA module. These components are detailed below, and the overall process can be mathematically described as follows:EQUATIONwhere LN represents LayerNorm and l ∈ [1, N ], F l is the l th block's input feature and F l out can be treated as F l+1 block's input, s 1 , s 2 , s 3 represent the learnable scales in the residual connection.(i) STMamba module. Previous video SCI reconstruction algorithms typically compute attention separately for the temporal and spatial dimensions, then fuse them using a residual network [50] . However, this approach may lack temporal-spatial consistency. To address this issue, we employ the STMamba model [49] , which integrates spatial-temporal information through structured SSMs. Specifically, As illustrated in Fig. 3(d) , the input F ∈ R H×W ×C×T is processed in two parallel branches. In the first branch, F is expanded to Ĉ = S × C channels via a linear layer, resulting in X ∈ R H×W × Ĉ×T , where S is expansion scale. Then, X is unfolded along frames T to form X s ∈ R T (HW )× Ĉ . Forward and backward scanned features, Xsf and X sb , are obtained by scanning X s in both directions, efficiently capturing spatial dependencies. Simultaneously, sequence X t ∈ R (HW )T × Ĉ is generated to explore temporal dependencies by forward scanning each pixel across the T frames.STMamba utilizes parallel SSMs to capture intra-frame and inter-frame correlations, enforcing time-space consistency constraints. This approach enables STMamba to capture both the spatial features within frames and the temporal dependencies between frames, accurately modeling dynamic changes in video data, facilitating efficient spatial-temporal feature extraction and preservation of consistency. As shown in Fig. 3 (d), the process can be formulated as:EQUATIONwhere ⊙ represents Hadamard product, LN represents LayerNorm and SiLU is an activation function.(ii) EDR and CA module. To achieve the fine reconstruction of edge details and to compensate for the missing inter-channel interaction capability in the Mamba model, we introduce the EDR and CA modules. The EDR module consists of linear layers and depthwise separable convolution (DW-Conv). By combining linear transformation with DWConv features, the EDR module gains the ability to perform multi-scale feature fusion, allowing the network to extract global information from local features and process both global and local information simultaneously. This enhances the model's capacity to understand complex edge structures. Additionally, through adaptive weight initialization, the model can capture finer details more effectively in the early stages of training, enabling efficient edge detail reconstruction in a lightweight manner while enhancing image edge information.On the other hand, the CA module compresses the spatial dimension of the feature map to 1 × 1 × 1 through an average pooling operation, which in turn maps the compressed features to the interval (0, 1) through a convolutional layer with an activation function to form channel attention weights. These weights are subsequently multiplied channel-by-channel with the original feature map to implement the channel attention mechanism. In addition, the CA module utilizes the convolution operation to further facilitate the fusion and exchange of information between channels, thereby enhancing the interaction effect between channels.Like transformer, Mamba encounters severe optimization and convergence challenges as network depth increases [40] . In the bottleneck layer, multiple Residual-Mamba-Blocks are concatenated, keeping the same number of feature channels and resolution. This maintains feature richness and clarity, enhances the model's spatial-temporal dependency capture, and improves MambaSCI's performance without significantly increasing computational burden.Decoding layer decodes features and recover image resolution. It receives two inputs: F e ∈ R Ĥ× Ŵ × Ĉ×T from the skip connection, which retains original spatial information, and F d ∈ R Ĥ× Ŵ × Ĉ×T from the previous decoding layer, containing higher-level spatial-temporal information. The decoding layer fuses features using element-wise addition to enhance expressiveness, applies DWConv with residual concatenation, an activation function and an upsampling operation. This process produces output features with richer semantics and higher spatial resolution.The color video reconstruction block reconstructs the desired video X out ∈ R H×W ×3×T . Instead of the computationally intensive remosaicing and demosaicing of raw quad-Bayer images, we use a three-layer convolution (kernel sizes 3×3×3, 3×3×3, and 1×1×1) to process the decoding layer's output and obtain the final RGB color video. STFormer 6HW T C 2 + 2G h G w HW T C + HW T 2 C EfficientSCI 1 2 HW T K 2 C 2 + 1 2 HW T C 2 + 1 2 HW T 2 C MambaSCI 8HW T CN + 2HW T CN 2To balance size and performance, we propose three versions of the MambaSCI model: MambaSCI-T (tiny), MambaSCI-S (small), and MambaSCI-B (base). Tab. 1 shows the network hyperparameters, model parameters, and computational complexity (FLOPS). By varying the number of channels from the initial DWConv, our method achieves significantly lower complexity than EfficientSCI [16] and STFormer [17] .We also calculate the computational complexity of the attention module n MambaSCI compared to other SOTA methods, as shown in Tab. 3, where C is the number of input features, K represents the kernel size, G h and G w are the spatial size of local window in Swin-transformer [51] , N is a fixed parameter in Mamba set to 16 . In MambaSCI, input features are unfolded into a sequence S ∈ R HW ×C×T . As seen in Tab. 3, while STFormer and EfficientSCI scale linearly with the spatial size (HW ), their complexity grows quadratically with video frames T and C, which is typically 64 or larger, resulting in high computational costs. MambaSCI scales linearly with the entire video sequence (HW T ) and C, which is capped at 64 in MambaSCI-B, enabling efficient reconstruction of longer video sequences. Inference time comparisons are shown under various methods in Tab. 5. 
experiment: 
 In this section, we evaluate MambaSCI against SOTA video reconstruction methods on multiple simulation datasets using PSNR, SSIM metrics, and visual comparisons.Following STFormer and EfficientSCI, we use DAVIS2017 [52] simulated datasets, six benchmark mid-scale color datasets [15] (Beauty, Bosphorus, Jockey, Runner, ShakeNDry and Traffic of size 512×512×3×32), and four benchmark large-scale color datasets [15] (Messi, Hummingbird, Swinger and Football). Since there is currently no real color video SCI dataset based on quad-Bayer pattern, our method is not tested on real datasets.We use PyTorch framework training on 4 NVIDIA RTX4090 GPUs and use random flipping, scaling, and cropping on DAVIS2017 for data augmentation. We use randomly generated masks as training input to enhance model robustness and optimize the model using the Adam [53] optimizer. Since MambaSCI is flexible in input size, we first train for 100 epochs at a learning rate of 0.0005 on data with a spatial size of 128×128. Then, we train for 50 epochs at learning rate of 0.0001, followed by fine-tuning on 256×256 data at learning rate of 0.00001 for an additional 50 epochs.To test the performance of our method for color video reconstruction, we perform experiments on a 32-frame simulation color RGB video dataset with size of 512 × 512 × 3 × 32. We compress the color video with compression rate of B = 8 and capture quad-Bayer pattern measurements using a camera with quad-Bayer CFA pattern.Since all current color video SCI reconstruction algorithms are designed based on Bayer pattern, which cannot be directly applied to quad-Bayer pattern. Meanwhile, re-training an E2E model requires much training time and memory. Thus we only re-train two of the latest SOTA E2E models (STFormer [17] , EfficientSCI [16] ) with quad-Bayer pattern. We compare with iterative optimization algorithm (GAP-TV [13] ), PnP algorithms (PnP-FFDnet [14] and PnP-FastDVD [15] ) and E2E algorithms (STFormer [17] and EfficientSCI [16] ). The number of parameters, FLOPS, and reconstruction results are shown in Tab. 2.Notably, there is no readily available quad-Bayer demosaicing package. Therefore, for model-based and PnP algorithms, we first upsample the reconstructed raw quad-Bayer video X ∈ R H×W ×1×T according to the quad-Bayer CFA pattern to obtain a three-channel video X ∈ R H×W ×3×T . We then demosaic it using the BJDD [54] algorithm to get the final RGB color video. See supplementary materials for details. Visual comparisons of the reconstruction results are shown in Fig. 5 .We summarize our observations: (i) Our MambaSCI model significantly outperforms SOTA methods with lower computational and memory resources. In addition, our proposed model demonstrates SOTA reconstruction performance even at higher frame rates and compression ratios. We tested it on beauty data across various compression ratios (B = 8, 16, 32), with results detailed in the Tab. 4. Notably, our method requires only 9.8% of the FLOPS needed by EfficientSCI, while also significantly outperforming it in the SSIM.Similar to previous studies, we conduct experiments on a large-scale color video dataset. Due to the significant time and memory required to retrain existing E2E models for Bayer patterns, we only compare with SOTA model-based methods (GAP-TV, PnP-FFDnet, PnP-FastDVDnet) and retrain STFormer-S and EfficientSCI-S for the quad-Bayer pattern. Tab. 5 shows the comparisons on PSNR and SSIM, and Fig. 6 provides visual comparisons.We summarize the observations: (i) MambaSCI-S outperforms other methods in PSNR and SSIM on football and swinger, achieving over 1.5 dB higher PSNR on football. (ii) In visual comparisons, GAP-TV and PnP algorithms exhibit artifacts, while STFormer and EfficientSCI suffer from color distortions. MambaSCI excels in reconstructing detailed information, resulting in superior reconstruction quality. (iii) The poorer performance on Messi and Hummingbird may be due to faster, more detailed motions that strained by limited parameters and FLOPS. However, the visual results remain superior to other SOTA methods. The Residual-STMamba-Block is the core customised module of our MambaSCI. Tab. 7 is experimentally demonstrated that the residual connections and the learnable scales are effective in improving the reconstruction quality enhancement. 
conclusion: 
 In this paper, we introduced quad-Bayer pattern into video SCI for the first time, enabling SCI to align with the fact that most current videos are captured by mobile phones with quad-Bayer cameras, thus avoiding artifacts and color distortions caused by existing algorithms. Specifically, we integrate Mamba model with an asymmetric UNet in video SCI, leveraging Mamba's linear complexity and the speed improvements from the non-symmetric architecture for efficient SCI reconstruction. Moreover, we customized Residual-Mamba-Blocks to connect STMamba, EDR, and CA modules through residual connections, ensuring efficient spatial-temporal consistency and detailed reconstruction. Experimental results on simulated color video datasets highlighted that MambaSCI outperformed SOTA methods with fewer parameters, lower computational complexity and better visual effects. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We describe all the training and testing details needed to understand the results in Sec. 4.2.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The train/test split, initialisation in our approach does not affect the overall run for a given experimental condition, while certain parameters are not randomly plotted.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provide the experimental platform GPU devices and memory, the number of model parameters, the amount of computation and the corresponding inference time in the paper Sec. 3.5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Broader impacts are in Sec. A.5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our papers pose no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited, and the licences and terms of use are are are clearly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not deal with crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not deal with crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the supplementary material, we provide more details that are not in out main paper: (e) Broader impacts in Sec. A.5Coded aperture compressive temporal imaging (CACTI) is one of the famous video SCI systems. Specifically, for color video SCI systems utilizing quad-Bayer arrays, the raw data is spatially structured that each pixel captures only red (R), green (G), or blue (B), creating a format similar to 'RGGB' where each color occupies adjacent 2×2 pixels in succession. Thus, the initial color video X ∈ R H×W ×3×T is multiplied pixel by pixel with the filter and superimposed in the channel dimension, ultimately producing the raw data X ∈ R H×W ×T . Given the mask M ∈ R H×W ×T , the modulation is:EQUATION′ denotes the modulated video data and ⊙ represents element-wise multiplication. 2D compressed measurement Y can be captured across time dimention, which can be expressed as:EQUATIONwhere N ∈ R H×W means the measurement noise.Vectorization. Given:EQUATIONEQUATIONEQUATIONEQUATIONwhere vec(•) means vectorization operation, x t = vec(X(:, :, t)) represents the vectorization of t-th frame of X and D t = Diag(vec(M(:, :, t))) is a diagonal matrix and its diagonal elements is filled by vec(M(:, :, t)). The vectorization forward process can be expressed as:EQUATIONAnd the reconstruction process is to investigate algorithms that can reconstruct x given y and Φ.As mentioned in the paper. Residual-Mamba-Blocks can complete high quality reconstruction by capturing spatial-temporal coherence while completing edge detail reconstruction and compensating for channel information interactions not available in the Mamba model. In Algorithm 1, we show the PyTorch style pseudo-code on how to construct a Residual-Mamba-Block.In this section, we will demonstrate how we employ certain techniques to apply existing demosaicing algorithms to the raw quad-Bayer images reconstructed by GAP-TV, PnP-FFDNet, and MambaSCI method achieves superior color fidelity and detailed reconstruction on middle-scale benchmark simulation color video dataset, offering a significant visual advantage over previous methods. Meanwhile, as shown in Fig. 11 , even though our proposed MambaSCI method may not surpass PnP-FastDVDFnet in terms of PSNR and SSIM metrics, it achieve superior reconstruction visual quality. In comparison to the artifacts introduced by GAP-TV and PnP-FastDVDFnet as well as the color distortions in STFormer-S and EfficientSCI-S, MambaSCI demonstrate significantly better visual results. As shown in Fig. 12 and Fig. 13 , our proposed method achieve pleasant visual results.STFormer-B EfficientSCI-B MambaSCI-S MambaSCI-B GAP-TV PnP-FFDNet PnP-FastDVDnet A.4 Limitation.The limitations of our approach are in two respects:(i) One limitation is the trade-off between computational complexity and performance. As shown in Table 6 , to achieve better performance, we incorporate EDR and CA modules. However, the use of Conv3d in the CA module significantly increases the number of parameters and computational complexity.(ii) The second limitation is that measurements based on quad-Bayer in real datasets are currently unavailable, making it impossible to evaluate the performance of our proposed model in real-world scenarios.Given these limitations, we aim to investigate ways to simplify the internal modules while maintaining performance, thereby reducing computational complexity and accelerating inference speed. Additionally, we will focus on collecting quad-Bayer-based SCI measurements in real-world scenarios to verify the reliability of our method under real scene conditions. Video reconstruction is a fundamental task in snapshot compressive imaging (SCI), an area with a research history spanning several decades. As artificial intelligence continues to evolve, the handling of high-quality, high-dimensional data has emerged as a significant challenge for large-scale deep learning models. Video SCI systems, which utilize low-speed cameras to capture high-speed video, present several advantages including low memory requirements, low transmission bandwidth, low cost, and low power consumption [17, 55] . Our proposed MambaSCI algorithm enables more efficient high-quality reconstruction of videos captured with quad-Bayer pattern, significantly broadening the application scenarios of video SCI.To date, video reconstruction techniques have not demonstrated any negative social impact. Similarly, our proposed MambaSCI algorithm does not present any foreseeable negative societal consequences.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We give the pytorch type pseudo-code of the core module in the supplementary material Sec. A.2, and provide the corresponding gif format dynamic image. The code will be publicly available soon.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: The research conducted in the thesis complies in all respects with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper received 4 mixed reviews with the scores spread across a spectrum - one 3 (Reject), one 5 (Borderline Accept), one 6 (Weak Accept), and one 8 (Strong Accept). 

There was a general appreciation for the addressed problem (video compressive imaging with quad Bayer sensors, which is timely given the increasing popularity of quad-Bayer sensors), quality of the exposition, the principled nature of the proposed solution which is well-motivated, and the overall quality of the results. 

There were several concerns raised, in particular about lack of real experiments with a quad Bayer sensor. The authors submitted a response and there was considerable discussion post-rebuttal. While the concern around lack of real experiments remain (due to the relative difficulty of obtaining large scale quad-Bayer datasets), most other concerns were reasonably addressed. Therefore, on balance, an accept decision is recommended. 

The authors are strongly urged to incorporate the reviewers' feedback and include the additional results / evaluations promised in the rebuttal when preparing the final camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper presents a method for compressive video image using Mamba-Unet for quad bayer sensors.
strength_weakness: strength_weakness: - The work is not making significant contribution in terms of method. Directly applying Mamba-Unet to this problem looks unnatural. Why not use a transformer or a CNN? I dont think there will be a much of a difference.

- The writing needs improvement and section 3 is not that clear.
questions: questions: What is the primary motivation to address quad bayer sensors when they are not that prevalent?

What is the need of using Mamba-Unet for this work?
limitations: limitations: Limitations are not addressed well.
paper_summary: paper_summary: This manuscript introduces the Mamba model and Quad-Bayer CFA pattern into color video snapshot compressive imaging (SCI) for the first time. Specifically, the proposed MambaSCI adopts a non-symmetric U-shaped encoder-decoder architecture, which includes DWConv, Residual-Mamba-Blocks, and ReConv. The Residual-Mamba-Blocks integrate several modules designed to enhance reconstruction quality and edge details. Experiments demonstrate that MambaSCI outperforms comparison methods in both quantitative and qualitative results, with fewer parameters and FLOPs.
strength_weakness: strength_weakness: + The proposed MambaSCI takes X_in as input, which is obtained from initialization Block. So the framework can be seen as a simple video enhancement task. Furthermore, MambaSCI does not consider the characteristics of Quad-Bayer pattern like previous work (STFormer, EfficientSCI), but instead directly adopts or integrates popular modules.The in-depth analysis appears to be lacking in certain modules, specifically within the passages from lines 185-187 and 193-197.
	
+The lack of an explanation for the initialization block raises questions about whether special processing was done to address issues introduced by the quad-Bayer pattern. It would be better to add further visualization of the X_in results for comparison.

+In Figure 6, the results of proposed method show artifacts and color distortion in the lower left corner of ``Swinger”. More analysis should be included.

+The manuscript lacks real experiments on videos captured by quad-Bayer patterns. All the results are obtained on  simulated data. Results on real data are expected to evaluate the effectiveness of the proposed method.
questions: questions: Please refer to weakness.
limitations: limitations: Please refer to weakness.
paper_summary: paper_summary: This paper investigate video snapshot compression imaging reconstruction task by Quad-Bayer CFA pattern into color video SCI. They design a Residual-Mamba-Block consisting of ST-Mamba, Edge-Detail-Reconstruction module and Channel-wise attention module to enhance reconstruction quality and edge details. Experimental results on many datasets validate its effectiveness.
strength_weakness: strength_weakness: The contribution (ii) and (iii) have limited novelty. It seems that the author simply adapt ST-Mamba to video SCI reconstruction task without elaborated designs. The proposed Residual-Mamba-Block is an incremental combination of ST-Mamba from Vivim, DWConv and Channel-wise Attention. The author should further clarify the novelty.

More description of EDR module should be given. Why EDR module can provide more edge details?

Observed from Table 4, the improvements on large-scale simulation color video are not consistent across datasets. More analysis on such phenomenon should be given. While the author claim the results on Messi and Hummingbird is strained by parameters and FLOPs, specific quantitative comparison should be provided.
questions: questions: Why we need Mamba instead of self-attention in this task?

Why quad-Bayer pattern is better than a single Bayer-patterned measurement?
limitations: limitations: The author is suggested to compare the efficiency of the proposed method and existing methods on long videos to validate the effectiveness on long sequence.
paper_summary: paper_summary: The paper proposes a method called MambaSCI for efficient reconstruction of quad-Bayer patterned color video snapshot compressive imaging.This method surpasses state-of-the-art algorithms with lower computational and memory costs, providing improved color accuracy and demosaicing.
strength_weakness: strength_weakness: 1) The reconstruction seems to be performed on fairly low resolution images(512 x 512). I would have liked to see the simulations being done on higher resolution.
2) The proposed method is tested only on simulated datasets (since there is no real SCI dataset based on quad-Bayer pattern). To me, it is unclear how the simulated datasets have been created. Was noise added while creating the simulated datasets? My biggest concern is the way the noise has been modeled while creating the simulated dataset might not match the actual data(If there was a way to do it). Therefore, currently the efficacy of the provided solution is somewhat hypothetical.
questions: questions: 1) The paper describes the computational benefits of the proposed approach over SOTA in terms of fewer parameters and FLOPS, however no concrete numbers have been provided around the performance. Is it possible to add some information around the time it would take to reconstruct each frame?
2)  There is no mention on how the masks were generated. Or the conditioning of the masking matrix. This very important information should be included.
3) What is the relationship between T (number of frames used during reconstruction) and the amount of motion between consecutive frames? For instance I would expect you could use more frames(T) if the object was moving slowly within the scene, compared to when it is moving fast. This might help in further improving the computational cost. 
4) There is no tiling done during reconstruction. I wonder why this is so? Especially if you want to deal with higher resolution images?
limitations: limitations: Limitations have been adequately addressed in the appendix, so nothing to comment upon here.

Comments: title: Summarization
Further Explanation
Further discussion
Further Explanation
Thank you for your review！
Further explanation
comment: As the discussion phase comes to a close, we would like to summarize our key points for the convenience of the esteemed reviewers and ACs.

**New Task:**  As far as I know, we are the first to introduce quad-Bayer arrays into the field of video SCI, which responds to the trend of smartphones becoming the mainstream video shooting devices and the mainstreaming of quad-Bayer arrays in smartphone cameras, and also compensates for the serious color distortion and ghosting problems that exist in the existing methods when facing quad-Bayer array videos. Meanwhile, numerous literatures have shown that quad-Bayer arrays can provide better image quality compared with Bayer, and we hope to broaden the research scope and reconstruction quality of video SCI.

**New Approach:** For lightweight video reconstruction, we have pioneered the use of an asymmetric U-shaped architecture with customized Residual-Mamba-Blocks modules. This design enables efficient and high-quality video reconstruction, achieving superior results with fewer parameters and FLOPS compared to previous transformer and CNN-based methods, as demonstrated in **Table 2** of the main manuscript.

**Real Dataset:** As the task is still in the exploratory phase, public datasets of this class are currently targeted. Additionally, constructing the necessary SCI encoding system requires specialized optical hardware and considerable time, which often exceeds the resources of researchers focused on decoding algorithms. We are committed to developing and refining this system to fully capitalize on the advantages of quad-Bayer over Bayer. We also extend our gratitude to reviewers **zshs** and **yWFS** for highlighting potential challenges in real-world applications, which will guide us in building a robust system.

Finally, we sincerely thank all reviewers for their time and thoughtful reviews. We will carefully consider your constructive suggestions and continue to improve our work.
Thank you for the rebuttal. Most of my concerns have been addressed. However, I strongly suggest the authors include these additional experiments and analyses in the revised manuscript, and clearly describe the difference against previous Mamba-related works. I tend to maintain my score.
Thanks for the explanation, from which I can feel their frank and honest sense. While this work investigate a topic that might be interested in the area of SCI, it leaves too many aspects unconsidered, such as camera response, optical transmission, noise modelinng (Commented also by Reviewer yWFS), and mask discretization. This suggests that the research is at its early stage, and I would encourage the authors further investigate these issues.
We sincerely appreciate your careful consideration and prompt response to our rebuttal. We are readily prepared to address any further questions or concerns you may have.
Thanks for your valuable comments.

**Q1: Real System.**

**A1:** Since we are exploring a new task in the SCI field, our focus has been on addressing issues like artifacts and color  distortion when dealing with quad-Bayer array video. We appreciate your insight into the potential challenges that might arise in real-world scenarios. We will take your suggestions into account and work on developing a quad-Bayer-based coding system as soon as possible.

**Q2: Reconstruction Quality.**

**A2:** Our proposed E2E network strikes a balance between performance and efficiency. As demonstrated in **Figure 5 of the main manuscript**, at a resolution of 512x512, our method produces superior visual results compared to the cascaded "PnP-FastDvDnet + BJDD" combination. Our approach maintains high color fidelity without introducing additional artifacts, whereas the "PnP-FastDvDnet + BJDD" combination suffers from severe artifacts and shape distortions across multiple datasets. Also, we provide a comparison of the corresponding performance metrics as well as a comparison of the reconstruction times in the table below:

*Table 1: A comparison of performance metrics (PSNR (dB), SSIM) and reconstruction time between PnP-FastDVDnet and MambaSCI.* (**Note: The time for PnP-FastDvDnet does not include the additional time required for the demosaicing process performed by BJDD.**)

|Method|Beauty|Bosphours|Runner|ShakeNDry|Traffic|Jockey|Running time(s)|
|-|-|-|-|-|-|-|-|
|PnP-FastDvDnet|34.29,0.967|33.07,0.947|34.18,0.928|30.11,0.883|23.74,0.811|32.70,0.921|14.60|
|MambaSCI|**36.95,0.979**|**38.62,0.982**|**40.02,0.977**|**34.55,0.950**|**27.52,0.904**|**36.54,0.960**|**5.12**|

Table 1 clearly highlights the superior performance of our approach compared to PnP-FastDVDnet, along with a significant advantage in the time required for reconstruction.

Additionally, we are actively exploring improvements to the pipeline and have identified the following areas for enhancement:

- **Raw Domain Reconstruction Performance:** While our method performs well on middle-scale datasets, it does not achieve superior metrics on large-scale datasets, primarily due to the significant difference in resolution sizes between training (128\*128, and 256\*256 for finetuning) and testing (1920\*1080). We are trying to further investigate multi-scale generalization approaches to address this issue.
- **Improved Demosaicing Algorithms:** Similar to previous Bayer-based reconstruction methods, and for lightweight considerations, our current approach utilizes 3D convolution for the final demosaicing operation. However, the distinct structure of the quad-Bayer pattern makes it more challenging to demosaic compared to traditional Bayer arrays. This complexity can often result in color confusion and artifacts during the demosaicing process. **Recognizing this limitation, we have designed a lightweight demosaicing network that has already shown promising results in preliminary experiments.** We will continue to refine and explore this approach further.
I appreciate the authors' clarification and the new result in the Raw domain. However, I am not sure whether my concerns have been addressed.

**Q1: Real System.**
For simulation as is, changing the Bayer pattern is a simple modofication by easily replacing the old Bayer mask by the new mask in the data generation and reconstruction. Moreover, the simulation is oversimplified without considering the camera response, optical transmission, and mask discretization ($\mathbf{M} \in \mathbb{R}$ in current work), which present significant gap against practical implementation.  Glad to know that the authors are constructing a real system for the proposed pipeline, which I feel that would significantly lift the contribution and quality of this research work.

**Q2: Reconstruction Quality.**
Intuitively, if trained properly, the proposed end2end reconstruction pipeline has a great potential to outperform the cascaded “PnP-FastDvDnet + BJDD” combination. If it is not the case, I feel that it would be worth to further improve the reconstruction net.
Thanks for your valuable comments.

**Q1: Clarification about comparison on long videos.**

**A1:** Regarding your questions about run-time and performance, we'll address them as follows:

- **Performance:** As the model's FLOPs increase with $T$, our method maintains only **9.8%** of the FLOPs required by EfficientSCI. While achieving comparable PSNR to EfficientSCI, our method significantly outperforms it in SSIM metrics. Additionally, as shown in the visual comparison in **Figure 4** of the submitted PDF, our method provides superior visualization compared to EfficientSCI.
- **Run-time:** The reason why our method's runtime is not as competitive as EfficientSCI's can be attributed to two key factors:
  - **Mamba Characteristics.** Mamba's acceleration is tied to the GPU's hardware characteristics and the underlying CUDA architecture, limiting our ability to enhance its speed directly.
  - **Network Framework Optimization.** To achieve high-quality reconstruction, our framework currently uses 3D convolution for channel feature interactions at the end of each Residual-Mamba-Block. However, this process becomes more time-intensive as $T$ increases. We plan to optimize the framework further in the future to develop a lightweight, high-quality, and faster reconstruction algorithm.


**Q2: Clarification of the innovation of DWConv.**

**A2:** The depth-wise convolution used by Vivim [1] is essentially a 3D depth-wise convolutional layer, which focuses solely on spatial feature extraction. **It lacks a pointwise convolution component, meaning it doesn't account for the relationships between channels**. In contrast, the DWConv we designed **integrates both depthwise and pointwise convolution**. First, it processes spatial information through depthwise convolution, then it combines the relationships between channels via pointwise convolution to produce the final output features. This design allows us to effectively extract local detailed features and combinations between channels while reducing computational costs and the number of parameters.

Thank you for your insightful feedback. We will update the DWConv section of Figure 4 in the main manuscript accordingly and provide a more detailed explanation of DWConv in lines 186-187. Additionally, we will include further analysis in the ablation study in Section 4.5 to enhance the clarity and robustness of our findings.

[1] Yang Y, Xing Z, Zhu L. Vivim: a video vision mamba for medical video object segmentation[J]. arXiv preprint arXiv:2401.14168, 2024.
We sincerely appreciate your thorough review of our paper, careful consideration of our rebuttals, and raising the score. If you have any further questions or concerns, we are always available to provide additional clarification as necessary.
1. About comparison on long videos, when increasing T to 16 and 32, I cannot see the great advantage in run-time and performance over EfficientSCI. Could you please further explain this point?
2. The author claims the global-local design in the proposed framework compared to STMamba. However, as far as I know, there is also Depth-wise Convolution in ST-Mamba. Thus, this cannot be one of the innovations.
Thanks for the detailed response. My doubts are cleared and such a revision should help. I have increased my rating after also reading other reviews. Good luck!
Thanks for your valuable comments.

**Q1: Clarification on the lack of real experiments using videos captured by quad-Bayer patterns.**

**A1:** As we are pioneering a **new task** in the field of video SCI, which is still in its **exploratory stage**, no public dataset currently exists for this class. Meanwhile, our main innovations include:

- **New task:** Quad-Bayer offers significant advantages over traditional Bayer sensors, such as higher resolution and HDR capabilities, and is widely used in smartphone cameras. However, there have been no studies on its application in video SCI. We are **the first** to introduce quad-Bayer into video SCI, aiming to expand the field and achieve higher quality reconstruction.
- **New method:** To enable lightweight reconstruction, we are **the first** to utilize an asymmetric U-shaped architecture, employing customized Residual-Mamba-Blocks modules to achieve efficient, high-quality video reconstruction.

Furthermore, constructing the corresponding SCI encoding system requires specialized optical encoding hardware and significant time, which is often beyond the capabilities of researchers focused on decoding algorithms. 

**We are working on building and refining this system** to fully leverage the advantages of quad-Bayer over Bayer, such as capturing video in low-light environments and compressing HDR video SCI by adjusting pixel exposure within the 2x2 color grid.

**Q2: Clarification of swinger's visual comparison.**

**A2:** As detailed in lines 494-500 of the Supplementary Material of main manuscript, we utilized the high-performance BJDD [1] model for joint denoising and demosaicing to display the RGB images reconstructed by PnP-FastDvDnet. Specifically, **PnP-FastDvDnet only handles Raw domain reconstruction**, while BJDD performs further denoising and demosaicing to obtain the final RGB video. In contrast, **our model integrates both Raw domain reconstruction and demosaicing**. As a result, the visual quality of PnP-FastDvDnet may appear superior due to the BJDD network's influence. Therefore, as described in the main text, to ensure a fair comparison, we converted the RGB video output from MambaSCI back to the Raw domain and calculated metrics such as PSNR and SSIM in the Raw domain. The performance metrics of MambaSCI vs. PnP-FastDvDnet on swinger in Raw domain are compared as follows:

|Methods|PSNR (dB)|SSIM|
|-|-|-|
|PnP-FastDvDNet|28.60|0.887|
|MambaSCI|**29.78**|**0.920**|

As seen in the table, our method outperforms PnP-FastDVDnet in both PSNR and SSIM in the Raw domain. Additionally, as shown in Figure 6 and Figure 15 of the main manuscript, our approach achieves superior reconstruction in other detailed areas compared to PnP-FastDVDnet.

[1] A Sharif S M, Naqvi R A, Biswas M. "Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor." *CVPR* 2021.
1. The quad-Bayer pattern has some advantages, e.g., higher resolution and HDR capability, in convential camera imaging. However, these advantages are not necessarily inherited by the SCI configuration due to the mask modulation. Moreover, the current simulation of SCI pipline is oversimplified without considering optical effects in real scenarios. Therefore, it is crucial to have real-scene evidence to verify the significance of the SCI  task with quad-Bayer.  
The manuscript lacks real experiments on videos captured by quad-Bayer patterns. All the results are obtained on simulated data. Results on real data are expected to evaluate the effectiveness of the proposed method.

2. Regarding the result of the "Swinger", I feel that the PnP-FastDvDnet method provides better  visual quality than the proposed MambaSCI, given that both introduced some artifacts. While PnP-FastDvDnet presents some ghosting artifacts, MambaSCI brings obvious geometrical distortion: line structures are curved locally with also some ringing artifacts.