Towards Real-time Video Compressive Sensing on Mobile Devices

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-26 07:35:29

Abstract: Video Snapshot Compressive Imaging (SCI) uses a low-speed 2D camera to capture high-speed scenes as snapshot compressed measurements, followed by a reconstruction algorithm to retrieve the high-speed video frames. The fast evolving mobile devices and existing high-performance video SCI reconstruction algorithms motivate us to develop mobile reconstruction methods for real-world applications. Yet, it is still challenging to deploy previous reconstruction algorithms on mobile devices due to the complex inference process, let alone real-time mobile reconstruction. To the best of our knowledge, there is no video SCI reconstruction model designed to run on the mobile devices. Towards this end, in this paper, we present an effective approach for video SCI reconstruction, dubbed MobileSCI, which can run at real-time speed on mobile devices for the first time. Specifically, we first build a U-shaped 2D convolution-based architecture, which is much more efficient and mobile-friendly than previous state-of-the-art reconstruction methods. Besides, an efficient feature mixing block, based on the channel splitting and shuffling mechanisms, is introduced as a novel bottleneck block of our proposed MobileSCI to alleviate the computational burden. Finally, a customized knowledge distillation strategy is utilized to further improve the reconstruction quality. Extensive results on both simulated and real data show that our proposed MobileSCI can achieve superior reconstruction quality with high efficiency on the mobile devices. Particularly, we can reconstruct a 256 × 256 × 8 snapshot compressed measurement with real-time performance (about 35 FPS) on an iPhone 15. Code of this paper will be released.

Main paper: 
 introduction: 
 To record high-speed scenes, researchers usually rely on high-speed cameras which suffer from high hardware cost and require large transmission bandwidth. Inspired by compressed sensing [7] , video Snapshot Compressive Imaging (SCI) provides a promising solution as it can capture high-speed scenes using a low-speed 2D camera with low bandwidth. As shown in Fig. 1(a) , there are two main stages in a video SCI system: hardware encoding and software decoding [35] . In the hardware encoding process, we first modulate the high-speed scene with different random binary masks, and then the modulated scene is compressed into a series of snapshot measurements which are finally captured by a low-cost and low-speed 2D camera. So far, many successful video SCI hardware encoders [6, 9, 21, 39] have been built. In the software decoding stage, the captured snapshot measurements and the modulation masks are fed into a reconstruction algorithm to retrieve the desired high-speed video frames. In this aspect, numerous video SCI reconstruction algorithms [4, 5, 28, [30] [31] [32] , mostly based on the deep neural networks, have been proposed with superior reconstruction quality. Therefore, it is time to consider the real-world applications of the video SCI system.Nowadays, mobile devices such as smartphones or embedded devices are fast evolving, which inspires us to seek the possibility of implementing a mobile video SCI system. When considering the mobile deployment, the biggest challenge is the limited computational resources in the mobile platforms. However, the inference of previous video SCI reconstruction algorithms such as BIRNAT [5] , RevSCI [4] , STFormer [27] , and EfficientSCI [26] is usually complex. In other words, it is difficult to deploy previous video SCI reconstruction algorithms on the low-end mobile devices. Therefore, existing video SCI reconstruction methods run on server-side, resulting in dependency on Internet connection, speed, and server usage costs for the potential users of the video SCI system.To the best of our knowledge, mobile video SCI reconstruction has never been explored within the research community. In this paper, we propose the first video SCI reconstruction network to be deployed on the mobile devices, dubbed MobileSCI which is computationally more efficient than previous state-of-the-art (SOTA) deep learning-based reconstruction methods. Specifically, we first revisit the network design of previous video SCI reconstruction methods to identify the unfriendly operations when deploying on the mobile devices. Particularly, we find that complex operations such as deep unfolding inference and 3D convolutional layers are the primary computational bottlenecks hindering the mobile deployment of previous video SCI reconstruction methods. Bearing these in mind, we build a U-shaped 2D convolution-based architecture which is much more efficient and mobile-friendly than previous SOTA reconstruction methods. Besides, to reduce model size and computational complexity and improve network capability of our proposed MobileSCI model, we introduce the channel splitting and shuffling mechanisms to build an efficient feature mixing block as a novel bottleneck block. Finally, knowledge distillation by a stronger teacher network also improves the reconstruction quality of the proposed MobileSCI. As shown in Fig. 1 (b), our MobileSCI-KD network can achieve comparable reconstruction quality with much faster inference speed than previous SOTA reconstruction methods on the mobile devices as well as the NVIDIA GPU platform. In particular, our MobileSCI-KD runs 45× and 10× faster than previous SOTA BIRNAT on an iPhone 15 and a NVIDIA Jetson Orin Nano platform, respectively.Our contributions can be summarized as follows:• We propose MobileSCI, to the best of our knowledge, the first mobile video SCI reconstruction network. • A U-shaped architecture is built with computational efficient and mobile-friendly 2D convolutional layers. Following this, an efficient feature mixing block, based on the channel splitting and shuffling mechanisms, is introduced as a novel bottleneck block of our proposed MobileSCI to further alleviate the computational burden. • We implement a customized knowledge distillation strategy which further improves the reconstruction quality.• Comprehensive experimental results show that our MobileSCI network can achieve superior performance with much better real-time performance than previous SOTA methods, especially on the mobile devices. 
related work 2.1 video sci reconstruction methods: 
 Current video SCI reconstruction methods can be divided into traditional iteration-based methods and deep learning-based ones. The traditional iteration-based methods formulate the video SCI reconstruction process as an optimization problem with regularization terms such as total variation [34] and Gaussian mixture model [33] , and solve it via iterative algorithms. The major drawback of these iteration-based methods is their time-consuming iterative optimization process. To improve the running speed, Yuan et al. propose to plug a pre-trained denoising model into each iteration of the optimization process [36, 37] . However, it still take a long time to reconstruct large-scale scenes.Recently, researchers begin to utilize deep neural networks in the video SCI reconstruction task. For example, BIRNAT [5] uses a bidirectional recurrent neural network to exploit the temporal correlation. MetaSCI [30] is designed to solve the fast adaptation problem of video SCI by introducing a meta modulated convolutional network. RevSCI [4] builds the first end-to-end 3D convolutional neural network, and adopt the reversible structure to save model training memory. Wang et al. build the first Transformer-based video SCI reconstruction method [27] with space-time factorization and local self-attention mechanism. After that, Wang et al. develop an efficient reconstruction network [26] based on dense connections and space-time factorization. Combining the idea of iteration-based methods and deep learning-based ones, several deep unfolding networks [31, 32, 40] are proposed.Due to their superior performance, we favor the deep learningbased reconstruction methods. However, it is challenging to deploy them on the mobile devices due to the existence of the complex operations. Therefore, in this paper, we mainly focus on developing mobile-friendly video SCI reconstruction algorithm with careful network design.So far, a variety of computational efficient architectures have been proposed which are more suitable for mobile deployment. The first well-known architectures are the MobileNet family. Among them, MobileNet V1 [11] first introduces the depth-wise separable convolutional layers as a basic unit to build lightweight deep neural networks. Following this, MobileNet V2 [22] proposes the inverted residual structure which is more memory efficient. Finally, MobileNetV3 [10] is tuned to mobile devices through a combination of hardware-aware network architecture search complemented by the NetAdapt algorithm. After that, numerous MobileNet Variants [2, 8, 24, 25] have been proposed with superior performance. Another emerging branch is the mobile Transformer design. For example, EdgeViTs [18] In the channel projection layer, we first split the input feature X 𝑖𝑛 along the channel dimension as X 1 and X 2 . Then, X 1 undergoes two 1 × 1 convolutional layer followed by a LeakyReLU function to get the output feature X 𝑐1 . Next, we concatenate X 𝑐1 and X 2 to obtain the output feature. Finally, a channel shuffling operation is implemented. Note that, to better propagate the extracted features, we add some shortcut connections in the proposed MobileSCI network.Transformer with a two-way bridge in between [3] . By rethinking the mobile block design for efficient attention-based networks, [14] and [38] present two novel lightweight vision Transformer architectures: EfficientFormer and EMO.The success of the above-mentioned mobile architectures and the fast evolving mobile devices motivate us to deploy the video SCI reconstruction algorithms on the mobile devices. Therefore, in this paper, we are the first to explore the mobile-friendly network design for video SCI reconstruction. 
preliminary: video sci system: 
 Here, we describe the forward model of the video SCI system. Let {X 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑛 𝑥 ×𝑛 𝑦 denotes a 𝑇 -frame high-speed scene to be captured in a single exposure time, where 𝑛 𝑥 , 𝑛 𝑦 represent the spatial resolution of each frame and 𝑇 is the compression ratio (Cr) of the video SCI system. Following this, the modulation process can be modeled as multiplying {X 𝑡 } 𝑇 𝑡 =1 by pre-defined masks{M 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑛 𝑥 ×𝑛 𝑦 , that is Y 𝑡 = X 𝑡 ⊙ M 𝑡 ,where {Y 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑛 𝑥 ×𝑛 𝑦 and ⊙ denote the modulated frames and Hadamard (element-wise) multiplication, respectively. Finally, a low-speed 2D camera is used to capture the modulated high-speed scene as a snapshot compressed measurement Y ∈ R 𝑛 𝑥 ×𝑛 𝑦 . Thus, the forward process of the video SCI system can be fomulated as,EQUATIONwhere N ∈ R 𝑛 𝑥 ×𝑛 𝑦 denotes the noise.In the decoding stage of the video SCI system, we can get the desired video frames { X𝑡 } 𝑇 𝑡 =1 by feeding the compressed measurement Y and the modulation masks {M 𝑡 } 𝑇 𝑡 =1 into a video SCI reconstruction network. 
our proposed methods 4.1 motivation: 
 In this section, we rethink previous video SCI reconstruction algorithm designs from a mobile efficiency perspective. First of all, iteration-based reconstruction methods are not applicable for mobile deployment due to their time-consuming property. Secondly, the deep unfolding framework is not feasible for the mobile devices due to the complex deep unfolding inference procedure. Finally, we turn our attention to the end-to-end (E2E) deep neural network-based reconstruction methods. On the one hand, several 2D convolution-based methods including MetaSCI [30] and U-net [20] enjoy high efficiency. However, the reconstruction quality is unsatisfactory (lower than 32𝑑𝐵). On the other hand, the E2E video SCI reconstruction methods such as RevSCI [4] and Eff-cientSCI [26] , can achieve superior reconstruction quality (higher than 33𝑑𝐵). However, they are build upon 3D convolutional layers or the non-local self-attention modules which are computational heavy and thus not friendly for the mobile devices. Now, we take EfficientSCI [26] as an example. It can present the best efficiency performance along with high reconstruction quality when testing on the NVIDIA GPU platform. However, due to the memory constrain, we cannot deploy EfficientSCI on the mobile devices. Base on the above analysis, we believe that an E2E deep neural network with 2D convolutional layers shall be a mobile-friendly and efficient video SCI reconstruction algorithm design.The overall architecture of our MobileSCI is shown in Fig. 2 (a). In the initialization stage, inspired by [5, 27] , we use the estimation module to pre-process measurement (Y) and masks (M) as follows,EQUATIONwhere ⊘ represents Hadamard (element-wise) division, Y ∈ R 𝑛 𝑥 ×𝑛 𝑦 is the normalized measurement, which preserves a certain degree of the background and motion trajectory information, and X 𝑒 ∈ R 𝑛 𝑥 ×𝑛 𝑦 ×𝑇 represents the coarse estimate of the desired video. We then take X 𝑒 as the input of the proposed network to get the final reconstruction result.Our proposed MobileSCI network is mainly composed of three parts: 𝑖) feature extraction module, 𝑖𝑖) feature enhancement module, and 𝑖𝑖𝑖) video reconstruction module. Firstly, the feature extraction module is composed of a 2D convolutional layers with a kernel size of 3 × 3, followed by a LeakyReLU activation function [17] . With the proposed feature extraction module, we can effectively project the input video frames into the high-dimensional feature space and produce the shallow features. After that, inspired by CST [1] which has been successfully applied on the hyperspectral SCI reconstruction task [13] , we adopt a U-shaped structure in the feature enhancement module of the MobileSCI model to generate the deep features. Finally, the video reconstruction module, composed of a 2D convolutional layer with a kernel size of 3 × 3, is able to conduct video reconstruction on the deep features output by the feature enhancement module.The proposed U-shaped feature enhancement module consists of an encoder E, a bottleneck B and a decoder D. Among them, E consists of two convolutional blocks and two downsample modules. The downsample module is a strided 3×3 convolutional layer which can downscale the feature maps and double the channel dimension. B is a convolutional block or an efficient feature mixing block. D contains two convolutional blocks and two upsample modules. The upsample module is a 1 × 1 convolutional layer followed by a PixelShuffle operation [23] . In each convolutional block, we stack several convolutional units. As shown in Fig. 2 (b), the convolutional unit is a residual style module which is composed of two 3 × 3 convolutional layers followed by a LeakyReLU activation function. Note that, the channel number of the features keeps consistent in the convolutional unit. Finally, to alleviate the information loss during rescaling, we add some shortcut connections between the encoder E and the decoder D.To further enhance our MobileSCI, we aim to reduce its model size and alleviate its computational burden. The key challenges are two-folds: 𝑖) Mobile attention modules [14, 38] have achieved superior performance on multiple vision tasks. However, when applying attention modules on the extracted features with large spatial size, the real-time performance will be largely worsen. 𝑖𝑖) The inverted residual structure [22] is not feasible for the U-shaped architecture because channel number of the bottleneck layers is not the constrain for improving the reconstruction quality. Bearing the above in mind, we propose an efficient feature mixing block, based on the channel splitting and shuffling mechanisms [16] , as a novel bottleneck structure of the proposed MobileSCI. As shown in Fig. 2(c ), our proposed feature mixing block is composed of two feature mixer layers with a shortcut connection. Feature Mixer Layer: As shown Fig. 2(d), The feature mixer layer contains a channel projection layers and a 3 × 3 depth-wise convolutional layer. To mix features at the channel locations, we adopt point-wise MLPs to perform channel projection. Besides, we introduce the channel splitting and shuffling mechanisms to reduce computational cost in the channel projection layer. Specifically, as shown Fig. 2 (e), in each channel projection layer, we first split the input feature X 𝑖𝑛 along the channel dimension as X 1 and X 2 . Then, X 1 undergoes two point-wise MLPs followed by a LeakyReLU function to generate the output feature X 𝑐1 . Among them, the first MLP expands the channel number twice, and the second MLP reduces the channel number by half. Next, we concatenate X 𝑐1 and X 2 to obtain the output feature. Finally, a channel shuffling operation is employed to enable the information exchange on the concatenated feature. The above procedure of the channel projection layer can be expressed as follows,[X 1 , X 2 ] = Split(X 𝑖𝑛 ),EQUATIONwhere Split(•) and Shuffle(•) denote the splitting and shuffling of features in the channel dimension. W 1 and W 2 are the pointwise convolutional layers. 𝜎 represents the LeakyReLU function. Concat(•) is the concatenation operation.As established in Sec. 3, our proposed method takes the measurement (Y) and the corresponding masks ({M 𝑡 } 𝑇 𝑡 =1 ) as inputs, and then generates the dynamic video frames ({ X𝑡 } 𝑇 𝑡 =1 ∈ R 𝑛 𝑥 ×𝑛 𝑦 ). Given the ground truth ({X 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑛 𝑥 ×𝑛 𝑦 ), we choose the mean square error (MSE) as our loss function, which can be expressed as,EQUATIONFor the knowledge distillation process, we choose a deeper Mo-bileSCI model as the teacher network due to its strong representation ability. Then we put a knowledge-distillation loss between the L 𝐾𝐷 = 1 𝑇𝑛 𝑥 𝑛 𝑦 𝑇 ∑︁ 𝑡 =1 ∥ X𝑘 -X𝑡 ∥ 2 2 .(5)Therefore, the final loss function for our MobileSCI network will be a weighted combination:EQUATIONwhere 𝜆 1 = 1, 𝜆 2 = 1 in our experimental settings.  
experiments: 
 Hardware Implementation: The optical setup of the real video SCI system is shown in Fig. 3 . The encoding process can be summarized as follows: First, the reflected light from the target scene is imaged onto the surface of the digital micromirror device (DMD) (TI, 2560 × 1600 pixels, 7.6𝜇𝑚 pixel pitch) via a camera lens (Sigma, 17-50/2.8, EX DC OS HSM) and the first relay lens (Coolens, WWK10-110-111). Then, the projected dynamic scene is modulated by the random binary masks loaded in the DMD. Finally, the encoded scene is projected onto a low-speed 2D camera (Basler acA1920, 1920 × 1200 pixels, 4.8𝜇𝑚 pixel pitch) with the second relay lens (Coolens, WWK066-110-110), which is captured in a snapshot manner. In our experiments, the camera works at 50 FPS, thus the equivalent sampling rate of the real built video SCI system is 50 × Cr FPS.Real Data Acquisition Procedure: The first essential step of the real data acquisition process is to record the modulated masks in a right way. Specifically, we first place a Lambertian white board at the objective plane of the camera lens. Then, each mask pattern (M 0 ) which is projected on the surface of DMD, will be recorded sequentially. Moreover, to eliminate the influence of background noise and nonuniform light distribution, we need to capture two extra images corresponding to a pure white (O with all '1's) and black mask (Z with all '0's), respectively. Finally, we can get the actual mask (M) with (7) .EQUATIONwhere ⊘ stands for Hadamard (element-wise) division.Training and Testing Datasets: Following BIRNAT [5] , we choose DAVIS2017 [19] with resolution 480 × 894 as the training dataset. We compare our proposed MobileSCI network with the traditional iteration-based methods (GAP-TV [34] , PnP-FFDNet [36] , PnP-FastDVDnet [37] , and DeSCI [15] ) and the deep learning-based ones (U-net [20] , MetaSCI [30] , BIRNAT [5] , RevSCI [4] , GAP-CCoT [28] , STFormer-S [27] , and EfficientSCI-T [26] ) on the simulated testing datasets. In this section, the peak-signal-to-noise-ratio (PSNR) and structured similarity index metrics (SSIM) [29] are used as the evaluation metrics of the reconstruction quality. Running time on the GPU platform is adopted as the evaluation metrics of the algorithm efficiency. Moreover, for a fair comparison, we test all the deep learning-based reconstruction methods on the same NVIDIA RTX 3090 GPU. We customize the MobileSCI model by setting the channel number of the embedding output feature, the number of the convolutional unit in each convolutional block and the number of the feature mixing block in the bottleneck block as 64, 6, and 1, respectively. Additionally, the final MobileSCI-KD model is obtained by distilling from a stronger network. We can observe from Tab. 1 that our proposed MobileSCI-KD can achieve much better real-time performance with comparable reconstruction quality than previous SOTA methods. In particular, i) The PSNR value of our MobileSCI-KD surpasses U-net and MetaSCI by a large margin on average (4.47𝑑𝐵 and 2.20𝑑𝐵, respectively), while reducing the testing time by more than 2.0×. ii) The PSNR value of our MobileSCI-KD model surpasses BIRNAT and GAP-CCoT by 0.61𝑑𝐵 and 0.39𝑑𝐵 on average, while reducing the testing time by about 6.7× and 5.3×, respectively. iii) Our MobileSCI-KD can achieve comparable reconstruction quality with RevSCI and STFormer-S, while the testing time is reduced by more than 12.7× and 9.3×, respectively. iv) Although EfficientSCI-T outperforms our MobileSCI-KD by about 0.3𝑑𝐵, it required more than 4.7× longer inference time on the NVIDIA RTX 3090 GPU. Besides, EfficientSCI-T cannot be deployed on the mobile devices.For visualization purposes, we plot several reconstructed video frames in Fig. 4 , where we can see from the zooming areas in each selected video frame that our MobileSCI-KD can provide comparable high-quality reconstructed images with previous SOTA STFormer-S, and EfficientSCI-T. Especially for the complex Traffic, Crash and Aerial scenes, we can observe sharp edges and more details in the reconstructed frames of our proposed MobileSCI-KD. We also measure the actual inference speed of our MobileSCI model on real mobile devices with input resolution as 256 × 256 × 8. Due to the extensive use of 3D convolutional layers or Transformer modules, which consume a large amount of memory, some of the previous SOTA methods including RevSCI, GAP-CCoT, STFormer-S, and EfficientSCI-T cannot be deployed to resource-limited mobile devices such as mobile phones and embedded devices. Thus, we compare our MobileSCI model with previous SOTA BIRNAT in this section. As shown in Tab. 2, our MobileSCI is more than 40× and 10× faster than BIRNAT on the mobile phones and NVIDIA embedded device, respectively. Particularly, our MobileSCI can achieve real-time (35 FPS) reconstruction on an iPhone 15.We further test our proposed MobileSCI on the real data captured by the video SCI system. Here, it is worth mentioning that due to the mismatch between the camera and DMD, the real captured masks cannot meet the binary distribution as that of the simulated random binary masks. Thus, we need to finetune the model which is pretrained on the simulated training data. The training data pairs for finetuning are generated with the original scenes in DAVIS2017 and the real captured masks. In the finetuning process, we first set the learning rate as 0.0001. Then we can finetune for 20 epochs on the training data with a 128 × 128 resolution, followed by 10 epochs on the training data with a 256 × 256 resolution.Effect of the feature mixing block: To evaluate the effectiveness of various efficient modules in our MobileSCI model, we conduct experiments on the efficient modules including the mobile attention module proposed in EfficientFormer V2 [14] , the inverted residual structure shown in MobileNet V2 [22] and our proposed feature mixing block. In the baseline model, we adopt the convolutional unit (shown in Fig. 2(b )) in the whole U-shaped architecture. Then, we replace the convolutional unit in the bottleneck block with the proposed feature mixing block to get our MobileSCI-Base model.We can get the following observations from Tab. 3 that: i) Ef-ficientFormer V2 outperforms the baseline model with slightly 0.04𝑑𝐵, while the inference time increases by about 2×. ii) Mo-bileNet V2 can help reduce the model size and slightly alleviate the computational burden. However, the reconstruction quality drops by more than 0.17𝑑𝐵 with no improvement on the real-time performance. iii) Our proposed feature mixing block is extremely lightweight and more efficient. Specifically, compared with the baseline model, the model size and computational complexity of our MobileSCI-Base model reduce by 2.14×, 18%, along with a 7.4% improvement on the inference speed. Moreover, the reconstruction quality is guaranteed. Thus, our proposed feature mixing block can achieve the best trade-off between accuracy and efficiency. Then, we replace the convolutional units at different parts of the U-shaped architecture with the proposed feature mixing block. Keeping the same testing time on the same NVIDIA RTX 3090 GPU, we sequentially replace the convolutional units at the bottleneck block ("Case 1"), Conv Block 2&3 ("Case 2") and Conv Block 1&4 ("Case 3") of our MobileSCI model. As shown in Tab. 4, "Case 1" outperforms "Case 2" and "Case 3" by about 0.23𝑑𝐵 and 1.04𝑑𝐵, respectively. Therefore, replacing the convolutional units only at the bottleneck block can better ensure the reconstruction quality. Effect of knowledge distillation: In this section, we study different knowledge distillation strategies. The teacher MobileSCI network is obtained by setting the channel number of the embedding output feature, the number of the convolutional unit in each convolutional block and the number of the feature mixing block in the bottleneck block as 64, 24 and 4, respectively. We first implement knowledge distillation on the randomly initialized student model ("Strategy 1"). However, we can see from Fig. 6 that this brings no performance improvement on the proposed MobileSCI-Base. Therefore, we customize a novel knowledge distillation strategy ("Strategy 2"). Specifically, we select one layer every four layers from the teacher model to initialize the student model. Following this, we conduct knowledge distillation on the initialized student model. As shown in Fig. 6 , "Strategy 2" brings 0.15𝑑𝐵 PSNR improvement of the reconstruction quality with much faster convergence speed than "Strategy 1".Figure 6 : Loss curve and PSNR value of different knowledge distillation strategies. In "Strategy 1", we implement knowledge distillation on the randomly initialized student model.In "Strategy 2", we first initialize the student model with selective layers from the teacher model. Then, we conduct knowledge distillation on the initialized student model. 
conclusion: 
 In this paper, we propose the first real-time mobile video SCI reconstruction method, dubbed MobileSCI, with effective network design. Specifically, a U-shaped architecture with 2D convolutional layers is built as the baseline. Following this, an efficient feature mixing block based on the channel splitting and shuffling mechanisms, is proposed to reduce computational cost and improve network capability of our proposed MobileSCI. Finally, a customized knowledge distillation strategy is introduced to further improve the reconstruction quality. Extensive experiments on both simulated and real testing data show that our proposed MobileSCI can achieve a good trade-off between accuracy and efficiency. More importantly, combining the proposed optical setup with our MobileSCI network, we contribute a promising way to build a whole mobile video SCI system with real-time performance. 
 
 Appendix:  


Decision: The paper received mixed reviews, but the rebuttal helped reviewers better understand the paper and its contributions. Based on the reviews, the rebuttal, and updates, it is leaning more towards acceptance than rejection. The paper focuses on "efficient on-device execution" using state-of-the-art techniques and optimizations. While it doesn't propose a new technique, it optimizes existing ones, and it is believed that this work deserves space at ACM MM and would lead to valuable discussions.

Reviews: review: paper_summary: paper_summary: The paper aims to deploy the above pipeline in mobile phones (here iPhone 15). However, the main problem is related to the reconstruction network. Some currently available reconstruction networks cannot be deployed on mobile phones, and if they do, there are issues related to reconstruction quality, latency, or memory.

The paper proposes a MobileSCI reconstruction network which is much more efficient. The innovation is done in three steps:

(i) A U-shaped architecture is built with 2D convolutional layers (**mobile friendly).
(ii) A feature mixing block is introduced into the architecture to reduce computational burden and model size.
(iii) The knowledge distillation strategy is utilized to improve the reconstruction quality further.
strength_weakness: strength_weakness: **The results are shown for iPhone 15, which I understand is not a low-end mobile device. The mobile phone has sufficient resources.

**Bottleneck: The term is introduced by the ResNet paper. It refers to a procedure where we reduce the number of channels, apply the convolution, and restore the channels again.

**The paper claims it is the first to explore the mobile-friendly network design for video SCI reconstruction.

The network has three components:

(i) Feature extraction module
(ii) Feature enhancement module
(iii) Video reconstruction module

Paper’s main component: Feature enhancement module!!

U-shaped Feature Enhancement Module

Components: Encoder, Bottleneck, and Decoder

Encoder: Two convolutional blocks and two downsampling modules (reduce dimension and double the channel dimensions)

Decoder: Two convolutional blocks and two upsampling modules
limitations: limitations: **From the results on the synthetic dataset, it is clear that in terms of latency, MobileSCI-KD outperforms every current SCI model on the NVIDIA GPU 3090.
Only the BIRNAT SCI model is compared on iPhone 15 with the MobileSCI-KD since we cannot deploy other SCI models on mobile phones (large memory need). 

**No quantitative analysis is shown for testing on real data. Only qualitative analysis is shown. Qualitatively it is showing comparable performance with other SCI models.


**After analyzing the RevSCI and EfficientSCI, I found the architecture of MobileSCI is slightly different due to the:

(i) 2D convolution blocks are there while the earlier papers have 3D convolutional blocks. The feature mixing block is itself present in the above EfficientSCI paper, but the only difference is that MobileSCI uses 2D convolutional units.

**The paper should address why the earlier methods were using the 3D convolutional blocks how they found that 2D convolutional blocks are more efficient and what will be the side effects of ignoring the 3D convolutional blocks.

(ii) The knowledge distillation strategy is famous. Here, it is used to improve the reconstruction quality. But no novelty is found.

**Further, the entire analysis provided by the paper has already been done. The paper is incomplete in terms of giving details about working reasons and mathematical equations for the efficiency of the proposed feature mixing block.
The paper should have to include a great analysis of the only thing it is proposing. 

**The supplementary material has no analysis. I think they have submitted it just for the formality.
paper_summary: paper_summary: In this paper, the authors propose the first mobile-friendly method for Video Compressive Sensing. The authors propose to use a UNet architecture with a shuffleNet block in the bottleneck layer for efficient video compressive sensing. The proposed method is 45x faster than previous state-of-the-art EfficientSCI-T while achieving comparable results. Moreover, the authors have shown that the proposed method can run on an iPhone 15 at 30 FPS.
strength_weakness: strength_weakness: 1) This paper proposes a mobile-friendly approach for video compressive sensing that is 45x faster than the earlier state-of-the-art method (EfficientSCI-T)
2) The authors have implemented the proposed method on several mobile devices and have benchmarked the execution time. The proposed method can run at 30 FPS on iPhone 15
3) The authors have also experimented with other efficient architectures such as MobileNetV2 and shown that using ShuffleNet-based architecture yields the best results
limitations: limitations: 1) The authors use a standard UNet architecture with a feature mixing block bottleneck layer. However, the proposed feature mixing block is a standard ShuffleNet unit. The use of knowledge distillation for efficient execution is also a known technique. Hence, the novelty of this paper is weak. 

2) In Fig.5 for the water balloon scenario, the authors have shown slight improvement in a static region. However, the main purpose of compressive sensing is to recover regions with fast motion (i.e. the balloon region). Hence the authors should ideally show examples where improvements can be observed in regions with fast motion

3) L573: "EfficientSCI-T cannot be deployed on mobile devices" - Can you elaborate more on this?

4) Can you show a peak memory analysis on a mobile device against RevSCI, which is a memory-efficient method? For efficient on-device execution, both execution time and memory need to be analyzed.
paper_summary: paper_summary: The paper primarily introduces MobileSCI, a real-time mobile video compression-aware sensing and reconstruction network based on the Spatial Compression Imaging (SCI) technique. This is the first application of video SCI technology on mobile devices. MobileSCI achieves real-time high frame rate video reconstruction on mobile devices while maintaining high reconstruction quality by optimizing the network structure, utilizing 2D convolutional layers, and employing an efficient feature fusion block based on channel splitting and shuffling mechanisms. The reconstruction performance is further improved through knowledge distillation strategy. MobileSCI achieves real-time performance, approximately 35 FPS, on mobile devices such as iPhone 15, surpassing existing methods.
strength_weakness: strength_weakness: 1. The accomplishment of real-time generation on mobile devices, as demonstrated in this article, is truly remarkable. Implementing inference on mobile devices poses significant challenges that require careful optimization and consideration of resource constraints.
2. The experimental section showcases the performance of MobileSCI on both simulated and real-world data. Compared to existing methods, MobileSCI achieves significant improvements in real-time performance while maintaining similar reconstruction quality. Particularly on mobile devices, MobileSCI outperforms BIRNAT by a wide margin, demonstrating its capability for real-time video compression-aware sensing and reconstruction on resource-constrained devices.
limitations: limitations: 1. In this article, when I saw the U-Shape network, the first thing that came to my mind was the structure of U-Net. But there is no discussion between them. The difference between the U-Shape 2D convolutional network to U-Net in this article needs to be emphasized more effectively. 
2. Writing needs to be improve, but it's ok to read for now.
paper_summary: paper_summary: In this work, a light-weight video Snapshot Compressive Imaging (SCI) method is proposed to achieve real time SCI reconstruction on mobile devices. A U-shaped 2D convolution-based architecture with 2D convolutional layers is adopted as the baseline. A feature-mixture model with channel splitting and shuffling mechanisms is proposed to reduce computational cost and improve network capacity. Knowledge distillation strategy is introduced to further improve the performance.
strength_weakness: strength_weakness: * The performance is good, the proposed method achieves comparable reconstruction quality with lower time consumption.
* The acquisition of the real SCI data is well designed and demonstrated. The authors could consider opening the collected real test data.
* The running speed comparison on mobile device is innovative, it could be instructive if the more test details could be opened.
limitations: limitations: * Novelty is somehow limited. The architecture of the network is common for low complexity applications and no specific network design/discussion neither insights oriented for SCI task are discussed in the manuscript. The superiority of the method compared to other low-complexity SCI methods, e.g., MetaSCI and UNet is not explained, neither. And the knowledge distillation is also a common strategy in deep learning methods.
* In table 3, the number of parameters of the bottleneck layer in the baseline method seems to occupy an excessive proportion. This is counterintuitive and makes the effectiveness of the proposed feature mixing block on complexity reduction less convincing.
* The writing of the manuscript is somehow slapdash and the review suggests using more polishing.