# An Overview on Deep Learning Techniques for Video Compressive Sensing
## 5 Video Single-Pixel Imaging and Video Snapshot Compressive Imaging
### Video Snapshot Compressive Imaging  
Compressing high-speed videos is already possible due to the huge research work done in video snapshot compressive imaging (SCI). The video SCI system is composed of two main networks: the hardware encoder and the software reconstruction (decoder) network [68]. The hardware decoder represents the optical imaging framework and the software decoder denotes the reconstruction algorithm. The hardware decoder aims to compress the 3D video signal into a 2D measurement and the compression is done across the third dimension (the temporal dimension in this case). This compression aims to avoid huge memory storage and transmission bandwidth. The optical system is called the coded aperture compressive temporal imaging (CACTI) [69] system. In this system ad during one exposure time, the video scene is gathered by an objective lens and then coded by a temporal-variant mask (shifting physical mask [69; 70] or different patterns on a Digital Micromirror Device (DMD) [7; 71]). Then, the output is detected by a Charge Coupled Device (CCD) and then integrated into one single measurement frame. From a mathematical perspective, a video SCI system captures a dynamic scene of B frames \(X\in\mathbb{R}^{h\times x\mapsto S}\) (h and w are the height and the weight of the frame, respectively) is modulated by B masks \(C\in\mathbb{R}^{h\times x\mapsto S}\) before being integrated into one single measurement frame \(Y\in\mathbb{R}^{h\times w}\) by a camera sensor in one exposure time (B frame). This operation is expressed as follows:  
\[Y=\sum_{k=1}^{B}X_{k}\circ C_{k}+G \tag{15}\]  
where \(\circ\) and \(G\in\mathbb{R}^{h\times x\mapsto}\) denote the Hadamard product and noise, respectively. Then, we define \(y=Vec(Y)\in\mathbb{R}^{hw}\) and \(g=Vec(G)\in\mathbb{R}^{hw}\). Correspondingly, we define \(x\in\mathbb{R}^{hw}\) as:  
\[x=Vec(X)=[Vec(X_{1})^{T},\dots,Vec(X_{B})^{T}]^{T} \tag{16}\]  
The measurement \(y\) can then be expressed as:  
\[y=[D_{1},...,D_{B}]x+g \tag{17}\]  
Figure 10: Model of Single Pixel Imaging. where \(D_{b}=diag(Vec(C_{b})\in\mathbb{R}^{hw\times hw}\), for \(b=1\dots B\). We have in this case a matrix \([D_{1},\dots,D_{B}]\) that is highly structured and sparse. Depending on the theoretical study in [72], the original video can be reconstructed from the single measurement frame \(y\) (Figure 11). The second important part of video SCI is the reconstruction process which aim to recover the original video from the 2D measurement frames and the masks. This process is crucial to have a practical and efficient video SCI system. In the literature, the reconstruction algorithms could be categorized into two categories: optimization based methods and Deep Learning based algorithms. The optmization based algorithms, such as GAP-TV [44], GMM [43], DeSCI [73], and PnP-FFDNet [74], require huge computational resources and large reconstruction time. For instance, DeSCI, that has led recently the state-of-the-art optimization based approaches, takes hours to generate a \(256\times 256\times 8\) video from one single measurement frame). However, GAP-TV is a fast algorithm but it can not provide a good reconstruction. In general, to use an algorithm in a real world application, we need a PSNR \(\geqslant\) 30 which is not the case for GAP-TV [74]. In Deep Learning based methods [34; 45; 48; 53; 75; 76; 77; 78; 79], these problems have been ameliorated. Indeed, [PERSON] et al. [75] proposed a bidirectional neural network based method to reconstruct the video frames from the measurement matrix and the masks by exploiting the correlation of sequential frames. The idea behind this approach, illustrated in Figure 12, is based on two main sub-networks: A deep convolutional neural network (CNN) with ResBlock [80] and a self attention module [81] in order to reconstruct the first frame (reference frame), and a bidirectional neural network to reconstruct the rest of the frames. To improve the quality of the reconstruction, an adversarial training is defined with the Mean Square Error (MSE) loss.