# An Overview on Deep Learning Techniques for Video Compressive Sensing
## 4 Video Compressive Sensing
### Temporal VCS
Then the measurement formed on the detector plane is given by the function \(g\):  
\[ \begin{split} g(x^{\prime},y^{\prime},t^{\prime})=\int_{1}^{N_{x}} \int_{1}^{N_{y}}\int_{1}^{N_{t}}[f(x,y,t)T(x,y,t)\\ \times p(\frac{x-x^{\prime}}{\Delta},\frac{y-y^{\prime}}{\Delta} )p_{t}(\frac{t-t^{\prime}}{\Delta t})]dxdydt\end{split}  \tag{8}\]  
where \(T\) is the time modulation introduced by the DMD, \(\Delta\) the pixel pitch, \(\Delta t\) the camera integration time, \(N_{x}\) and \(N_{y}\) the spatial dimensions space, \(N_{t}\) the temporal dimension, \(p\) and \(p_{t}\) the functions of spatial and temporal pixel sampling. The sampling of the pixel is discrete and follows the following equation:  
\[Y=\sum_{k=1}^{B}X_{k}\circ C_{k}+G \tag{9}\]  
where \(B\) is the number of pixels, \(X\) is the high speed frames, \(C\) is the coding patterns, \(G\) represents the noise and \(\circ\) is the Hadamard product. Figure 5: Video SCI. Let \((i,j)\) the position of the pixel and thus the above equation becomes:  
\[y_{i,j}=\sum_{k=1}^{B}c_{i,j,k}x_{i,j,k}+g_{i,j} \tag{10}\]  
We define: \(x=[x_{1}^{T},\dots,x_{B}^{T}]^{T}\) where \(x_{k}=Vec(X_{k})\). We have \(D_{k}=diag(Vec(C_{k}))\) for \(k=1,\dots,B\). It is obvious that our problem is a compressive sensing problem:  
\[y=\phi x+g \tag{11}\]  
where \(\phi\in\mathbb{R}^{n\times nB}\) is the detection matrix (which is only dense when \(n=n_{x}n_{y}\)), the signal \(x\in\mathbb{R}^{nB}\) and \(g\in\mathbb{R}^{n}\) the noise vector. The matrix \(\phi=[D_{1},\dots,D_{k}]\) consists of diagonal matrices. It is now clear that the goal of this problem is to reconstruct the signal \(x\) from the measurements \(y\). As a result, the E2E-CNN model has been proposed. However, this model needs a large database and huge execution time. In addition, if we change the matrix \(\phi\), the neural network must execute another learning process which needs another temporal data. To cope with this, PnP framework is needed to use pre-trained data in an optimization framework in order to establish an equilibrium between the flexibility of the algorithm and its running time. E2E-CNN architecture, represented in Figure 6, is based on convolutional encoder-decoder architecture. It consists of 5 residual blocks for the encoder and 5 other blocks for the decoder and the two structures are connected by 2 convolutional layers. Each convolution is followed by ReLU activation function and a batch normalization. In addition, the output of a residual block of the decoder is added to the input of the residual block of the mapped decoder. In this architecture, the authors did not use pooling layers nor the oversampling in order not to lose the details of the images. The loss function of this model is:  
\[L_{CNN}=\alpha||x-\hat{x}||_{2}^{2}+\beta[1-MS.SSIM(x,\hat{x})] \tag{12}\]  
where \(MS.SSIM\) is multiscale structural similarity index between the output of the network. The actual values of \(x\), \(\alpha\) and \(\beta\) are predetermined. Figure 6: E2E-CNN Architecture. It has been said before that E2E-CNN suffers from a problem of flexibility (for different tasks and different compression ratios) which means that when we change the measurement matrix \(\phi\), we are forced to retrain our model which requires other databases and more execution time. This problem will be corrected by the PnP algorithm that allows to reconstruct \(x\) from \(y\) and \(\phi\):  
\[\hat{x}=\operatorname*{argmin}_{x}\frac{1}{2}||y-\phi x||_{2}^{2}+\tau R(x), \tag{13}\]  
where \(\tau\) is an equilibrium parameter between the \(l_{2}\) norm and the deep denoising prior \(R(x)\) used to solve the minimization problem without re-training the model which enables the flexibility of the algorithm. To solve Equation (13), the ADMM technique could be applied [48]. In addition, a denoising problem could be faced and then FFDNet algorithm is needed to solve it. The only drawback with the FFDNet is the undesirable artifacts produced with high compression ratios.