# End-to-End Video Snapshot Compressive Imaging using Video Transformers

Wael Saideni, Fabien Courrèges, David Helbert, Jean-Pierre Cances

► To cite this version:

Wael Saideni, Fabien Courrèges, David Helbert, Jean-Pierre Cances. End-to-End Video Snapshot Compressive Imaging using Video Transformers. International Conference on Image Processing Theory, Tools and Applications (IPTA 2022), Apr 2022, Salzburg, Austria. 10.1109/IPTA54936.2022.9784128. [hal-03593050]

HAL Id: hal-03593050

https://hal.science/hal-03593050v1

Submitted on 16 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# End-to-End Video Snapshot Compressive Imaging using Video Transformers

Wael SAIDENI

XLIM Research Institute
UMR CNRS 7252

Limoges, France

wael.saideni@xlim.fr

Fabien COURREGES

XLIM Research Institute
UMR CNRS 7252

Brive-la-Gaillarde, France

fabien.courreges@unilim.fr

David HELBERT

XLIM Research Institute
UMR CNRS 7252

Poitiers, France

david.helbert@univ-poitiers.fr

Jean Pierre CANCES

XLIM Research Institute
UMR CNRS 7252

Limoges, France

cances@ensil.unilim.fr

**Abstract**—This paper presents a novel reconstruction algorithm for video Snapshot Compressive Imaging (SCI). Inspired by recent research works on Transformers and Self-Attention mechanism in computer vision, we propose the first video SCI reconstruction algorithm built upon Transformers to capture long-range spatio-temporal dependencies enabling the deep learning of feature maps. Our approach is based on a Spatiotemporal Convolutional Multi-head Attention (ST-ConvMHA) which enable to exploit the spatial and temporal information of the video scenes instead of using fully-connected attention layers. To evaluate the performances of our approach, we train our algorithm on DAVIS2017 dataset and we test the trained models on six benchmark datasets. The obtained results in terms of PSNR, SSIM and especially reconstruction time prove the ability of using our reconstruction approach for real-time applications. We truly believe that our research will motivate future works for more video reconstruction approaches.

**Index Terms**—Vision Transformers, deep learning, optimization, computer vision, video compressive sensing

## I. INTRODUCTION

Today, the huge demand for high dimensional data acquisition and processing is faced with the lack of sensors able to directly capture it [1]. To deal with this problem, Video Snapshot Compressive Imaging (SCI), a fundamental concept in signal processing, uses 2D detectors to indirectly capture high dimensional data. In fact, the original video, considered as a high dimensional signal, is coded by different coding patterns to finally generate one single 2D measurement matrix. This compression technique enables to optimize the memory storage and the transmission bandwidth of the acquisition system. Then, an advanced reconstruction algorithm is required to recover the high dimensional signal from the compressed measurement.

Recently, and inspired by deep learning, many reconstruction approaches have been proposed [12]–[20]. While these approaches achieve practical performances, the recovery process remains very challenging in terms of flexibility, scalability and the speed of the training and the testing phases [1].

In order to address the different challenges related to the existing reconstruction models, mainly based on convolutional and recurrent networks, we propose a new video reconstruction algorithm built upon Transformers [28]. Indeed, Transformers have dominated Natural Language Processing (NLP) applications in recent years. And, because of various high level

analogies between video processing and NLP, we decided to take advantage of this architecture for our video reconstruction purposes, especially when Transformers have already been exploited for various computer vision purposes such as classification [31], [32], video segmentation [33], object detection [34], and video inpainting [35].

In fact, we decided to move towards Transformers because of the extraordinary performance that have been recently demonstrated on a wide range of applications in comparison with convolutional and recurrent networks. In contrast to recurrent neural networks, Transformer based architectures can handle the complete input sequence parallelly which enable to easily learn long range relationships and accelerate the training process. While convolutional models enable the parallel processing, they are unable to learn different dependencies among all possible combinations of the input elements. That's why it is practical to process long sequences as a whole using transformers. In addition, Transformers based models are scalable to high-complexity architectures and large datasets [43].

However, different vision transformers already proposed are facing several challenges. In fact, famous vision transformers (e.g. ViT [31]) divide the 2D input images into several patches which may threaten the local spatial information [38] because some low level visual features (e.g. shapes and edges) are divided into different patches. Then, global fully-connected self-attention is applied to extract the global interactions between different tokens which ignore local details.

Bearing the above problems in mind, we design and implement a new reconstruction algorithm for video snapshot compressive imaging based on a vision Transformer with an attention layer that exploit local and spatio-temporal data information.

In a nutshell, our contributions are summarized as follows:

* To the best of our knowledge, the proposed algorithm is the first video SCI reconstruction method built upon Transformers.

* We used a convolutional attention mechanism in order to exploit spatiotemporal information instead of global fully connected attention layers used in recent vision transformers.

* We provide detailed explanation of our architecture with
detailed results and analysis which may be used as reference in future research works, especially on video transformers.

* Our model achieves strong results on DAVIS2017 training dataset, in comparison with other video SCI reconstruction algorithms based on Deep Learning or optimization methods.

## II. BACKGROUND AND RELATED WORKS

In this section, we introduce the video SCI framework and the tremendous research work done in the reconstruction phase. The video SCI framework is based on two major subsystems: the optical imaging system [2] to encode the 3D videos and the reconstruction algorithm [1]. Mathematically, the acquisition system captures the 3D video $X \in \mathbb{R}^{h \times w \times B}$ ($h, w$ are the dimensions of one frame and $B$ is the number of frames in the video) after being modulated by $B$ coding patterns $C \in \mathbb{R}^{h \times w \times B}$ having the same dimensions of the original frames. The idea behind this process is to have one single measurement matrix, called the compressed measurement $Y$:

$$Y = \sum_{k=1}^{B} X_k \circ C_k + G \quad (1)$$

Where $X_k, C_k$ represent the $k^{th}$ frame and the $k^{th}$ coding pattern, respectively. $\circ$ is the Hadamard product and $G$ denotes the noise. So, $Y$ can be defined by:

$$y = Dx + g \quad (2)$$

Where $y = \text{Vec}(Y) \in \mathbb{R}^{hw}$, $g = \text{Vec}(G) \in \mathbb{R}^{hw}$ and $x = \text{Vec}(X) = [\text{Vec}(X_1)^T, ..., \text{Vec}(X_B)^T] \in \mathbb{R}^{hwB}$. $\text{Vec}$ denotes the vectorization operator. The sparse sensing matrix $D = [D_1, ..., D_B]$ is composed of $B$ diagonal matrices. In this case, the sampling rate in video SCI is $\frac{1}{B}$.

The reconstruction process, aiming to reconstruct the original video from one single compressed measurement frame and the coding patterns $\{C_k\}_{k=1}^B$ [7], is crucial to build an efficient video SCI system and it can be categorized into two main categories: optimization based approaches and deep learning based algorithms.

The optimization based algorithms are aiming to solve a regularized optimization problem via iterative approaches. Among these algorithms, we find GAP-TV [8], GMM [9] and DeSCI [10], leading the state-of-the-art performances. These optimization based methods require huge computational resources (CPU) and large reconstruction time. DeSCI, for example, may need hours to generate a 256 x 256 x 8 signal from one single measurement matrix. However, the main drawback of these approaches is the need to run the reconstruction process of every new measurement frame without even automatically fine-tune the different parameters, which a huge time-consuming operation.

In order to overcome this problem, Deep Learning algorithms [12]–[20] have been widely exploited. However, the main drawback of these approaches is that they require large-scale

datasets and huge memory and computational resources for the training.

## III. OVERVIEW OF THE PROPOSED ARCHITECTURE: ViT-SCI

In this section, we introduce our reconstruction approach using Transformers for video SCI. The main architecture is presented in Figure 1. We start by explaining the pre-processing phase using measurement energy normalization, then we introduce our three main modules: two feature extraction modules and a reconstruction module.

### A. Video pre-processing

The input of our reconstruction model is a pre-processed video scene generated by using the compressed measurement $Y$ and the coding patterns $\{C_k\}_{k=1}^B$ used. A measurement energy normalization is applied on $Y$ [15], [21], [22] as expressed in the following equation:

$$\bar{Y} = Y \circ \sum_{k=1}^{B} C_k \quad (3)$$

Where $\circ$ denotes the matrix dot division. In order to generate an estimated average of the ground truth video scene $X \in \mathbb{R}^{h \times w \times B}$, we process the following concatenation along the 3rd dimension:

$$Inp = [\bar{Y} \circ C_1, ..., \bar{Y} \circ C_B]_3 \in \mathbb{R}^{h \times w \times B} \quad (4)$$

This pre-processing technique aims at providing an estimated video tensor that conserves the background and some objects of the original frames which will increase the performances of our model and guide the learning process.

### B. Feature Extraction Module

The pre-processed video is passed through 5 convolutional residual blocks to enable low frequency feature extraction as expressed in the following equation:

$$F_{low} = N_{f_e}(Inp) \quad (5)$$

Where $N_{f_e}$ denotes the residual blocks network aiming to extract low frequency features from the input scene.

### C. 3D Positional encoding

Since our Transformer based model is permutation-invariant, it requires an extra layer to provide a position information. This technique is highly recommended [28] because all tokens of the input sequence are fed to the network with no special order or position. Inspired by [34], the original positional encoding used in [28] is adapted for our 3D application. The spatio-temporal positional encoding is expressed as follows:

$$PE_{3D}(pos, i) = \begin{cases} \sin(\beta_k \cdot pos) & \text{for } i = 2k \\ \cos(\beta_k \cdot pos) & \text{for } i = 2k + 1 \end{cases}$$

Where $\beta = \frac{1}{10000^{\frac{6k}{d_c}}}$, pos denotes the position information in the corresponding dimension, $d_c$ is the size of the channel dimension and $k \in \mathbb{N}$ s.t. $k \in [0, \frac{d_c}{6}]$.
<figure><img src="image_2.png" /><figcaption>Fig. 1. The proposed Transformer based architecture for video reconstruction</figcaption></figure>

### D. Deep Feature Extraction Module

The Transformer based module, illustrated in Figure 2, aims to lean non-linear mapping which enable video reconstruction. The transformer architecture is inspired from [28] and it is composed of two main networks: the Spatiotemporal Convolutional Multi-head Attention (ST-ConvMHA) and the Feed Forward Network (FFN).

Fully-connected Self Attention mechanism used in NLP works [28] is designed to learn global interactions between the different tokens of the input video. In addition, this attention mechanism neglects the temporal dimension. So, proved in [23], [24], fully-connected self-attention is not suitable when dealing with vision Transformers. As a result, ST-ConvMHA is designed to overcome the previous limitations. In fact, it is a stack of multiple convolutional self-attention layers used to understand the multiple aspects of the feature maps $F_{low}$. As shown in Figure 3, we start by applying three convolutional projections instead of position-wise linear projections used in [28]. The embedding step is expressed in the following equation system:

$$ 
\begin{aligned}
Q &= W_Q \otimes F_{low} \\
K &= W_K \otimes F_{low} \\
V &= W_V \otimes F_{low}
\end{aligned} 
\qquad (6) 
$$

Where $W_Q$, $W_K$ and $W_V$ are the trained parameters related to the Query, Key and Value, respectively. $\otimes$ denotes the convolution operation. Then, we process the unfolding operation to extract sliding local tokens:

$$ 
\begin{aligned}
Q_1, \dots, Q_N &= \Theta(Q) \\
K_1, \dots, K_N &= \Theta(K) \\
V_1, \dots, V_N &= \Theta(V)
\end{aligned} 
\qquad (7) 
$$

Where $\Theta$ is the unfolding operator, $N = \frac{BWH}{W_{patch} \times H_{patch}}$ denotes the number tokens generated by the unfolding operation.

$Q_i, K_i, V_i \in \mathbb{R}^{BN \times c \times W_{patch} \times H_{patch}}, \text{s.t.} i \in \mathbb{N} \text{ s.t. } i \in [0, N]$. Then, we calculate the attention layer as follows:

$$ \text{Attention}(Q_i, K_i, V_i) = \text{softmax}\left(\frac{Q_i K_i^T}{\sqrt{\frac{\text{dim}_{patch}}{n_{\text{heads}}}}}\right) V_i \quad (8) $$

At the end, the $N$ tensors generated by the attention operation pass through a folding layer to merge the sliding local blocks into one large feature map. Then, a convolutional layer processes the feature map.

The ST-ConvMHA implemented allows to deeply learn the temporal and spatial features much better than the self-attention mechanism used in [28].

<figure><img src="image_3.png" /><figcaption>Fig. 2. Deep Feature Extraction Module</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Spatiotemporal Convolutional Multi-head Attention</figcaption></figure>

After the attention layer, we pass each vector into a feed-forward unit, which is in our case a linear layer, as in [28]. The purpose of this layer is to process the output of the attention layer in order to better fit the reconstruction module.
### E. Video Reconstruction Module

This module aims to reconstruct the video scene by passing the feature maps generated by the transformer and the feature extraction module to stabilize the training process:

$$O_f = N_{rec}(F_{deep} + F_{low}) + Inp \quad (9)$$

Where $O_f$ is final output of our reconstruction approach and $N_{rec}$ denotes the reconstruction module.

## IV. PERFORMANCE EVALUATION, COMPARISON AND DISCUSSION

### A. Datasets

Following the previous research works [13], [15], we train our algorithm on Densely Annotated Video Segmentation (DAVIS2017) [26] dataset which is originally used for video segmentation. DAVIS2017 contains 90 video clips in various scenes. To efficiently train our greedy transformer based approach in terms of training dataset, data preparation remains a paramount step. Firstly, we generate 6516 video sequences of size 8×256×256. Then, we process for a data augmentation in order to increase the performances of our model. We generate 417024 video sequences of size 8 × 64 × 64.

### B. Compared methods and performance metrics

To evaluate the performance of our proposed reconstruction model, we compare it with two well-known iterative algorithms: GAP-TV [8] and DeSCI [10] and one very recent Deep Learning approach: BIRNAT [15]. We use two famous metrics to measure the quality of reconstructed frames in comparison with the original ones: Peak Signal to Noise Ratio (PSNR) and Structural SIMilarity (SSIM). The result is calculated frame by frame and we report their mean value.

### C. Implementation details

The proposed reconstruction model is implemented with Pytorch-1.10, trained with Adam optimizer [40] with a learning rate of $3 \times 10^{-4}$. Our approach is trained for 100 epochs, with an increasing learning rate of 5% every 5 epochs. The loss function used for the implementation is Mean Square Error (MSE). For evaluating different reconstruction approaches, we use NVIDIA RTX 2080 GPU. Deep Learning based models (ours and BIRNAT) are trained in the same implementation conditions. Our approach takes about 190 hours. However, BIRNAT, that is implemented in the same conditions as our implementation (e.g. loss function, number of epochs, optimizer), takes about 400 hours. In order to reduce the complexity of the model and the training time, the number of channels in one frame is reduced to 1 (gray-scale frames).

### D. Main simulation results

We compare our proposed reconstruction method against some previous well-known models in video SCI on six benchmark datasets in Table I. The comparison is based on the reconstruction quality and the recovery speed. We notice that our Transformer based approach slightly outperforms BIRNAT on the Aerial (+1.49% and +0.5% in terms of

PSNR and SSIM, respectively) and largely outperforms GAP-TV(+32.23% and +10.19% as to PSNR and SSIM, respectively) and DeSCI(+28.33% and +8.37% as to PSNR and SSIM, respectively) on the same dataset in terms of PSNR and SSIM. We observe a limited improvement using our approach compared with DeSCI (+0.55%) in terms of PSNR on the Kobe dataset. However, SSIM average results remains much better using DeSCI on this dataset. Also, These results prove the fact that our algorithm based on ST-Conv Multi-head Attention provide better results on datasets with complex background. Also, we notice that DeSCI works better on the Drop, Runner and Traffic in terms of PSNR and SSIM. These performance results can be obviously explained: our trained dataset rarely include high speed motions which is the case for these three datasets. Our algorithm is not effectively trained to process this type of datasets. However, it outperforms BIRNAT, the other Deep Learning based state-of-the-art approach which is already trained on the same dataset. As a whole, our approach slightly outperforms DeSCI, the leader of the state-of-the-art results for video SCI on the average PSNR of the six datasets. Figure 4 proves the quantitative results. Our approach is able to reconstruct finer details and sharper edges not only on the Aerial and Kobe that have better quantitative results but also on the other test datasets, which demonstrate the impact of the Spatiotemporal Attention on the recovery process. Another interesting performance result concerns the reconstruction time: our algorithm is faster than DeSCI by 730 000 times and faster than BIRNAT by 12 times. Our reconstruction algorithm can obviously be used for real-time applications due to its reconstruction speed and good quality performances.

### E. Discussion

In this work, it is proved that vision Transformers can be used in the recovery process of video SCI using the 2D compressed measurement as an input. Although our approach achieves good qualitative and quantitative results, it presents some limitations. In fact, the algorithm has been trained on video clips of size 8 × 64 × 64, which may affect the performances of the attention mechanism since high speed motion are rarely captured and the main objects are not very well identified. In future work, we believe to come up with an optimized Transformer based architecture in terms of memory consumption to be able to train large-scale datasets (e.g. video clips of size 8 × 256). The major drawback of our algorithm is its non scalability since it requires more memory resources when we increase the dimensions of the training dataset.

## V. CONCLUSION

In this work, we have proposed the first video SCI reconstruction algorithm based on Transformers, which views the recovery model as an end-to-end decoding problem. We achieve good image quality performances compared to the well known iterative algorithms: GAP-TV and DeSCI and a recent Deep Learning approach: BIRNAT. In addition, we achieve the highest speed among all existing approaches.
<table><caption>TABLE I
AVERAGE PSNR(DB), SSIM AND RUN TIME(IN SEC) PER MEASUREMENT FOR DIFFERENT APPROACHES ON 6 EVALUATION DATASETS. BEST RESULTS ARE IN **BOLD**, SECOND BEST RESULTS ARE IN **BLUE**</caption><thead><tr><th>Algorithms</th><th>Aerial</th><th>Drop</th><th>Kobe</th><th>Runner</th><th>Traffic</th><th>Vehicle</th><th>Average</th></tr></thead><tbody><tr><td rowspan="3">GAP-TV [8]</td><td>22.09</td><td>27.73</td><td>25.74</td><td>31.29</td><td>24.17</td><td>24.72</td><td>25.95</td></tr><tr><td>0.8719</td><td>0.9141</td><td>0.7909</td><td>0.9177</td><td>0.7515</td><td>0.8700</td><td>0.8526</td></tr><tr><td>8.0</td><td>8.0</td><td>8.1</td><td>8.1</td><td>8.3</td><td>8.2</td><td>8.12</td></tr><tr><td rowspan="3">DeSCI [10]</td><td>22.76</td><td><b>36.51</b></td><td><b>31.08</b></td><td><b>38.48</b></td><td><b>31.59</b></td><td><b>26.05</b></td><td><b>31.07</b></td></tr><tr><td>0.8866</td><td><b>0.9840</b></td><td><b>0.9278</b></td><td><b>0.9609</b></td><td><b>0.9138</b></td><td><b>0.9140</b></td><td><b>0.9311</b></td></tr><tr><td>6168.2</td><td>6336.9</td><td>6396.5</td><td>6331.5</td><td>6215.3</td><td>6258.8</td><td>6284.5</td></tr><tr><td rowspan="3">BIRNAT [15]</td><td>28.74</td><td>32.77</td><td>28.96</td><td>35.41</td><td>26.49</td><td><b>28.23</b></td><td>30.10</td></tr><tr><td>0.9560</td><td>0.9626</td><td>0.8594</td><td>0.9337</td><td>0.8199</td><td>0.9019</td><td>0.9056</td></tr><tr><td>0.1050</td><td>0.1097</td><td>0.1056</td><td>0.1057</td><td>0.1087</td><td>0.1132</td><td>0.1079</td></tr><tr><td rowspan="3">Ours</td><td><b>29.21</b></td><td>35.40</td><td><b>31.25</b></td><td>37.67</td><td>28.15</td><td>25.40</td><td><b>31.18</b></td></tr><tr><td><b>0.9608</b></td><td>0.9759</td><td>0.9047</td><td>0.9509</td><td>0.8696</td><td>0.8161</td><td>0.9130</td></tr><tr><td><b>0.0092</b></td><td><b>0.0090</b></td><td><b>0.0089</b></td><td><b>0.0079</b></td><td><b>0.0089</b></td><td><b>0.0080</b></td><td><b>0.0086</b></td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Reconstructed frames of GAP-TV, DeSCI, BIRNAT and our proposed approach on six simulated video SCI datasets</figcaption></figure>

This is achieved by using our Transformer model based on ST-ConvMHA mechanism. This research can be effectively enhanced using optimized Transformer architecture especially in terms of memory consumption.

REFERENCES

[1] X. Yuan, D. Brady, and A. K. Katsaggelos, "Snapshot compressive imaging: Theory, algorithms and applications," IEEE Signal Processing Magazine, 2020

[2] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, "Coded aperture compressive temporal imaging," Opt. Express, vol. 21, no. 9, pp. 10,526-10,545, May 2013. doi: 10.1364/OE.21.010526

[3] P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, "Coded aperture compressive temporal imaging," Opt. Express, vol. 21, no. 9, pp. 10,526-10,545, May 2013.

[4] R. Koller, L. Schmid, N. Matsuda, T. Niederberger, L. Spinoulas, O. Cossairt, G. Schuster, and A. K. Katsaggelos, "High spatio-temporal resolution video with compressed sensing," Opt. Express, vol. 23, no. 12, pp. 15,992-16,007, 2015.

[5] D. Reddy, A. Veeraraghavan, and R. Chellappa, "P2c2: Programmable pixel compressive camera for high speed imaging," in CVPR 2011, June 2011, pp. 329-336.

[6] Y. Sun, X. Yuan, and S. Pang, "Compressive high-speed stereo imaging," Opt Express, vol. 25, no. 15, pp. 18, 182-18, 190, 2017.

[7] S. Jalali and X. Yuan, "Snapshot compressed sensing: Performance bounds and algorithms," IEEE Transactions on Information Theory, vol. 65, no. 12, pp. 8005-8024, Dec 2019.

[8] X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing," in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2016, pp. 2539-2543.

[9] J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, "Video compressive sensing using Gaussian mixture models," IEEE Trans. Image Process., vol. 23, no. 11, pp. 4863-4878, Nov. 2014.

[10] Y. Liu, X. Yuan, J. Suo, D. J. Brady and Q. Dai, "Rank Minimization
for Snapshot Compressive Imaging,” in IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 41, no. 12, pp. 2990-3006, 1
Dec. 2019, doi: 10.1109/TPAMI.2018.2873587.

[11] X. Yuan, Y. Liu, J. Suo and Q. Dai, "Plug-and-Play Algorithms for Large-Scale Snapshot Compressive Imaging," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020, pp. 1444-1454, doi: 10.1109/CVPR42600.2020.00152.

[12] M. Iliadis, L. Spinoulas, and A. K. Katsaggelos, "Deep fully-connected networks for video compressive sensing," Digit. Signal Process., vol. 72, pp. 9-18, Jan. 2018.

[13] Qiao, Mu Meng, Ziyi Ma, Jiawei Yuan, Xin. (2020). Deep learning for video compressive sensing. APL Photonics. 5. 030801. 10.1063/1.5140721.

[14] K. Xu and F. Ren, "Csvideonet: A real-time end-to-end learning framework for high-frame-rate video compressive sensing," Dec. 2016, arXiv:1612.05203. [Online]. Available: https://arxiv.org/abs/1612.05203

[15] Cheng, Z., Lu, R., Wang, Z., Zhang, H., Chen, B., Meng, Z., Yuan, X.: BIRNAT: Bidirectional recurrent neural networks with adversarial training for video snapshot compressive imaging. In: European Conference on Computer Vision (ECCV) (2020)

[16] Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. Reconnet: Non-iterative reconstruction of images from compressively sensed random measurements. In CVPR, 2016.

[17] Ziyi Meng, Shirin Jalali, and Xin Yuan. Gap-net for snapshot compres-
sive imaging. arXiv: 2012.08364, December 2020.

[18] Xin Yuan and Yunchen Pu. Parallel lensless compressive imaging via deep convolutional neural networks. Optics Express, 26(2):1962–1977, Jan 2018.

[19] J. Ma, X. Liu, Z. Shou, and X. Yuan, "Deep tensor admm-net for snapshot compressive imaging," in IEEE/CVF Conference on Computer Vision (ICCV), 2019.

[20] M. Iliadis, L. Spinoulas, and A. K. Katsaggelos, "Deepbinarymask: Learning a binary mask for video compressive sensing," Digital Signal Processing, vol. 96, p. 102591, 2020.

[21] Z. Cheng, B. Chen, G. Liu, H. Zhang, R. Lu, Z. Wang, and X. Yuan. Memory-efficient network for large-scale video compressive sensing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.

[22] Z. Wang, H. Zhang, Z. Cheng, B. Chen, and X. Yuan. Metasci: Scalable and adaptive reconstruction for video compressive sensing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.

[23] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021

[24] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. In ICCV, 2021.

[25] C. Mou, J. Zhang, X. Fan, H. Liu, and R. Wang. Cola-net: Collaborative attention network for image restoration. IEEE Transactions on Multimedia, 2021.

[26] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbelaez, P., Sorkine-Hornung, A., Gool, L.V.: The 2017 DAVIS challenge on video object segmentation. CoRR abs/1704.00675 (2017), http://arxiv.org/abs/1704.00675

[27] A. Horé and D. Ziou, "Image quality metrics: PSNR vs. SSIM," 2010 20th International Conference on Pattern Recognition, Istanbul, 2010, pp. 2366-2369, doi: 10.1109/ICPR.2010.579.

[28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.

[29] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.

[30] Alec Radford and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.

[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weisenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsbay. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.

[32] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.

[33] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020.

[34] Y. Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia, "End-to-end video instance segmentation with transformers," arXiv preprint arXiv:2011.14503, 2020.

[35] Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, Hongsheng Li;FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 14040-14049

[36] Hochreiter, S. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(2):107–116, 1998. Hochreiter, S. and Schmidhuber, J. Feature extraction through LOCOCODE. Neural Computation, 11(3): 679–714, 1999.

[37] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106.13230, 2021

[38] . Li, K. Zhang, J. Cao, R. Timofte, and L. Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707, 2021.

[39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019

[40] D. Kingma and J. Ba. Adam: a method for stochastic optimization. In ICLR, 2015

[41] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve J´egou. Training ´data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012.12877, 2020.

[42] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pages 14014–14024, 2019.

[43] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey. arXiv:2101.01169.