# Transformer-Based Cascading Reconstruction Network for Video Snapshot Compressive Imaging
## 3 Transformer-Based Cascading Reconstruction Network
### Overall Structure Reconstruction  
The overall structure reconstruction stage is designed as a multi-scale Transformer network with an encoder and decoder for extracting multi-scale and long-range spatiotemporal features from the initial reconstruction. Figure 3 shows the diagram of the multi-scale Transformer network. The encoder and decoder are also linked with skip connection. Figure 2: The diagram of the Transformer-based Cascading Reconstruction Network for Video Compressive Snapshot Imaging. The encoder first employs a convolution layer to adjust the number of channels of \(X_{lo}\in\mathbb{R}^{n_{x}\times n_{y}\times B}\) and outputs the feature map \(x_{0}^{e}\in\mathbb{R}^{n_{x}\times n_{y}\times C}\) with \(C\) channels. The encoder has three scales, and each scale is configured with the Residual Window Transformer (RWT) module and the down-sampling operation. The processing of each scale can be defined as  
\[x_{i}^{e}=H_{douni}(f_{RWT}(x_{i-1}^{e})),i=1,2,3, \tag{6}\]  
where \(x_{1}^{e}\in\mathbb{R}^{\frac{n_{x}}{2}\times\frac{n_{y}}{2}\times C}\), \(x_{2}^{e}\in\mathbb{R}^{\frac{n_{x}}{2}\times\frac{n_{y}}{2}\times C}\), \(x_{3}^{e}\in\mathbb{R}^{\frac{n_{x}}{2}\times\frac{n_{y}}{2}\times C}\), \(H_{douni}\) represents the \(i\)-th down-sampling and \(f_{RWT}\) denotes the decoding feature extraction mapping of the RWT module. Each RWT module contains a residual link, four TransBlocks and one convolution layer. TransBlock adopts non-overlapping local window attention in TransBlock, as in [26]. In detail, we divide the input feature maps of size \(H\times W\times C\) into non-overlapping local windows of size \(N\times M\times M\times C\). Therefore, there are a total of \(N=(H\times W)/M^{2}\) windows. Then, we calculate the correlations in each window through the self-attention mechanism. The features of the local window are reshaped into tokens \(X\in\mathbb{R}^{M^{2}\times C}\). With the embedding projection, the Query, Key and Value matrices \(Q,K,V\in\mathbb{R}^{M^{2}\times d}\) can be calculated as  
\[Q=XW^{q},K=XW^{k},V=XW^{v}, \tag{7}\]  
where \(W^{q},W^{k},W^{v}\in\mathbb{R}^{C\times d}\) represent the projection matrices, and they are shared across different windows. After that, we can calculate the self-attention matrix of each local window and update the features as  
\[Attention(Q,K,V)=SoftMax(QK^{T}/\sqrt{d}+P)V, \tag{8}\]  
where \(P\) denotes the learnable positional encoding. TransBlock uses Window Multi-head Self-Attention (W-MSA) to calculate self-attentions in parallel and concatenates the results of multi-head. In addition, the outputs of W-MSA undergo the processing of Multi-Layer Perception (MLP) and the LayerNorm (LN) layer. Two residual connections are used inside TransBlock. The whole calculation formula of TransBlock is as follows:  
\[X_{MSA}{}^{l}=W-MSA(LN(X^{l-1}))+X^{l-1}, \tag{9}\]  
\[X^{l}=MLP(LN(X_{MSA}{}^{l}))+X_{MSA}{}^{l}, \tag{10}\]  
where \(X^{l}\in\mathbb{R}^{M^{2}\times C}\) represents the \(l\)-th output of TransBlock. Figure 3: The diagram of the multi-scale Transformer network for overall structure reconstruction. The decoder has a structure symmetry with the encoder. Each scale of the decoder is configured with an RWT module and an up-sampling operation. The features of the same scale in the encoder are transmitted to the decoder through a skip connection and a concatenation operation. The calculation of each scale in the decoder can be written as  
\[{x_{i}^{d}}^{\prime}=H_{up}(f_{RWT}(x_{i+1}^{d})),i=1,2, \tag{11}\]  
\[x_{i}^{d}=Conv_{(2,c)}(Concat({x_{i}^{d}}^{\prime},x_{i}^{e})),i=1,2, \tag{12}\]  
where \({x_{2}^{d}}^{\prime},{x_{2}^{d}}\in\mathbb{R}^{\frac{m_{x}}{4}\times\frac{m_{ x}}{4}\times C}\), \({x_{1}^{d}}^{\prime},{x_{1}^{d}}\in\mathbb{R}^{\frac{m_{x}}{2}\times\frac{m_{ x}}{2}\times C}\) and \(x_{i}^{d}=x_{i}^{e}(i=3)\). \(H_{up}\) represents the up-sampling, \(Conv_{(2,C,C)}\) denotes a convolution layer that reduce the number of channels from \(2C\) to \(C\).