## 2 Coded Aperture Compressive Temporal Imaging (CACTI)
### Spatiotemporal Super-Resolution Reconstruction of Image Sequences by Belief Propagation  
Given the current motion vector and preliminary estimation of image sequence, the estimation can be conducted on HR images with the following equation as:  
\[\hat{I}=\underset{l}{\text{argmin}}\sum_{p}\left(\|\mathbf{D}\mathbf{K}I-f_{0 }\|+\mu(\|\text{TV}(I)\|)\right)+\sum_{(p,q)\in NS}(\beta_{i}\|\mathbf{D} \mathbf{K}\mathbf{M}_{i}I-f_{i}\|) \tag{33}\]  
where the first term is the data term incorporating the data distortion and TV total variation regularization, and the second is the penalty term. \(\mathbf{D}\) is the down-sampling matrix, \(\mathbf{K}\) is the blur kernel matrix, \(\text{TV}(I)\) represents the total variation operator of \(I\), \(\mathbf{M}_{i}\) the matrix of flow \(\mathbf{w}\) which can be calculated by optical flow or SIFT Flow methods, \(f_{0}\) the image on current frame, \(f_{i}\) the preliminary estimation of image, and \(NS\) is the weighted neighborhood system where Point \(p\) exists. \(\mu\) is the weight parameter which are set to values as \(\mu=0.7\). However, there should be a way to reflect the contribution of different frames to the current super-resolution reconstruction, to demonstrate the spatiotemporal relationship in the process. Accordingly, the weighted neighborhood system is proposed herein based on the motion vector. The image patches in Figure 3 represent these image patches from different frames. The space vector \(\mathbf{v}_{0}^{k}\), which represents the motion vector between the \(k\) frame and its adjacent frame \(k+N\) (\(N\) is the number of frames for SR reconstruction, and \(N\) can be larger than \(N_{f}\) in order to utilize the information between multi-observations \(y_{i}\)), can be obtained by the fitting of the motion vectors among different frames, such as \(\mathbf{v}_{1}^{k}\), \(\mathbf{v}_{2}^{k}\), \(\mathbf{v}_{3}^{k}\) and \(\mathbf{v}_{4}^{k}\):  
\[\mathbf{v}_{0}^{k}=\mathbf{v}_{1}^{k}+\mathbf{v}_{2}^{k}+\mathbf{v}_{3}^{k}+ \mathbf{v}_{4}^{k} \tag{34}\]  
The inter-frame motion vectors of different frames can be projected onto the motion vectors between the current frame and the frame \(N\), to mark the spatiotemporal proximity between the pixels. In this way, the weight coefficient can be settled for spatiotemporal SR reconstruction:  
\[w_{ij}=\frac{\|\mathbf{v}_{0}^{k}\mathbf{v}_{j}^{k}\|}{\|\mathbf{v}_{0}^{k}\| \|\mathbf{v}_{j}^{k}\|} \tag{35}\]Therefore, \(\beta_{i}\) is the weight factor of WNS and can be expressed as:  
\[\beta_{i}=\omega_{fk}(p,q)=1-\xi\exp\left(-\gamma\left|\frac{\|\overrightarrow{v _{0}}\overrightarrow{v_{f}}\|}{\|\overrightarrow{v_{0}}\|\|\overrightarrow{v _{f}}\|}\right|}{\|\overrightarrow{v_{f}}\|\|\overrightarrow{v_{f}}\|}\right|\right) \tag{36}\]  
where \(\xi\) is the normalization coefficient ensuring \(\beta_{i}\subset[0,1]\); \(\gamma\) is the attenuation coefficient, which can be set as 1 for most application. MAP-MRF can be adopted for the estimation on multi-frame reconstructed HR images and results in the energy minimization problem being suitable for inference algorithms based on graph cuts or belief propagation [29]. As an iterative method for the probabilistic inference of the probability graph model, the BP (Belief Propagation) algorithm is one of the key algorithms in solving MRF problems. In the BP algorithm, all the information is passed on in a parallel way, as shown in Figure 3. The nodes exchange information between one another to update the entire status of MRF. The algorithm works as the robust iterative approximate computation of MRF, which can reach the optimal solution through convergence.