# The Paradigm Shift in Hyperspectral Image Compression: A Neural Video Representation Methodology
## 3 Proposed Methods
### The Processes of the Proposed Method  
The overall framework of the proposed method is illustrated in Figure 2b, where a framework of a joint spatial coordinate index with a frame index is adopted. Figure 2: Video-based hyperspectral image compression framework. (**a**) Frame-index-only framework (FIO-F). (**b**) Joint spatial coordinate index with frame index framework (JSCIFI-F). MLP denotes the multilayer perceptron. In Figure 2a, hyperspectral image \(I\) is first segmented into \(k\) three-channel tensors, which can be grouped into a list as \(V=[v_{0},v_{1},v_{2},\ldots,v_{k-1}]\). Then, according to the theory of NeRV [15], \(V\) can be represented by a neural network \(f_{\theta}:\mathbb{R}^{1}\rightarrow\mathbb{R}^{3\times W\times H}\), where the input of \(f_{\theta}\) is frame index \(t\), and the output is the corresponding frame, namely the three-channel image. Based on this, the storage capacity taken by the neural network parameters represents the size of the compressed hyperspectral image. For the classical deep learning-based compression issue, \(Loss=R+\lambda D\), the optimization of \(R\) is then decoupled from the loss function, and the neural network only needs to learn how to use a predetermined number of parameters to learn how to recover a higher-quality image output. To achieve interdomain conversion, the frame index number needs to be positionally encoded, because directly feeding the parameters into the network for overfitting training does not achieve satisfactory results [16; 21]. Specifically, the position encoding in [33] is employed:  
\[\mathcal{P}(t)=\left(\sin\left(b^{0}\pi t\right),\cos\left(b^{0}\pi t\right), \ldots,\sin\left(b^{l-1}\pi t\right),cos\left(b^{l-1}\pi t\right)\right) \tag{9}\]  
where \(b\) and \(l\) are pre-defined hyperparameters. \(t\) is the normalized input between [0, 1] using the max number of indexes. \(\mathcal{P}(t)\) is then input into the following network architecture to obtain the fitting function. This can be formulated as follows:  
\[M_{o}=MLP(\mathcal{P}(t)) \tag{10}\]  
\[N_{o}=NeRV_{N-1}(\cdots(NeRV_{0}(M_{o}))) \tag{11}\]  
\[\hat{v}_{t}=Conv(N_{o}) \tag{12}\]  
where \(\hat{v}_{t}\) denotes the reconstructed \(t\)-th frame, and \(N_{o}\) is generated by using \(N\) NeRV blocks. Then, the final output \(\hat{V}\) can be characterized as follows:  
\[\hat{V}=Concat(\hat{v}_{0},\hat{v}_{1},\ldots\hat{v}_{k-1}) \tag{13}\]  
where \(Concat(\cdot)\) is the concatenate operation for the channel dimension. In Figure 2b, the input of JSCIFI-F employs the same input \(V\) with FIO-F. However, since FIO-F directly employs the frame index to directly obtain the full spatial-temporal information of hyperspectral image \(I\), this leads to parameter redundancy in the setup of the network architecture, which fails to achieve a more compact information representation. Therefore, in JSCIFI-F, the input is decomposed into temporal and spatial contexts for separate coding to achieve more compact parameter learning. The embedding \(E_{m}\) generation process of JSCIFI-F can be formulated as follows:  
\[\gamma=F_{1}(\mathcal{P}(t))\odot\mathrm{S} \tag{14}\]  
\[E_{m}=F_{fuse}(\gamma,F_{2}(\mathcal{P}(t))) \tag{15}\]  
where \(F_{fuse}\) denotes a feature extraction network with a mixture of a transformer block and an MLP block, and \(F_{1}\) and \(F_{2}\) are MLP layers with fewer parameters which output a \(d\)-dimensional vector. S is a spatial context embedding with a size of \([d\times h\times w]\). The position embedding is decomposed into a temporal vector, \(F(\mathcal{P}(t))\), and a spatial vector, S. Finally, a feature fusion layer is adopted to fuse the spatial and temporal vectors into the position embeddings. To enable vector S to contain the spatial information, the same position encoding with FIO-F is adopted; then, a transformer layer with single-head attention is adopted for feature extraction.