# A Spatio-Temporal Encoding Neural Network for Semantic Segmentation of Satellite Image Time Series
## 5 Discussion  
Comparison Analysis. In Table 3 and Figure 8c, the performance of ConvLSTM and ConvGRU demonstrates that LSTM performs better in handling time series data. Furthermore, the combination of using U-Net as the backbone for spatial feature extraction and extracting temporal features through ConvLSTM or Bidirectional ConvLSTM does not yield satisfactory results. The 3D U-Net can simultaneously handle both 2D spatial and 1D spectral data, providing shorter inference times and good performance. Our proposed model has less than half the number of parameters compared to U-TAE. Furthermore, under the same OA conditions, our model's mIoU outperforms the U-TAE by 8 percentage points. This indicates that, when dealing with complex land parcels, our model excels in the delineation of land object boundaries, exhibiting superior robustness and generalization capabilities. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Model** & **Param (\(\times\)1000)** & **OA** & **mIoU** & **IT (ms)** \\ \hline STENN (Ours) & **447** & **79.37** & **55.8** & 322 \\ U-TAE & 1087 & 79.31 & 47.82 & **88** \\ ConvLSTM & 1009 & 77.47 & 54.06 & 163 \\ ConvGRU & 956 & 70.31 & 36.93 & 141 \\ U-ConvLSTM & 1521 & 75.27 & 36.19 & 110 \\ U-BiConvLSTM & 1210 & 76.14 & 42.32 & 104 \\
3D U-Net & 1554 & 76.92 & 47.21 & 96 \\ \hline VGG Backbone & 401 & 78.18 & 55.26 & 193 \\ No dense connection & 403 & 77.26 & 53.16 & 245 \\ No Transformer encoder & 289 & 73.41 & 31.91 & 153 \\ Single Date (August) & 289 & 58.75 & 31.12 & 151 \\ Single Date (May) & 289 & 55.16 & 30.43 & 151 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Segmentation result of different approach. The best performance is displayed in bold, and the second best performance is underlined. Based on the comparison in Figure 9, it is evident that our model has shown significant improvements in recognizing many land cover categories, especially Grapevine (8), Winter durum wheat (11), Fruits, vegetables, flowers (12), Potatoes (13), and Sorghum (18). This improvement primarily stems from U-TAE harnessing spatial features predominantly, while training attention weights solely on the temporal dimension. Instances of misclassification occur when the target land cover shares visual similarities with the background in crucial temporal segments, leading to erroneous categorization into the background. According to Figure 10, it is visually evident that our model's segmentation results outperform other methods significantly. Other methods exhibit clear instances of small object omissions, large object edge dissolution, and missed areas (highlighted in red circles). Additionally, comparing the patch with the ground truth, it is apparent that the accuracy of the PASTIS dataset requires improvement. Ablation Analysis. First, we replaced the spatial encoding with VGG, and the different resolution feature maps were simply up-sampled without the gradual resolution restoration used in FCN or U-Net. We observed a 1% drop in OA. We also verified the effectiveness of the dense connections. The experimental results show that models without dense connections decreased by 2% in both OA and mIoU. Extracting temporal features proved to be necessary, as removing the Transformer encoder resulted in a significant performance  
Figure 8: During the training and validation process, the OA (**a**) and mIoU (**b**) of the STENN model. The change in mIoU for all models on the validation set during the training process (**c**). All models using the same learning rate may lead to oscillations during the training process for some models (red circles). Figure 9: The confusion matrices for STENN (**a**) and U-TAE (**b**) display the classification accuracy of these two models when predicting crop categories. Specifically, each entry in the matrix at the \(i\) row and \(j\) column represents the proportion of samples belonging to class \(j\) that were classified as class \(i\). The darker the color, the larger the proportion, indicating better model performance.