# A Spatio-Temporal Encoding Neural Network for Semantic Segmentation of Satellite Image Time Series
## 3 Methodology
### STENN Architecture
#### 3.2.2 Temporal Encoding with Transformer Encoder  
After multiple layers of dilated convolution encoding, we obtain a feature map time series with consistent spatial resolution and different depth levels of semantic information, as well as multi-scale information. Concatenating the feature map time series outputs from all layers along the channel dimension results in the feature map time series F, with a shape of \(T\times C^{\prime}\times H\times W\), where \(C^{\prime}=\sum_{n=1}^{N}C^{\prime}_{n}\). The concatenated feature map time series is then reshaped into a time series for each pixel P, with a shape of \(T\times C^{\prime}\), totaling \(H\times W\) pixels, as expressed in the following formula:  
\[F=concat[f^{\prime}_{n}],n\in[1,N] \tag{3}\]  
Finally, feature extraction is performed on each pixel's time series using the Transformer encoder, as illustrated in Figure 7.  
The images in SITS are arranged in chronological order based on their acquisition dates, and their position indices can directly correspond to their order in the sequence. Alternatively, position indices can be determined by calculating the number of days between each image's acquisition date and a fixed reference date. However, in order to capture the annual phenological patterns of crops within the image time series,  
Figure 6: Spatial encoding with dilated convolution. The feature map at each layer is obtained by concatenating the input and output along the channel dimension from the previous layer.  
Figure 7: Temporal encoding with Transformer encoder. After spatial encoding, the feature map time series is organized into a feature sequence for each pixel. Following positional encoding, this sequence is fed into the Transformer encoder, and the obtained results are reshaped to the original dimensions.  
is calculated based on the image's acquisition date \(t\) and the New Year's Day \(t_{newyear}\) of the same year. This ensures that the position of the images in the sequence corresponds to the growth stages of the represented crops. In other words, the image sequence is arranged based on the chronological order of crop growth stages, rather than the sequence of collection dates. The formula for calculating \(pos\) is as follows:  
\[pos=(t-t_{newyear})\cdot days \tag{4}\]  
As shown in Table 1, the process begins with data preprocessing to obtain an SITS \(X\in\mathbb{R}^{T\times C\times H\times W}\) for each land parcel, arranged by acquisition time, where \(T\) represents the number of image acquisitions, and the lengths may vary. Following this, the feature maps \(f_{n},n\in[1,N]\) are obtained through the dilated convolution encoder \(e^{n},n\in[1,N]\). The feature maps from each layer are then concatenated along the channel dimension to form the feature map \(F\in\mathbb{R}^{T\times C^{\prime}\times H\times W}\). Subsequently, the feature map F is reshaped into a time series for each pixel, resulting in \(P\in\mathbb{R}^{T\times C^{\prime}}\), where the time series has a length of \(T\) and a feature dimension of \(C^{\prime}\). The pixel-wise time series data \(P\), along with position encoding, is fed into the Transformer encoder, which processes all pixel-wise data to generate spatio-temporal features for the entire image, producing the spatio-temporal feature map \(F^{\prime}\in\mathbb{R}^{T\times C^{\prime}\times H\times W}\). Finally, the spatio-temporal feature map \(F^{\prime}\) undergoes temporal averaging and convolution to yield the final segmentation result \(R\).