## 2 Proposed Method
### (3+2)D Features Pyramid Network  
Recent advances in computer vision suggest that Features Pyramid Networks (FPNs) [10] are very effective in detecting objects at different scales, but they are also a great tool for many image segmentation problems [17; 18]. However, traditional FPNs are designed for 2D images. Here, we propose an FPN network that uses 3D kernels on the input time series and on almost the entire network, except on the last layers near the output where we use 2D kernels to be able to segment a map. Figure 2 shows a graphical representation of the proposed end-to-end model. The leftmost column represents the path from the bottom (input data) to top, i.e., the feed-forward computation of the backbone ConvNet. For this bottom-up pathway, a known model is used and in Figure 2, we used a modified version of the ResNet 50, but we conducted experiments with others models. Unlike [10] where the network is designed for large images, we change the model in some places to be able to work with small image in general (i.e., \(48\times 48\) pixels size in the specific case). For example, as input, we use a kernel size equals to \((7,3,3)\), with \(\text{stride}=(1,1,1)\) and \(\text{padding}=(3,1,1)\), to avoid arriving at the blocks \(c4\) or \(c5\) (see Figure 2) with an image size \(=1\times 1\). We removed the smooth layers after the blocks \(p4\), \(p3\) and \(p2\) (see Figure 2) to avoid destroying the signal too much since we use very small images. Figure 2: Summary scheme of the proposed convolutional model. The green layers on the left represent the bottom-up pathway, here represented by a modified version of the ResNet 50. The output of the last layer of each block is used to enrich the top-down path (yellow layers) by lateral connection. The feature maps from the bottom-up pathway and the top-down pathway are merged by element-wise addition. This set of feature maps is called \(\{P2,P3,P4,P5\}\), corresponding to \(\{C2,C3,C4,C5\}\) that are, respectively, of the same shapes. To obtain the last block for prediction \(\{52,53,54,55\}\), some 3D kernels are used which lead to features maps with a dimension of size 1 that will be removed. The outputs of the last two 2D convolutions are used by the MSE Loss and the Cross-Entropy Loss to calculate the segmentation map and the class activation intervals. Let \(I\in\mathbb{R}^{B\times T\times H\times W}\) be the time series having \(T\) images \(H\times W\) with \(B\) channels, and let \(\hat{y}=f(I)\) be the part of the network that transforms the image \(I\) into the NDVI index [19] (see Equation (2)) associated with each pixel. That is, \(\hat{y}\in\mathbb{R}^{T\times H\times W}\) is the output of the second last layer called NDVI layer. The final output \(y_{i,j,c}\) of the model, calculated for each pixel \((i,j)\) of the segmentation map and for each class \(c\in C\), can be written with the following convolution operation between a kernel \(T\times K\times K\) and the \(\hat{y}\) values:  
\[y_{i^{\prime},j^{\prime},c}=\sum_{t=1}^{T}\sum_{j=1}^{K}\sum_{i=1}^{K}w_{i,j,t }^{c}\cdot\hat{y}_{i^{\prime}+i,j^{\prime}+j,t} \tag{1}\]  
Our bottom-up and top-down blocks are of the same size to avoid losing too much information from time series. Furthermore, in the semantic block before the output, we switch from 3D kernels to 2D kernels. Finally, in the NDVI layer we use an MSE loss function to predict the following Normalized Difference Vegetation Index (NDVI)  
\[NDVI= \frac{NIR-RED}{NIR+RED}=\frac{B8-B4}{B8+B4} \tag{2}\]  
where \(B8\) and \(B4\) are two spectral bands of the Sentinel-2 satellite in which each pixel corresponds to a geographical area of \(10\times 10\) m. NDVI is a radiometric measure of the photosynthetically active radiation absorbed by chlorophyll in green leaves and is, therefore, a simple indicator for assessing the presence and quantity of green vegetation in the observed target. In agriculture monitoring, time series of NDVI allows to follow the planting period, plant growing and harvesting of crops.