## 4 Convolutional and Recurrent Neural Networks for Pixel-Based Crops Classification  
### Formulation  
A single multi-temporal, multi-spectral pixel can be represented as a two-dimensional matrix \(X^{(i)}\in\mathbb{R}^{tsb}\) where \(t\) and \(b\) are the number of time steps and spectral bands, respectively. Our goal is to compute from \(X^{(i)}\) a probability distribution \(F(X^{(i)})\) consisting of \(K\) probabilities, where \(K\) is equal to the number of classes. To achieve this objective, we propose a compact representation learning architecture composed of three main building blocks:  
* **Time correlation representations**--this operation extracts temporal correlations from multi-spectral, temporal pixels \(X^{(i)}\) exploiting a sequence-to-sequence recurrent neural network based on long short-term memory (LSTM) cells. A final time-distributed layer is used to compress and maintain a sequence like structure, preserving the multidimensionality nature of the data. In this way, it is possible to take advantage of temporal and spectral correlations simultaneously. * **Temporal pattern extraction**--this operation consists of a series of convolutional operations followed by rectifier activation functions that non linearly maps each elaborated temporal and spectral patterns onto high dimensional representations. So, RNN output temporal sequences are processed by a subsequent cascade of filters, which in a hierarchical fashion, extracts essential features for the successive stage. * **Multiclass classification**--this final operation maps the feature space with a probability distribution \(F(X^{(i)})\) with \(K\) different probabilities, where \(K\), as previously stated, is equal to the number of classes. The comprehensive pipeline of operations constitutes a lightweight, compact architecture able to non-linearly map multi-temporal information with its intrinsic nature, achieving results considerably better than previous state-of-the-art solutions. Human brain mental imagery studies [70], where images \begin{table}
\begin{tabular}{l l c c} \hline \hline
**Bands Used** & **Description** & **Central Wavelength (\(\upmu\)m)** & **Resolution (m)** \\ \hline Band 2 & Blue & 0.49 & 10 \\ Band 3 & Green & 0.56 & 10 \\ Band 4 & Red & 0.665 & 10 \\ Band 8 & Near-infrared & 0.705 & 10 \\ NDVI & (Band8-Band4)/(Band8+Band4) & - & 10 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Bands used in this study. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Date** & **Doy** & **Sensing Orbit \#** & **Cloud Pixel Percentage** \\ \hline
7/4/2015 & 185 & 22-Descending & 0 \\
8/3/2015 & 215 & 22-Descending & 0.384 \\
9/2/2015 & 245 & 22-Descending & 4.795 \\
9/12/2015 & 255 & 22-Descending & 7.397 \\
10/22/2015 & 295 & 22-Descending & 7.606 \\
2/19/2016 & 50 & 22-Descending & 5.8 \\
3/20/2016 & 80 & 22-Descending & 19.866 \\
4/29/2016 & 120 & 22-Descending & 18.61 \\
6/18/2016 & 170 & 22-Descending & 15.52 \\
7/18/2016 & 200 & 22-Descending & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Sentinel-2 data acquisition. are a form of internal neural representation, inspired the presented architecture. Moreover, the joint effort of RNN and CNN distributes the knowledge representation through the entire model, exploiting one of the most powerful characteristics of deep learning known as distributed learning. An overview of the overall model, dubbed Pixel R-CNN, is depicted in Figure 3. Each pixel is extracted contemporary from all images taken at different time steps \(t\) with all its spectral bands \(b\). In this way, it is possible to create an instance \(X^{(i)}\), which can feed the first layer of the network. Firstly, the model extracts temporal representations from the input sample. Subsequently, these temporal features are further enriched by the convolutional layers that patterns in a hierarchical manner. The overall model act as a function \(F(X^{(i)})\) that map the input sample with its related probabilities \(K\).