## 4 Convolutional and Recurrent Neural Networks for Pixel-Based Crops Classification
### Formulation
#### 4.1.1 Time Correlation Representation  
Nowadays, a popular strategy in time series data analysis is the use of RNNs that have proven excellent results in many fields of the application over the years. Looking at the simplest possible RNN shown in Figure 4, composed of just one layer, it looks very similar to a feedforward neural network, except it also has a connection going backward. Indeed, the layer is not only fed by an input vector \(x^{(i)}\), but it also receives \(h^{(i)}\) (cell state), which is equal to the output neuron itself, \(y^{(i)}\). So, at each time step \(t\), this recurrent layer receives an input 1-D array \(x^{(i)}_{t}\) as well as its own output from the previous time step, \(y^{(i)}_{(t-1)}\). In general, since the output of a recurrent neuron at time step \(t\) is a function of all inputs from previous time steps, it has, intuitively, a sort of memory that influences all successive outputs. In this example, it is straightforward to compute a cell's output, as shown in Equation (1). \[y^{(i)}_{t}=\phi(x^{(i)}_{t}\cdot W_{x}+y^{(i)}_{(t-1)}\cdot W_{y}+b), \tag{1}\]  
Figure 3: An overview of the pixel recurrent-convolutional neural networks (R-CNN) model used for classification. Given a multi-temporal, multi-spectral input pixel \(X^{(i)}\), the first layer of long short-term memory (LSTM) units extracts sequences of temporal patterns. A stack of convolutional layers hierarchically processes the temporal information. where, in the context of this research, \(x_{t}^{(i)}\in\mathbb{R}^{(1*b)}\) is a single time step of a pixel with \(n_{inputs}\) equal to the number of spectral bands b. \(y_{t}^{(i)}\) and \(y_{(t-1)}^{(i)}\) are the output of the layer at time \(t\) and \(t-1\), respectively, \(W_{x}\) and \(W_{y}\) are the weights matrices. It is important to point out that \(y_{t}\) as \(x_{t}^{(i)}\) are vectors and they can have an arbitrary number of elements, but the representation Figure 4 does not change. Simply, all neurons are hidden in the depth dimension. Unfortunately, the basic cell just described suffers from major limitations, but most of all it is the fact that, during training, the gradient of the loss function gradually fades away. For this reason, for the time correlation representation, we adopted a more elaborated cell known as the peephole LSTM unit, see Figure 5. That is an improved variation of the concept proposed in 1997 by [PERSON] and [PERSON] (1997). The key idea is that the network can learn what to store in a long-term state, \(c_{(t)}\) what to throw away and what to use for the current state \(h_{(t)}\) and \(y_{(t)}\) that, as for the basic unit, are equal. That is performed with simple element-wise multiplications working as \"_valves_\" for the fluxes of information. Those elements, \(V_{1}\), \(V_{2}\), and \(V_{3}\) are controlled by fully connected (FC) layers that have as input the current input state \(x_{(t)}\) and the previous short-term memory term \(h_{(t-1)}\). Moreover, for the peephole LSTM cell, the previous long-term state \(c_{(t-1)}\) is added as an input to the FC of the forgot gate,\(V_{1}\), and the input gate, \(V_{2}\). Finally, the current long-term state \(c_{t}\) is added as an input to the FC of the output gate. All \"gates controllers\" have sigmoid as activation functions (green boxes) instead of tanh ones to process the signals themselves (red boxes). So, to summarize, a peephole LSTM block has three signals as input and output; two are the standard input state \(x_{(t)}\) and cell output \(y_{(t)}\). Instead, \(c\) and \(h\) are the long and short-term state, respectively, that the unit, utilizing its internal controllers and valves, can feed with useful information. Formally, as for the basic cell seen before, Equations (2) and (7) summarizes how to compute the cell's long-term state, its short-term state, and its output at each time step for a single instance.