# Methods for Mid-Term Forecasting of Crop Export and Production
## 3 Materials and Methods
### Modeling of Wheat Export
#### 3.2.1 Regression Models
The recurrent models have the same inputs, country embeddings and outputs as in the NARX model, except that they do not require past output values explicitly (Figure 2b). First, the inputs of the model are processed with the recurrent (LSTM, GRU, or simple recurrent [15]) network with \(\mathit{elu}\) activation and four neurons, which builds a single embedding for all the past observations. Then we transform that embedding with a multilayer feed-forward network \(d_{1}..d_{m}\). As in the NARX model, the network has four layers (\(m=4\)) with alternate _linear_, \(\mathit{elu}\) and \(\mathit{relu}\) activations and four neurons in each layer Similar to the NARX-based model, we apply the self-attention layer to consider  
Figure 2: Neural network models to predict wheat export and production: (**a**) NARX model with self-attention. (**b**) recurrent network (LSTM/GRU/simple) with self-attention. (**c**) Transformer network with temporal and spatial attention. the dependencies between indicators of the countries. The network architecture after the recurrent layer is the same as for the NARX model. The recurrent networks utilize backpropagation through time and have a gradient vanishing problem, affecting the prediction in a mid- and long-time series analysis. Instead of using backpropagation through time, the Transformer-like model (Figure 2c) applies positional encoding and multi-head attention mechanisms, which address the problem [20,29]. The Transformer network has the same inputs and outputs as the recurrent ones. First, the model generates embeddings for the inputs; they encode a country and a time step. We use the same embedding layer as for the previous models to encode a country, and apply a standard position encoding approach from [29] to generate the time step related information (positional embedding size is 3). Second, the model applies the multi-head attention to catch the dependencies between past time steps and passes the results through the residual network (ResNet). The all residual networks have four layers with alternate _linear_ and _elu_ activations, four neurons in each layers, and one residual connection between the input and the output of the RestNet. Eventually, the model uses multi-head attention to reveal dependencies between the series from different countries and generates the outputs with the network \(d_{1}.d_{m}\) (\(m=4\)). We use four-head self-attention for both countries and time steps. We added _12_ regularization to all the layers of the models to make the models more stable. We also used a dropout mechanism to regularize the networks (dropout levels were chosen from the interval [0..0.2]). Adam optimizer with learning rate _0.001_ was used to perform the models training. We also utilized the early stopping mechanism to prevent overfitting. We used the MSE loss function during the training because the prediction error was usually distributed normally. All the network hyper-parameters (number of network cells at each layer, number of layers, dropout level, regularization weights) were fine-tuned with the 3-fold cross-validation.