# Prediction of Harvest Time of Apple Trees: An RNN-Based Approach
## 3 Materials and Methods
### Evaluation Methodology
#### 3.3.4 Evaluation Scenarios  
To evaluate the PredHarv model, tests were performed in four evaluation scenarios. The use of these scenarios aims to perform a performance estimate and provide information about the operation of the model and how it can be improved. The evaluation of the model was carried out considering the following scenarios:  
* Scenario 1: The first evaluation scenario is defined using the validation methodology called K-fold cross-validation (CV). CV is a basic form of cross-validation to assess a model's generalizability from a dataset [34]. The CV divides the systematic measures of data into k groups and performs a cross-validation that produces a more robust evaluation set than just the model once. Figure 7 illustrates how the method works. For each sample, performance is evaluated according to the metrics described in Section 3.3.5.
* Scenario 2: In the second evaluation scenario, we run tests with real data from dataset1 and dataset2. We use samples from the datasets to analyze the actual values against the predicted values returned by the model. The samples were selected according to the validation scheme used in the evaluation scenario 1. The datasets were divided into subsets with part of the data for training and another for testing, respecting the temporal order of the data. For each sample, the RMSE and MAE metrics were calculated.  
Figure 6: Feature correlation heatmap.  
* Scenario 3: In the third evaluation scenario, we use synthetic data. Variations were made in some time windows in certain periods, with an increase or decrease in the values of the temperature variables. The objective is to visualize the behavior of the method with guided changes in the input data.
* Scenario 4: In the fourth evaluation scenario, we conducted tests with other ML algorithms applied to the problem. This evaluation scenario focuses on the selection of the best supervised ML algorithm and its hyperparameters tuning. We use Automated Machine Learning (AutoML) methods to select the best models from a set of models and optimize their hyperparameters. In order to produce a just comparison between the models, the assessments were divided into categories: General ML (GML) algorithms selection and Deep Leaning (DL). In the GML category, we use Auto-Sklearn (available at: [[https://www.automml.org/automl/auto-sklearn/](https://www.automml.org/automl/auto-sklearn/)]([https://www.automml.org/automl/auto-sklearn/](https://www.automml.org/automl/auto-sklearn/)) accessed on 13 February 2022), which is an AutoML library built on top of the Scikit-Learn ML framework. The choice of algorithms and hyperparameters implemented by Auto-Sklearn takes advantage of recent advances in Bayesian optimization, meta-learning, and Ensemble Learning [35]. We use Auto-SkLearn version 0.14.4. In the DL category, we use Auto-Keras (available at: [[https://autokeras.com/](https://autokeras.com/)]([https://autokeras.com/](https://autokeras.com/)) accessed on 13 February 2022), which is a Python library based on the Keras module and that is focused on an automatic DL Neural Architecture Search (NAS) [36]. The search is performed by using a Bayesian optimization, with the tool automatically tuning the number of dense layers, units, type of activation functions used, dropout values, and other DL hyperparameters. In this work, we adopt Auto-Keras version 1.0.18. Each tool is measured in terms of its predictive performance using K-fold cross-validation with k = 10. The best model result of each AutoML tool is presented.