dataesr/scientific-paragraphs-categorization huggingface dataset:
The maturation of new information technologies, namely mobile technology and remote sensing, and the implementation of new initiatives, such as crowdsourcing, could help solving this situation  (Janssen et al. , 2017)  at the cost of accuracy. An assessment of suitable datasets for crop modelling in terms of number of observations and measurement error may bring light to the potential benefits of these technologies to improve crop impact projection performance. The learning curve approach evaluates in a progressive manner the impact of the size and measurement error of the calibration dataset on model performance. Learning curves are graphs displaying the evolution of simulation errors with the size of the training dataset  (Perlich et al. , 2003; Perlich, 2011). Errors usually evolve asymptotically with the size of the training dataset, increasing for the training dataset and decreasing for the testing dataset.
The shape of the curves can reveal, for instance, when the model is considered to have a sufficiently large calibration dataset. The size is considered large enough when greater observations produce small changes in the simulation skills. However, defining when the changes are small enough depends on the model application. The learning curve approach has been used in the past with statistical models in the field of machine learning (e. g. Perlich, 2011 or Figueroa et al. , 2012). To our knowledge, the method has not been applied yet for the assessment of dataset features in crop modelling. Drawing the learning curves requires calibrating and evaluating the model repeatedly, changing the size of the calibration dataset. This makes the process computationally demanding and data intensive. Phenology combines its relevance for yield  (Craufurd and Wheeler, 2009)  with its simple mathematical formulation and fast execution (e.
Our study aims to analyse the influence of datasets on model simulation performance. More specifically, we seek to elucidate the impact of number and measurement error of crop state variables on the prediction skills of a phenology model intended for climate change applications. We apply the learning curve approach which allows the progressive assessment of properties of datasets and brings the opportunity to compare the evolution of model performance with the scoring rules specified in the data classification system. Additionally, we inspect possible compensations between size and measurement error thanks to their joint analysis.
Methods The generation of learning curves is a two-step process repeated multiple times. The first step is the calibration and evaluation of the models against the training (or calibration) dataset. The second step is the evaluation of the predictive skills of the model against the testing (or evaluation) dataset. The training dataset varies in number of observations (quantity of observations) and levels of measurement error (quality of observations). Long series of records (greater than 10 seasons) of flowering dates required to construct the learning curves are scarce. Hence, data is replaced by the simulations of a "perfect model" with structure and parameter values considered to be true. The simulations from such perfect models are masked with different levels of noise. This perfect model approach gives us full control over the number of seasons and errors introduced in the datasets.
Perfect models and artificial flowering date records A "perfect model" will be used in subsequent steps in substitution of the lacking long series of records of flowering dates. The "perfect model" has a structure and parameter values considered to be true. Parameter values for this "perfect model" were derived from calibration using actual data. These data were collected and used in simulations of the Agricultural Model Inter-comparison Project  (Asseng et al. , 2015). The information available covered the average flowering date during 1980-2010 ( ), the average sowing date, daily maximum and minimum temperatures for the same period, latitude and longitude and qualitative descriptions of the sensitivities to vernalization and photoperiod of the varieties being grown. A subset of 8 locations (Table  1 ) was selected among the 60-major wheat producing regions worldwide available.
The locations are Netherlands, Argentina, USA, China (with continental and oceanic climates), Russia, Turkey and Canada, showing a wide diversity of environmental conditions. The "perfect model" was calibrated independently for each location using Ordinary Least Squares (OLS). The calibration concerned the parameters related to vernalization, photoperiod and thermal responses (see section 2. 1. 5). The OLS method searched iteratively for those parameter values that minimize the squared distance between the actual flowering date ( ) and the simulation for every season ( ) between 1980 and 2010 (Eq. 15). The calibration was carried out in R (version 3. 3. 1) using the optim function (R Core Team, 2016). (Eq. 15) Then, we used the calibrated "perfect model" to generate two artificial datasets: (  1  Climate Model  (Asseng et al.
, 2015). We assume that there is no adaptation to climate change, hence sowing dates and cultivars were fixed for both time periods in each location. (Table  1 ) To mimic the sampling error that exists in field measurements  (Kersebaum et al. , 2015) , we added noise ( ) to the flowering time datasets created with the "perfect model" (Eq. 16 and 20 in Fig. 2 ). Noise values were sampled from normal distributions with mean at zero and variance. We assume hereinafter that the resulting values ( or ) represent the long series ( ) of records of anthesis dates under baseline and future climate. The artificial datasets generated for the simulation experiment are listed in Table  2. (Table  2 ).
Steps to generate the learning curves The models were recalibrated (Fig. 2 ) using OLS (Eq. 17 We repeated the calibration-evaluation process multiple times (Fig. 2 ), changing the number of measurements (n) and noise levels ( ) in the training dataset. The number of measurements ranged from 5 up to 30 seasons, in steps of 2. The lower limit in the number of seasons was set just above the minimum number required to calibrate 3 parameters from a mathematical point of view. We also increased the noise in training set from 0 to 0. 25, 1, 2. 25 and 4 days 2. We consider that the upper limit in the level of noise is a rare situation when observations are taken by well-trained experimentalists. A represents a 4. 6% chance to have a measurement error greater than 4 days. The result of the calibrations and evaluation may vary depending on the seasons and errors sampled in every combination of n and.
Model performance, number of measurements, noise and data requirements In statistics, it is known that the MSEP reacts to the size of the training dataset (n) following Eq. 21 for linear regressions models  (Wallach et al. , 2013). The magnitude of MSEP depends on model errors ( ) and the number of parameters being calibrated ( ). The theory is valid when (1) the linear regressions represent suitably the system and (2) the training and testing datasets belong to the same population. (Eq. 21) Phenology models in climate impact assessments contradict both premises; (1) they are far from linear and (2) the baseline (training datasets) and future climate flowering dates (testing dataset) represent different populations. Instead of Eq. 21, the relationship will be expressed according to the power law (Eq.
22). In Eq. 22, a and b represent the learning rate and learning limit, respectively. The learning rate (a) represent the portions of the MSEP that is reducible with larger training datasets (n). Conversely, the learning limit (b) constitutes the unreducible part of MSEP. Eq. 22 is a more general form of Eq. 21 since a and b can adopt the values and. (Eq. 22) Based on Eq. 22, we explore the model data requirements by estimating the smallest calibration dataset that does not trigger significant improvement in the prediction errors under future climate, i. e. the lower value of n that makes crossing a threshold. We will consider that is trivial when the error is reduced less than 1 day in one of the 30 seasons under climate change (t = 1 2 /30 ≈ 0. 03). The use of to determine the data requirements focuses on the role of the size of the dataset rather than any other factor affecting the MSEP.
"Perfect model" calibration, training and testing datasets The calibration of the "perfect model" yielded good representation of the observed average flowering date under baseline climate (Table  1  and Fig. 3 ). The 30-year means of the annual flowering date simulated by the Continuous Curvilinear (CC) model were nearly equal the actual averages (Table  1 ). The simulations carried out with the "perfect model" under climate change conditions (Fig. 3 ) led to earlier flowering dates. Flowering dates with the CC model occurred between 6-17 days earlier than in the baseline. Russia was the only location where the model predicted a later flowering (3 days). (Fig. 3 ) error in both calibration and evaluation (R 2 =0. 99) (Fig. 5a ). 3. 2. We adjusted Eq. 22 by estimating the learning rate (a) and learning limit (b) that fitted best the median MSEs and MSEPs among locations (solid lines in Fig.
The non-equidistant contour lines in Fig. 8a  depict the non-linearities between MSEP and n captured in Eq. 21 and 22. The straightness of the contour lines reflects the linear relationship between MSEP and represented in Eq. 21. We inspected whether larger but less precise datasets could lead lower MSEPs than smaller but more precise datasets. The dashed black line in Fig. 7a  shows one case where the MSEP is reduced from 5 day 2 to 4 day 2 (in steps of 0. 25 day 2 ) by using training datasets with size n equal to 4, 6, 9, 13 and 30 and noise levels equal to 2. 22, 2. 25, 2. 37, 2. 41 and 2. 51 days 2 , respectively. Eqs. 22-23 and Fig. 7a  confirm that it is possible in theory to compensate the lack of precision in the measurements with more seasons observed. However, the equations and the results in Fig.
7a  highlight two major limitations for this type of compensations; (1) the noise imposes a minimum limit of the MSEP ( lim ) and (  2 ) n changes very quickly with MSEP and (n a ), becoming rapidly very large and practically unfeasible. (Fig. 7 ) Data required to reach the threshold ∆MSEP < 0. 03 was calculated using Eqs. 21-22. The improvements in model performance were not significant when the size of training dataset reached the number of observations appearing in Table  3  (column Situation S1). For instance, models showed no meaningful improvement in prediction skills with training datasets larger than 11(±1) measurements when noise was = 1. The data required increased with growing levels of noise. (Table  3 ) Every square dot in Fig. 4  represents the squared error (MSE/MSEP) of a particular location.
The dispersion of the MSEP values reveals that the variation between locations is large. To explore the reasons behind these differences, Eq. 22 was adjusted independently for the results of each location. We inspected whether the variance of the training population (flowering dates 1980-2010) might be behind the differences in the location-specific learning rates (a) and limits (b) of the MSEPs. Fig. 8  displays the a and b obtained from the MSEPs for each location and noise level on the x-axis. On the y-axis, the graph shows the a' and b' obtained from a regression based on noise ( ) and the variance of the training dataset ( ). We found that the variance of the training dataset and the variance of noise in the measurements explained most of the variability in the learning rates (Fig. 8a ).
The regression of a' based on and shows a good fit between the actual and the estimated learning rates (R 2 =0. 85). The variance of training dataset and its product with the variance of noise ( • ) were highly significant (p < 0. 01) to explain the variations in learning rates. The variability in b' (Fig. 8b ) was only significantly explained (p < 0. 01) by the noise (R 2 =0. 98). (Fig. 8 ) 3. 3. Size of the training dataset, measurement error and model performance -S2: model structures are approximations ( ) The entire process was repeated, but this time the true model and the simulation model were different. In Fig. 9 , the CC model represents the true mechanism ( ), and the BS model is used as an approximation ( ). Curves with the shape of Eq. 22 were adjusted to the results of the MSE and MSEP (Fig.
9 ). MSEs and MSEPs evolved asymptotically with the size of the training dataset as in S1. Eq. 22 represented well the variations of the MSEs (grey dots in Fig. 9 ); R 2 ranged between 0. 96 and 0. 99 for the BS model simulations (black lines in Fig. 9 ) and dropped to 54-90% for the MSEPs with the BS model (red lines in Fig. 9 ). The results show that the prediction error increased linearly with the noise (R 2 =0. 99) (Fig. 5b ). The values of MSEs and MSEPs were well represented by a linear regression with an intercept ( ) greater than zero. This intercept shows the average cost of an approximated model structure, which was 1. 10 and 3. 68 days 2 for the MSE and MSEP, respectively. The influence of model structure is also illustrated by a wider spread of MSEPs among locations in S2 than in S1 (red dots in Fig.
9 ). Structural model errors worsened prediction performance to a greater or lesser extent depending on the location. For instance, the MSEPs were high and roughly decreased with the size of training dataset (n) when applying the BS model in Turkey (outliers in Fig. 9 ). The flat evolution of the error represents the need of structural model improvements. (Fig. 9 ) The impact of structural error on MSEP was removed by subtracting the location-specific minimum prediction error obtained with zero noise training datasets ( ). As in S1, the differences among MSEPs caused by noise were eliminated by dividing MSEP by ( )). We adjusted Eq. 22 to by calibrating a and b (Fig. 6b ). The model was significant ( ) and explained a high portion of the variability in (R 2 = 0. 84). The estimated values of the coefficients and were 4.
46(±0. 56) and 1. 25(±0. 05), so was slightly greater than the value in S1 and was similar to S1 and its theoretical value. Therefore, the model structure hampered the parameter estimation, since is the portion of MSEP attributed to parameter estimation error. We estimated the n (contour lines in Fig. 7b ) based on a given MSEPs and. The specific version of Eq. 21-22 to S2 was rearranged ( ). Compared to S1, contour lines in S2 are offset to the lower right corner of the graph. This indicates that the number of observations needed to reach a prediction performance in S2 is larger than in S1. The contours lines are more horizontal than in S1, representing a lower response of n to the noise in the training dataset. Results suggest (black dots in Fig. 7b ) that the training datasets of n equal to 5, 7, 12 and 32 can reduce the prediction error from 5 days 2 to 4.
25 days 2 (in steps of 0. 25 day 2 ) with increasing noises (1. 06, 1. 07, 1. 09 and 1. 10 days 2 ). Data requirements were estimated by finding the smallest n that surpassed the threshold with the learning rates and limits specific to each location. The models stopped significantly improving model predictions at the n's specified in Table  3  under the column for Situation S2. There is an increase in data requirements when the model structure changed from perfect to approximate (Table  3 ). As in S1, Eq. 22 was fitted independently to the results from each location, extracting the values of a and b. To understand the differences between locations, we explored the relationship between the learning rate and limits with the training population variance ( ) and level of noise ( ). Fig. 10  is similar to Fig.
8 , but with the results from S2. The results showed a worse approximation between actual and estimated learning rates (a vs. a') (R 2 = 0. 69) and learning limits (b vs. b') (R 2 = 0. 60) than in S1 (Fig. 10 ). The terms and ( • ) were highly significant (p < 0. 01) for explaining the variations of the learning rates among locations. The variation of the learning limit among locations was significantly explained by the terms and. Fig. 10b  shows that and alone did not represent well the learning limits in locations such as Turkey (green squares). The shift of the points towards the right while remaining parallel to the 1:1 line indicates existence of an additional locations-specific constant term explaining the learning limit. (Fig. 10 ) As in other disciplines (e. g. ,  Figueroa et al. , 2012) , the learning curves have proved to be useful for assessing crop phenology models in terms of elucidating the relationship between datasets and prediction performance and defining the suitable size of the calibration datasets given a prediction error target.
We explored the interaction between the number of measurements in the calibration dataset and the prediction skills of two phenology models. The results show a nonlinear relationship between prediction error and the size of the calibration dataset. The system developed by  Kersebaum et al. (2015)  scores the quality of modelling datasets in a linear fashion with the number of seasons observed. The existing statistical theory and our results suggest that a nonlinear power-law scoring system would be more representative. According to the effect of noise on model squared error, we observed that prediction performance improves proportionally with reductions in measurement error. The relationships between size, noise of datasets and model skills (Eq. 21-22) indicate that it could be possible to improve the predictions skills using less precise but more abundant datasets ( ).
Therefore, satellite images, for instance, could help observing ground-based phenology  (Sakamoto et al. , 2005)  to improve climate change impact assessments. Their spatial and temporal coverage (large n) may compensate the errors arising from calibration and atmospheric disturbances (high )  (Studer et al. , 2007). However, compensations between noise and size of datasets might be limited by the non-linear growth in size needed to compensate for measurement error. Further assessments investigating these synergies are needed. We estimated that 5-7 observations of flowering dates were enough to conduct impact assessments under 2050's climate change conditions. These results correspond to 0. 25 day 2 measurement error and perfect model structures. However, model structures are known to be imperfect representations of the agricultural systems  (Rötter et al.
(2017). Strictly, neither structures can be considered correct, nor are parameter values true. For these reasons, the results obtained with this kind of assessment are merely theoretical and advisory. These recommendations can vary among locations: the data required depends on the learning rate and results show that it varies with the inter-annual flowering variability of the training population (Fig. 8  and 10). Therefore, the suitable size of the dataset could be larger in places where there is greater variability among seasons. The estimates of data requirements made in this assessment concern phenology models used on their own for climate change impact assessments for 2050's under the RCP8. 5 scenario. Results cannot be extended to phenology models embedded in crop models, even when phenology parameters are independently calibrated as the initial process of model calibration (e.
Conclusions To our knowledge, there is no study to date giving statistical evidence about the effects of the size and measurement error of the datasets on crop modelling for climate impact assessment. Here we applied the learning curve approach to crop modelling, using phenology models varying the dataset features in a progressive manner. Learning curves might be promising tools to explore the balance between the size of the dataset, measurement error and model performance to provide practical guidance. Prediction skill reacted non-linearly to the size of the training dataset according to power-law. Approximate phenology models required at least 7-9 observations to reach negligible improvements with larger datasets to predict the flowering time for the 2050's under the RCP8. 5 scenario. The analysis based on learning curves also suggested that improvements in predictions can be achieved with less precise but more abundant datasets.
Based on the theory, these compensations follow. Therefore, new satellite-based monitoring techniques could potentially improve simulations despite their errors. The extent of improvement will depend on the noise and number of seasons used as a training set and more studies are needed. The estimates made in this study concern the phenology models used independently for impact studies of flowering in 2050's under RCP8. 5. We encourage further efforts to adapt the learning curve approach to complete crop models and explore the requirements for projecting different time horizons. ). The black dots represent the paths to improve the prediction skills of the models (decreasing MSEP) by using less precise (i. e. , higher ) but larger datasets (i. e. , greater n). Tables  Table 1:  Details of the locations used in the analysis.
) and n randomly sampled seasons from the training dataset (Eq. 16). The resulting model ( ) was used to simulate the n seasons of the calibration dataset (baseline) and the 30 seasons of the testing dataset (i. e. 2050's anthesis dates under RCP8. 5). The assessment of the performance of was based on its Mean Square Error (MSE) (Eq. 18) and the Mean Square Error of Prediction (MSEP) (Eq. 20).
Size of the training dataset, measurement error and model performance -S1: model structures are correct ( ) Several calibrations and evaluations of the CC model were carried out following the algorithm described above. The calibration dataset was changed with respect to the number of seasons (n) and levels of noise ( ) and the model performance was tested in terms of mean squared errors (MSE and MSEP). The squared errors of the CC models can be seen in Figs. 4-5. In general, Fig. 4 shows an increase of MSE and a decrease of MSEP with greater sizes of the calibration dataset (n). The MSE and MSEP tend to the variance of noise, i. e. 0. 25, 1, 2. 25, and 4 days 2 , without reaching it for the range of n explored. It should be noted that the graphs differ in the range of squared errors displayed on the y-axis for visualization purposes.
Acknowledgements The present study was carried out in the context of CropM within the  FACCE-MACSUR  knowledge hub. The contributions of MMSM, JEO and JRP were funded by  Innovation Fund Denmark RPR and MPH  received financial support from the ' Limpopo Living Landscapes' project (SPACES program ; 794 grant number  01LL1304A ) funded by the  German Federal Ministry of Education and Research  (http://www. bmbf. de/en/),and from the  IMPAC3  project, funded by the  German Federal Ministry of Education and Research , grant number  031A351A. AKK and AJC are also supported by the  CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) , which is carried out with support from  CGIAR.