dataesr/scientific-paragraphs-categorization huggingface dataset:
AgTrials uses the Crop Ontology initiative as the basis for standardizing data and making it interoperable, further development of which is crucial for supporting open trial data  (Matteis et al. , 2013). The open data movement's emphasis on metadata is another development that could lead to better use of agricultural trial data. As the AgTrials data resource is adopting the CGIAR metadata schema, new opportunities for data discovery will likely become more apparent. Open data will also promote principles such as clear designation of intellectual property and digital identifiers for interoperability and proper citation, developments that could substantially improve the use of agricultural trial data. Finally, the survey and our experience in developing an agricultural trial database suggests the difficulties in developing a network of data providers and users.
Private sector actors, such as large seed and agricultural input companies, have very well developed trial and phenotyping databases. They have a much more direct line between the data and its impact because understanding their trials and using the data directly affects their profits. As private sector companies tend to be vertically integrated hierarchical organizations, they can enforce discipline among their employees in developing and using agricultural trial data. Public sector efforts, on the other hand, depend on a networking model to organize themselves. They must all agree and be motivated to contribute to a data initiative. There are few negative consequences if they do not pursue an open data policy towards their trial data. Given the international agricultural research environment we are working in, public efforts to build agricultural trial information resources need a combination of carrots and sticks, incentives to participate and disincentives to go it alone.
Conclusions This evaluation considered the development of a global agricultural trial database that can be established and used by a large number of stakeholders interested in crop improvement. Our experience in the initiative to date has shown that this type of effort has great potential for a number of applications. There is very likely a large number of agricultural trials that could be part of a global database. An initiative of this type requires development of well documented data and systems that facilitate ease-of-use. Barriers and incentives to participate could be addressed using a carrot and stick approach, where data providers and users work within an enabling environment to advance the initiative. Changes in practice are necessary for documenting and providing trial data as it is collected from the field or greenhouse.
Open Peer Review Current Referee Status: It would be helpful if the paper had provided a clearer introduction to the AgTrials project perhaps with some examples of data that AgTrials holds. It would also be useful to understand what similar projects or initiatives are being undertaken elsewhere in the world and how they have dealt with the issues raised in the paper. For example the paper notes that 17% of respondents published their data on a different platform. As funding bodies, donors and governments push for greater open access/open data the important issues raised in this paper (at both the technical and sociological levels) will need to be dealt with and most importantly coordinated at the international level to help ensure the effective use of agricultural trial data in improving agricultural and agricultural practices.
) to round out the top five. The countries with the largest number of trials in AgTrials were Mexico, India, Colombia, Guatemala and Ethiopia -making up over half the trials in the database. Cotaxtla and Tlaltizapan in Mexico, Palmira in Colombia and Las Vegas, Guatemala were among four trial sites that contributed more than 500 trials to the database. Other sites that were large contributors of trials include Agua Fria, Mexico, Hyderabad and Bangalore in India, Nioro in Senegal and Bako in Ethiopia. The top research centers contributing data were the International Maize and Wheat Improvement Center (CIMMYT), the International Center for Tropical Agriculture (CIAT), the International Institute of Tropical Agriculture (IITA) and the International Crop Research Institute for the Semiarid Tropics (ICRISAT); all centers of the CGIAR network.
The Integrated Breeding Platform is one such initiative (https://www. integratedbreeding. net/). This CGIAR-supported program is aimed at developing methods and tools for public sector breeding. The initiative does make some data available, although providing data is not a key focus of the program. This type of program could work interoperably with AgTrials providing public sector breeders an outlet for sharing their information. The AgMIP initiative mentioned above also makes available trial data (https://data. agmip. org/cropsitedb). The datasets could be from variety trials, but could also be field experiments with multiple factors such as N, water, temperature, CO2, cultivar, etc. The AgMIP datasets are associated with field observations for management and production. For example, even the simplest farm survey data will include lat-long, crop, recorded planting date, fertilizer applied, harvest date, and some estimation of final yield.
Field experiments include much richer recordings of all management activities in the field and measurements of crop, soil, and weather variables. Some AgMIP datasets also include simulated results, in addition to the field records. Other such initiatives include the USDA's Long Term Agro-ecological Research program and an Australian initiative called Online Farm Trials (https://www. farmtrials. com. au/). The former initiative does not yet provide open access data. Both of these initiatives are focused on their respective countries. European researchers have proposed a pan-European trial data initiative, but it has not yet been implemented.
Acknowledgements The AgTrials initiative has been primarily supported by the  CGIAR Research Program on Climate Change and Agricultural Food Security (CCAFS). Additional support for the initiative and the development of the user survey was provided by the  CGIAR System Management Office via the CGIAR Fund and the Bill and Melinda Gates Foundation. We thank  Nora Casteneda  for comments and suggestions on an earlier draft of this paper. We especially thank to those colleagues who responded to our survey on user perceptions and experience of the AgTrials information resource.
Data availability The survey data that was the basis of this paper is available on Survey Monkey at the following URL: https://en. surveymonkey. net/ results/SM-YLGDFPQG/. Readers are also encouraged to visit the AgTrials website where metadata and data on agricultural trials can be accessed (www. agtrials. org). Dataset 1: Anonymous individual responses and survey questions. doi, 10. 5256/f1000research. 11179. d155255 (Hyman et al. , 2017).
Statistics in the Big Data world Big Data is one of the major challenges of statistical science and a lot of recent references start to think about the numerous consequences of this new context from the algorithmic viewpoint and for the theoretical implications of this new framework (see  [1, 2, 3] ). Big Data always involve massive data but they also often include data streams and data heterogeneity (see  [4]  for a general introduction), often characterized by the fact that data are frequently not structured data, properly indexed in a database and that simple queries cannot be easily performed on such data. These features lead to the famous three V (Volume, Velocity and Variety) highlighted by the Gartner, Inc. , the advisory company about information technology research 1. In the most extreme situations, data can even have a too large size to fit in a single computer memory.
Then data are distributed among several computers. For instance, Thusoo et al. [5]  indicate that Facebook c had more than 21PB of data in 2010. Frequently, the distribution of such data is managed using specific frameworks dedicated to For statistical science, the problem posed by this large amount of data is twofold: first, as many statistical procedures have devoted few attention to computational runtimes, they can take too long to provide results in an acceptable time. When dealing with complex tasks, such as learning of a prediction model or complex exploratory analysis, this issue can occur even if the dataset would be considered of a moderate size for other (simpler tasks). Also, as pointed out in  [6] , the notion of Big Data depends itself on the available computing resources. This is especially true when relying on the free statistical software R  [7] , massively used in the statistical community, which capabilities are strictly limited by RAM.
In addition to the parallel construction of a lot of models (the trees of a given forest) RF include intensive resampling and, it is natural to think about using parallel processing and to consider adapted bootstrapping schemes for massive online context. Even if the method has already been adapted and implemented to handle Big Data in various distributed environments (see, for instance, the libraries others), a lot of questions remain open. In this paper, we do not seek to make an exhaustive description of the various implementations of RF in scalable environments but we will highlight some problems posed by the Big Data framework, describe several standard strategies that can be use for RF and describe their main features, drawbacks and differences with the original approach. We finally experiment five variants on two massive datasets (15 and 120 millions of observations), a simulated one as well as real world data.
One variant relies on subsampling while three others are related to parallel implementations of random forests and involve either various adaptations of bootstrap to Big Data or to "divide-and-conquer" approaches. The fifth variant relates to online learning of RF. Since the free statistical software R  [7] , is de facto the esperanto in the statistical community, and since the most widely used programs for designing random forests are also available in R, we have adopted it for numerical experiments as much as possible. More precisely, the R package randomForest, implementing the original RF algorithm using Breiman and Cutler's Fortran code, contains many options together with a detailed documentation. It has then been used in almost all experiments. The only exception is for online RF for which no implementation in R is available.
We then use a python library, as an alternative tool in order to provide the means to compare this approach to the alternative Big Data variants. The paper is organized as follows. After this introduction, we briefly recall some basic facts about RF in Section 2. Then, Section 3 is focused on strategies for scaling random forests to Big Data: some proposals about RF in parallel environments are reviewed, as well as a description of online strategies. The section includes a comparison of the features of every method and a discussion about the estimation of the out-of-bag error in these methods. Section 4 is devoted to numerical experiments on two massive datasets, an extensive study on a simulated one and an application to a real world one. Finally, Section 5 collects some conclusions and discusses two open perspectives.
Scaling random forests to Big Data This section discusses the different strategies that can be used to scale ran- In addition, the section will use the following notations: RF will denote the random forest method (in a generic sense) or the final random forest classifier itself, obtained from the various approaches described in this section. The number of trees in the final classifier RF is denoted by Q, n is the number of observations of the original dataset and, when a subsample is taken in this dataset (either with or without replacement), it is denoted by τ l (l identifies the subsample when several subsamples are used) and its size is usually denoted by m. When different processes are run in parallel, the number of processes is denoted by K. Depending on the method, this can lead to learn smaller RF with q < Q trees that are denoted by RF dom (q) l , in which l is an index that identifies the RF.
Random subsampling are usually adequate for such tasks, providing the fact that the sampling fraction is large enough. However, in the Big Data world, datasets are frequently not structured and indexed. In this situation, random subsampling can be a difficult task (see  [14]  for a discussion on this point and a description of a parallel strategy to overcome this problem). Section 4 provides various insights on the efficiency of subsampling, on the effect of the sampling fraction and on the representativeness of the subsample on the accuracy of the obtained classifier. The next section investigates approaches which try to make use of a wider proportion of observations in the dataset using efficient computational strategies.
Alternative bootstrap schemes for RF (moonRF and blbRF) To avoid selecting only some of the observations in the original big dataset as it is done in sampRF (Figure  3 ), some authors have focused on alternative bootstrap schemes aiming at reducing the number of different observations of each bootstrap samples. [29]  propose the m-out-of-n bootstrap that consists in building bootstrap samples with only m observations taken without replacement in {1,. , n} (for m n). This method is illustrated in Figure  5. moonRF t 1 t 2 t Q RF = ∪ Q l=1 t l Q samples (without replacement) trees in parallel aggregate trees Initially designed to address the computational burden of standard bootstrapping, the method performance is strongly dependent on a convenient choice of m and the data-driven scheme proposed in  [30]  for the selection of m requires to test several different values of m and eliminates computational gains.
All the trees are then gathered into a final forest RF. It consists in two steps: in a first step, K subsamples, (τ l ) l=1,. ,K , are obtained, with m observations each, that are taken randomly without replacement from the original observations. In a second step, each of these subsamples is used to obtain a forest, RF (q) l with q = Q K trees. But instead of taking bootstrap samples from τ l , the method uses over-sampling and, for all i ∈ τ l , computes weights, n l i , from a multinomial distribution with parameters n and 1 m 1 m , where 1 m is a vector with m entries equal to 1. These weights satisfy i∈τ l n l i = n and a bootstrap sample of the original dataset is thus obtained by using n l i times each observation i in τ l. For each τ l , q such bootstrap samples are obtained to build q trees.
Divide-and-conquer RF (dacRF) Standard alternative to deal with massive datasets while not using subsampling is to rely on a "divide-and-conquer" strategy. The large problem is divided into simpler subproblems and the solutions are aggregated together to solve the original problem. The approach is illustrated in Figure  7 : the data are split into small sub-samples, or chunks, of data, (x i , y i ) i∈τ l , with ∪ l τ l = {1,. , n} and τ l ∩ τ l = ∅. dacRF RF (q) 1 RF (q) 2 RF (q) K RF = ∪ K l=1 RF (q) l partition into K samples forests with q trees in parallel aggregate forests τ K τ 2 τ 1 Figure  7 : divide-and-conquer RF (dacRF). In this method, the original dataset is partitionned into K subsets. A random forest with q trees is built from each of the subsets and all the forests are finally aggregated in a final forest, RF.
Mismatches with original RF In this section, we want to stress the differences between the previously proposed parallel solutions and the original algorithm. Two methods will be said "equivalent" when they would provide similar results when used on a given dataset, up to the randomness in bootstrap sampling. For instance, seqRF and parRF are equivalent since the only difference between the two methods are the sequential or parallel learning of the trees. sampRF and dacRF are not equivalent to seqRF and are both strongly dependent on the representativity of the dataset. This is the standard issue encountered in survey approaches for sampRF but it is also a serious limitation to dacRF even if this method uses all observations. Indeed, if data are thrown in the different chunks with no control on the representativity of the subsamples, data chunks might well be specific enough to produce very heterogeneous forests: there would be no meaning in simply averaging all those trees together to make a global prediction.
The data stream is then randomly partitioned in two streams: the structure stream and the estimation stream. Data from structure stream only participate on the splits optimization, while data from estimation stream are only used to allocate a class label to a node. Thanks to this partition, the authors manage to obtain consistency results of onRF. [19] also describes an online estimation of the OOB error: since a given observation is OOB for all trees for which the Poisson random variable used to replicate the observation in the tree is equal to 0, the prediction provided for such a tree t is used to update errTree t. However, since the prediction cannot be re-evaluated after the tree has been updated with next data, this approach is only an approximation of the original errTree t. Moreover, as far as we know, this approximation is not implemented in the python library RFTK  5  which provides an implementation of onRF used in experiments of Section 4.
Experiments The present section is devoted to numerical experiments on a massive simulated dataset (15 millions of observations) as well as a real world dataset (120 There are strong reasons to carry out experimentations in a unified way involving codes in R. This will be the case in this section except for onRF in Section 4. 4. Due to their interest, onRF are considered in experimental part of the paper, even if, due to the lack of available program implemented in R, an exception has been made using a python code. To allow fair comparisons between the other methods and to make them independent from a particular software framework or a particular programming language, all methods have been programmed using the following packages: millions • the package readr  [35]  (version 0. 1. 1), which allows to read more efficiently flat and tabular text files from disk; • the package randomForest  [36]  (version 4.
6-10), which implements RF algorithm using Breiman and Cutler's original Fortran code; • the package parallel  [7]  (version 3. 2. 0), which is part of R and supports parallel computation. To address all these issues, simulated data are studied in this section. They correspond to a well controlled model and can thus be used to obtain comprehensive results on the various questions described above. The simulated dataset corresponds to 15,000,000 observations generated from the model described in  [37] : this model is an equiprobable two class problem in which the variable to predict, Y , takes values in {-1, 1} and the predictors are, for 6 of them, true predictors, whereas the other ones (in our case only one) are random noise. The simulation model is defined through the law of Y (P (Y = 1) = P (Y = -1) = 0.
5) and the conditional distribution of the (X j ) j=1,. ,7 given Y = y: • with probability equal to 0. 7, X j ∼ N (jy, 1) for j ∈ {1, 2, 3} and X j ∼ N (0, 1) for j ∈ {4, 5, 6} (submodel 1); • with probability equal to 0. 3, X j ∼ N (0, 1) for j ∈ {1, 2, 3} and X j ∼ N ((j -3)y, 1) for j ∈ {4, 5, 6} (submodel 2); • X 7 ∼ N (0, 1). All variables are centered and scaled to unit variance after the simulation process, which gave a dataset which size (in plain text format) was equal to 1. 9 Go. Compared to the size of available RAM, this dataset was relatively moderate which allowed us to perform extensive comparisons while being in the realistic Big Data framework with a large number of observations. This 15,000,000 observations of this dataset were first randomly ordered. Then, to illustrate the effect of representativeness of data in different subsamples in both divide-and-coquer and online approaches, two permuted versions of this same dataset were considered (see Figure  8  for an illustration): unbalanced Y values: gray: -1 white: 1 x-biases X values: gray: submodel1 white: submodel2 • x-biases will refer to a permuted dataset in which X values arrive with a particular pattern.
Four RF methods for Big Data involving parallel implementations The aims of the simulations of this subsection were multiple: firstly, different approaches designed to handle Big Data with RF were compared. The comparison was made on the point of view of the computational effort needed to train the classifier and also in term of its accuracy. Secondly, the differences between the OOB error estimated by standard methods corresponding to a given approach (which generally uses only a part of the data to be computed) was compared to the OOB error of the classifier estimated on the whole data set. All along this subsection we use a simulated dataset corresponding to 15,000,000 observations generated from the model described in Section 4. As illustrated by the left-hand side of Figure  9 , the OOB error of seqRF (with a total number of trees equal to 500) stabilizes between 100 and 200 trees.
The training of the RF with 500 trees took approximately 18 hours. Hence, we chose to keep a limited number of trees of 100, which seems a good compromise between accuracy and computational time. The choice of a maximum number of leaves of 500 was also motivated by the fact that maximal trees did not bring much improvement in accuracy, as shown in the right-hand side of Figure  9. On the contrary, it increases the final RF complexity significantly (maximal trees contain approximately 60,000 terminal nodes). We designed experiments to compare this sequential forest (seqRF) to the  four variants introduced in Section 3, namely: sampRF, moonRF, blbRF and dacRF (see Table  2  for definitions). In this section, the purpose is only to compare the methods themselves so all subsamplings were done in such a way that the subsamples were representative of the whole dataset from the X and Y distributional viewpoint.
On the contrary, for dacRF, the number of trees built in each chunk must be quite high to get an unbiased BDerrForest, at a cost of a substantially increased computational time. In q q q q q q q q q q q q q q q q q q q q q q q q blbRF dacRF q (number of trees) prediction error q q q q q q q q blbRF dacRF other simulations for dacRF, q was also set to 100 and K was increased but this did not give any improvement (not shown). Due to these conclusions, the values K = 10 and q = 50 were chosen for blbRF and the values K = 10, q = 100 were chosen for dacRF in the rest of the simulations. Second, the impact of the sampling fraction, f = m n was studied for sam-pRF and moonRF, with a number of trees set to 100. More precisely, for sampRF, a subsample containing m observations was randomly drawn for the entire dataset, with f ∈ {0.
1% 1% 10% 0. 1% 1% 10% The number of trees is set to 100. shows that BDerrForest gives quite unbiased estimations of the prediction error. Moreover, the computational time for training remains low. The increase of the prediction error when f = 0. 1% is explained by the fact that subsamples contain only 150 observations in this case. Based on these experiments, the total sampling fraction was set to 1% for both sampRF and moonRF in the rest of the simulations. Several conclusions can be driven from these results. First, the computational time needed to train all these Big Data versions of RF is almost the same and quite reduced (about a few minutes) compared to the sequential RF. The fastest approach is to extract a very small subsample and the slowest is the dacRF approach with 10 chunks of 100 trees each (because the number of observations sent to each chunk is not much reduced compared to the original dataset).