dataesr/scientific-paragraphs-categorization huggingface dataset:
Constant: Using the same threshold t(C i ) = T for each cascade classifier t(C i ); i = 0. 1 2) Increasing: Monotonously increasing the classifiers threshold. t(C i ) = T + c × 0. 01. 3) Recall: Estimating thresholds during training such that recall equals to r. This is the method most commonly used in the literature. 4) Regular: Using the same purposely low threshold t(C i ) = T for each cascade classifier t(C i ) and insuring that d% of the remaining data are removed at each step of the cascade. the parameter T, and d are manually set to the best performing value. However, the same value is employed for all datasets. T=0. 58 is utilized for experiment 1, T=0. 55 for experiment 2, and d=40% for experiment 4. Results are displayed in table in table VIII.
1 ) 1 Constant: Using the same threshold t(C i ) = T for each cascade classifier t(C i ); i = 0. 1 2) Increasing: Monotonously increasing the classifiers threshold. t(C i ) = T + c × 0. 01. 3) Recall: Estimating thresholds during training such that recall equals to r. This is the method most commonly used in the literature. 4) Regular: Using the same purposely low threshold t(C i ) = T for each cascade classifier t(C i ) and insuring that d% of the remaining data are removed at each step of the cascade. the parameter T, and d are manually set to the best performing value. However, the same value is employed for all datasets. T=0. 58 is utilized for experiment 1, T=0. 55 for experiment 2, and d=40% for experiment 4. Results are displayed in table in table VIII.
We present BreizhCrops, a novel benchmark dataset for the supervised classification of field crops from satellite time series. We aggregated label data and Sentinel-2 top-of-atmosphere as well as bottom-of-atmosphere time series in the region of Brittany (Breizh in local language), north-east France. We compare seven recently proposed deep neural networks along with a Random Forest baseline. The dataset, model (re-)implementations and pre-trained model weights are available at the associated GitHub repository (https://github. com/dl4sits/breizhcrops) that has been designed with applicability for practitioners in mind. We plan to maintain the repository with additional data and welcome contributions of novel methods to build a state-of-the-art benchmark on methods for crop type mapping.
INTRODUCTION The Earth's surface is governed by spatio-temporal processes that are measured by various satellites on discrete temporal intervals. Extracting knowledge from this data at a large scale is a key objective to modern remote sensing research. The related field of machine learning has demonstrated that direct model comparisons through application-specific benchmarks are a central driver for rapid development in the field. So far, a direct comparison of models has been difficult in remote sensing due to the diverse nature of remote sensing data, the partly proprietary access to satellite data and labels, and the exclusive expertise on data-processing in remote sensing. Hence, proposed time series methods have been tested predominately on self-compiled datasets rather than on a common benchmark.
The spatial component in remote sensing data has been explored through its methodological relation to computer vision and a variety of related benchmark datasets have been proposed, such as DeepGlobe2018  (Demir et al. , 2018) , SEN12MS  (Schmitt et al. , 2019)  or BigEarthNet  (Sumbul et al. , 2019). The temporal component has received less attention. MediaEval Benchmarking Initiative has recently run a task on Emergency Response for Flooding Events that included imagery from multiple dates 1 and a benchmark dataset for change detection has been proposed  (Caye Daudt et al. , 2018). For the temporal task of crop type mapping from satellite image time series, novel approaches have predominantly been tested on self-created datasets and only partly compared to other state-of-the-art methods  (Pelletier et al.
, 2019 , Rußwurm, Körner, 2017a , Rußwurm, Körner, 2018). Time series datasets involving land cover classification labels have been proposed  (Ienco, 2017 , Bailly, 2017). However, and to the best of our knowledge, no public benchmark for satellite time series classification that comprehensively compares existing models is available to this date. In this work, we propose a novel large-scale satellite image time series dataset for crop type mapping termed BreizhCrops from the region of Brittany, France. We extracted time series from Sentinel-2 at two processing levels (top-and bottom-ofatmosphere) which yielded more than 600k multivariate time  The remainder of this paper is organized as follows: section 2 presents the BreizhCrops dataset. Then section 3 briefly describes the seven classification algorithms that are evaluated and compared in section 4.
Crop Type Labels The Common Agricultural Policy of the European Union subsidizes farmers based on their cultivated crops. Each member country is required to gather geographical information about the geometry and cultivated crops. This information is obtained from the farmers themselves by surveys within the subsidy application process. We decided to keep "well-defined" classes and avoided broad categories, such as diverse or fodder crops. We also made the choice of keeping two minority classes (sunflower and nuts), which have only a few occurrences in RPG, as a challenge to the classification models. It also reflects the strong class imbalance in real-world crop-type-mapping datasets, as shown in fig. 2b.
Satellite Data The dataset is composed of Sentinel-2 image time series extracted from January 1, 2017 to December 31, 2017 3. We aggregated the satellite time series data in two processing levels: the raw reflectances at the top-of-atmosphere (level 1C) and the atmospherically corrected surface reflectances at the bottom-of-atmosphere (level 2A). For both processing levels, we average reflectance values over the bounds of the field geometry retrieved from the dataset. Each spectral band is mean-aggregated over one field parcel to a feature vector xt ∈ R D , with D the number of features and t a timestamp. This aggregation strategy requires known field geometries that are accessible for most of the fields in Europe  (Léo, Lemoine, 2001). In case no geometry data is available, a trained model can be inferred with feature vectors from each pixel of a Sentinel-2 image time series.
, 2015)  from PEPS -a French portal for Sentinel-2 data 4. Only images with a cloud-cover below 80 % are processed by MAJA. Hence, we collected an average of 53 images per tile. We show an example of a field parcel in fig. 3c. After MAJA processing, only 10 spectral bands are availablethe three Sentinel-2 spectral bands at a 60-meter spatial resolution serve only to apply the atmospheric correction and to detect clouds. Note that the process of several Sentinel-2 tiles results in samples representing reflectance values on different timestamps that might be disturbed by clouds (and shadows). This temporal sampling inhomogeneity between observation requires usually an additional preprocessing such as an interpolation, a subsampling or padding.
Data organisation The data is organized at a regional level by the Nomenclature of Territorial Units for Statistics (NUTS) which forms the European standard for referencing authoritative districts. Brittany is the NUTS-2 region FRH0, as highlighted in fig. 1a. It is further divided into the four NUTS-3 regions (fig. 1b ): Côtes-d'Armor (FRH01), Finistère (FRH02), Ille-et-Vilaine (FRH03), and Morbihan (FRH04). We partitioned all acquired field parcels according to the NUTS-3 regions and suggest to subdivide the dataset into training (FRH01, FRH02), validation (FRH03), and evaluation (FRH04) subsets based on these spatially distinct regions.
Random Forest Random Forests are one of the most used shallow algorithms for the classification of satellite image time series  (Gómez et al. , 2016 , Belgiu, Drȃgut ¸, 2016)  at large scale  (Inglada et al. , 2017 , Defourny et al. , 2019). They are able to handle the high dimensionality of satellite image time series datasets, are robust to some class label noise, and are generally insensitive to the ρ × 10 4 ρ B1 ρ B2 ρ B3 ρ B4 ρ B5 ρ B6 ρ B7 ρ B8 ρ B8A ρ B9 ρ B10 ρ B12 Sentinel 2 Satellite Spectral Bands choice of hyperparameters  (Pelletier et al. , 2016 , Pelletier et al. , 2017). Random Forests are an ensemble approach that trains a set of binary decision trees  (Breiman, 2001). Each tree is built by using a bootstrap sample (sampling with replacement the training instances). The optimal split at each node is determined using an effectiveness test (usually the maximization of the decrease in node impurity) on only a subset of randomly selected variables.
InceptionTime  (Fawaz et al. , 2019b ) is an ensemble of five Inception networks that currently obtained the state-of-the-art deep learning results on 85 classification problems of the UCR archive  (Dau et al. , 2019). Each network is composed of a series of six Inception modules followed by a global average pooling operation and a dense layer with a softmax activation. It also makes the use of residual connections every three Inception modules. OmniscaleCNN  (Tang et al. , 2020)  is composed of three convolutional layers followed by a global average pooling and a dense layer with a softmax activation. Its specificity is to concatenate the outputs of several convolution filters whose length is one plus all the prime numbers between two and a quarter of the time series length. Please note that only TempCNN architecture has been used for land cover mapping from Sentinel-2 image time series  (Pelletier et al.
Random Forest Following the lead of  (Inglada et al. , 2017) , we applied a linear temporal interpolation on a regular temporal grid with a time gap of 5 days. The linear temporal interpolation is applied for the non-cloudy values: we use cloud masks available in L1C products for the top-of-atmosphere dataset and we use cloud masks computed by the MAJA processing chain for the bottom-of-atmosphere dataset. After the gap-filling operation, each sample is described by a total of 71 ×D variables where 71 represents the number of interpolated dates, and D the number of spectral features used (D = 13 for top-of-atmosphere data and D = 10 for bottom-of-atmosphere data). For the Random Forests hyperparameters, it has been shown that tuning their values results only in a slight performance improvement  (Cutler et al.
Deep learning To obtain fixed-length time series that are required for training deep learning methods with batches, we decided to randomly sub-sample each time series to a fixed length of 45 observations for the deep learning models while maintaining the sequential topology. For the model selection, we trained models on FRH01 and FRH02 regions and chose hyperparameter values that gave the lowest validation loss calculated on the FRH03 region. All the model section procedure is conducted on top-of-atmosphere time-series data (L1C). More precisely, we followed a two-step process where we first tuned the values of model-specific and optimization-specific hyperparameters for five epochs and then determined the number of optimal training epochs in a second step. For the optimization-specific parameters, we sampled learning rate ν and weight decay λ from the log-uniform distributions ∼ U log ([10 -2 , 10 -4 ]) and ∼ U log ([10 -2 , 10 -8 ]) for all models.
The model-specific hyperparameters varied per model. For convolution-based models, we tested TempCNN models with convolution filter sizes K ∈ {3, 5, 7} and a number of hidden representations H ∈ {2 5 , 2 6 , 2 7 }. We also evaluated different dropout rates d ∼ U ([0, 0. 8]). The only model-parameter to select for MS-ResNet is the number of hidden representations that we searched over H ∈ {2 5 , 2 6 , 2 7 , 2 8 , 2 9 }. Regarding the Incep-tionTime model, we evaluated the stacking of L ∈ {1, 2, 3, 4} Inception modules with the following number of hidden representations H ∈ {2 5 , 2 6 , 2 7 }. Finally, we did not tune any model-specific parameter for the OmniscaleCNN model since the authors do not recommend any tuning of the network. For both LSTM and StarRNN recurrent-based networks, we evaluated models with several cascaded layers L ∈ {1, 2, 3, 4}, hidden vector dimensionalities of H ∈ {2 5 , 2 6 , 2 7 }, and a dropout rate d ∼ U([0, 0.
8]). For the LSTM approach, we also tested mono-directional and bidirectional models. Finally, we search over H ∈ {2 5 , 2 6 , 2 7 , 2 8 , 2 9 } hidden representation with N head ∈ {1, 2,. , 8} self-attention heads with a number of stacked layers L ranging from 1 to 8 for Transformer. As per the other models, the dropout rate is drawn from a uniform distribution d ∼ U ([0, 0. 8]). This hyperparameter search was done via a random search for 12 hours on a DGX-1 server with a P-100 GPU for each model where three runs were trained in parallel per GPU with a batch-size of 256. (b) Results L2A Table  2. Accuracy metrics of all models benchmarked on the Breizhcrops dataset, considering L1C (a) and L2A (b) data. For each table, the top part displays the performance obtained when testing on the FRH04 region while training on the three remaining areas, whereas the bottom part displays average performance (plus one standard deviation) when models were tested on FRH01, FRH02 and FRH04 regions.
Bold values show the highest performance. 80 LSTM, 84 StarRNN and 86 InceptionTime trained models. For completeness, we detail below the selected configuration (i. e. , the one that obtained the minimum validation loss) for each approach. TempCNN uses a kernel size K of 7 with 128 hidden units H. The dropout rate is set to 18%, the learning rate ν to 2. 38 • 10 -4 and the weight decay λ to 5. 18 • 10 -5. A MSResNet model composed of 32 hidden units H with ν = 6. 27 • 10 -7 and λ = 4. 75 • 10 -6 was selected. InceptionTime stacks three Inception modules where each convolutional filter has a hidden vector dimensionality of H = 128. The learning rate ν equals to 8. 96 • 10 -3 and the weight-decay λ to 2. 22 • 10 -6. For OmniscaleCNN, we evaluated an optimal learning rate of ν = 1. 06 • 10 -3 and a weight decay of λ = 2.
Results For the final model evaluation, we re-trained the models from scratch with their respective hyperparameter configuration. We present the results in two formats in table 2. In the top rows, we show the model performances on the FRH04 region that was completed in the previous hyperparameter tuning procedure. For the latter rows, we followed a cross-evaluation scheme where we trained on three regions and evaluated the accuracy of the remaining one. From the four possible folds, we discard the evaluation on FRH03 since the hyperparameters have been determined based on the performance of this region. We present mean and standard deviations from FRH01, FRH02, and FRH04 for the top-of-atmosphere and bottom-of-atmosphere data. The evaluation is performed on a DGX-1 machine with one model per P-100 GPU which gave us the possibility to compare the runtime in iterations per second of one batch of 1024 data samples of each model, as summarized in table 1.
Both experimental setting choices (applying a temporal subsampling and using cloudy information) could lead to a nonoptimal learning process of discriminative convolution filters. Moreover, TempCNN that obtains higher overall performance than MS-ResNet, Inception Time, and OmniscaleCNN is the only convolution-based model that has been specifically developed for land cover mapping. We leave a detailed evaluation of this systematic difference between model architectures to future research since it is beyond the scope of this paper. We observed surprisingly little difference in model performances between top-of-atmosphere L1C data and bottom-of-atmosphere L2A data. However, we would like to emphasize that the hyperparameters have been determined on L1C data evaluated on the FRH03 region which may bias the results towards better L1C accuracies.
CHALLENGES In the following, we outline a series of challenges associated with the dataset that pose demanding questions to the time series community and likely need to be addressed to improve the accuracy of methods trained on this dataset. Classification of noisy time series. Clouds cover the Earth's surface at irregular intervals and are inherent to all optical imagery. Their large reflectance introduces positive outliers to the data at single intervals which can be seen in the reflectance data over the time scale (see fig. 3 ). Existing approaches classify, mask, and interpolate values from cloudy observations in a pre-processing step. Regional variations in the class distributions. Regional variances in soil quality, elevation, temperature, and precipitation lead to a spatial correlation in the frequency of dominated agricultural crops.
This autocorrelation can introduce a dependence between training and validation datasets that may disguise overfitting and impede generalization. To counteract this, several researchers  (Rußwurm, Körner, 2017b , Jean et al. , 2019 ) have adopted a training/validation/evaluation partitioning that groups spatially distant parcels. Hence, we organized the data in their respective NUTS-3 regions to encourage training on these spatially separate regions.
REPRODUCIBILITY With BreizhCrops, we aim at comparing the state of the art in crop type mapping. We thus release a curated code repository of labeled data, models, experiments, and evaluations in https: //github. com/dl4sits/breizhcrops. This Python package can be easily installed with pip install breizhcrops. We provide here a minimal working example from breizhcrops import BreizhCrops, models x, y, field_id = BreizhCrops("frh04")[0] model = models. pretrained("Transformer") y_pred = model(x. unsqueeze(0)) that downloads the FRH04 dataset and retrieves the first sample. Then, it loads a pretrained model (on FRH01, FRH02 and FRH03) and performs a prediction for this sample. We believe that the accessibility to crop type data, classification models, and evaluation routines will accelerate developments in the scientific community and we welcome code contributions to include novel developments in the field of crop type mapping.
CONCLUSION In this work, we presented a novel benchmark dataset, Breizh-Crops, for crop type mapping with top and bottom-of-atmosphere reflectance Sentinel-2 time series. We evaluated seven recently developed state-of-the-art deep learning models on time series classification for crop type mapping along with a Random Forest classifier. The attention-based Transformer model slightly outperformed the recurrent neural networks followed by the convolutional neural networks. We release the dataset, model implementations and pre-trained model weights in an associated Python package that can be installed and run with few lines of code. We hope that the accessibility of the dataset and deep learning models will encourage the community to benchmark novel crop type mapping methods on BreizhCrops. We encourage active code contributions to reflect the state of the art in crop type mapping with this dataset.
a field crop and corresponds to a multivariate time series obtained by averaging at the crop field level reflectance values extracted from Sentinel-2 images. Breizh in the local language) located in the northwest of France and covering 27,200 km², as shown in fig. 1. The region is dominated by a temperate oceanic climate (Köppen classification) with an annual average temperature ranging from 5. 6°in winter to 17. 5°in summer and mean annual precipitation of 650 millimeters. The dataset comprises about 610k labeled observations per processing level. Each observation describes the temporal profile arXiv:1905. 11893v2 [cs. LG] 10 May 2020 of.
National agencies monitor the correctness either by gathering control samples in-situ or by means of remote sensing and Earth observation. In France, the National Institute of Forest and Geography Information (IGN) is responsible for gathering this information, the so-called Agricultural Land Parcel Information System (Registre Parcellaire Graphique)-RPG. The IGN institute recently started releasing anonymized parcel geometries and types of cultivated crops with an open license policy 2. The raw crop type categories contain 328 unique crop labels grouped into 23 groups. For BreizhCrops dataset, we selected the 9 following crop categories: barley, wheat, rapeseed, corn, sunflower, orchards, nuts, permanent meadows and temporary meadows. The field labels have been gathered in the year 2017.
The data of germination, appressoria, and haustoria were statistically contrasted with a T-Student test by Infostat R software. Additionally, the uredospore morphology was analyzed by scanning electron microscopy (SEM) in a JEOL JSM-IT300 microscope. At 24 hai, the leaf disks were fixed in a 4% glutaraldehyde solution for 24 h and washed with phosphate buffer (pH = 7. 2). Samples were then dehydrated in a gradient of ethanol (70%, 80%, 90% and absolute ethanol) and subsequently dried with liquid CO 2 for 15 min in a Toussimis Autosamdri R -815, Series A, incorporated to aluminum stubs and sputter coated with 10-nm gold layer using Denton Desk V sputter coater. The scanning was performed with an accelerating voltage of 20 kV.
, 2011)  gene with respect to the constitutive gene CaUbiE2, as it has been performed with other pathogen models  (Eshraghi et al. , 2011). We designed the specific oligonucleotides with software Primer3Plus R  (Untergasser et al. , 2007) , (Table  1 ) by using the National Center for Biotechnology (NCBI) database. Primers were synthesized by the T4Oligo Lab in Irapuato, Mexico.
qRT-PCR Each reaction was performed with 3 µl cDNA, 1 × of reaction mixture (20 µl) ''SYBR Green PCR Master Mix (Applied Biosystems)'' and 5 pmol of each primer. Amplification conditions were: 94 • C for 10 min, 40 cycles at 94 • C for 30 s, 60 • C for 30 s and 72 • C for 40 s. The qRT-PCR amplifications were performed with a thermal cycler 7500 Fast Real-Time PCR System (Applied Biosystems)  (Cruz et al. , 2009; Ramiro et al. , 2009; Borges, Tsai & Caldas, 2012). Four independent replicates were obtained with a standard error less than 0. 1 for each sample. Each expression value is the average of these replicas. Calculations were performed with the 7500 Software v2. 0. 1 (Applied Biosystems). The amplification efficiency for each set of oligonucleotides was determined by performing dilution series (1: 5).
ACKNOWLEDGEMENTS We thank  Dr. Guillermo Ángeles  ( INECOL ) for permission to use the light microscope and  Dr. Orlando Hernández-Cristóbal  ( ENES, UNAM Morelia ) for technical support with SEM. We also are thankful for the support of  Mónica Ramírez-Vázquez  and  Lorena María Luisa López Sánchez  in microscopic analyses at INECOL facilities. Finally, we thank  Centro Internacional de Capacitación en Cafeticultura y Desarrollo Sustentable, A. C. (CICADES)  for kindly providing us with plant material.
Minor issues Text Page 5: "and less ambiguity makes data more Findable, Accessible, Interoperable, and Reusable (FAIR)": less ambiguity doesn't result in "more Accessible". Page 6: "(i. e. Data production, Images, Source Code etc). ": Set these to lower case. Page 6: "as well as cutting edge libraries for modern data science (such as Pandas, scikit-learn, TensorFlow and Numpy/SciPy). " Unclear why this aspect of python is needed for a data annotation platform, which is usually hosted on platforms not suited for heavy computation. Page 6: "COPO runs on a Python 3. 5. 2 interpreter with Django 2. 0. 2 as the web framework. ": According to your docker and requirements file, is the python version 6 and Django is 2. 1. 4. Related to that, I would recommend upgrading to Django 2. 2, which has long term support at Pre-population of data in COPO.