dataesr/scientific-paragraphs-categorization huggingface dataset:
The accuracy of a map is dependent on the reference dataset used in its construction. Classification analyses used in thematic mapping can, for example, be sensitive to a range of sampling and data quality concerns. With particular focus on the latter, the effects of reference data quality on land cover classifications from airborne thematic mapper data are explored. Variations in sampling intensity and effort are highlighted in a dataset that is widely used in mapping and modelling studies; these may need accounting for in analyses. The quality of the labelling in the reference dataset was also a key variable influencing mapping accuracy. Accuracy varied with the amount and nature of mislabelled training cases with the nature of the effects varying between classifiers. The largest impacts on accuracy occurred when mislabelling involved confusion between similar classes.
Introduction Maps are widely used in scientific research. Their accuracy can, however, be critical, with the effect of map error being dramatic in a range of applications (e. g. ,  [1] ). For example, the estimated value of ecosystem services for the conterminous USA determined using the National Land Cover Database (2006) changes from $1118 billion/y to $600 billion/y after adjustment for the known error in the maps used  [2]. It is essential, therefore, that maps be as accurate as possible and the accuracy information is conveyed usefully to map users. One initial source of error in mapping is the reference data used to construct the map. It is, for example, normally assumed that the reference dataset used is from an authoritative source and can be treated as a gold standard. This is, however, often unlikely to be true.
In addition, there may be other concerns about the reference data. These data may, for example, have been generated from samples that are small, biased, and unrepresentative. Moreover, in some large international databases the sampling issues may vary from region to region (e. g. , due to different national data acquisition policies). The databases may also contain errors of varying nature and magnitude such as mislabelling arising from confusion between classes  [3] , which may also vary regionally if, for example, the skills and expertise of data collectors vary. These various sources of error (e. g. , mislabelled cases) and uncertainty (e. g. , ambiguous class membership) may degrade mapping and the effect may vary between mapping methods. As a result, it is important to know the sensitivity of mapping methods to error in the data used to generate them.
Reference Data Quality and Mapping Reference data may sometimes be obtained from databases that bring together data from a variety of sources. While this is useful, there may also be a range of problems with such resources. One key issue is that the contributed data may have been acquired using very different methods. For example, different sample designs may have been used, and if this variation is not addressed in later analyses it could cause problems (e. g. , imbalanced samples, etc. ). The quality of the labelling of cases in a database may also vary. This is a major concern in common applications such as mapping land cover from remotely sensed data because the reference dataset is typically used as if it is perfect yet even a small deviation can be a problem. For example, in assessing the accuracy of maps or making estimates of class areal extent from them, small reference data errors can be a source of large error  [4].
Ground reference data play a fundamental role in supervised image classification. The ground dataset used is typically assumed to be perfect (i. e. , ground truth) but in reality is normally imperfect. Datasets such as the Global Biodiversity Information Facility (GBIF,  [7] ) for example, hold valuable information on species observations that could be used to aid mapping species directly or from remotely sensed data. However, the data contained in the database are highly variable. The contents include data aggregated from many sources, ranging from authoritative, systematic plot censuses and field surveys to casual observations contributed by "citizen scientists. " Standardizing the data in terms of factors, such as sampling effort or labelling quality, is a challenge. Mislabelling is, for example, a common error in ground data, even that acquired by authoritative sources  [3].
The importance of this type of issue will also vary between users and their planned use of the thematic map; for any specific use case, some errors will be more critical than others  [18]. As a general starting point, however, mislabelled cases in a ground dataset will be expected to degrade the training statistics and so ultimately the accuracy of a supervised digital image classification. The specific effects of mislabelled cases would, however, depend on the details of the approach to classification adopted. Classifiers, for example, can differ greatly in how they use a training set (e. g. , some focus upon summary statistical features such as the class centroid while others rely directly upon subsets of the individual cases available)  [16, [19] [20] [21] [22]  and so their sensitivity to mislabelling will be expected to vary.
The SVM has also been claimed to have a range of attributes that make it particularly attractive for use in mapping land cover from remotely sensed data. In particular, it has been claimed that the SVM is insensitive to the Hughes effect  [27] , that it only requires a small training set  [28, 29] , and that it is insensitive to error in the training set  [30]. The first claim, about freedom from the Hughes effect, has been shown to be untrue  [31]. The second claim, about the potential for accurate classification from small training sets, has been demonstrated but the training cases have to be collected with care to fulfil this potential  [32]. The focus of this article is on the final attribute that is claimed: that is, the low sensitivity of the SVM to error in the training dataset. The literature does include studies that show that the accuracy of classification by SVM can be affected by error in the training set  [33, 34] , and this issue is explored in this article from a remote sensing perspective.
For example, in many studies some of the land cover classes are defined in such a way that sites on the ground that are very similar belong to different classes. For example, the class forest is often defined using a variable such as the percentage canopy cover  [36]. Two sites on the ground made up of the same species and having similar environmental conditions could belong to completely different classes due to miniscule differences in their canopy cover if close to the threshold value used in the definition of the classes. As a result, error disproportionately affects cases that one would expect to be similar both on the ground and spectrally. This paper will briefly highlight imbalances in databases, often linked to sampling, which may require attention prior to a classification before focusing, in more detail, on the effects of mislabelled training cases on map accuracy.
Variation in Sampling In many mapping studies the available data are simply used without explicit accommodation for their detailed nature. For example, in land cover mapping from remotely sensed data it is common for a proportion of the available reference data to be used for training a classifier and the remainder used for validation. However, in some datasets there may be problems with such an approach. One problem with large international databases is that the data contributed may have been acquired following very different methods. Critically, for example, the sampling effort may vary greatly. This could lead to substantial problems with, for example, sampling being more intensive in some regions than others, leading to datasets that are artificially imbalanced in terms of class composition if the geographical distributions of the classes differ.
This latter issue can be a major problem as popular mapping methods such as the SVM can sometimes be highly sensitive to imbalanced training sets  [26]  and a failure to account for sampling variations may hinder the use of advanced machine learning classifiers. In this section the aim is to simply illustrate the magnitude of sampling problems, using a major database as an example. An important and increasingly used source of species field observations is the GBIF  [7]. GBIF data comprises a large range of species occurrence observations collected with a wide variety of sampling approaches. In addition, there may be differences in the methodologies used to observe and record occurrences per taxon. Plots, and plots within transects, are common practice in vegetation censuses, while transects, point counts, and live traps are preferred in the case of animals.
Using this approach, maps showing the differences in sampling effort (number of different survey dates in the database) and occurrences (observations counts) for a set of plant species over an equal sized grid of Europe were generated (Figure  1 ). The cartograms were developed using the free and open source software ScapeToad (http://scapetoad. choros. ch/). The cartograms were generated based on two metrics, number of field surveys (proxy: dates) and number of observations per grid cell. The size of error is given by the size the grid cell should have in terms of the real spatial area it covers, over the actual proportion, as calculated by the number of observations/area. Uncertainty is shown at the per grid cell scale and corresponds to the deformation of the original cell size, that is, cells bigger than their original size required strategies to reduce the effect of oversampling on the products derived from the GBIF data, while cells displayed as smaller than their original size required more sampling efforts.
Mislabelled Training Cases The effect of mislabelled cases in training datasets was explored using a set of classifiers. The set of classifiers used included contemporary. State-of-the-art approaches such as SVM, RVM, and SMLR, together with a conventional statistical classifier, a discriminant analysis (DA), as a benchmark. Training sets of varying nature were generated and the key aspects of the study design and results are presented in the following sub-sections.
Data and Methods A series of supervised classifications were undertaken using airborne thematic mapper (ATM) data acquired for a test site near Feltwell in the United Kingdom. This is a topographically flat test site that is composed mainly of large agricultural fields, each of which had been planted with a single crop type at the time of the ATM data acquisition (Figure  2 ). The ATM is a standard multispectral scanning system that acquires data in 11 spectral wavebands. Here, the spatial resolution of the imagery was much smaller than the typical field size, reducing aspects of the mixed-pixel problem and so the potential for ambiguous class membership. Samples of the data were input to a series of supervised image classifications using a range of classifiers, from conventional statistical classifiers to contemporary machine learning methods.
The initial training dataset was assumed to be error-free and a series of training sets of variable quality was obtained from it by controlled degradation following two strategies. In both strategies the class label for the training cases that lay closest to the border position between two classes in feature space, identified from the set of Mahalanobis distances to class centroids for each case  [22] , was altered; the focus is, therefore, on the border area between classes in feature space that is likely to furnish support vectors. Specifically, the difference between the Mahalanobis distance to the two spectrally closest classes was used as a simple means to identify border cases that lie between classes  [22]. The training cases for each class were ordered by the difference in this distance and a percentage of the cases with the smallest distance relabelled to form the imperfect training sets.
In the first strategy, the class label was altered from the actual class to that of the second most likely class of membership and so the error is between relatively similar classes. In the second strategy, the label was altered from the actual class to that of a class chosen at random. Both strategies were used to form a series of training sets in which the magnitude of mislabelled cases was 5%, 10%, and 20% of the total training set size. Throughout, therefore, the focus is on mislabelling of what may be thought of as border cases rather than, for example, randomly selected cases. The accuracy of each classification was assessed using a single testing dataset. This testing set was formed using stratified random sampling with 75 cases per class (n = 450). Note that this size of testing set exceeds the widely used suggestion that at least 50 cases per class be used.
The accuracy of each classification was assessed and expressed as the proportion of correctly allocated cases obtained from the confusion matrix. The statistical significance of differences in the magnitude of the estimated overall accuracy of classifications was also evaluated using the McNemar test at the 95% confidence level  [40]. Additionally, in recognition of the need to accommodate for the sample design used, for individual classes, the confidence interval around estimates of accuracy obtained was used to evaluate the statistical significance of differences in accuracy  [41]. A series of classifications were undertaken using training sets of variable quality. In each classification the size of the training set was constant. The initial training dataset was assumed to be error-free and a series of training sets of variable quality was obtained from it by controlled degradation following two strategies.
In the second strategy, the label was altered from the actual class to that of a class chosen at random. Both strategies were used to form a series of training sets in which the magnitude of mislabelled cases was 5%, 10%, and 20% of the total training set size. Throughout, therefore, the focus is on mislabelling of what may be thought of as border cases rather than, for example, randomly selected cases. The accuracy of each classification was assessed using a single testing dataset. This testing set was formed using stratified random sampling with 75 cases per class (n = 450). Note that this size of testing set exceeds the widely used suggestion that at least 50 cases per class be used. The accuracy of each classification was assessed and expressed as the proportion of correctly allocated cases obtained from the confusion matrix.
Classifiers Four classification algorithms were used: discriminant analysis, SVM, RVM, and SMLR. The salient details of each of the classifiers are provided below. This discussion draws, in part, on a previous article  [35]  that also provides fuller details on the SVM, RVM, and SMLR. In the discussion about different classification algorithms, a training dataset (x i , y i ) , i = 1,. , n, having n number of samples, where x = [x 1 , x 2 ,. , x f ] T ∈ R f is input vector with f spectral features and y = y 1 , y 2 ,. , y q T ∈ R q is the class vector with q classes, is used.
Conclusions Reference datasets used in map production are typically imperfect in some way. In this article it has been stressed that the reference data may have a heterogeneous nature in relation to issues such as sampling effort and may contain errors such as mislabelling. These imperfections can be expected to impact negatively on a mapping project. This is especially the case with the use of contemporary classifiers such as machine learning techniques like SVM. Imbalanced training samples can, for example, impact on SVM, but if the nature of the samples contributed to a reference dataset are known it may be possible to reduce the problem. Mislabelling has been proposed to be less of an issue (e. g. ,  [30] ), but here was given particular focus. Here, it was shown that the quality of datasets, in terms of the accuracy of their class labelling, is important in the production of land cover maps from remotely sensed data.
We will need to collect prospective data of circulating mutation emergence, including analysis during progression and non-progression to determine the clinical relevance before an interventional study. Such a prospective study is currently ongoing in our centre (NCT02473120). We also analyzed plasma samples at t p+3 , while patients were receiving various treatments. In contrast to a homogeneous evolution of ESR1 circulating kinetics before progression, after the end of AI exposure, we observed that ESR1 mutation variation is more heterogeneous. Indeed, we found that an increase in the amount of mutation was related to progression for 100% of the patients, as also reported by Takeshita et al. (11 progressions in 13 patients)  [10]. Due to the low number of patients analyzed, this value may be overestimated.
To assess the background noise of our method, the allele burden was measured in 10 preamplified cfDNA extracted from healthy control heparinized plasma samples collected in the same conditions as the patient samples. Using the method of Hindson et al. [26] , we defined the relevant threshold for each mutation to discriminate positive versus negative samples according to the reaction conditions (Supplementary Table  3 ). All data were analyzed using the QuantaSoft software (Bio-Rad, Hercules, CA, USA) and were manually reviewed to provide precise interpretation of data points. The variant allele fraction (VAF) was defined as the proportion of mutant DNA copies relative to the sum of mutant and wild-type DNA copies obtained by ddPCR. Samples at t p were considered mutated if at least two independent ddPCR analyses by two independent operators found a VAF above the mutation threshold.
The LFMIP experiments are of high relevance for the assessment of key feedbacks and systematic biases of land surface processes in coupled mode  (Dirmeyer et al. , 2015) , and are particularly focusing on two of the main feedback loops over land: the snow-albedo-temperature feedback involved in Arctic Amplification, and the soil moisture-temperature feedback leading to major changes in temperature extremes  (Douville et al. , 2016). In addition, LS3MIP will allow the exchange of data and knowledge across the snow and soil moisture research communities that address a common physical topic: terrestrial water in liquid and solid form. Snow and soil moisture dynamics are often interrelated (e. g. ,  Hall et al. , 2008; Xu and Dirmeyer, 2012)  and jointly contribute to hydrological variability (e.
GSWP3 forcing data are generated based on a dynamical downscaling of 20CR. A simulation of the Global Spectral Model (GSM), run at a T248 resolution (∼ 50 km) is nudged to the vertical structures of 20CR zonal and meridional winds and air temperature using a spectral nudging dy-namical downscaling technique that effectively retains synoptic features in the higher spatial resolution  (Yoshimura and Kanamitsu, 2008). Additional bias corrections using observations, vertical damping  (Hong and Chang, 2012)  and single ensemble member correction  (Yoshimura and Kanamitsu, 2013)  are applied, giving considerable improvements. Weedon et al. (2011)  provide the meteorological forcing data for the EU Water and Global Change (WATCH) programme 4  , designed to evaluate global hydrological trends and impacts using offline modeling.
The half-degree resolution, 3-hourly WATCH Forcing Data (WFD) was based on the ECMWF ERA-40 reanalysis and included elevation correction and monthly bias correction using CRU observations (and alternative GPCC precipitation total observations). WATCH hydrological modeling led to the WaterMIP study  (Haddeland et al. , 2011). The WFD stops in 2001, but within a follow-up project  EMBRACE Weedon et al. (2014)  generated the WFDEI data set that starts in 1979 and was recently extended to 2014. The WFDEI was based on the WATCH Forcing Data methodology but used the ERA-Interim reanalysis (4D-var and higher spatial resolution than ERA-40) so that there are offsets for some variable in the overlap period with the WFD. The forcing consists of 3-hourly ECMWF ERA-Interim reanalysis data (WFD used ERA-40) interpolated to half degree spatial resolution.
Historical offline simulations: Land-Hist The Tier 1 experiments of the offline LMIP experiment consist of simulations using the GSWP3 forcing data for a historical (1831-2014) interval. The land model configuration should be identical to that used in the DECK and CMIP6 historical simulations for the parent coupled model. The atmospheric forcing will be prepared at a standard 0. 5 × 0. 5 • spatial resolution at 3-hourly intervals and distributed with a package to regrid data to the native grids of the global climate models (GCMs). Also vegetation, soil, topography and land/sea mask data will be prescribed following the protocol used for the CMIP6 DECK simulations. Spinup of the land-only simulations should follow the TRENDY protocol 8  which calls for recycling of the climate mean and variability from two decades of the forcing data set (e.
, -2015) )  records of terrestrial water storage from the GRACE satellite  (Rodell et al. , 2009; Reager et al. , 2016; Kim et al. , 2009) , the multi-product LandFlux-EVAL evapotranspiration synthesis  (Mueller et al. , 2013) , multi-decadal satellite retrievals of the Fraction of Photosynthetically Absorbed Radiation (FPAR, e. g. ,  Gobron et al. , 2010; Zscheischler et al. , 2015) , and upscaled Fluxnet based products  (Jung et al. , 2010). Several details of snow and soil moisture dynamical processes can be indirectly inferred through the analysis of river discharge  (Orth et al. , 2013; Zampieri et al. , 2015). Variables simulated by the routing schemes included in the land surface models can be compared with the station data available from the Global Runoff Database (GRDC 15  ). Combined use of in situ discharge observations and terrestrial water storage changes observed by GRACE will verify how the land surface simulations partition the terms in the water balance equation (i.
, 2014). This metric provides a precise, overarching measure of the snow-induced perturbation to solar absorption in each model, integrating over the variable influences of vegetation masking, snow grain size, snow cover fraction, soot content, etc. SSRE is analogous to the widely used cloud radiative effect diagnostic, and its time evolution provides a measure of snow albedo feedback in the context of changing climate  (Flanner et al. , 2011). We recommend that the diagnostic snow shortwave radiative effect (SSRE) calculation be implemented in standard LS3MIP simulations (Tiers 1 and 2). This will enable us to evaluate the integrated effect of model snow cover on surface radiative fluxes. -Complementary snow-related offline experiments -Additional offline experiments are enabled by the provision of a collection of localized forcing data in the Land-Hist experiment (see above).
Time line, participating models and interaction strategy The offline land surface experiments (Land-Hist) are expected to be completed in early 2017. Future time slices can only be performed when the Scenario-MIP results become available. All coupled LS3MIP simulations and their subsequent analyses will be timed after the completion of the DECK and historical 20th century simulations, expected by mid-2017. Table  2  lists the participating Earth system modeling groups. The organizational structure of LS3MIP relies on active participation of modeling groups. Coordination structures are in place for the collection and dissemination of data and model results  (Eyring et al. , 2016) , and for the organization of meetings and seminars (by the core team members of LS3MIP, first six authors of this manuscript).