dataesr/scientific-paragraphs-categorization huggingface dataset:
The rates of snowfall were used directly from the reanalysis, since reliable measurements are lacking. However, the snow depth is limited to the approximate height of the polygons (Sect. 4. 1), which ensures realistic maximum annual snow depths independent of the snowfall rates. The largest uncertainties can hence be expected during the buildup phase of the snow cover, before the threshold height is reached. CryoGrid 3 was initialized to steady-state conditions of the forcing for the first 10 model years by re-running the model with the 1979-1989 forcing until the change of temperatures in the uppermost 100 m was less than 0. 05 • C between subsequent 10-year runs. The first initial condition was determined as described in  Westermann et al. (2013). Long-term thaw susceptibility run: in order to force the model from 1901 to 2100 with a consistent data set, model-derived forcing data building on an anomaly approach were employed which were subsequently downscaled for Samoylov Island similar to the validation run.
Past and present-day forcing data were taken from the CRU-NCEP data set (http://dods. extra. cea. fr/data/p529viov/ cruncep/), which combines high-frequency variability from the National Centers for Environmental Prediction/National Center for Atmospheric Research reanalysis  (Kalnay et al. , 1996)  with the monthly mean climatologies from the Climatic Research Unit temperature and precipitation data sets  (Harris et al. , 2014). The meteorological forcing for future climate change experiments was constructed following the Representative Concentration Pathway (RCP) 4. 5 and 8. 5 climate change projections obtained with the CCSM4 coupled climate model  (Meehl et al. , 2012). Monthly climate anomalies (for the meteorological state variables: air temperature, specific humidity, surface pressure, and wind speed) and scale factors (for the meteorological flux variables: precipitation, incoming shortwave and long-wave radiation) were extracted from the "Permafrost Carbon RCN forcing data"  (ESFG, 2015).
The high-frequency variability in the future meteorological forcing was obtained by looping over the 1996-2005 CRU-NCEP forcing data. The base climatological period, which is used to calculate the monthly anomalies/scale factors, is also 1996-2015. These forcing data were also used in  Koven et al. (2015)  and in an ongoing international intercomparison exercise within the framework of the Permafrost Carbon Network (http://www. permafrostcarbon. org/;workinprogress). As for the validation run, the data from the closest model grid cell were downscaled for Samoylov Island using the in situ record from the AWS. Since the match with the measured data was significantly worse compared to the ERA-Interim reanalysis (in particular for the CCSM4 data, but also for CRU-NCEP), the entire data set for the overlap periods 2010-2012 for CRU-NCEP and 2012-2014 for CCSM4 was used to determine a linear regression for L in , T air , and q.
For the incoming shortwave radiation S in , which follows a defined diurnal pattern, the procedure was conducted separately for each of the time step 00:00, 06:00, 12:00, and 18:00. Values with S in = 0 were excluded from the regression. Downscaling of wind speeds was challenging, since there was no statistically significant correlation to measured data. Therefore, the distribution of wind speeds from CRU-NCEP/CCSM4 was scaled to fit the distribution obtained from the in situ record. For this purpose, the wind speeds were sorted in ascending order in bins containing 20 % of the available data each. The CRU-NCEP/CCSM4 data were then scaled linearly so that the average wind speed of each bin matched the average wind speed of the in situ data in the respective bin. As for the validation run, the snowfall data from CRU-NCEP/CCSM4 were used without modification, but as in the validation run, the maximum height of the snow cover is limited to the approximate height of the polgon rims.
Long-term thaw susceptibility runs with CryoGrid 3 Xice The CRU-NCEP/CCSM4 data set facilitates simulations of the ground thermal state over a period of 200 years from 1901 to 2100 for different future warming scenarios. Since these forcing data are derived from a coupled climate model, they feature considerable biases for some forcing variables, although partly corrected as described in Sect. 4. 2. Therefore, the ground thermal regime modeled for Samoylov Island deviates to a certain extent from the observations. Using the stratigraphy for Samoylov Island (Table  2 ), the modeled maximum thaw depths for the period 2002-2014 is on average 0. 66 m, approximately 0. 1 m more than for the more realistic forcing of the validation run (Fig. 5 ). The stratigraphy of the ground has a pronounced influence in particular on the thaw depth: for the ice complex stratigraphy, a significantly lower average maximum thaw depth of 0.
CryoGrid 3 -an experimental platform for permafrost process parameterizations The structure of CryoGrid 3 is designed to be similar to ESM land-surface schemes, so that algorithms and process parameterizations can easily be transferred. While an adaptive time step is employed in the time integration procedure (Sect. 2. 6), constant time steps as typical in large-scale modeling approaches are feasible, but the numerical stability cannot be guaranteed with the simple forward Euler scheme in this case. Nevertheless, this choice guarantees maximum simplicity and readability of the CryoGrid 3 code, so that the model can be easily modified to test the effect of adding different process parameterizations to the basis module. As such, the goal of CryoGrid 3 is different from existing land-surface schemes, such as GEOtop  (Endrizzi et al.
, 2014) , COUP  (Jansson and Karlberg, 2004)  or SURFEX  (Masson et al. , 2013) , which are aimed at users running the models with a predefined set of model options, largely without modifying the program code itself. In contrast, CryoGrid 3 is optimized to encourage users with the necessary programming skills to modify and extend the program code, thus providing an open platform to explore the effect of different process parameterizations on model results. In this study, CryoGrid 3 has been applied to cold, continuous permafrost in Siberia. While the satisfactory agreement of model results and in situ observations is encouraging, it is likely that the model performance would be less satisfactory for some environmental conditions encountered in permafrost areas. We see three major shortcomings of the current version of CryoGrid 3, which should be addressed prior to application in a wider range of permafrost environments: 1.
Acknowledgements. This work was funded by  CryoMet  (project no. 214465 ;  Research Council of Norway) ,  Page21  (contract no. GA282700 ;  European Union   FP7-ENV) ,  COUP  (project no. 244903/E10 ;  JPI Climate ;  Research Council of Norway) ,  SatPerm  (project no. 239918 ;  Research Council of Norway ), a  Feodor-Lynen Grant  awarded to  M. Langer  by the  Alexandervon-Humboldt Foundation ,  Germany , and the  Department of Geosciences, University of Oslo, Norway. We gratefully acknowledge the support of the Russian-German research station on Samoylov Island. We thank  Günther Stoof ,  Sofia Antonova , and  Sascha Niemann  for their contributions to fieldwork, and  Isabelle Gouttevin  for valuable discussions in the planning stages of CryoGrid 3. Three anonymous reviewers greatly contributed to improving the quality of the original manuscript.
The BioCreative Interactive text mining Task (IAT) introduced in BioCreative III  (8, 9)  has served as a means to observe the approaches, standards and functionalities used by state-of-the-art TM systems with potential applications in the biocuration domain. The IAT task also provides a means for biocurators to be directly involved in the testing of TM systems. IAT consists of demonstration and evaluation of web-based systems addressing literature curation tasks, evaluated by biocurators on performance and usability. One of the main goals is to collect data from biocurators testing the systems, and provide useful feedback to developers on possible enhancements and how to better tailor their system for biocuration. The benefits are multifold, including: direct communication and interaction; exposure to new TM tools that can be potentially adapted and integrated into the biocuration workflow, contribution to the development of systems that meet the needs of the biocuration community and dissemination of findings in proceedings and peerreviewed journal articles.
A User Advisory Group (UAG, http://www. biocreative. org/about/biocreative-v/user-advis ory-group), representing a diverse group of users with literature-based curation needs, has been assisting in the design and assessment of the IAT. In particular, in BioCreative V the UAG was engaged in multiple aspects of the task, including preparing the requirements for the systems, reviewing the user survey, recruiting biocurators and testing the systems. Some tasks proposed in IAT build on previous BioCreative outcomes. For example, BioCreative IV included a panel session, sponsored by the Department of Energy (DOE), on the TM needs of the metagenomics community which studies the genetic material recovered directly from environmental samples. The discussions from this session inspired the participation of a team in the IAT task specifically to address the extraction of environmental and species metadata from free text.
Similarly, three teams that participated in BioCreative IV, participated again in BioCreative V with their systems presenting enhancements based on feedback from previous participation. It is worth noting that the interactive activities have gained traction in the last few years, beyond BioCreative context. For example, in recognition of potential barriers that may inhibit the widespread adoption of biomedical software, the 2014 i2b2 Challenge introduced a special track, Track 3-Software Usability Assessment, which highlighted usability issues and therefore limitation of use/ adoption of biomedical software  (10). Also, in parallel to IAT track, BioCreative V has introduced the Collaborative Biocurator Assistant Task (BioC)  (11) , which explores the integration of the BioC format  (12)  output from different TM modules to provide a system for literature curation of protein-protein interactions tailored for the BioGrid Database  (13).
Systems and user recruitment Seven international teams participated in the IAT. Table  2  summarizes some aspects of the participating systems. The systems cover a variety of curation steps in the literature curation workflow, namely triage (selection of relevant articles), entity detection (identification of relevant bioentities in text with/without linking to entities in corresponding databases) and relation/evidence extraction (capture the relations between bioentities and evidence for such relations). Gene and disease/phenotype annotation theme was prevalent (5 out of 7 systems). However, there was a great variability in the complexity of the systems in terms of the task proposed (Table  2  under column 'Description') and the overall system capabilities. In the former case, the simplest task included the detection of bioentities in text with links to appropriate vocabularies [e.
g. EXTRACT  (17)  and Ontogene  (18) ], others also captured relations between the bioentities [e. g. Argo  (19, 20) , MetastasisWay  (21) , egas  (22, 23) ] and the most complex one included, in addition, the representation of the extracted information in structure language (e. g. BELIEF). Also, the system capabilities varied even within a similar type of task, some offered workflow design options, where the user can build customized TM workflows [e. g. Argo  (19, 20) ]; management systems for curation, where a user can monitor and compare the different annotations [e. g. , egas  (22, 23)  and BELIEF], plug-ins/ bookmarklets for the web browser [e. g. EXTRACT and MetastasisWay  (21) ] and network visualization [e. g. GenDisFinder  (24)  and MetastasisWay]. In terms of the text selected for text mining, three of the seven systems offered full-text processing for the task.
However, other systems also have the capability to use full-text (egas) or even any text as long as it is presented in a browser (EXTRACT). Finally, in terms of the browser compatibility, which is important at the time of testing the systems, majority of the systems were reported to be compatible with Chrome (all systems) and Firefox (6 out of 7). With the help of the UAG and the teams, we were able to recruit a wide variety of biocurators worldwide. A total of 43 biocurators participated in the IAT in different capacities. Figure  1  shows the distribution by geographical location (Figure  1A ), examples of type of database or institution represented (Figure  1B ), and distribution by system and level of participation (Figure  1C ). All systems were inspected by at least seven biocurators at some level (full/partial).
Evaluation It should be noted that the IAT activity is a demonstration task, which yields qualitative rather than quantitative  The columns from left to right indicate: (i) name of the system, (ii) description of the system in relation to the biocuration task proposed, (iii) bioconcepts (what entities are detected, e. g. gene, disease), (iv) standards adopted by the system to link the bioconcepts detected to corresponding databases and ontologies, (v) what step of the literature curation workflow the system helps with, (vi) the relations captured include relations between bioconcepts (e. g. relation between gene-disease), (vii) the text column lists what type of text the systems are able to process and (viii) the browser column indicates the system compatibility with web browsers. results.
Full curation task In the full curation task, some teams reported equal or slight improvement of throughput with the system over the manual task. One system reported important differences between novice and expert users. Another system reported lower throughput using the tool, due to differences in level on annotation between manual and system-assisted (in the latter the user needs to check all mention and normalization results). One system reported >10 times the throughput using the TM system (Table  10 ). It is worth noting that this system was exclusively tested by curators from one database working closely with the developer throughout the task. The results from the SUS score (Figure  4 ) shows that most of the systems scored around average. In one case, a system showed lower than average SUS score but average learnability score, pointing to issues specifically in the usability of the system.
The usability score is just slightly higher than the average; the learning component seems to have the highest variability. System comparison to BioCreative IV: Argo offers high flexibility and customization of annotation workflows. In BioCreative IV, the user was asked to build the TM workflow to facilitate the curation of metabolic processrelated concepts  (20). However, this was a real bottleneck to the biocurators, as they were not familiar with all the different TM modules and file formats. In BioCreative V, the user was presented with the text mined results with a pre-built workflow. In addition, the ability to view or edit already saved manual annotations was included. These features seem to have improved the usability of the system for the biocurators. BELIEF (URL: http://belief. scai.
All users worked with the same set of data divided into two sets (Set1 and Set2), containing 10 documents each. Results: There were two distinct groups of users: those who had previous experience with BEL coding and those who were new to both BEL and the annotation interface. Annotators in the first group had a higher throughput per hour (approximately 5 documents) than the novice (1-2 documents). As mentioned before, we provided 10 documents each for the two curation tasks (TM-assisted and non-TM-assisted curation) to each annotator. Due to time constraints, most of the annotators curated less than ten documents for each of the tasks. Overall 25 documents were curated in 392 min with the BELIEF Dashboard. The manual curation produced in total 24 documents in 374 min out of which eight documents (33.
This is in agreement with the results shown for the pre-designed tasks. It should also be mentioned that the task had the highest complexity-extraction of complex relationships between different entity classes as well as to understand and acquire the BEL syntax. egas (URL: https://demo. bmd-software. com/egas/, Team 286: Matos, Campos, Pinho, Silva, Mort, Cooper, and Oliveira) Description: Egas is a web-based platform for TMassisted literature curation, supporting the annotation and normalization of concept mentions and relations between concepts. Egas allows the definition of different curation projects with specific configuration in terms of the concepts and relations of interest for a given annotation task, as well as the ontologies used for normalizing each concept type. Egas may be described as an 'annotation-as-a-service' platform.
Document collections, users, configurations, annotations and back-end data storage, are all managed centrally, as are the tools for document processing and TM. This way, a curation team can use the service, configured according to the annotation guidelines, to take advantage of a centrally managed pipeline. Task: This task was identification of human, inherited gene mutations and associated clinical attributes, such as inheritance mode and penetrance, described in PubMed abstracts. Seven curators were selected and were asked to annotate documents that were pre-analyzed by an automatic concept recognition tool (half of the corpus), and raw documents (the remaining corpus), in order to evaluate the added benefit of TM-assisted curation. Three curators annotated the complete corpus, two curators followed a 4 h time-limited work plan, and the other two curators annotated a small portion of the corpus (13 and 9 documents).
Corpus: A classifier, trained using articles previously selected as relevant for the Human Gene Mutation Database (HGMD)  (28) , was used to prioritize the results from a PubMed query (about 28 000 articles). Then the top 100 articles were selected for the annotation task. Results: In general, it took a shorter time to curate documents that had been previously annotated by the concept recognition tool, although the results are not The upper half of the table shows the number of curators involved in the evaluation, and the throughput (average time per article or per concept annotated) without or with the assistance of TM. The lower half of the table shows the central tendency of the survey results for the pool of questions related to ability to complete the task (Task), Design of the interface (Design) and Usability.
Behind the web-based user interface, the system combines components from published systems for Named Entity Recognition (NER) of environments, organisms, tissues and diseases. Task: The two full evaluators were asked to investigate if the EXTRACT bookmarklet can help them locate sampling environment information in a document and if it can accelerate the metagenomics record metadata annotation process. In particular, they were asked to annotate samples as recommended by the standards. Annotated metadata included filling in the 'environmental feature, environmental material and biome' describing a sample's source environment. The evaluators performed this task with and without the assistance of EXTRACT and compared the time taken in both cases. The goal was to assess the curation acceleration that EXTRACT offers when evaluators work as closely as possible to their actual workflow.
variability as it is the result of two users using EXTRACT in their own different curation pipelines. GenDisFinder (URL: http://biominingbu. org/GenDis Finder, Team 294: Subramani and Natarajan) Description: GenDisFinder is a web-based TM tool that aids in the extraction of known/novel/inferred human gene-disease associations from biomedical literature and further categorizes them using networks analysis. GenDisFinder has four different modules for the above tasks: (1) gene mention and normalization of gene/protein names with NAGGNER and ProNormz  (33) , respectively, (2) disease mention identification and normalization using OMIM-based normalized disease phenotype dictionary, (3) identification and extraction of semantic relations between genes and diseases using a relation keyword dictionary and (4) construction of gene-disease association networks and further categorization.
To the best of our knowledge, GenDisFinder is the first tool that integrates TM with network analysis to discover novel genes associated with a disease and provides an interface to view the interaction network. Task: Curate a set of abstracts for gene-disease association. Curate genes, disease and gene-disease association relations. Also, validate the categorization of the abstract into novel, unknown or known gene-disease associations. Corpus: This is an in-house, curated, gene-disease association corpus called the Human Gene-Disease Association (HGDA) corpus, which is available on-line from the website. From the GeneRIF database a randomly selected set of 500 sentences were manually annotated with gene name, disease name relation type and gene-disease association information by three domain experts.
Corpus: To collect a set of articles related to metastasis and its regulation, we searched PubMed with the query term 'EMT[title/abstract] AND TGF-b[title/abstract]' (Note: EMT is epithelial-to-mesenchymal transdifferentiation). From the result, 300 abstracts were randomly selected as the curation dataset for the interactive TM task. The table shows the central tendency of the survey results for the pool of questions related to ability to complete the task (Task), Design of the interface (Design) and Usability. The responses were converted to a numeric scale from 1 (most negative response) to 5 (most positive response). To give an idea of the response distribution, the central tendency is described with the median along with minimum (min) and maximum (max) values, respectively, and the lower (Q1) and upper (Q3) quartiles, respectively.
BioCreative has served a catalyst of interactions between the NLP and the biocuration community since 2010. As a result, the systems participating repeatedly in the BioCreative IAT task (e. g. Argo and Ontogene) have improved their performance over time. In addition, in the annual International Society for Biocuration (ISB) meetings, the subject of TM for biocuration has moved from isolated talks within 'Literature collection & curation' session  (ISB 2008 (ISB -2010) )  to its own session/workshops 'Text/literature Mining'  (ISB 2011 (ISB -2016)). The biocuration community is now very actively exploring and evaluating tools, especially for the need of scaling up curation  (37). Moving forward the interactive task will work on refining and documenting the metrics for assessing interactive systems, and exploring TM use in several other real-word and large-scale applications beyond database curation.
participated in the task and the discussion during the BioCreative V Panel 1 on text mining applications, the latter reason has come up as a real bottleneck. The integration of the TM tools in the curation pipeline in wellestablished and mature databases might be difficult to achieve. The best opportunities exist in small and/or newly established databases. Therefore, TM developers should consider interaction with biocuration groups at an early stage of database development. Alternatively, effective ways to consume the TM data in ways that do not significantly interfere with the established database pipelines need to be developed, such as web-based TM services (26).