dataesr/scientific-paragraphs-categorization huggingface dataset:
The Dinners from Hell corpus  (Rudinger et al. , 2015a ) is a similar dataset centered around the EATING IN A RESTAURANT scenario. In the past, script modeling systems have been evaluated using intrinsic tasks such as event ordering  (Modi and Titov, 2014) , paraphrasing  (Regneri et al. , 2010; Wanzare et al. , 2017) , event prediction (namely, the narrative cloze task)  (Chambers and Jurafsky, 2008, 2009; Rudinger et al. , 2015b; Modi, 2016)  or story completion (e. g. the story cloze task Q1 Where did they sit inside the sauna? a. on the floor b. on a bench Q2 How long did they stay in the sauna? a. about ten minutes b. over thirty minutes Figure  1 : An example for a text from MCScript with 2 reading comprehension questions. (Mostafazadeh et al. , 2016) ). These tasks test a system's ability to learn script knowledge from a text but they do not provide a mechanism to evaluate how useful script knowledge is in natural language understanding tasks.
Our shared task bridges this gap by directly relating commonsense knowledge and language comprehension. The task has a machine comprehension setting: A machine is given a text document and asked questions based on the text. In addition to what is mentioned in the text, answering the questions requires knowledge beyond the facts mentioned in the text. In particular, a substantial subset of questions requires inference over commonsense knowledge via scripts. For example, consider the short narrative in (1). For the first question, the correct choice for an answer requires commonsense knowledge about the activity of going to the sauna, which goes beyond what is mentioned in the text: Usually, people sit on benches inside a sauna, an information that is not given in the text. The dataset also comprises questions that can just be answered from the text, as the second question: The information about the duration of the stay is given literally in the text.
Related Work Recently, a number of datasets have been proposed for machine comprehension. One example is MCTest  (Richardson et al. , 2013) , a small curated dataset of 660 stories, with 4 multiple choice questions per story. The stories are crowdsourced and not limited to a domain. Answering questions in MCTest requires drawing inferences from multiple sentences from the text passage. In our dataset, in contrast, answering requires drawing inferences using knowledge not explicit in the text. Another recently published multiple choice dataset is RACE  (Lai et al. , 2017) , which contains 100,000 questions on reading examination data. Rajpurkar et al. (2016)  have proposed the Stanford Question Answering Dataset (SQuAD), a data set of 100,000 questions on Wikipedia articles collected via crowdsourcing.
In that dataset, the answer to a question corresponds to a segment/span from the reading passage. Since Wikipedia articles mostly contain factual knowledge, SQuAD does not assess how in practice, language comprehension relies on implicit and underrepresented knowledge about everyday activities i. e. script knowledge. Weston et al. (2015)  have created the BAbI dataset. BAbI is a synthetic reading comprehension data set testing different types of reasoning to solve different tasks. In contrast to our dataset, the artificial texts in BAbI are not reflective of a typically occurring narrative text. Two recently published datasets that also have a larger focus on commonsense reasoning are NewsQA and TriviaQA. NewsQA  (Trischler et al. , 2017)  contains newswire texts from CNN with crowdsourced questions and answers.
During the question collection, workers were only presented with the title of the text, and a short summary. This method ensures that literal repetitions of the text are avoided and the generation of non-trivial questions requiring background knowledge is supported. The NewsQA text collection differs from MCScript in domain and genre (newswire texts vs. narrative stories about everyday events). Knowledge required to answer the questions is mostly factual knowledge and script knowledge is only marginally relevant. TriviaQA  (Joshi et al. , 2017)  contains automatically collected question-answer pairs from 14 trivia and quiz-league websites, together with webcrawled evidence documents from Wikipedia and Bing. While a majority of questions require world knowledge for finding the correct answer, it is mostly factual knowledge.
Script and Commonsense Knowledge Data We also encouraged participants to make use of existing script data collections. Thus, we provided several existing collections of script data together with the machine comprehension corpus: DeScript  (Wanzare et al. , 2016) , RKP  (Regneri et al. , 2010)  and the OMCS stories  (Singh et al. , 2002). The three datasets contain sequences of short, telegraphstyled descriptions of all events that need to be conducted in a scenario (event sequence descriptions, ESDs). The data sets contain ESDs for different numbers of scenarios, with a total coverage of over 200 scenarios. The complexity of scenarios varies from simple activities, such as opening a window, to more complex ones, such as attending a wedding. For 90 of the 110 scenarios in MCScript, there exist multiple ESDs per scenario in at least one of the 3 script data collections.
, 2017). One participating system made use of script knowledge in the form of event sequence descriptions. Resources commonly used by participants include pretrained word embeddings such as GloVe  (Pennington et al. , 2014)  or word2vec  (Mikolov et al. , 2013) , and preprocessing pipelines such as NLTK 4. In the following, we provide short summaries of the participants' systems and we give an overview of models and resources used by them (Table  1 ). Non-neural methods IUCM  (Reznikova and Derczynski, 2018 ) applied an unsupervised approach that assigns the correct answer to a question based on text overlap. Text overlap is computed based on the given passage and text sources of the same topic. Different clustering and topic modeling techniques are used to identify such text sources in MCScript and DeScript.
The best performance for each column is marked in bold print. Significant differences in results between two adjacent lines are marked by an asterisk (* p<0. 05) in the upper line. The last line shows the human upper bound  (Ostermann et al. , 2018)  as comparison. (based on ConceptNet). The model is pretrained on another large machine comprehension dataset, namely the RACE corpus. MITRE  (Merkhofer et al. , 2018)  use a combination of 3 systems -two LSTMs with attention mechanisms, and one logistic regression model using patterns based on the vocabulary of the training set. The two neural models use different word embeddings -one trained on GoogleNews, another one trained on Twitter, which were enriched with word overlap features. Interestingly, the simple logistic regression model achieves competitive performance and would have ranked 4th as an individual system.
Jiangnan  (Xia, 2018)  applies a BiLSTM over GloVe and CoVe embeddings  (McCann et al. , 2017)  with an additional attention mechanism. The attention mechanism computes soft word alignment between words in the question and the text or answer. Manual features, including part-of-speech tags, named entitity types, and term frequencies, are employed to enrich word token representations. ELiRF-UPV (José -Ángel  González et al. , 2018)  employs a BiLSTM with attention to find similarities between texts, questions, and answers. Each word is represented based on Numberbatch embeddings, which encode information from ConceptNet. YNU Deep  (Ding and Zhou, 2018)  test different LSTMs and BiLSTMs variants to encode questions, answers and texts. A simple attention mechanism is applied between question-answer and text-answer pairs.
The final submission is an ensemble of five model instances. ZMU  (Li and Zhou, 2018 ) consider a wide variety of neural models, ranging from CNNs, LSTMs and BiLSTMs with attention, together with pretrained Word2Vec and GloVe embeddings. They also employ data augmentation methods typically used in image processing. Their best performing model is a BiLSTM with attention mechanism and combined GloVe and Word2Vec embeddings. ECNU  (Sheng et al. , 2018)  use BiGRUs and BiLSTMs to encode questions, answers and texts. They implement a multi-hop attention mechanism from question to text (a Gated Attention Reader  (Dhingra et al. , 2017) ). YNU AI1799  (Liu et al. , 2018)  submitted an ensemble of neural network models based on LSTMs, RNNs, and BiLSTM/CNN combinations, with attention mechanisms. In addition to word2vec embeddings, positional embeddings are used that are generated based on word embeddings.
YNU-HPCC  (Yuan et al. , 2018)  use an ensemble of neural networks with stacked CNN and LSTM layers and attention. CSReader  (Jiang and Sun, 2018)  use GRUs to encode questions and texts. Answer and text are combined by using an attention mechanism that models soft word alignments, inspired by work on Natural Language Inference  (Bowman et al. , 2015). Two answer classifiers based on these representations are ensembled for prediction.
In contrast, only one participant used crowdsourced script data from the DeScript corpus in their final submission, IUCM. They found that the use of script data, instead of or in addition to texts, improved performance by up to 0. 3% absolute. CSReader tried to extend their neural model with script data from OMCS, but report that it did not result in an improvement. No participant made use of narrative chains or other forms of structured/learned representations of scripts or events (such as event embeddings). Pretraining. Most participants made use of pretraining in the form of word embeddings such as word2vec or GloVe, that were build on large data collections. Yuanfudao used the RACE dataset, which is the largest available multiple-choice machine comprehension corpus, for pretraining the complete model for several epochs.
In their ablation analysis, they found pretraining to have the largest effect on model performance, with improvements in accuracy of up to 1. 4% absolute. This result underlines that the comparably small size of MCScript naturally restricts how much neural approaches can learn from the data without overfitting. Word representations. For representing tokens, most participants used word2vec embeddings, GloVe embeddings, or combinations thereof. The participating teams used different dimensionality sizes, and some of them refitted the vectors while others did not, leading to differing outcomes for both embedding types. In summary, none of both representations seems to clearly outperform the other. In contrast, participants consistently found that extending word representations with additional features improves results: For example, Yuanfudao and Jiangnan use predicted part-of-speech tags and named entity information, as well as term frequency, and report improvements of up to 1% absolute in accuracy.
Conclusions This shared task provides an evaluation framework for commonsense knowledge in a machine comprehension setting. We create the MCScript corpus, which provides 2,119 stories and 13,939 answers for 110 everyday activities of different complexity. In contrast to previous datasets, MCScript was created in a way that results in a relatively large amount of common sense questions, i. e. questions which can not be answered directly from the text but require some form of common-sense knowledge about the scenario under consideration to be answered correctly. 24 teams submitted systems during the the evaluation period of the shared task, of which 11 teams submitted task description papers. The bestperforming system achieves an overall accuracy of 84%, which outperforms the two baselines by a large margin; yet, there gap to the human upper bound (98%) is still relative large.
Although participants were explicitly encouraged to use additional common-sense knowledge resources like DeScript of OMCS, only 3 systems (including the best-performing system) actually used such additional resources. The evaluation results suggest that additional common-sense knowledge is in fact beneficial for overall accuracy. However, the positive effect is relatively small, which might be due to the fact that our dataset has been created in a way that leads to relatively "easy" stories, and that the systems are able to learn a certain amount of common sense knowledge directly from the stories. In future work, it would be interesting to see if the results of our shared task carry over to other, presumably more complex stories like for instance personal blog stories from the Spinn3r corpus  (Burton et al.
doe. gov/data-andtools/bbtools/), and chimeras were identified and removed with VSEARCH_-UCHIME_REF  55  using Greengenes Release 13_5 (at 97%)  56. Singletons were removed and the resulting sequences were clustered into operational taxonomic units (OTUs) with VSEARCH_CLUSTER 55 at 97% sequence identity  57. Representative sequences for each OTU were taxonomically assigned by RDP Classifier with the bootstrap threshold of 0. 8 or greater  58  using the Greengenes Release 13_5 (full)  56  as the reference. Unless stated otherwise, default parameters were used for the steps listed. The fungal ITS sequences were analysed using PIPITS  59  with default parameters as outlined in the citation. Briefly, this involved quality filtering and 97% clustering of the ITS2 region as indicated above for the 16S processing, using the UNITE database for chimera removal and taxonomic identification of representative OTUs.
Subsequent analyses of taxon abundances were conducted in R using principally the vegan library (https://github. com/vegandevs/vegan) for rarefaction and ordination. Both bacterial and fungal OTU abundance tables were resampled to a minimum of 4000 reads per sample, and samples with a Shannon diversity higher than 6 and 3. 2 for 16S and ITS, respectively, were removed prior to further analyses. We generated non-metric multidimensional scaling (NMDS) plots to visualise and determine the effect of drought on fungal and bacterial community composition using the functions decostand and adonis in the R library vegan. We also performed principal components analysis (PCOA) based on Bray-Curtis dissimilarities on both communities using the function cmdscale; PCOA scores were used as proxies for community composition in subsequent structural equation modelling.
For calculating indicator taxa and generating circle plots all taxa with fewer than ten reads across all samples were also excluded to ease presentation. We analysed fungal and bacterial networks for each sampling and drought and control treatment separately. Thus, each network was based on 36 communities, but only OTUs that occurred in at least 8 communities were included in the analysis, as in Shi et al. 28. All network analyses were done in R, using the package igraph, as in Williams et al. 26  ; we adapted the code available at https://github. com/ ryanjw/co-occurrence. Interactions consisted of Spearman's rank correlations and co-occurrence networks were constructed using only significant correlations (P < 0. 01 as in Barberan et al. 21 ) of ρ > 0. 6; this cutoff was chosen to include a range of interactions strengths (not only strong interactions).
Random networks were constructed as in Williams et al. 26. Subsequently, we detected network modules using the edge. betweenness. community function and calculated clustering coefficients using the transitivity function, analysed the centrality of network nodes using the betweenness function, and analysed the connectedness of network nodes using the degree function. All bacterial and fungal networks were significantly more clustered than random networks. Networks were visualised in the R library igraph. General statistical analyses. All data were checked for normality and logtransformed if necessary. Gas fluxes, N cycling gene data and measures derived from sequencing data were analysed using repeated measures ANOVA taking into account the split-plot design (error term with block, drought and mesocosm), soil nutrient data were analysed using ANOVA (error term with block and drought).
All analyses were performed in R version 3. 3. 2  60. Structural equation modelling. We constructed our a priori models based on current knowledge on plant-microbe-functioning interactions and tested whether the data fit these models using the standard and the multigroup modelling approach in the R library lavaan. We used model modification indices and stepwise removal of non-significant relationships as in De Vries and Bardgett  61. We used a minimum set of parameters to assess model fit, including root mean square error of approximation (RMSEA), and comparative fit index (CFI). We used PCOA axis 1 scores as a proxy for fungal and bacterial community composition, PCA axis 1 scores as a proxy for plant community composition, and the ratio between the sum of nirS and nirK and the sum of nosZI and nosZII gene abundances to indicate relative changes in genetic potential for the soil functions denitrification and N 2 O reduction.
Acknowledgements This study was funded as part of the  European  project  EcoFINDERS  ( FP7-264465 ). F. T. d. V. is supported by a  BBSRC   David Phillips Fellowship  ( BB/L02456X/1 ). We thank  Caley Brown ,  Melanie Hartley ,  Neil Mullinger ,  Helen Quirk ,  Deborah Ashworth  and  Victor van Velzen  for help in the lab and field.
Code availability. Code for network analysis was adapted from, and is available at, https://github. com/ryanjw/co-occurrence. Code for indicator analysis is available on request. Data availability. OTU tables and metadata are available from figshare (https:// doi. org/10. 6084/m9. figshare. 6548999. v1). Raw sequence data have been deposited at the EBI European Nucleotide Archive under study accessions ERP109485 (fungal amplicons) and ERP109472 (16S amplicons).
RT data We analyzed 42 genome-wide RT datasets of Repli-chip previously published  (Rivera-Mulia et al. , 2015) , representing 25 cell types and differentiation intermediates of human development. The genome was first divided into bins of 50 kb, then RT value was calculated for each bin. The 42 datasets include human embryonic stem cells (hESC), primary cells and established cell lines representing intermediate stages of endoderm, mesoderm, ectoderm and neural crest development (Supplementary Table  S1 )  (Hiratani et al. , 2010; Pope et al. , 2011; Rivera-Mulia et al. , 2015; Ryba et al. , 2011b; Sexton et al. , 2012). Telomeric and centromeric regions, as well as sex chromosomes were removed in this analysis. The RT values used in our analysis cover 85% of the human genome. The distributions of the RT values aggregated across all samples are shown in Figure  1A.
HMM Like the chromHMM model proposed by Ernst and Kellis to call chromatin states  (Ernst and Kellis, 2010, 2012; Ernst et al. , 2011) , we use a multivariate HMM to capture the underlying structure of the 42 RT programs. Our model assumes a fixed number-15, of hidden states K (the selection of K is discussed in detail in a subsequent section). For each hidden state, we use a distinct normal distribution (emission probabilities) to characterize the RT values (at the bin level) from each RT program. Specifically, for each of the K hidden states, there are two parameters: l k and r 2 k , denoting the mean and variance of the normal distribution in hidden state k ðk¼1;. ;KÞ that the RT values follow. Although many of the datasets and cell types are closely related, for the sake of mathematical simplicity, following chromHMM, we treat the RT values obtained from different cell types as independent condition on the assigned RT state at the locus.
Let s ct denotes the unobserved state on chromosome c at location t for state sequence s c. The model has a similar structure as chromHMM method  (Ernst et al. , 2011)  but customized for fitting RT data. We use R package depmix4 to conduct inference on the HMM  (Visser, 2010). Once the HMM parameters are estimated and 50 kb bins along the genomes are assigned to different RT states. We group all bins belong to the same RT states together and obtain average RT values for each RT state in every cell type. We next perform a two-way clustering on these average RT values, and ordered the 15 RT states such that the first one has the earliest RT values and the last one has the latest RT values. By doing this, the 15 RT states are ordered from early RT to late RT.
, 2012)  annotation to produce an integrated annotation  (Hoffman et al. , 2013)  with seven states for six ENCODE cell lines (GM12878, H1-hESC, HepG2, HELA, K562 and HUVEC, files available on http://ftp. ebi. ac. uk/pub/databases/ ensembl/encode/integration_data_jan2011/byDataType/segmentations/ jan2011/hub). Libbrecht et al. developed GBR (Libbrecht et al. , 2015)  to decompose the genome into five different types of domains. GBR annotations on eight different cell lines are obtained from https://noble. gs. washington. edu/proj/gbr/. To compare with these alternative annotations, we again apply the method described above for comparing chromatin states with RT states.
Transcriptome analysis on protein -coding gene Expression level measured by the Reads Per Kilobase of transcript per Million mapped reads (RPKM) of RNA sequencing data of proteincoding genes of 17 cell lines were downloaded from REMC (http:// www. roadmapepigenomics. org/)  (Bernstein et al. , 2010). The 17 cell types include the nine ENCODE cell types used in the chromHMM analysis plus additional eight cell types overlapping with the 42 cell lines used to construct the RT states: H1 BMP4 derived trophoblast, H1 derived mesenchymal stem, H1 derived neuronal progenitor, hESC derived CD184 endoderm, hESC derived CD56 ectoderm, hESC derived CD56 mesoderm and HELA. The RPKM value of the protein -coding genes that located inside regions corresponding to each of the 15 RT states were calculated, then their average expression levels in is computed in log scale as log10(RPKM average ).
Three-dimensional chromosomal organization analysis We use high -resolution Hi-C data (GEO accession number: GSE63525) obtained from human lymphoblastoid cells (GM12878) produced by Rao et al. (Rao et al. , 2014). We count the number of contact domains found in regions corresponding to each of the 15 RT states. The same method was used to analyze the LADs in fibroblasts. The LADs data were obtained from ENCODE  (Guelen et al. , 2008).
Alignment of RT values from 42 datasets along a chromosome shows the global preservation of RT during development and also some deviations among cell types (Fig. 2C ). As examples, we show some regions with early RT values in hESC cells but late in other cell types (the green rectangle in Fig. 2C ). The opposite pattern can be observed in the blue rectangle (Fig. 2C ). Such patterns illustrate the importance of statistical modeling of RT program to identify recurrent and biologically meaningful combinatorial patterns of RT values.
Discussion Genome-wide datasets constitute a valuable tool for genome annotation of distinct chromatin functional properties, especially the   S5 ). non-coding part of the genome. Histone modification, DNA methylation, chromatin accessibility and disease association have all been used to provide functional annotation  (Boyle et al. , 2012; Chen and Qin, 2016; Dunham et al. , 2012; Fu et al. , 2014; Ward and Kellis, 2012, 2016). Since RT is a fundamental BP, closely related to chromatin organization and transcription regulation dynamics during cell fate commitment, we believe RT can supplement the current functional annotation of the genome. Our RT state concept is inspired by the chromHMM work of Ernst and Kellis  (Ernst and Kellis, 2010, 2012; Ernst et al. , 2011). They define chromatin states by applying HMM on ChIP-seq data of histone modifications, together with histone variants (H2A.