dataesr/scientific-paragraphs-categorization huggingface dataset:
Selecting Content To create data units, we adapted the procedure outlined by Perez-Beltrachini et al. (  2016 ) and sketched in Figure  2. This method can be summarised as follows. First, DBPedia category graphs are extracted from DBPedia by retrieving up to 500 entity graphs for entities of the same category. 7 For example, we build a category graph for the Astronaut category by collecting, graphs of depth five for 500 entities of types astronaut. Next, category graphs are used to learn bi-gram models of DBPedia properties which specify the probability of two properties co-occuring together. Three types of bi-gram models are extracted from category graphs using the SRILM toolkit (Stolcke,  2002): one model (S-Model) for bigrams occurring in sibling triples (triples with a shared subject); one model (C-Model) for bigrams occurring in chained triples (the object of one triple is the subject of the other); and one model (M-Model) which is a linear interpolation of the sibling and the chain model.
An input is a set of triples produced by the content selection module. The number of input (#Inputs) is thus the number of distinct sets of triples produced by this module. In contrast, input patterns are inputs where subject and object have been abstracted over. That is, the number of input patterns (#I. Patterns) is the number of distinct sets of properties present in the set of inputs. The number of properties (#Properties) is the number of distinct RDF properties occurring in the dataset. Similarly, the number of entities (#Entities) is the number of distinct RDF subjects and objects occurring in each given dataset.
Associating Content with Text We associate data with text using the Crowdflower platform. 8 We do this in four main steps as follows. 1. Clarifying properties. One difficulty when collecting texts verbalising sets of DBPedia triples is that the meaning of DBPedia properties may be unclear. We therefore first manually clarified for each category being worked on, those properties which have no obvious lexicalisations (e. g. , crew1up was replaced by commander).
Getting verbalisations for single triples. Next, we collected three verbalisations for data units of size one, i. e. single triples consisting of a subject, a property and an object. For each such input, crowdworkers were asked to produce a sentence verbalising its content. We used both a priori automatic checks to prevent spamming and a posteriori manual checks to remove incorrect verbalisations. We also monitored crowdworkers as they entered their input and banned those who tried to circumvent our instructions and validators. The automatic checks comprise 12 custom javascript validators implemented in the CrowdFlower platform to block contributor answers which fail to meet requirements such as the minimal time a contributor should stay on page, the minimal length of the text produced, the minimal match of tokens between a triple and its verbalisation and various format restrictions used to detect invalid input.
Comparing Benchmarks We now compare a dataset created using our dataset creation framework (henceforth WEBNLG) with the dataset of Wen et al. (2016)   9  (henceforth, RNNLG). Example 3 shows a sample data-text pair taken from the RNNLG dataset. (3) Dialog Moves recommend(name=caerus 33;type=television; screensizerange=medium;family=t5;hasusbport=true) The caerus 33 is a medium television in the T5 family that's USB-enabled. As should be clear from the discussion in Section 2 and 3, both datasets are similar in that, in both cases, data is built from ontological information and text is crowdsourced from the data. An important difference between the two datasets is that, while the RNNLG data was constructed by enumerating possible combinations of dialog act types and attribute-value pairs, the WEBNLG data is created using a sophisticated content selection procedure geared at producing sets of data 9 https://github.
com/shawnwun/RNNLG units that are relevant for a given ontological category and that are varied in terms of size, shape and content. We now investigate the impact of this difference on the two datasets (WEBNLG and RNNLG). To assess the degree to which both datasets support the generation of linguistically varied text requiring complex micro-planning operations, we examine a number of data and text related metrics. We also compare the results of an out-of-the-box sequence-to-sequence model as a way to estimate the complexity of the learning task induced by each dataset.
Data Comparison Terminology. The attributes in the RNNLG dataset can be viewed as binary relations between the object talked about (a restaurant, a laptop, a TV or a hotel) and a value. Similarly, in the WEBNLGdataset, DBpedia RDF properties relate a subject entity to an object which can be either an entity or a datatype value. In what follows, we refer to both as attributes. Table  3  shows several statistics which indicate that, while the RNNLG dataset is larger than WEBNLG, WEBNLG is much more diverse in terms of attributes, input patterns and input shapes.
Number of attributes. As illustrated in Example (4) below, different attributes can be lexicalised using different parts of speech. A dataset with a larger number of attributes is therefore more likely to induce texts with greater syntactic variety. (4) Verb: X title Y / X served as Y Relational noun: X nationality Y / X's nationality is Y Preposition: X country Y / X is in Y Adjective: X nationality USA / X is American As shown in Table  3 , WEBNLG has a more diverse attribute set than RNNLG both in absolute (172 attributes in WEBNLG against 108 in RNNLG) and in relative terms (RNNLG is a little more than twice as large as WEBNLG). Number of input patterns. Since attributes may give rise to lexicalisation with different parts of speech, the sets of attributes present in an input (input pattern) 10  indirectly determine the syntactic realisation of the corresponding text.
Hence a higher number of input patterns will favour a higher number of syntactic realisations. This is exemplified in Example (  5 ) where two inputs with the same number of attributes give rise to texts with different syntactic forms. While in Example (5a), the attribute set {country, location, start-Date} is realised by a passive (is located), an apposition (Australia) and a deverbal nominal (its construction), in Example (5b), the attribute set {almaMater, birthPlace, selection} induced a passive (was born) and two VP coordinations (graduated and joined). ( Again, despite the much larger size of the RNNLG dataset, the number of input patterns in both datasets is almost the same. That is, the relative variety in input patterns is higher in WEBNLG.
Number of input / Number of input patterns. The ratio between number of inputs and the number of input patterns has an important impact both in terms of linguistic diversity and in terms of learning complexity. A large ratio indicates a "repetitive dataset" where the same pattern is instantiated a high number of times. While this facilitates learning, this also reduces linguistic coverage (less combinations of structures can be learned) and may induce over-fitting. Note that because datasets are typically delexicalised when training NLG models (cf. e. g. , Wen et al. 2015 and  Lampouras and Vlachos 2016), at training time, different instantiations of the same input pattern reduce to identical input. The two datasets markedly differ on this ratio which is five times lower in WEBNLG. While in WEBNLG, the same pattern is instantiated in average 2.
40 times, it is instantiated 10. 31 times in average in RNNLG. From a learning perspective, this means that the RNNLG dataset facilitates learning but also makes it harder to assess how well systems trained on it can generalise to handle unseen input. Input shape. As mentioned in Section 3, in the RNNLG dataset, all inputs can be viewed as trees of depth one while in the WEBNLG dataset, input may have various shapes. As a result, RNNLG texts will be restricted to syntactic forms which permit expressing such multiple predications of the same entity e. g. , subject relative clause, VP and sentence coordination etc. In contrast, the trees extracted by the WEBNLG content selection procedure may be of depth five and therefore allow for further syntactic constructs such as object relative clause and passive participles (cf.
Text Comparison Table  4  gives some statistics about the texts contained in each dataset. As illustrated by the contrast between Examples (  6 ) and (7) above, text length (number of tokens per text) and the number of sentences per text are strong indicators of the complexity of the generation task. We use the Stanford Part-Of-Speech  Tagger and Parser version 3. 5. 2 (dated 2015-04-20, Manning et al. 2014) to tokenize and to perform sentence segmentation on text. As shown in Table  4 , WEBNLG's texts are longer both in terms of tokens and in terms of number of sentences per text. Another difference between the two datasets is that WEBNLG contains a higher number of text per input thereby providing a better basis for learning paraphrases. The size and the content of the vocabulary is another important factor in ensuring the learning of wide coverage generators.
Neural Generation Richer and more varied datasets are harder to learn from. As a proof-of-concept study of the comparative difficulty of the two datasets with respect to machine learning, we compare the performance of a sequence-to-sequence model for generation on both datasets. We use the multi-layered sequence-to-sequence model with attention mechanism described in (Vinyals et al. , 2015). 12 The model was trained with 3-layer LSTMs with 512 units each with a batch size of 64 and a learning rate of 0. 5. To allow for a fair comparison, we use a similar amount of data (13K data-text pairs) for both datasets. As RNNLG is bigger in size than WEBNLG, we constructed a balanced sample of RNNLG which included equal number of instances per category (tv, laptop, etc). We use a 3:1:1 ratio for training, developement and testing.
Conclusion We presented a framework for building NLG datato-text training corpora from existing knowledge bases. One feature of our framework is that datasets created using this framework can be used for training and testing KB verbalisers an in particular, verbalisers for RDF knowledge bases. Following the development of the semantic web, many large scale datasets are encoded in the RDF language (e. g. , MusicBrainz, FOAF, LinkedGeoData) and official institutions 13 increasingly publish their data in this format. In this context, our framework is useful both for creating training data from RDF KB verbalisers and to increase the number of datasets available for training and testing NLG. Another important feature of our framework is that it permits creating semantically and linguistically diverse datasets which should support the learning of lexically and syntactically, wide coverage micro-planners.
We applied our framework to DBpedia data and showed that although twice smaller than the largest corpora currently available for training data-to-text microplanners, the resulting dataset is more semantically and linguistically diverse. Despite the disparity in size, the number of attributes is comparable in the two datasets. The ratio between input and input patterns is five times lower in our dataset thereby making learning harder but also diminishing the risk of overfitting and providing for wider linguistic coverage. Conversely, the ratio of text per input is twice higher thereby providing better support for learning paraphrases. 13 See http://museum-api. pbworks. com for examples. We have recently released a first version of the WebNLG dataset in the context of a shared task on micro-planning 14.
This new dataset consists of 21,855 data/text pairs with a total of 8,372 distinct data input. The input describes entities belonging to 9 distinct DBpedia categories namely, Astronaut, University, Monument, Building, ComicsCharacter, Food, Airport, SportsTeam and WrittenWork. The WebNLG data is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4. 0 International and can be downloaded at http://talc1. loria. fr/ webnlg/stories/challenge. html. Recently, several sequence-to-sequence models have been proposed for generation. Our experiments suggest that these are not optimal when it comes to generate linguistically complex texts from rich data. More generally, they indicate that the data-to-text corpora built by our framework are challenging for such models. We hope that the WEBNLG dataset which we have made available for the WEBNLG shared task will drive the deep learning community to take up this new challenge and work on the development of neural generators that can handle the generation of KB verbalisers and of linguistically rich texts.
We used the TensorFlow code available at https://github. com/tensorflow/models/ tree/master/tutorials/rnn/translate. Alternatively, we could have used the implementation of Wen et al. (2016)  which is optimised for generation. However the code is geared toward dialog acts and modifying it to handle RDF triples is non trivial. Since the comparison aims at examining the relative performance of the same neural network on the two datasets, we used the tensor flow implementation instead.
The development of new imaging and optogenetics techniques to study the dynamics of large neuronal circuits is generating datasets of unprecedented volume and complexity, demanding the development of appropriate analysis tools. We present a comprehensive computational workflow for the analysis of neuronal population calcium dynamics. The toolbox includes newly developed algorithms and interactive tools for image pre-processing and segmentation, estimation of significant single-neuron single-trial signals, mapping event-related neuronal responses, detection of activity-correlated neuronal clusters, exploration of population dynamics, and analysis of clusters' features against surrogate control datasets. The modules are integrated in a modular and versatile processing pipeline, adaptable to different needs.
Introduction Brain function relies on the interaction of large neuronal populations. Anomalies of these complex neuronal circuits are associated with diverse brain disorders  [1]. Therefore, to understand brain function both in health and disease, it is necessary to explore the activity dynamics of neuronal networks. Recent advances in optical methods and optogenetics provide unprecedented possibilities for functional imaging of large neuronal populations  [2] [3] [4] [5] , and even whole brains  [6, 7] , with high spatial (e. g. , single-neuron) resolution. These imaging datasets can be analyzed to reveal the responses of spatially distributed neuronal populations to sensory, motor, or task variables  [5, [8] [9] [10] [11]. In contrast to the important experimental advances in imaging techniques, computational analysis tools are still incipient, consisting of processing pipelines that widely vary across labs, leading to poor standardization  [12, 13].
Few integrated software packages are currently available, and they all focus on different aspects of the initial data pre-processing stages of the analysis workflow  [14] [15] [16]. Nevertheless, the study of large neuronal circuits imposes new challenges to the processing and analysis of the complex, high-dimensional datasets typically acquired. In order to make sense of neuronal population dynamics, a popular approach used is based on statistical methods of dimensionality reduction  [17]. These methods project the high-dimensional data into a low-dimensional space that preserves or reveals underlying features of the data. Remarkably, they have been applied to expose organizing principles of neuronal networks dynamics  [18] [19] [20]  and to detect neuronal assemblies  [9, [21] [22] [23].
For a recent study, we developed an algorithm to characterize the spatio-temporal structure of neuronal activity patterns observed in large neuronal populations imaged using two-photon microscopy  [9]. The framework applies dimensionality reduction and clustering techniques for the analysis of calcium imaging datasets, outperforming other traditional algorithms in the effective detection of neuronal assemblies embedded within neuronal networks  [9]. This method is able to detect non-exclusive assemblies that are engaged and disengaged on a moment-to-moment basis, compatible with the distributed and dynamic nature of brain processing  [24]. Here, we present a computational toolbox that integrates this method for detecting neuronal assemblies into a complete data processing pipeline designed for the comprehensive analysis of fluorescence imaging data, from the raw images to interpretable results on neuronal population dynamic.
It consists of modules for video pre-processing, morphological image segmentation into regions of interest (ROIs) corresponding to single neurons, extraction of fluorescence signals, analysis of ROI responses to stimulus and/or behavioral variables, detection of assemblies of ROIs, exploratory analysis of network dynamics and the automatic generation of surrogate shuffled datasets to act as controls for statistical purposes. Typically, the full protocol can be completed on a workstation computer in ~1 hour, depending on the size of the dataset. The different processing algorithms that we now implement in this toolbox have been previously used in studies for the analysis of in vivo imaging data from zebrafish larvae expressing genetically encoded calcium indicators (GCaMPs)  [7, 9, 10, 25, 26].
Here, we demonstrate that our integrated toolbox can be successfully applied for the analysis of the fluorescent dynamics of different calcium reporters, in different animal models and brain regions, for single-plane and multi-plane volumetric imaging, and for different imaging approaches, including largefield two-photon microscopy and single and two-photon light-sheet imaging. Moreover, we show that the module for detecting assemblies is remarkably efficient in finding biologically meaningful assemblies in datasets of whole-brain light-sheet imaging in zebrafish larvae, consisting of more than 40,000 ROIs. These examples demonstrate that the analysis framework implemented in this toolbox can effectively shed light on the neuronal interactions that underlie brain computations.
Overview Analytical demands vary greatly depending on the scientific questions and the nature of the datasets. Therefore, the pipeline is organized in flexible sub-modules that can be bypassed, replaced or used in a stand-alone manner, by allowing the user to import and integrate data and/or results from other preferred methods at different critical points along the pipeline. We provide the source Matlab code for the toolbox, along with a detailed tutorial  [27]  and a raw imaging dataset for a hands-on case study, to guide users in the utilization of the toolbox and demonstrate its capabilities. The workflow of the toolbox includes five main modules (Fig  1 ). The first pre-processing module contains sub-modules for: 1) smooth registration; 2) automatic detection and interactive manual curation of motion artifacts; 3) morphological single-neuron ROIs; 4) automatic detection of significant fluorescence events associated with neuronal calcium transients.
The PCA-promax implements a fully automated method that searches for ROIs with correlated calcium dynamics. Since a given ROI could belong to more than one functional group (assemblies), the PCA-promax is tailored to define non-exclusive ROI assemblies (i. e. , it allows for potential overlap between the detected clusters), in contrast to classic k-means and hierarchical clustering. The procedure relies on previously proposed techniques  [23, 34]. Briefly, it consists of two processing steps. First, it z-scores the activity of each ROI and reduces the dimensionality of the complete z-scored dataset of ROI activities through principal component analysis (PCA). The initial z-scoring homogenizes the variance across ROIs, allowing PCA to reveal the global structure of ROI activity covariance. To define the assemblies, it then uses a second algorithm to partition this space of reduced dimensionality, by means of non-orthogonal factor rotation, promax  [35].
This latter step extends the simpler PCA-clustering method  [34]  to non-exclusive assemblies. Dataset dimensionality reduction is obtained by only keeping principal components (PCs) with eigenvalues greater than λ max , a theoretical lower bound to the eigenvalues of informative PCs given by the Marčenko-Pastur distribution  [34, 36]  (Eq 1) l ¼ ð1 þ ffiffiffiffiffiffiffiffiffi ffi N=T p Þ 2 þ N 2=3 ð1Þ where N and T are the number of ROIs and imaging frames, respectively. Since we are looking for non-exclusive assemblies, we relax the PCA orthogonality condition  [23]. Therefore, to delineate the assemblies, the algorithm works on a space of obliquely rotated components (promax), that sparsely concentrates the PC loadings along non-orthogonal rotated PCs. Hence, after standardizing the loadings on rotated PCs by means of a z-score, a given ROI is included in a particular assembly if its z-scored loading on that rotated PC exceeds a threshold value.
), and compare it to the pooling of the original assemblies. Certainly, the choice of control datasets that serve as null models will depend on the particular scientific question being statistically tested. We focused on two particular kinds of control datasets. We preserve the imaged fluorescence dynamics of all ROIs, and regroup ROIs according to: i) surrogate shuffled assemblies (where the ROIs of any given original assembly are randomized); ii) surrogate assemblies that preserve the original assemblies' spatial features (i. e. , conserving the distribution of the relative pair-wise physical distances between ROIs in the original assemblies).
Results Given the large scope of the pipeline, here we focus on the typically obtained results of the most relevant processing stages. The processing of all datasets was performed according to the tutorial provided. As described in the tutorial, the initial data pre-processing consisted in registering in-plane x-y drifts of the imaged optical section, followed by the automatic detection of movement-related imaging artifacts. Then, the toolbox was used to automatic detection and interactive manual curation of morphological single-neuron ROIs, as explained next.
Segmentation in single-neuron ROIs As expected, the success of this module depends on the image's spatial resolution, the anatomy of the imaged region, and the cellular labeling by the fluorescent reporter. Here, we demonstrate that it can successfully process remarkably different datasets. We provide 5 examples of single-neuron ROI detection performance obtained with twophoton imaging: i) for an injected synthetic dye that labels the entire volume of the neurons (OGB-1 AM) in mouse visual cortex  [37].
For the light-sheet imaging datasets in Figs  2D  and 8 , σ was also estimated with the Gaussian model, but due to the large size of the datasets, significant transients were obtained with the faster Static threshold method. We quantified the toolbox performance in detecting neuronal activity from imaging data. For this, we applied the dynamic threshold method to a dataset  [28, 39]  consisting of simultaneous imaging and loose-seal cell-attached recordings in GCaMP6f expressing neurons (37 recordings from 11 different neurons; Fig  4 ; dataset cai-1 from crcns. org). Using this groundtruth data, we observed that the significant fluorescence transients inferred by the algorithm correlated well with the firing rate of the neurons (r = 0. 44±5; top panel in Fig  4B ). This is comparable to the best performance obtained by more complex spike-inference algorithms applied to the slower GCaMP6s signal (compared to that of GCaMP6f), albeit imaged in more challenging conditions  [40].