dataesr/scientific-paragraphs-categorization huggingface dataset:
Distributional semantic models (DSMs) are often evaluated on artificial similarity datasets containing single words or fully compositional phrases. We present a large-scale multilingual evaluation of DSMs for predicting the degree of semantic compositionality of nominal compounds on 4 datasets for English and French. We build a total of 816 DSMs and perform 2,856 evaluations using word2vec, GloVe, and PPMI-based models. In addition to the DSMs, we compare the impact of different parameters, such as level of corpus preprocessing, context window size and number of dimensions. The results obtained have a high correlation with human judgments, being comparable to or outperforming the state of the art for some datasets (Spearman's ρ=. 82 for the Reddy dataset).
, 2013a). Larger units can often be assumed to have their meanings derived from their parts according to the language's grammar, but this is not always the case  (Sag et al. , 2002). Many multiword units are associated with idiomatic interpretations, unrelated to the meaning of the component words (e. g. silver bullet, eager beaver). Precision-oriented NLP applications need to be able to identify partly-compositional and idiomatic cases and ensure meaning preservation during processing. Compositionality identification is a first step towards complete semantic interpretation in tasks such as machine translation (to translate non-compositional compounds as a unit), word sense disambiguation (to avoid assigning a sense to parts of non-compositional compounds), and semantic parsing (to identify complex predicates and their arguments).
Related Work We define nominal compounds as conventional noun phrases composed by two or more words, such as science fiction  (Nakov, 2013). In English, they are often expressed as noun compounds but their syntactic realization may vary for different languages. For instance, one of the equivalent forms in French involves a denominal adjective used as modifier (e. g. cell death and the corresponding mort cellulaire). 1 In this paper, we focus on 2-word nominal compounds involving modifiers that are nouns (e. g. word embedding) or adjectives (e. g. hard time). Semantically, nominal compounds may display a wide range of idiomaticity, from compositional cases like access road to idiomatic or noncompositional cases like gravy train, whose meaning is unrelated to its parts. 2 Even when there is a level of compositionality in the compound, the contribution of each word may vary considerably, independently from its status as a syntactic head or modifier, as cash in cash cow versus tears in crocodile tears.
Indeed, various annotation scales have been proposed as means to collect human judgments about compositionality. Particularly for nominal compounds,  Reddy et al. (2011)  used a 6-point scale to collect judgments on the literal or figurative use of nominal compounds and its components in English. Similar judgments have also been collected for 244 German compounds, for which an average of 30 judgments on a scale from 1 to 7 were gathered through crowdsourcing  (Roller et al. , 2013). An alternative to multi-point scales is the binary judgment adopted by  Farahmand et al. (2015) , for a dataset of English nominal compounds. There has been much interest in creating semantic representations of larger units, such as phrases  (Mikolov et al. , 2013b) , sentences and documents  (Le and Mikolov, 2014) , and in examining whether it is possible to accurately derive the semantics of a compound or multiword expression from its parts  (McCarthy et al.
, 2013c). However, the particular effects may be heterogeneous and depend on the task and model  (Lapesa and Evert, 2014). In this paper, we examine the impact of both corpus and context parameters for a variety of models, for the task of nominal compound compositionality prediction in English and French. For the choice of particular DSM, contradictory results have been published showing the superiority of neural models  (Baroni et al. , 2014)  and of more traditional but carefully designed models  (Levy et al. , 2015). The former were also reported as a better fit to behavioral data on semantic prim-ing tasks  (Mandera et al. , 2016). Moreover, these evaluations are often performed on single-word similarity tasks  (Freitag et al. , 2005; Camacho-Collados et al. , 2015)  and little has been said about the use of word embeddings for the compositionality prediction of multiword expressions.
Two notable exceptions are the recent works of  Salehi et al. (2015)  and  Yazdani et al. (2015). Salehi et al. (2015)  show that word embeddings are more accurate in predicting compositionality than a simplistic count-based DSM. Yazdani et al. (2015)  focus on the composition function, using a lightly supervised neural network to learn the best combination strategy for individual word vectors. In order to consolidate previous punctual results, we present a large-scale and systematic evaluation, comparing DSMs and their parameters, on several compositionality datasets.
Compositionality Datasets For evaluation, we use nominal compound compositionality datasets for English  (Reddy, Reddy++ and Farahmand)  and for French (FR-comp). They provide annotations as to whether a given compound is more idiomatic or more compositional. Reddy contains compositionality judgments for 90 compounds and their individual word components, in a scale of literality from 0 (idiomatic) to 5 (literal), collected with Mechanical Turk  (Reddy et al. , 2011). For each compound, compositionality scores are averaged over its annotators. Compounds included in the dataset were selected to balance frequency range and degree of compositionality (low, middle and high). We use only the global compositionality score, ignoring individual word judgments. With a few exceptions (e. g. sacred cow), most compounds are formed exclusively by nouns.
Reddy++ is a new resource created for this evaluation  (Ramisch et al. , 2016). It extends the Reddy set with an additional 90 English nominal compounds, in a total of 180 entries. Scores also range from 0 to 5 and were collected through Mechanical Turk and averaged over the annotators. The extra 90 entries include some adjective-noun compounds and are balanced with respect to frequency and compositionality. We focus our evaluation on this combined dataset, since it includes Reddy. However, to allow comparison with state of the art, we also report results individually for Reddy. Farahmand contains 1042 English compounds extracted from Wikipedia with binary noncompositionality judgments by four experts  (Farahmand et al. , 2015). We consider a compound as non-compositional if at least two judges agree that it is non-compositional, following  Yaz-dani et al.
(2015). In our evaluations, we use the sum of all judgments in order to have a single numeral compositionality score, ranging from 0 (compositional) to 4 (idiomatic). FR-comp is also a new resource created for this evaluation  (Ramisch et al. , 2016). It contains 180 adjective-noun and noun-adjective compounds in French, such as belle-mère (mother-in-law, lit. beautiful-mother) and carte bleue (credit card, lit. blue card). This dataset was constructed in the same manner as the extension to Reddy, that is, using crowdsourcing and average numerical scores. Special care was taken to guarantee that annotators were native speakers by asking them to provide paraphrases along with compositionality scores. The new datasets Reddy++ and FR-comp are similar to Reddy. For instance, the average standard deviation of compound scores given by different annotators is σ = 1.
We use the default minimum word count threshold of 5. glove We use the count-based DSM of Pennington et al. (  2014 ), which implements a factorization of the co-occurrence count matrix. The configurations are the default ones, except for the following: internal cutoff parameter x max = 75; builds co-occurrence matrix in 15 iterations. Due to the large vocabulary size, we use a minimum word count threshold of 5 for lemma-based models, 15 for surface and 20 for surface +. For each DSM, we evaluate the influence of a set of parameters. By varying the values of these parameters, we build a total of 408 models per language. The parameters are: • WORDFORM: Refers to one of the four variants of each corpus: surface + , surface, lemma, and lemmaPOS. • WINDOWSIZE: Indicates within how many words to the left/right we are searching for target-context co-occurrence pairs.
Evaluation Setup We evaluate the compositionality models and their parameters on the datasets described in Section 3. 2. For Reddy, Reddy++ and FR-comp, we report Spearman's ρ correlation between the ranking provided by humans and those calculated from the models. We follow  Yazdani et al. (2015)  and report the best F1 score (BF1) obtained for the Farahmand dataset, by calculating the F1 score for the top k compounds classified as positive (noncompositional), for all possible values of k. Given the high number of experiments we performed, we report the best performance of each model type. For instance, the performances reported for w2v-cbow using different values of WINDOWSIZE are the best configurations across all possible values of other parameters such as DIMENSION and WORDFORM. This avoids reporting local maxima that can arise if one fixes all other parameters when evaluating a given one  (Lapesa and Evert, 2014).
For Reddy++ and Farahmand, we distinguish between strict evaluation, reported in the form of wider bars in the figures, and loose evaluation, shown as narrow blue bars in the figures. Strict evaluation corresponds to the performance of the model only on those compounds that have a vector representation in all underlying DSMs, 175 out of 180 for Reddy++ and 913 out of 1042 for Farahmand. Loose evaluation considers the full dataset, using a fallback strategy for the imputation of missing values, assigning the average compositionality score to absent compounds  (Salehi et al. , 2015). This is particularly important for Farahmand, which contains more rare compounds such as universe human and mankind instruction so that 129 compounds are missing in the corpus. Only strict evaluation is reported for FR-comp, as all compounds are frequent enough in FRWaC.
Reddy++ and Reddy Datasets Figure  1  summarizes the results for Reddy++ dataset. 10 Overall, w2v-cbow (ρ = 0. 73), w2v-sg (ρ = 0. 73), PPMI-SVD (ρ = 0. 72) and PPMIthresh (ρ = 0. 71) obtain similar results. In spite of this, except for the two best w2v models, all differences were deemed statistically significant (Wilcoxon rank correlation test, p < 0. 05). Figure  1 (b) shows the influence of the degree of corpus preprocessing (shown as WORDFORM in these figures). The results are heterogeneous, as the best w2v models seem to profit from the presence of stopwords, unlike the other models for which more preprocessing (lemma and lemma-POS) leads to better results. One exception is PPMI-SVD for which the use of lemmaPOS drastically reduces performance. 11 or WINDOWSIZE, Figure  1 (c), although increasing context size seems to help DSMs (at least up to 4), for the best w2v models, a better result is obtained with limited context of 1 word left/right.
Probably the interaction between the subsampling strategy and randomized window size explains why increasing this value does not improve the  The results for context vector dimensionality, Figure  1(d) , show, as expected, that the best results are obtained with larger dimensions (DIMENSION=750) for all models, except for glove, which displays very similar results independently of the number of dimensions. Examining the Reddy dataset alone, the same trends for all parameters were found, but with higher results. The overall best performances on Reddy were quite similar: w2v-cbow (ρ = 0. 82), w2v-sg (ρ = 0. 81), PPMI-SVD (ρ = 0. 80) and PPMI-thresh (ρ = 0. 79), and the differences are significant except for the two best w2v models. The 90 compounds added to Reddy++ seem to be more difficult to assess than the original ones, probably because they include many adjectives, which have been found harder to judge for compositionality than nouns  (Ramisch et al.
Farahmand Dataset Figure  2 (a) shows the overall best model for the Farahmand dataset. PPMI-SVD reached a BF1 score of 0. 52, with DIMENSION=750, WINDOW-SIZE=4, using lemma, and both w2v (BF1=0. 51) obtain comparable results with similar configurations. These results show a marked difference between the loose (the narrower bars in the figures) and the strict evaluation (wider bars). The former uses a fallback strategy for the imputation of missing values that does not accurately reflect how the compositionality scores vary. Indeed, we observed that compounds that do not appear very often in our corpora tend to be non-compositional, whereas most of the compound occurrences are compositional, increasing average compositionality. For instance, the 10 most compositional compounds in Reddy++ occur an average of 26551  times in the UKWaC vs 1096 times for the 10 least compositional ones.
Spearman rank correlation between frequency and compositionality in Reddy++ is ρ = 0. 43. 12 In short, even if a fallback strategy is adopted as the means to obtain a lowerbound for performance, it may be unrelated to the real performance for the missing compounds. For most models, corpus preprocessing resulted in better scores, with WORDFORM=lemma outperforming all other forms of preprocessing, especially for French. Concatenating lemmas and POS tags does not seem to help, probably due to decreasing word frequencies without substantial gain in informativeness (Figure  2(b) ). The impact of WINDOWSIZE has a similar trend to the one found for the Reddy++ and Reddy datasets (Figure  2 (c)). That is, the larger window was preferred by most models, but the average difference between the best and the worst size for each DSM is only 0.
FR-comp Dataset Globally, for the FR-comp dataset, PPMI-thresh (ρ = 0. 70) outperforms glove (ρ = 0. 68) and w2v (ρ = 0. 66), as can be seen in Figure  3(a). 13 or morphologically rich languages like French, Figure  3 (b) indicates that working on lemmatized data often yields better results than working on surface forms. Lemmas conflate the frequencies for all the many morphologically inflected variants which would otherwise be dispersed in different surface forms. Therefore, it is not surprising that the best results concerning WORDFORM are achieved by lemma. These results differ from English, where a corpus without any preprocess-.
Discussion When comparing DIMENSION across languages and datasets, larger values often bring better performance. Likewise, the lemma is usually the better WORDFORM. The recommended WINDOW-SIZE depends on the model and language, but for the best models in all datasets, a window of 1 outperforms the others. This may be a consequence of the linear decay context weighting process, which assigns higher weights to closer words as window size increases. As an overall conclusion, in combination with a large dimension and a small 14  For w2v, the same parameters used for English were adopted also for French. As a sanity check, we tested a range of negative sampling values  [5, 15, 25, 35, 50] , as well as subsampling rates for powers of 10 in [10 -3 to 10 -7 ]. Variations in ρ are minor and do not show any clear trend.
Model & Parameters Result  Reddy et al. (2011). 71 Salehi et al. (2014). 74 Salehi et al. (2015). 80 Best w2v (sg, WF=surface, D=750, W=1). 82 (. 80) Best PPMI (thresh, WF=surface, D=750, W=8). 80 (. 80) Best glove (WF=lemmapos, D=250, W=8). 76 (. 76) thresh, and this was also observed for the other datasets. In short, both types of models can obtain good results. While PPMI-thresh is a simple, fast and inexpensive model to build, w2v has a free and push-button implementation, and requires less hyper-parameter tuning, as is it seems more robust to parameter variation. More generally, the best results obtained for Reddy and Farahmand are comparable and even outperform the state of the art, as shown in Tables  1  and 2 , when strict evaluation is adopted (that is, when not using a fallback strategy for missing compounds).
Conclusions In this paper we presented a multilingual, largescale evaluation of DSMs for compound compositionality prediction. We have built 816 DSMs and performed 2,856 evaluations, examining the impact of corpus and context parameters, namely the level of corpus preprocessing, the context window size and the number of dimensions. Evaluation on 3 English datasets and a French one revealed that a large dimension is consistently better, and corpus preprocessing is usually beneficial. The choice of window size varies according to language and dataset, but a small window can often provide a good performance. The DSMs w2v and PPMI alternated in providing the best results. Moreover, the results obtained were comparable and even outperformed the state-of-the-art. As future work, we plan to examine the use of a voting scheme for combining the output of complementary DSMs.
Acknowledgments This work has been partly funded by projects  PARSEME  (Cost Action  IC1207 ),  PARSEME-FR  ( ANR-14-CERA-0001 ),  AIM-WEST  ( FAPERGS-INRIA 1706-2551/13-7 ),  CNPq   482520/2012-4 ,  312114/2015-0 , " Simplificacão Textual de Expressões Complexas ", sponsored by  Samsung Eletrônica da Amazônia Ltda. under the terms of Brazilian federal law No. 8. 248/91.
Post-conference papers were solicited from participants at any of the events held in 2015. All submissions were peer-reviewed in the Brainhack Proceedings Github repository [https://github. com/ Brainhack-Proceedings-2015] using an innovative open-review process. In keeping with the culture of Brainhack, we took advantage of the open platform provided by Github [http://github. com] to encourage a productive dialogue between authors and reviewers. This first issue of Brainhack Proceedings includes 23 project paperspresenting an overview of the broad range of interests, content, and achievements that converged at Brainhack events this past year. With at least four international events scheduled for 2016 [http:// events. brainhack. org], we hope that this publication format will provide an ongoing record of the growth within our community.
Introduction The past several decades have seen an explosive growth in the number of published neuroimaging studies. In concert, the demand for freely available and openly accessible 'study data' , that would facilitate future reanalysis, meta-analysis, hypothesis testing and repurposing has also soared. Here we report on developments made to Brainspell  [1]  one of several web-based initiatives (e. g. BrainMap  [2] , NeuroVault  [3] , Neurosynth  [4] ) that allow individuals to search through and organize massive numbers of neuroimaging studies and results in meaningful ways. Distinct from other databases, Brainspell [http://brainspell. org] is the first web-based initiative to allow users to manually annotate and curate machine-parsed data, as well as manually extend the database via its crowdsourcing user interface.
Meaningful organization of data Potential shortcomings of neuroimaging databases employing automatic coordinate data extraction is their inability to segregate (i) multiple contrasts (e. g. within group, inter-group), and (ii) significant versus nonsignificant coordinates, when present in a single table. The following options were added to facilitate non-ambiguous data organization (see Fig. 1 ): Addition of a Split link to each table. Fine-tuning the Split link enhancement to allow more than ten splits.
Conclusion We performed ten enhancements to Brainspell and provided instructions of use in Brainspell's wiki. We tested these enhancements on Safari, Firefox and Chrome. Moreover, 25 articles were manually added to Brainspell as part of our extended beta testing phase. Our goal with these enhancements was to extend the functionality, and ease of use of Brainspell for curating machine-parsed neuroimaging data from a wide database of studies. During January 15 to February 5, 2016 alone, Brainspell was used in 282 sessions by 133 users who watched 1421 pages. Moreover, Brainspell was forked to "BIDS-collaborative/Brainspell" which itself was forked by approximately 10 data-science students to extend the platform.
Introduction The goal of this project is to improve accessibility of open datasets by curating them. "NiData" aims to provide a common interface for documentation, downloads, and examples to all open neuroimaging datasets, making data usable for experts and non-experts alike. Approach Open datasets promise to allow more thorough analysis of hard-tocollect data and re-analysis using state-of-the-art analysis methods. However, open datasets are not truly open unless they are easy to find, simple to access, and have sufficient documentation for use. Currently, publicly available data in neuroscience are scattered across a number of websites and databases, without a common data format, no common method for data access, and varying levels of documentation. Datasets are being uploaded to public databases through a number of initiatives, including OpenFMRI  [1]  and NITRC  [2]  In addition, there are funded efforts for collecting data explicitly for the.
© The Author(s). 2016 Open Access This article is distributed under the terms of the Creative Commons Attribution 4. 0 International License (http://creativecommons. org/licenses/by/4. 0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons. org/publicdomain/zero/1. 0/) applies to the data made available in this article, unless otherwise stated.
pasteur. fr/labex/milieu-interieur). Direct comparison with conventional RNA extraction protocols indicated excellent correlation in gene expression counts between the two extraction methods (Spearman's rank-order correlation, r s > 0. 99, Figure  S1B ). Expression data were normalized with nSolver Analysis Software (NanoString), using four housekeeping genes: RPL19, TBP, POLR2A, and HPRT (Figures  S1C-S1F ). These four housekeeping genes were selected following the application of the geNorm method  (Vandesompele et al. , 2002) , an established algorithm for identifying stable housekeeping genes. The selection of these genes is supported by their strong correlations preand post-stimulation (r s > 0. 9) across the 25 donors, in contrast with those housekeeping genes that were discarded (r s < 0.
Comparison between the scores that Whole-blood stimulation was performed on 25 healthy donors using TruCulture systems pre-loaded with IFN-b (pale green), IFN-g (gray), IL-1b (purple), and TNF-a (turquoise). Principle component analysis (PCA) was used to project mRNA expression data from 572 genes employing Qlucore Omics Explorer v3. 1. Prior to applying PCA, values for each of the 572 mRNA were log transformed, centered to a mean value of zero across each donor, and scaled to unit variance. The four cytokine stimuli are indicated by the colored circles and the vector position of each of the 25 donors is represented. (A) Left: PC1 versus PC2. Right: PC2 versus PC3. The percentage of variance captured by each PC is indicated. (B) PCA on filtered gene expression data; first for differential gene expression (paired t test comparing each cytokine with null and a q value cut-off of 10 À3 ); followed by the classification of samples using linear support vector machine (SVM) approaches, and genes ranked according to a paired t test, yielding a union gene set of 44 genes.