dataesr/scientific-paragraphs-categorization huggingface dataset:
The involvement of Andres Sandoval-Hernandez (one of the co-authors of this chapter) with ICCS began in 2010 when he started working at the Research and Analysis Unit of IEA's Data Processing Centre (DPC), in Hamburg, Germany. The activities at this unit consisted of carrying out analysis on IEA datasets across subject areas and offering training focused on secondary analyses of IEA's data. At that time, the data from ICCS 2009 was freshly released and many of the training activities focused on this study. These activities created opportunities for several individuals who are now part of the network of researchers on ICCS across Latin American countries. For example, Ernesto Treviño, Juan Carlos Castillo, Diego Carrasco, Silvia Diazgranados, and Daniel Miranda (the co-author of this chapter), participated in one or more of the workshops organized by IEA and currently participate or have participated in one or more of the multi-year research projects based on analyses of ICCS data.
de Evaluación y Desarrollo de Competencias Ciudadanas 2010). Results were also discussed in a round table with specialists in civics and citizenship education in the Dominican Republic. From these conversations, a book titled Civic and Citizenship Education in the Dominican Republic was published in 2012 (Pepén Peguero 2012). This book had the support of the Latin American Faculty of Social Sciences (Facultad Lationamericana de Ciencias Sociales) and the Program to Promote Educational Reform in Latin America (Programa de Promoción de la Reforma Educativa en América Latina). The intention of the publication was to disseminate the results, analyses, concepts, and implications of ICCS 2009 in regards to education and society (Pepén Peguero 2012). (Pérez 2018)the Dominican Institute of Evaluation and Research of Educational Quality (Instituto Dominicano de Evaluación e Investigación de la Calidad Educativa[IDEICE]) is analyzing the existing civic datasets from ICCS 2016.
The source of such a successful transfer is still largely unexplained. Pires et al. (2019)  hypothesize that these models learn shared multilingual representations during pretraining. Focusing on syntax,  Chi et al. (2020)  recently showed that the multilingual version of BERT (mBERT)  (Devlin et al. , 2019) , encodes linguistic properties in shared multilingual sub-spaces. Recently, Gonen et al. (2020)  suggest that mBERT learns a language encoding component and an abstract cross-lingual component. In this work, we are interested in understanding the mechanism that leads mBERT to perform zero-shot cross-lingual transfer. More specifically, we ask what parts of the model and what mechanisms support cross-lingual transfer? By combining behavioral and structural analyses  (Belinkov et al. , 2020) , we show that mBERT operates as the stacking of two modules: (1) A multilingual encoder, located in the lower part of the model, critical for cross-lingual transfer, is in charge of aligning multilingual representations; and (2) a task-specific, language-agnostic predictor which has little importance for cross-lingual transfer and is dedicated to performing the downstream task.
Experimental Setup Tasks, Datasets and Evaluation We consider three tasks covering both syntactic and semantic aspects of language: Part-Of-Speech Tagging (POS), dependency parsing, and Named-Entity Recognition (NER). For POS tagging and parsing we use the Universal Dependency (Nivre et al. , 2018) treebanks, and for NER, we use the WikiANN dataset  (Pan et al. , 2017). We evaluate our systems with the standard metrics per task; word-level accuracy for POS tagging, F1 for NER and labeled attachment score (LAS) for parsing. All the reported scores are computed on the test set of each dataset. We experiment with English, Russian and Arabic as source languages, and fourteen typologically diverse target languages, including Chinese, Czech, German and Hindi. The complete list can be found in the Appendix A.
1. 2. In SRC-TRG, SRC indicates the source language on which we fine-tune mBERT, and TRG the target language on which we evaluate it. SRC-X is the average across all 17 target language with X = SRC. Detailed results per target language are reported in tables 6, 7 and 8 in the Appendix. Coloring is computed based on how mBERT with RANDOM-INIT performs compared to the REF model. ≥ REF < REF ≤ -2 points ≤ -5 points The results of a model that is fine-tuned and evaluated on the same language are referred to as same-language and those evaluated on distinct languages are referred to as cross-language.
77 points on other languages (EN-X). Furthermore, we show that applying RANDOM-INIT to the upper layers does not harm samelanguage and cross-language performances (e. g. when training on parsing for English, the performance slightly decreases by 0. 09 points in the same-language while it increases by 1. 00 in the cross-language case). This suggests that the upper layers are task-specific and language-agnostic, since re-initializing them have minimal change on performance. We conclude that mBERT's upper layers do not contribute to cross-language transfer.
Cross-Lingual Similarity in mBERT The results from the previous sections suggest that the lower layers of the model are responsible for the cross lingual transfer, whereas the upper layers are language-agnostic. In this section, we assess the transfer by directly analyzing the intermediate representations and measuring the similarities of the hidden state representations between source and target languages. We compute the CKA metric (cf. §2. 2) between the source and the target representations for pretrained and fine-tuned models using parallel sentences from the PUD dataset  (Zeman et al. , 2017). In Figure  1 , we present the similarities between Russian and English with mBERT pretrained and fine-tuned on the three tasks. 4  The cross-lingual similarity between the representations constantly increases up to layer 5 for all the three tasks (reaching 78.
Conclusion The remarkable performance of multilingual languages models in zero-shot cross-lingual transfer is still not well understood. In this work, we combine a structural analysis of the similarities between hidden representation across languages with a novel behavioral analysis that randomlyinitialize the models' parameters to understand it. By combining those experiments on 17 languages and 3 tasks, we show that mBERT is constructed from: (1) a multilingual encoder in the lower layers, which aligns hidden representations across languages and is critical for cross-language transfer, and (2) a task-specific, language-agnostic predictor that has little effect to cross-language transfer, in the upper layers. Additionally, we demonstrate that hidden cross-lingual similarity strongly correlates with downstream cross-lingual performance suggesting that this alignment is at the root of these cross-lingual transfer abilities.
A. 1. 2 Data Data Sources We base our experiments on data originated from two sources. The Universal Dependency project  (McDonald et al. , 2013)  downloadable here https://lindat. mff. cuni. cz/ repository/xmlui/handle/11234/1-2988 and the WikiNER dataset  (Pan et al. , 2017). We also make use of the CoNLL-2003 shared task NER English dataset https://www. clips. uantwerpen. be/conll2003/ Languages For all our experiments, we use English, Russian and Arabic as source languages in addition to Chinese, Czech, Finish, French, Indonesian, Italian, Japanese, German, Hindi, Polish, Portuguese, Slovenian, Spanish, and Turkish as target languages. Fine-tuning Data For all the cross-lingual experiments, we use English, Russian and Arabic as source languages on which we fine-tune mBERT. For English, we take the English-EWT treebank for fine-tuning, for Russian the Russian-GSD treebank and for Arabic the Arabic-PADT treebank.
Domain Analysis Datasets We list here the datasets for completing our domain analysis experiment in Section 4. 1 reported in Table  2. To have a full control on the source domains, we use for fine-tuning the English Partut treebank for POS tagging and parsing  (Svizzera, 2014). It is a mix of legal, news and wikipedia text. For NER, we keep the WikiANN dataset  (Pan et al. , 2017). For the same-language and out-of-domain experiments, we use the English-EWT, English-Lines and English Lexnorm (van der Goot and van Noord, 2018) treebanks for Web Media data, Literature data and Noisy tweets respectively. For the cross-language French evaluation, we use the translation of the English test set, 6  as well as the French-GSD treebank. For NER, we take the CoNLL-2003 shared task English data as our out-of-domain evaluation extracted from the News domain.
Cross-Lingual Similarity Analysis For a given source language l and a target language l , we collect a 1000 pairs of aligned sentences from the UD-PUD treebanks  (Zeman et al. , 2017). For a given model and for each layer, we get a single sentence embedding by averaging token-level embeddings (after excluding special tokens). We then concatenate the 1000 sentence embedding vectors and get the matrices X l and X l. Based on these two matrices, the CKA between the language l and the language l is defined as: CKA(X l , X l ) = ||X T l X l || 2 F ||X T l X l || F ||X T l X l || F with ||. || F defining the Frobenius norm. We do so for each source-target language pairs using the representation of the pretrained mBERT model as well as for mBERT fine-tuned on each downstream task. In addition to the results presented in §4.
Acknowledgments We thank  Hila Gonen ,  Shauli Ravfogel  and  Ganesh Jawahar  for their careful review and insightful comments. We also thank the anonymous reviewers for their valuable suggestions. This work was partly funded by two  French National  funded projects granted to Inria and other partners by the  Agence Nationale de la Recherche , namely projects  PAR-SITI  ( ANR-16-CE33-0021 ) and  SoSweet  ( ANR-15-CE38-0011 ), as well as by the third author's chair in the  PRAIRIE institute  funded by the  French national agency ANR  as part of the " Investissements d'avenir" programme  under the reference  ANR-19-P3IA-0001. Yanai Elazar  is grateful to be partially supported by the  PBC fellowship  for outstanding Phd candidates in Data Science.
Robust statistical data modelling under potential model mis-specification often requires leaving the parametric world for the nonparametric. In the latter, parameters are infinite dimensional objects such as functions, probability distributions or infinite vectors. In the Bayesian nonparametric approach, prior distributions are designed for these parameters, which provide a handle to manage the complexity of nonparametric models in practice. However, most modern Bayesian nonparametric models seem often out of reach to practitioners, as inference algorithms need careful design to deal with the infinite number of parameters. The aim of this work is to facilitate the journey by providing computational tools for Bayesian nonparametric inference. The article describes a set of functions available in the R package BNPdensity in order to carry out density estimation with an infinite mixture model, including all types of censored data.
1. Introduction R (RCoreTeam 2019) is often cited by Bayesian statisticians as their favorite programming language due to the many packages that provide tools for Bayesian inference. The general program for Bayesian inference BUGS  (Gilks, Thomas & Spiegelhalter 1993)  has been available for a couple of decades, with interfaces in R. Since then, additional software has been developed to make that language more accessible to the users, for instance OpenBUGS  (Thomas et al. 2006) , JAGS  (Plummer 2003) , and Stan (Stan Development Team & Stan Developement Team 2019). All three can be accessed directly from R by respectively using R2OpenBUGS/R2WinBUGS  (Sturtz, Ligges & Gelman 2005) , rjags  (Plummer 2019) , runjags  (Denwood 2016) , and rstan  (Stan Development Team 2018). Programs for specific fields of Bayesian statistics have appeared in recent years, for instance bspmma  (Burr 2012)  for meta-analysis using Dirichlet Process Mixture (DPM) models, DPpackage  (Jara 2007; Jara et al.
Finally, we added several new functions for graphical model checking, assessing Markov chain Monte Carlo (MCMC) convergence and parallel computation. The paper is organised as follows. We start with a concise overview of Bayesian nonparametric mixture models for density estimation in Section 2, along with our strategy for posterior inference and a description of the recent improvements to BNPdensity. We then describe the package and its general syntax in Section 3, including some simple examples, and provide in Section 4 a comprehensive comparison of the features and functionalities offered in three R packages dedicated to BNP density estimation, namely: BNPdensity, BNPmix, and DPpackage. We then conclude with a case study in Section 5.
Bayesian nonparametric density estimation This section aims at providing a concise review of the statistical model used in the BNPdensity package. As the name suggests, the focus of the package is density estimation based on BNP priors, including all types of censored data. The density model used is a mixture model  (Frühwirth-Schnatter, Celeux & Robert 2018) , where the mixing measure is a BNP prior, thus leading to an infinite mixture model. The most widely used BNP mixture model for density estimation is the Dirichlet Process Mixture (DPM) model due to  Lo (1984). Generalisations of the DPM correspond to allowing the mixing distribution to be any discrete nonparametric prior. A large class of such prior distributions is obtained by normalising increasing additive processes  (Sato 1999 ).
Package description The implementation of BNPdensity package is available from the Comprehensive R Archive Network (CRAN) at https://CRAN. R-project. org/package= BNPdensity. Fitting a model with BNPdensity starts with calling one of the two functions, MixNRMI1 or MixNRMI2, or their versions for censored data. The function MixNRMI1 fits a semiparametric mixture model where all components have a common scale parameter σ with an independent parametric prior, σ ∼ P σ , while MixNRMI2 is devoted to fully nonparametric mixtures of location and scale parameters: • distr. py0: Integer number identifying the base measure P 0 on the location parameters. Three choices are available, which are constrained by the conjugate prior we place on the hyperparameters of P 0 : Gaussian, gamma and beta. Additional arguments can be used to tune the shape of the base measure.
Inference algorithm Efficient posterior computation for BNP mixture models relies on two types of approaches: marginal or conditional. Marginal methods incorporate analytic integration of infinite dimensional parts of the parameter, which is the case of DPpackage and BNPmix. Instead, BNPdensity relies on a conditional sampler that directly samples trajectories of the processes. More specifically, the Ferguson & Klass algorithm is employed (see Section 2. 1), with the crucial merit of ensuring that largest weights in the series representation are not left out. This is to be compared to the stick-breaking representation where the weights sequence is decreasing only stochastically (that is, in expectation).
Non-conjugacy Mixture models present the difficulty that the likelihood goes to infinity for infinitely small clusters located exactly on one observed data point. This may induce numerical problems and instabilities, and such tiny clusters are almost invariably undesirable in practical applications. A reasonable solution in the Bayesian framework is to use a prior distribution on scale parameters with little mass on very small values, i. e. a gamma distribution with shape parameter larger than 1 or a truncated distribution. We might also want to provide a different kind of information on cluster scales: for instance, for a dataset whose variance has been scaled to 1, there is no reason to find clusters with a variance much larger than one. This would suggest using a prior with an upper bound, or with light tails for large values.
Finally, flexibility in the choice of the kernel k is a clear asset when modelling real data, to choose a reasonable error model. These three examples suggest that we might need a certain flexibility in the specification of the prior distribution on scale parameters or in the choice of the kernel. The inference algorithm used in BNPdensity and presented in Section 2. 1 does not rely on conjugacy between the base measure and the kernel of the mixture, as do standard algorithms for sampling from a Dirichlet mixture process such as that presented in  Escobar & West (1995). In contrast, DPpackage and BNPmix are limited to using conjugate couples of base measure and the mixture kernel. Not being bounded to conjugacy allows us first to use any relevant kernel for the mixture. Moreover, even in the case of the normal kernel, this removes the dependence imposed in the conjugate case between the location of the clusters and their variances.
More precisely, this allows a full flexibility on specifying priors based on external knowledge, and proves particularly useful concerning the scale parameters of the kernels. Indeed, half-Cauchy or half-Gaussian priors for hierarchical variance parameters have recently become popular  Gelman (2006) ;  Chung et al. (2015). The illustration on Species Sensitivity Distribution (SSD) (Section 5), where the data is scaled, offers such an example where both an upper bound and lower bound on the cluster variances are useful.
Censored data BNPdensity can deal with left, right and interval-censored data by using the functions MixNRMI1cens and MixNRMI2cens. The same holds true for DPpackage, while BNPmix does not handle censored data at all. Censored data usually emerge from imperfections of the measurement process, such as detection limits (high or low) or saturation, low measurement precision, or binning of the data. Improper treatment of censored data is clearly a source of bias  (Helsel 2005) : in the case of right-censored data due to a detection limit for high values, for instance, data are not censored at random and discarding them or substituting them deteriorates the dataset. We deal with censored data by using a version of the likelihood  (Helsel 2005)   This is done by running multiple chains starting from different initial conditions, potentially in parallel, and converting them into an mcmc object that can be processed by coda.
2014; Zhao & Chen 2016)  have sought to find the best parametric distribution by model comparison using goodness-of-fit measures. The general understanding is that no single distribution seems to provide a superior fit and that the answer is dataset dependent  (Forbes & Calow 2002). Therefore, the log-normal distribution has become the customary choice, notably because it readily provides confidence intervals on the HC 5 , and because model comparison and goodness of fit tests have relatively low power on small datasets, precluding the emergence of a definite answer to the question. The availability of a package such as BNPdensity allows to move beyond this customary assumption very easily. NRMIs offer a flexible nonparametric mixture model, which can accommodate distributions very different from a normal distribution.
Using a BNP approach via BNPdensity allows generalising this approach while circumventing the theoretical and technical difficulties of estimating the right number of components in a mixture. It is also important to use a method which may be applied to small datasets. This is another motivation for using a BNP approach, where model complexity adapts to the number of data points, and will tend to suggest simple or even univariate mixtures when few data points are present. On the contrary, many classical nonparametric approaches to modelling species sensitivity distribution  (Wang et al. 2015; Verdonck et al. 2001 ) only work well on large datasets. To model species sensitivity distribution, we carefully select the parameters in the package BNPdensity. Given that concentrations vary on a wide range, it is common practice to work on log-transformed concentrations.
(2013)  for more details). For the prior on the scale of the clusters, we want to use two pieces of information: first, since the data has been scaled, scale parameters are likely to be smaller than 1, the extreme case being a mixture with a single component. Second, we want to avoid the possibility of extremely small clusters centred on a data point, because they are not very interesting from an interpretation point of view, and because they cause numerical problems (the likelihood diverges when a cluster scale goes to 0). Therefore, we choose a uniform distribution between 0. 1 and 1. 5 for the prior on the cluster scales. In keeping with the traditional assumption of normality of the species sensitivity distribution, we choose to use a normal kernel for the mixture (distr. k = 1). We now compare three approaches to modelling Species Sensitivity Distribution (SSD): the most standard and recommended approach of  Wagner & Lokke (1991) ;  Aldenberg & Jaworska (2000) , which is a simple normal model, the most recent proposal by  (Wang et al.
2015)  which is a normal kernel density estimate and the BNP normal mixture made available with BNPdensity that we presented above. As already stated, a quantity of interest is the 5th percentile of the distribution. We choose as an estimator the median of the posterior distribution of the 5th percentile, while the 95% credible bands are formed by the 2. 5% and 97. 5% quantiles of the posterior distribution of the 5th percentile. The 5th percentile of the Kernel Density Estimate (KDE) is obtained by numerical inversion of the cumulative distribution function, and the confidence intervals using the nonparametric bootstrap. The 5th percentile of the normal SSD and its confidence intervals are obtained following the classical method of  Aldenberg & Jaworska (2000). We use data from an ecotoxicity research database as pre-processed in  Hickey et al.
(2012). We extract data for the insecticide Carbaryl. 0. 8 q q q q q q 0. 4 0. 8 1. 2 q q q q q 0. 03 0. 10 0. 30 BNP_mixture_ssd_mu0. 1_sigma1. 5_uniform_Gama0. 4 normal_kernel_mixture_ssd normal_ssd 0. 0 0. 2 0. 4 0. 0 2. 5 5. 0 7. 5 q q q q 0. 5 1. 0 1. 5 2. 0 q q q q 0. 01 0. 10 1. 00 An added value of the BNP-SSD is that on top of being more flexible than the classic normal SSD and more robust than the nonparametric approach of  Wang et al. (2015) , as a mixture model it naturally induces a clustering of the data which may contain some biologically interesting information. We implemented functions to estimate the optimal clustering from the MCMC sample and visualise it, potentially including a label on each point to reflect available meta data for interpretation. In the context of SSD, it is interesting to know what drives species sensitivity: it might be taxonomy, in the sense that taxonomically close species will tend to respond in the same way and belong to the same cluster, but other drivers have been suggested such as habitat, feeding behaviour or respiration, which may not coincide with taxonomy.
Dowse et al. (2013);Kon Kam King et al. (2014)oximately 40% have censored data. We obtain a non censored version of this dataset by excluding right or left censored data, and replacing interval censored data by the midpoint of the interval. Helsel (2006);Dowse et al. (2013);Kon Kam King et al. (2014)have shown that transforming censored data risks inducing bias, hence the ability of BNPdensity to accommodate censoring is particularly valuable for SSD. There does not appear to be any easily available approach to use KDE methods on all types of censored data.
We would like to thank  a Referee  for several interesting suggestions and  Matti Vihola  for useful advice on adaptive MCMC. J. Arbel  is partially supported by  Grenoble Alpes Data Institute  ( ANR-15-IDEX-02 ). A. Lijoi  and  I. Prünster  are partially supported by  MIUR ,  PRIN  Project  2015SNS29B.