dataesr/scientific-paragraphs-categorization huggingface dataset:
g. Rich et al. , 2018; Hopman et al. , 2018). The Duolingo 4  data set was built with extractions from the Duolingo app  5  , retracing initial learner productions. The data set was complex, using learner input for three tasks (reverse translation and reverse translation with provided words, including distractors and translation of heard phrases) and three data sets from different learner groups were created for the competition: Learners of English (with Spanish as L1), learners of Spanish (with English as L1), and learners of French (with English as L1). Participants had to identify errors from a longitudinal sam-pling of learner productions presented in the CONNL format. Results at CONNL-2014 showed 0. 3733 F 0. 5 while HOO 2012 showed 0. 4147 F 1 -score. The most recent Duolingo shared task reported results of 0.
Data set description The data were derived from extracts of EFCAMDAT, an open access corpus developed by linguists at Cambridge University in collaboration with Education First (EF), an international school of English as a foreign language  (Geertzen et al. , 2013). The corpus 11 consists of written assignments submitted to Englishtown, the online school of EF Education First, summing 1,180,309 individual scripts by 174,743 learners. The curriculum of Englishtown 12 covers all proficiency levels, from CEFR A1 to C2 organised along 16 EF teaching levels. Each EF level contains 8 teaching units ending with an open-ended writing task, yielding 128 distinct writing assignments. Writing tasks cover a variety of communicative functions, e. g. writing a complaint, contributing to a blog discussion, completing a movie story, making a job application, writing a brochure for a product etc.
Such tasks cover a variety of genres, e. g. descriptive, argumentative, narrative. Writing tasks and types are broadly aligned with the teaching goals at different proficiency levels, and, therefore, tasks in higher levels are more variable and potentially more com-9. Compared with NLP tasks, not so many gold standards  (Paroubek et al. , 2007)  are available to the Learner Corpus Research Community. 10. By simulating human decision on the basis of observable evidence, the task is a formalization of human decision. It could be argued that this kind of competitions using human judgments on learner productions could be interpreted as a way to flesh out the CEFR levels with real observations in terms of regularly observed errors for given levels. 11. Learning and test data were selected and manipulated independently of the participation of the Cambridge and Education First research teams.
It can be accessed at corpus. mml. cam. ac. uk/efcamdat2/ public_html/. 12. The curriculum description corresponds to an earlier version of the teaching materials withdrawn in 2013 and no longer in use. plex, aiming to elicit longer texts than tasks in lower levels, in line with the CEFR descriptors reviewed in the previous section. Thus, texts vary in length from 20-40 words (lower levels) to 150-180 words (higher levels). Information on learner nationality is used as a proxy to estimate L1 background. Regarding the process of CEFR alignment, Englishtown learners are allocated to a teaching level by EFL teachers, after a placement test. Learners are always allocated at the first level of a stage, that is, levels 1, 4, 7 and 10. As the data come from a real life learning environment, the data are more variable and potentially dynamic as learners progress from lesson to lesson and as individual performance might vary from lesson to lesson.
As machine learning systems are very dependent on the training data, the EFCAMDAT task prompts are bound to have an impact on the systems trained on this data set. It is, therefore, possible that systems trained on this test set might show diminished performance on classifying writings elicited through a different set of task prompts. At the same time, the logic of the EFCAMDAT tasks is not uncharacteristic of task-based language learning systems. For machine learning, a data set is needed. A data set can be described as a table with one observation per line and features as columns. In our case, the data were hybrid, including a column for texts (one learner production for a given task), a column for the assigned CEFR level and other columns for the numerical values produced by linguistic tools (lexical and readability metrics).
Features provided Fifty-nine feature variables are available. The first (called fulltext) is a text value: The full text produced by the person to be assessed (with an average length of 70 words). The other 58 variables are metrics calculated from the text using X. Lu's Lexical Complexity Analyzer (LCA)  (Lu, 2014)  and the koRpus R package  (Michalke, 2017). They describe the degree of vocabulary diversity and the complexity of the text. Among these metrics we find: The number of sentences, words, letters, and syllables, the Type-Token ratio (and derived measures), readability measures calculated from the correlation between the number of words and length of words used, and lexical sophistication, which measures the richness of the lexicon using reference inventories. A precise description of the characteristics is available on the website 14  and in Appendix A.
Classification of learners' texts for lexical proficiency is usually based on features such as vocabulary size  (Crossley et al. , 2011). Recent papers have also incorporated readability metrics  (Lissón, 2017). We have not used syntactic metrics  (Lu, 2010)  but we have taken lexical sophistication into account. Table  9  in Appendix B gives a detailed description of each metric. Lexical sophistication gives information on how unusual the words used by a given learner are. Sophistication is defined as the percentage of low-frequency words belonging to specific morpho-syntactic categories, i. e. verbs, nouns, adjectives and adverbs. Frequency thresholds in lexicons Features computed with the R package koRpus  (Michalke, 2017)  include indices of lexical sophistication (e. g. type-token ratio, HD-D/vocd-D, MTLD) and readability metrics (Flesch-kincaid, SMOG, LIX, Dale-Chall).
Training labels The classes which were to be predicted correspond to the six reference levels of the CEFR. The learner data shared on the EFCAMDAT corpus are of 16 different levels. Table  1  shows the conversion between the levels estimated by EFCAMDAT and the CEFR, and the number of each class in the training set. From a machine learning perspective, this is an unbalanced ordinal classification problem -that is a multiclass classification problem with ranked target classes. "Unbalanced" means that the number of examples for each label is far from equal. "Multi-class" implies that more than one boundary has to be designed to separate the data into six classes (as opposed to a binary classification, which is easier). The ranking nature of the problem is due to the fact that the labels are ordered, assuming A1 < A2 < B1 < B2 < C1 < C2.
By construction it is not symmetric. Indeed, for instance when considering classes A1 and C2 with respectively 11,361 and 50 learners, the error at predicting C2 when the true label is A1 is more frequent and less severe than making a mistake the other way round by predicting A1 instead of C2. In that case, C 1,5 = 6 and C 5,1 = 44. Note that the perfect classification corresponds to the diagonal of zeros and would correspond to a penalty score of zero. used for the evaluation (in R, Python and Matlab). It shows the details of how the matrix cost is calculated for each submission. The competition received 37 submissions from 14 different teams, with an average slightly above 2. 5 submissions per team and a maximum of six. Table  4  shows the final leader board of all participating teams together with their affiliation and their best performance measure E (Equation [1]) computed on the test set.
), 'O' for other linguistic indicators, 'W' for words, 'C' for characters and 'S' for syntactic features. First, some participants were not satisfied by the provided indices, and they therefore recomputed some linguistic indicators and proposed others. For instance, the winning team used spaCy 20 and textstat 21 to recompute the number of sentences and of difficult words and report a slight improvement in performance by adopting these metrics. Also, the team Haral-Lenca used 547 values provided by TAALES software 22. Second, use of the full text allowed participants to design semantic features based on word representations, topic and language models. To this end, some usual preprocessing tools such as NLTK 23 have been used to clean the text data including capitalization, numbers and characters normalisation and other details using regular expressions.
Representation Once features are determined, the choice of representation mode is critical for obtaining good results. Representation often comes with dimensionality reduction using principal component analysis (PCA) or feature selection (see for instance  Flach, 2012). Interestingly, Rufino's team performs poorly by selecting the 13 most 20. spacy. io 21. github. com/shivam5992/textstat 22. kristopherkyle. com/taales. html 23. nltk. org significant features, while Terislepacom reports very good results, and even an improvement, when selecting only the top 10 most relevant features. To deal with natural language, different representations were proposed. The most popular, which also gave the best results, is the bag of words (BoW) that counts how many times a word or a bigram appears in a text.
Transfer learning refers to the knowledge transfer between tasks  (Thrun & Pratt, 1998) , that is, the use of a model solving one problem (such as language modelling on a WikiText data set) to tackle another related task (such as text classification) using significantly less data than if training from scratch. A common transfer learning strategy is to specialise and fine-tune the first model using data from the application of interest. This popular approach in image classification is now fully lifted into NLP tasks (see ULMFiT for instance). Finally, at least one competitor (ICSI) used an AutoML tool, an off-the-shelf machine learning method that automatically performs algorithm selection and hyperparameter tuning, obtaining good performances at a low implementation cost. Such an AutoML function is now available in some machine learning frameworks such as H2O 31 (the one used by ICSI) and auto-sklearn 32.
The software used For this competition, Python was the most popular programming language. The tools the participants used are described in Table  6. The software components 33 range from libraries to preprocess natural language to core machine learning frameworks, including tools for word/sentence/text embedding or language model query. Automatic machine learning 6. 2. 5. Methodological concerns First, as pointed out by Igor Axinti, Yannis Haralambous and Philippe Lenca, it was possible to download the EFCAMDAT corpus in full and retrieve all the texts from test data sets and thus the correct associated labels. For this reason, details of the algorithm used by each participant were requested. Second, as reviewed in Section 3, the emphasis of the CEFR on the communicative functions that a learner ought to be able to perform at each proficiency level potentially introduces a correlation between task types and proficiency.
Indeed, task effects have been demonstrated on elicited language in EFCAMDAT by  Alexopoulou et al. (2017). Since each EFCAMDAT task comes with a topic variable, it was possible that competitors might use this variable to predict proficiency. To prevent the use of this variable and encourage identification of proficiency, irrespective of task/topic, we removed this variable information from the data set. However, in some cases where a specific task prompt elicited very specific lexical items (e. g. proper nouns, such as 'Tour Eiffel'), some competitors were able to retrieve words that were strongly correlated with the given task prompts. This suggests that we should have removed entries containing text related to some specific task prompts. Note also that competitors suggested metrics which are relevant for distinguishing early stages (e.
g. A1, A2) based on repetition and spelling error scores, with no use of the topic. Finally, we realised during the competition that we made the mistake of not removing the text variable (the 46th additional variable). This variable (not to be confused with fulltext) included the ID numbers of the texts. It was created to help us manipulate the data and was actually correlated with the labels. For this reason we asked the two winning teams to run their solutions without the text variable. Results reported in Table  7  show a clear deterioration in terms of performance but not in ranks since the score of the winning team without the text variable still outperforms all others. ) is obtained for the C2 samples. In one case out of 5, the system predicts these to be C1. This can be explained by the fact that the C2 class has a very small number of training samples and that the boundary between C1 (level 13-15 in EFCAMDAT) and C2 (level 16) is difficult to establish or somehow fuzzy.
Note that among these five misclassified samples two of them are short sentences, which is unusual for the C2 sample. More costly errors also occurred, such as mislabelled samples whose true label is several classes apart. This is the case, for example, for two samples labelled A1 and predicted as C1, and three others, labelled A2 and predicted as C2. In the first case, we note the use of several proper nouns in a single long sentence. In the second case, the sentences are also quite lengthy and tokenisation issues led to an over-interpretation of the proper nouns as rare words. The use of quotes artificially elevated lexical sophistication metrics as well. All the available EFCAMDAT scripts for advanced learners (C2) of French nationality were included, but the small number of texts for this proficiency level is a clear limitation to the effectiveness of machine learning techniques.
The strong classification performance of transfer learning methods suggest that these kinds of techniques could be applied to learner data, even for A1 / A2 levels. However rudimentary the task might seem, this competition may have been the first ever learner data challenge based on the French component of the EFCAM-DAT corpus. Perhaps more interesting tasks and formats could be designed. A more refined version of the competition could be organised, taking into account several tracks for several tasks. In one of the tracks, several first languages could be mixed (French was the only one used with this data set) in order to see if specific features could help indentify L1-based errors. For example, the Duolingo challenge had three tracks (three competition subsets) for three language learning trajectories (English to French, English to Spanish, Spanish to English).
Conclusions This challenge based on learner data was a success at the CAp2018 Machine Learning Conference and generated real interest among the machine learning community. The solution suggested by the winner solved the task as it was formulated, with the proviso that the trained model probably suffers from overfitting in the sense that the lexicon used is too heavily dependent on classes. There are 128 topics in the data set (see full distribution in Figure  4 ). The topic distribution in the test set was similar to the topic distribution in the training set (see Figure  4 ). With hindsight, we consider that we should have used different topics in the training and test sets since, as Murakami (2016) pointed out, there is a correlation between the level of the learner 34. nlp. fast. ai/category/classification.
html 35. http://www. ctapweb. com/ and the task instruction in the EFCAMDAT. It is worth noting that the variety of 128 tasks in EFCAMDAT contrasts with corpora like the MERLIN corpus 36 which involve writings from exams and where CEFR levels A and B involve a writing task of the same genre (writing a letter). This means that machine learning systems could not exploit the task or topic characteristics for CEFR alignment, at least not as readily as with our EFCAMDAT task. However, since CEFR descriptors require performance on a variety of tasks, it seems more appropriate to set competitions where systems are trained on one (sub)set of tasks and are able to 'generalise' to a different (sub)set of tasks. We have highlighted some of the benefits for the learner corpus research community, but they might be made more obvious if the data sets were universally available and potentially processed and discussed by other scientific communities.
Such a step within the learner corpus community needs to be balanced against the need to build corpora that can support wider research in language learning and the need to obtain informed consent from learners regarding the use of their data. In the case of EFCAMDAT, the terms of use have taken into account a wider range of academic users in the SLA and LCR communities, while ensuring use in accordance with obtained informed consent inevitably leads to managed access. So far, the data culture has been somewhat hindered in the learner corpus research community due to copyright restrictions or the lack of precise metadata about learner levels. Learner corpora collected in the early days of learner corpus research did not use placement tests systematically, resulting in publicly available learner data sets which often lack a crucial variable assessing learner levels.
Learner corpora have been used with data-mining techniques  (Jarvis, 2011)  36. https://merlin-platform. eu/index. php 37. https://dataverse. harvard. edu/dataset. xhtml?persistentId=doi:10. 7910/DVN/8SWHNO 38. https://www. kaggle. com/kumawatmanish/deep-learning-sign-language-dataset 39. https://www. kaggle. com/c/shorttext-langid/data and can be used as material for data sets calling for more elaborate methods for automatic treatment and analysis of learner corpus data  (Díaz-Negrillo et al. , 2013) , provided labels are assigned for supervised classification tasks. For the learner corpus research community, money and time should be invested in analyzing properly scored essays on specifically described tasks. Discoverable, accessible and retrievable data is key in the FAIR paradigm  (Mons, 2018).
29. cs224d. stanford. edu/syllabus. html 30. nlp. fast. ai/category/classification. html 31. docs. h2o. ai/h2o/latest-stable/h2o-docs/automl. html 32. automl. github. io/auto-sklearn 33. The code produced for the competition by Terislepacom team is available at github. com/ AlexouGarcia/CAP2018_SharedTask_2nd.
It should be noted that  Han and Sun (2018)  found the November index to be linked to a negative NAO and colder Eurasian near-surface temperatures, whereas the October index was correlated with warmer than usual temperatures over Eurasia and a southward-shifted jet. However, since many studies focus on northern Eurasian October snow cover as the predictor for winter climate, we will include it nonetheless. MERRA2 and 20CRv2c offer snow cover as well as snow depth as a post-process output; however, ERA20C only offers snow depth. We refrained from converting it to snow cover ourselves but found the index based on snow depth to be extremely similar (also see Supplement Fig. S1 ) to the same index using snow cover. Moreover, comparing snow indices from reanalyses with snow indices using the NOAA Climate Data record of Northern Hemisphere snow cover extent  (Robinson et al.
This hints at decadal variations in the strength of the regression but could also be due to biases in the reanalyses. To check for such biases we compared all reanalyses with the SLP reconstruction dataset HadSLP2r (Allen and Ansell, 2006) and found that for the regression analysis using the time period 1901-2010, 20CRv2c overestimates the polar sea level pressure response, whereas ERA20C is much closer to HadSLP2r (see Supplement Fig. S6 ). This would indeed support the notion of decadal variations in the strength of the relationship between predictor and predictand. However, it is worth highlighting that this overestimation for 20CRv2c is not visible for the 1851-2010 period, during which regression anomalies resemble HadSLP2r much more closely. We investigate other possible predictors for wintertime NAO via regressed anomalies onto the November BKS ice concentration, November-December mean polar GPH at 10 hPa, October AMO and October ENSO indices (Fig.
SLP anomalies regressed onto the AMO index show significant positive SLP regions for large parts of Eurasia as well as positive anomalies over the North Atlantic west of Great Britain. Interesting to note in 20CRv2c is the very strong high-pressure anomaly reaching from the BKS to the southern part of the Ural mountains, a prominent feature often found for years with positive AMO and negative sea ice concentration and frequently linked to a high frequency of Ural blockings (UBs). SLP distribution after El Niño events does not change considerably, irrespective of the dataset and time period used. A strong Pacific signal shows the northern part of the Pacific North American pattern (PNA), with negative SLP anomalies over the eastern North Pacific. Given the autocorrelation in the AMO and BKS sea ice index, the significance in Fig.
Compared to  Gastineau et al. (2017)  and  Han and Sun (2018) , we could extend the reanalysis study period from 35 to 150 years and highlighted the consistently negative sign of the snow-NAO relationship in the 20CRv2c dataset. Partial correlations for 110 years show that reduced BKS sea ice shows a similar response in DJF SLP anomalies; however, its statistical importance, and therefore its quality as being the prime predictor, is less than the November snow index (see Supplement Table  S2  for partial correlations). This is also found in simple multiple regression prediction models, whereas the November snow cover index incorporated the major share of the prediction power. Extending the analysis of  Gastineau et al. (2017)  to 150 years further underlines the lack of snow-atmosphere feedback in most of the CMIP5 models and reduces the probability that the snow-NAO link is due to random internal variability at the end of the 20th century.
20CRv2c shows negative correlation throughout the whole 20th century, whereas ERA20C flips the sign of the correlation in the late 1930s and late 1970s. The same relationship but using October snow shows high agreement between the two datasets, and the same is the case for the correlations between snow and stratospheric GPH. We therefore conclude that the information stored in the November snow cover in 20CRv2c is slightly different to the information stored in the ERA20C snow depth. Wegmann et al. (2017b) found that Eurasian November snow depth shows much larger disagreement between 20CRv2c and ERA20C than the same snow depth in October. In the same study, the authors found decadal trends (although linear trend subtraction for all predictor time series was done for this study) in ERA20C snow depth, which might impact the running correlations.
Conclusions Several reconstruction and reanalysis datasets were used to examine the link between autumn snow cover, ocean surface conditions and the NAO pattern in winter for the whole 20th century and into the 19th century. We found evidence for a manifestation of a negative NAO signal after November with a strong west-to-east snow cover gradient, with this relationship being significant for the last 150 years. Interdecadal variability for this relationship seems to be linked to Arctic warm periods, which increase the variability of the cryospheric predictors considerably. As a result, increased variability in the predictors helps to generate a better seasonal prediction estimation. Furthermore, our analysis of centennial time series supports studies pointing out the link of autumn snow to stratospheric circulation as well as the co-occurrence between reduced BKS ice concentration and increased snow cover in eastern Eurasia.