dataesr/scientific-paragraphs-categorization huggingface dataset:
The full dataset For the challenge, Duolingo released the data in three phases. 2 The first phase of the challenge afforded training on the first 80% of the data (TRAIN set). The second phase came with a release of the next 10% of the data (DEV set), for which participants could evaluate and fine-tune their models. Finally, the third phase allowed for predictions on the remaining 10% of the data to be submitted (TEST set). The task, as set out by Duolingo, was to predict which tokens, defined as distinct user-item pairs, users answered incorrectly in the TEST set. In the published challenge, the user's true performance was omitted from the TEST set and was not available on the website. As a result, we were not able to evaluate our modeling efforts on the same TEST set used in the SLAM challenge.
To remedy this without taking an unfair advantage over participants in the original competition, we combined the TRAIN and DEV sets and assigned the first 90% of instances for each learner to the train set, while the last 10% were assigned to the test set. This way, we emulated the SLAM challenge as closely as possible. The full dataset used here is thus based on 90% of the complete SLAM dataset published by Duolingo and follows a 90/10 train/test split. The training set contains 2,347,874 observations from 2593 users studying 2164 tokens (745,459 distinct user-token pairs). Accuracy in the training set is high (87. 6% overall), increasing as a user repeats a token (from 83. 5% on the first repetition and plateauing around 91% after the eighth repetition). However, repetitions are rare: 49% of user-token pairs are only repeated once (9% were repeated more than five times) and 71.
Slices of the dataset To investigate the potential benefits of enhancing an ML model with insights from a cognitive model, we explored two different ways of limiting the amount of information available to train the models. Thus, the approaches differ in what slice of the dataset was used, both with regard to the training and test sets. Furthermore, we decided to use only a limited set of input features that would be available for any comparable dataset (user, token, and timing information). This will impair the model overall but isolates the effects that are most generalizable and important (Table  1 ). The training set was sliced to limit the available data in two ways: First, by gradually exposing the models to more data day-by-day. In this restricted learning history approach, we made iterative predictions by fitting both models to the data up to day x and then making and comparing predictions made for day x + 1.
And second, making a prediction for a novel token is not a task a cognitive model would typically be leveraged for (collaborative filtering/recommender systems are better suited for this task,  Su & Khoshgoftaar, 2009). For all approaches based on slices of the dataset, we simplified the model by using only a small subset of input features, namely user and token identifiers, and timing information (expressed in days). These turned out to be the most important features (Table  1 ). Regardless of their importance, we believe that these are features that would be available in virtually all comparable prediction tasks in the context of language learning. Hence, restricting our explorations to a small number of near-universal features hopefully makes the results more generalizable.
Models For all approaches and subsets of the dataset outlined above, we always fit two models. Their predictive accuracies were evaluated using the area under the receiver operating characteristics curve (AUC;  Fawcett, 2006). The two models evaluated in this paper were GBDT and differed only with regard to how the timing related information was made available as input features. Both models had access to the "raw" timing information. The default GBDT had 10 additional input features that explicitly coded the lag (elapsed time) since the last, second-to-last, and so on repetition for each user-token pair. The PPE-enhanced GBDT only had two additional input features: The model time T and the stability component of the decay rate equation (see Eqs. 2 and 4, respectively). The PPE transformations imposed theoretical assumptions about how memory traces accumulate and decay over time.
Results To evaluate the default GBDT and the PPE-enhanced GBDT, we will report the area under the ROC curve (AUC;  Fawcett, 2006)  to quantify a model's predictive accuracy. AUC values range from 0. 5 (chance performance) to 1. 0 (perfect predictions) and were chosen because they are widely used as a measure of predictive accuracy in binary classification tasks and, specifically, they were the primary outcome measure in the SLAM challenge. The raw data and analysis scripts are available in an online Supplement at https://osf. io/vk8jr/, which contains the code to reproduce all numbers and figures reported here along with additional descriptives and visualizations. The rest of this section will follow the distinction made in the previous section and present the outcomes of our explorations, first on the full dataset and then on slices of the dataset.
Full dataset The two GBDTs were fit to the complete training set using both the provided and additionally engineered features (see online Appendix A at https://osf. io/fj8mr/). For the default GBDT, the AUC for the predictions was 0. 8530, which compares favorably to the published models reported in Settles and colleagues (2018, table 2)-slightly lower than the top three AUCs, which were 0. 859 and 0. 861 (tied), and comfortably outperforming the baseline (0. 774). The PPE-enhanced GBDT resulted in a slightly higher AUC of 0. 8538. The correlation between the two models' predictions is extremely high (r = 0. 985) and a Bayesian paired t-test suggests the data are 17. 6 times more likely to occur under the null model  (Morey & Rouder, 2018). However, DeLong's test reveals a statistically significant difference between the AUC values (z = -3.
, 2009, pp. 309-10). In this case, we used the Gini index for the measure of impurity  (Breiman, Friedman, Stone, & Olshen, 1984). 6 The ranking of the features in Table  1  suggest that in order to make accurate predictions in this task, the exact timing of practice is not nearly as important as who is expected to give a response to which token in which context (i. e. , prompt, previous, and next token). Taken together, the analyses on the full dataset suggest that the PPE-enhanced model performs at least as well or marginally better even though the dimensionality of the input is reduced relative to the default GBDT. Hence, providing the GBDT with PPE's preprocessed timing information did not impair the model. It appears that both GBDTs made very similar but not identical predictions that slightly favored the PPE-enhanced model.
Restricting number of users and tokens Besides restricting the learning history along the temporal dimension, we also explored limiting the amount of data available to train the models. Both models were trained on 100 samples for each combination of user/token pairings we explored (see Methods for details). This yielded a set of predictions on the test set for each model, from which AUC scores were calculated. The results are summarized in Fig. 2 , which shows in each cell the median AUCs for the default and PPE-enhanced GBDT (top and bottom values, respectively) as well as the percentage change. We see an overall advantage of the PPE-enhanced model with more data. In the lower left quadrant of Fig. 2 , the advantage of the PPE-enhanced GBDT is markedly larger relative to the default GBDT, with various user-token combinations resulting in advantages of 3-8% points.
Discussion Here, we attempted to enhance the performance of an ML model by incorporating insights from a cognitive model. We used second language acquisition data published by Duolingo to compare two GBDTs  (Friedman, 2001; Ke et al. , 2017) -one using raw timing information and one using cognition-inspired transformations of said information as input features. These transformations were based on the PPE  (Walsh et al. , 2018) , a cognitive model developed to capture theoretically grounded measures of human learning. When evaluated on the full dataset, the PPE-enhanced GBDT performed marginally better than the default model but produced largely comparable performance predictions. This suggests that with a sufficiently large dataset, transforming the timing information does not impair the GBDT and may even provide a small benefit.
Cognition-enhanced ML In many applied learning domains, however, datasets are not sufficiently large to effectively train state-of-the-art ML models. Since cognitive models are readily fit on small datasets, we expected the PPE-enhanced GBDT to prove particularly beneficial with limited data. To this end, we pitched the two models against each other in several smaller slices of the full dataset and found that, indeed, the PPE-based transformation of the timing-related input features can improve the GBDT's predictions. Specifically, training the models on restricted learning histories impairs predictive accuracy overall, but this impairment is consistently lower for the PPE-enhanced variant (Fig. 1 ). Furthermore, restricting the number of users and tokens available in the training set is less debilitating for the PPE-enhanced model (Fig.
Thus, these data points do not constitute a learning history. While this issue is present in this particular dataset due to the instructional learning design of Duolingo itself, it may be nonexistent in many realworld datasets. We anticipate that the modest improvement in accuracy of the PPE-enhanced model may actually underrepresent the potential for this technique to improve accuracy on human learning datasets in general, where study repetitions would normally be plentiful. Future extensions of this work should, therefore, explore the approach presented here in other datasets-from both naturalistic and experimentally controlled settings-where repetitions of individual study items are more frequent.
Conclusion It is uncommon to have a human learning dataset as vast and as varied as the ones published by Duolingo. 7 In many applications, the number of users is much smaller and the material set smaller (e. g. , an undergraduate class). Such circumstances make it difficult to train powerful ML models but a hybrid, cognition-enhanced model might be feasible. The work presented here should be understood as but one instantiation of a promising, more general approach: the appropriate cognitive model does not have to be the PPE; the chosen ML method does not have to involve gradient boosting. Using cognitive models to produce features to help train ML models could be applied in other domains.
Notes 1 For a list of the features available in the dataset (as well as those engineered by us), see online Appendix A at https://osf. io/fj8mr/ 2 https://dataverse. harvard. edu/dataset. xhtml?persistentId=doi:10. 7910/DVN/8SWHNO 3 Two additional rules were observed to ensure usable slices of the data: (1) The users and tokens were uniformly sampled (without replacement) from the 1000 users and tokens with the most observations. This avoided including users or tokens with very few observations but does bias the analysis slightly toward users that have used Duolingo more and tokens that appear more frequently. Please see the online Supplement for additional descriptive statistics pertaining to this split: https://osf. io/54ry7/ (2) Samples in which the test set did not include at least one correct and one incorrect response were discarded to ensure the AUC can be computed.
The online Supplement contains a figure that shows the distribution of sample sizes in the training and test sets, see https://osf. io/yg9ps/ 4 https://github. com/microsoft/LightGBM 5 These were: max number of leaves per tree: 50; learning rate: 0. 05; minimum number of observations per leaf: 10; and categorical smoothing: 50. 6 Online Appendix B (https://osf. io/fj8mr/) showcases an attempt to visualize the differences between the models by approximating the models with decision trees. This could be another way to illustrate feature importance but we believe it is less informative in the current context. 7 See the section Data & Tools at https://research. duolingo. com/.
Introduction Genomic studies generate a "data deluge"  [13]. Public data banks providing sequencing data or assembled genome sequences are growing at an exponential rate  [1]. Alongside, a fundamental need consists in comparing sequences at large scale, for instance for species identification  [15] , metagenome similarity estimation  [3, 11] , or any generic need for estimating the presence of a sequence among available datasets  [10]. Given the overwhelming amount of data to compare, these large scale sequence comparisons are made through alignment-free methods  [16]  mainly based on the number of words of fixed length, usually called k-mers for words of length k, shared between the two (set of) sequences to compare. So then, indexing kmers for fast and low memory membership queries is a fundamental need.
Preliminary definitions A k-mer is a word of length k over an alphabet Σ. Given a sequence S, |S| denotes the length of S. In the current framework, we consider a dataset as composed of one sequence or a set of sequences. We consider that a k-mer occurs in a dataset if it occurs at least once in any of the sequences composing the set. An AMQ data structure represents a set of elements D. It can be queried with any element d. If d ∈ D, then the AMQ answer is positive (there is no falsenegative). If d / ∈ D, the AMQ answer may be negative or positive, in this last case it is a false-positive call. The false-positive rate, denoted by F P R AM Q , is defined by F P R AM Q = #F P #F P +#T N with #F P and #T N denoting respectively the number of false-positive calls and the number of true negative calls.
Sequence similarity estimated by the number of shared k-mers Given two sequences, a k-mer occurring in both sequences is called "shared". The number of shared k-mers between a queried sequence q and a dataset B provides an insight of the presence of q in B, or at least of a sequence similar to q in B  [5]. In practice, for scaling Terabyte-sized sets B, a static AMQ indexes all (overlapping) k-mers from B. At query time, all (overlapping) k-mers from the query q are read on the fly and, for each of them, the k-mer is queried using the AMQ. Note that for each position p on q ∈ [0, |q| -k + 1] the k-mer starting at this position is queried. Thus, it overlaps by k -1 characters with the previously queried k-mer if it exists (if p > 0).
Querying K-mers with findere As exposed previously, after QTF, a false-negative call arises only when a positive k-mer belongs to a positive stretch of length ≤ z. Conversely, if a sequence of length K (with K > k) is shared between q and B, then it generates K -k + 1 successive true positive k-mers, thus defining a (K -k + 1)-stretch. Conceptually, we take advantage of this remark by proposing an algorithm that we call findere in which we query K-mers based on indexed k-mers filtered by the QTF algorithm, using z = K -k, hence looking for stretches of length ≥ z + 1. More precisely, given two integer values K and k, with K > k > 0, a bank dataset B and a query sequence q, the findere strategy consists in indexing all the k-mers from B using an AMQ. In contrast to QTF that calls k-mers, findere calls K-mers based on their k-mer content.
findere implementation We propose an implementation of findere, available at https://github. com/ lrobidou/findere. This implementation uses a Bloom filter as its inner AMQ. However, any other AMQ implementation can be used through a simple wrapper (provided with the findere implementation). The BF chosen for this implementation is a fork of the original https://github. com/mavam/libbf, which was modified to add the support of serialization. Although findere can index and query any alphabet, its implementation proposes a specialisation for genomic sequences: as such, one can index not only natural language, but also fasta and fastq files (gzipped or not) representing genomes or any sequencing read files. In this genomic context, a function to index and query canonical K-mers is also available.
Results We propose results on real biological data and on natural texts. The aim is to show the practical advantages offered by findere, both in terms of query precision, index size, and query time. Being developed to be used on top of any AMQ, we do not compare findere with any of those. Remind that findere may be used on filtering the results from any such data structure, including Cuckoo Filters for instance. To the best of our knowledge, the only tool comparable to findere is kBF. We compared findere and kBF on biological data only as kBF is not designed to work on a generic alphabet. Executions were performed on the GenOuest platform on a node with 4x8cores Xeon E5-2660 2,20 GHz with 200 Go of memory. A complete description of tool versions, data acquisition, command lines, and numerical results are provided in the Github repository https://github.
Experimental data Metagenomic data. In order to measure the impacts of the findere algorithm on real genomic data, we used two HMP  [8]  fastq files, indexing reads1 from sample SRS014107 and querying reads1 from sample SRS016349, both downloaded from the NCBI Sequence Read Archive. Theses samples contain respectively 4. 2 million reads of average size 92 characters and 2. 3 million reads of average size 96 characters. We simply refer to this dataset as the "hmp" dataset. Natural language data. In order to test the findere implementation on natural language, we used a dump of Wikipedia, from which we extracted two subsets overlapping with 10 Mo. They have each a size of 100 Mo, leading to about 10 8 31-mers each. We refer to this dataset as the "natural language dataset".
Query time Thanks to the optimization detailed section 2. 3, the query time decreases when z increases. As shown Table  1 , with discouraged values z = 0 or z = 1, the query time of findere is slightly higher than querying the original BF. This is due to additional conditional tests. With recommended z values (z = 2 to 5), compared to the query time of the BF, the findere query time is divided by a factor to 3. With the default z = 3 value, query time is divided by 2. 4, while query time still decreases when z increases: with z = 10, the query time is divided by ≈ 5. z 0 1 2 3 4 5 10 BF 42. 4 findere 42. 9 43. 7 24. 3 17. 5 14. 1 12. 0 8. 6 Table  1. BF and findere query time in seconds on the hmp dataset, depending on the z value. BF result does not depend on z and is reported only for z = 0.
Comparisons with kBF To compare findere and kBF, we created a fork of kBF, available at http:// github. com/lrobidou/kbf/. This fork enables to specify the amount of memory to be used by kBF and more importantly, it enables to index a set of reads and query another set (the original implementation does not allow that). kBF comes with two versions: 1-kBF and 2-kBF. 1-kBF uses less space at higher FPR than 2-kBF. The kBF strategy imposes to query up to nine times the BF when asking for the membership of a single k-mer. At the same time, findere is ≈ 2. 4 times faster than a Bloom filter with the recommended value z = 3. Moreover, kBF dumps all queried k-mers in RAM, and 2-kBF uses an additional hash set. Hence, kBF is much slower than findere. For instance, on the hmp data, with a BF FP of 5%, 1-kBF (resp.
Memory gain. As in Section 3. 2, we computed the FPR in function of the space used. From those results, we can deduce that, also on natural languages, for an FPR of 0. 1%, the findere space usage is two orders of magnitude less than BF. Indeed, findere needs 0. 023 Go, while a BF requires 3. 38 Go. Query time. As described in section 2. 3, the query time decreases when z increases. It holds when findere is used on the natural language dataset as well. With recommended z values (z = 2 to 5), compared to the query time of the Bloom filter, the findere query time is divided by a factor 1. 6 to 3. With the default z = 3 value, query time is divided by 2. 2 compared to the raw Bloom filter query.
Conclusion We propose a method filtering results of any approximate member query (AMQ) data structure when used for querying words of length K from a query. Despite its amazing simplicity, applied on metagenomics and natural text data, compared to the non-filtered results: findere 1/ makes queries two times faster, 2/ enables to decrease by two orders of magnitude the false-positive rate or enables to decrease the space allocated to each element by two orders of magnitudes, and 3/ has no drawback when used with recommended values. We are expecting an important impact of the findere tool, for which we propose an implementation. Indeed, AMQ data-structure are essential for indexing large datasets. In particular their usage is fundamental for indexing the genomic sequencing data, for which findere offers a new scaling breakthrough.
S1. 1 QTF false positive analyses In this section, we show qualitative results obtained thanks to the QTF strategy that motivated the findere algorithm. Results shown Table  S1  were obtained by applying the QTF approach, using k = 31, with various z values on an AMQ datastructure (a bloom filter) with 5% false positive rate (F P AM Q = 5%). Indexed and queried data are those from the hmp project, as described Section 3. 1. S1. Qualitative results using the QTF strategy, applied on the hmp dataset with F PAMQ = 5%. These results highlight the dramatic decrease of the false positive rate of the original AMQ data structure. Indeed, the false-positive rate falls from 5% with z = 0 where QTF has no effect, to ≈ 0. 51% when using z = 1 (removing all 1-stretches). With z = 2, the false positive rate falls to ≈ 0.
S1. 2 Results varying K We varied the value of K, and we report the false positive rate as well as the query time. Tests were performed on the hmp dataset using the default z = 3 value. For each K value, we determined size K being the size of the BF such that the FPR of the BF for K is 5%. Then, for each K value, we computed the findere FPR and query time, using the size K BF size for indexing k-mers. Results, also comparing results that would be obtained with the non filtered BF, are presented Fig S1.
S1. 4 Comparison with kBF on the hmp dataset To date, in the kBF implementation, the k-mers are read from a file, stored in a vector, and indexed. A fraction of these k-mers is mutated before querying all the   S2. BF and findere query time in seconds on the natural language dataset, depending on the z value. BF result does not depend on z and is reported only for z = 0. stored k-mers. Thus, the kBF implementation does not propose a way to index a set of reads and to query another set of reads. Furthermore, its implementation does not enable to serialize the index, and does not enable to set or get the size of the index. Finally, it is not possible to directly the amount of space to kBF. Instead, the user must provide an integer value (called "size factor"), which is the desired number of bits per element in the bloom filter used by kBF.
FPR of Bloom filter, 1-kBF, 2-kBF and findere with respect to the size of the filter, for K=31, on the hmp dataset. findere was used with default z = 3 value. From Table  S3 , we can observe that findere outperforms Bloom filters, 1-kBF and 2-kBF. For a high number of bit per element, the FPR of findere and 2-kBF seem close, however, the memory is heavily underestimated in the case of 2-kBF, and as such, 2-kBF is likely to take much more space (e. g. more than 8 Go when using 26 bits per element in the BF). In Table  S4 , we show that, for the indexation and the query steps, both 1-kBF and 2-kBF are slower than the Bloom filter used in the implementation of kBF. 1 The size factor only takes into account the bloom filter used by kBF, not the adjacent hashSet Table  S4. Time taken (in second) by kBF to index and query all of the 31-mers in the hmp dataset.