dataesr/scientific-paragraphs-categorization huggingface dataset:
The province of Ontario was chosen as a sample because it is the most populous Canadian province, with approximately 15 million residents  [29]. In some cases, there was a parent company that owned multiple clinics across Canada. When this occurred, we looked through available pages on the parent company website to identify that there was a physical clinic location in Ontario. Then, we analyzed data from the parent company website. The medical claims and cited supporting evidence included were not limited to Ontario, and thus could be broadly applicable across the country, where cannabis has been legalized nationally. One team member conducted Google searches in August 2021 using the keywords cannabis, clinic, medical marijuana, dispensary, and pain clinic, combined with the OR Boolean operator.
Evidence cited included information directly from the website or references to external sources. We compiled all health claims, qualitatively grouped them by content and theme when possible, and determined the following: number of total clinics, number of claims per clinic, number of specific indications identified, and number of claims per specific indication. Data collected included data on quality of evidence, study characteristics, cannabinoids used, routes of cannabis administration, and strain of the cannabis plant. We also tracked which clinic websites mentioned potential harms related to medical cannabis. Two team members independently in parallel appraised the quality of "supporting evidence" for all health claims across all cannabis clinic websites using the Oxford Levels of Evidence  [33].
Although the importance of generalization is almost undisputed, systematic generalization testing is not the status quo in the field of NLP. At the root of this problem lies the fact that there is little understanding and agreement about what good generalization looks like, what types of generalization exist, how those should be evaluated, and which types should be prioritized in varying scenarios. Broadly speaking, generalization is evaluated by assessing how well a model performs on a test dataset, given the relationship of this dataset with the data on which the model was trained. For decades, it was common to exert only one simple constraint on this relationship: that the train and test data are different. Typically, this was achieved by randomly splitting the available data into training and test partitions.
Shift source. On the shift source axis (Fig. 4 ) we see that almost half of the reviewed generalization studies consider naturally occurring shifts: natural corpora that are not deliberately split along a particular dimension. As discussed later in this section, this type of data source is most prevalent in cross-task and cross-domain generalization studies, for which such naturally different corpora are widely available. The next most frequent categories are generated shifts, where one of the datasets involved is generated with a specific generalization property in mind, and artificially partitioned natural data, describing settings in which all data are natural, but the way it is split between train and test is controlled. Fully generated datasets are less common, making up only 10% of the total number of studies.
Which types of data shift occur across different loci? Another interesting interaction is the one between the shift locus and the data shift type. Figure  6  (centre left) shows that assumed shifts mostly occur in the pretrain-test locus, confirming our hypothesis that they are probably caused by the use of increasingly large, general-purpose training corpora. When such pretrained models are further finetuned, experiments often consider either a shift between pretraining and finetuning where new labels are introduced, or a covariate shift in the finetuning stage; as such, they do not require an in-depth understanding of the pretraining corpus. The studies that do investigate covariate or full shifts with a pretrain-train or pretrain-test are typically not studies considering large language models, but instead multi-stage processes for domain adaptation.
Conclusion In this Analysis we have presented a framework to systematize and understand generalization research. The core of this framework consists of a generalization taxonomy that can be used to characterize generalization studies along five dimensions. This taxonomy, which is designed based on an extensive review of generalization papers in NLP, can be used to critically analyse existing generalization research as well as to structure new studies. The five nominal axes of the taxonomy describe why a study is executed (the main motivation of the study), what the study intends to evaluate (the type of generalization it aims to solve) and how the evaluation is conducted (the type of data shift considered, the source of this data shift, and the locus in which the shift is investigated). To illustrate the use and usefulness of our taxonomy, we analysed 543 papers from the ACL anthology about generalization.
We advise that this should be improved, and that inspiration might be taken from how generalization is evaluated in experiments with human participants, where control and access to the 'pretraining' data of a participant are unattainable. Along with this Analysis we also launch a website, with (1) a set of visualization tools to further explore our results; (2) a search tool that allows researchers to find studies with specific features; (3) a contributions page, allowing researchers to register new generalization studies; and (4) a tool to generate GenBench evaluation cards, which authors can use in their articles to comprehensively summarize their generalization experiments. Although the review and conclusions presented in this Analysis are necessarily static, we commit to keeping the entries on the website up to date when new papers on generalization are published, and we encourage researchers to engage with our online dynamic review by submitting both new studies and existing studies we might have missed.
Shift type-what kind of data shift is considered? We have seen that generalization tests differ in terms of their motivation and the type of generalization that they target. What they share, instead, is that they all focus on cases in which there is a form of shift between the data distributions involved in the modelling pipeline. In the third axis of our taxonomy, we describe the ways in which two datasets used in a generalization experiment can differ. This axis adds a statistical dimension to our taxonomy and derives its importance from the fact that data shift plays an essential role in formally defining and understanding generalization from a statistical perspective. We formalize the differences between the test, training and potentially pretraining data involved in generalization tests as shifts between the respective data distributions: p (x tst , y tst ) test (1) p (x tr , y tr ) training/finetuning/adaptation p (x ptr , y ptr ) pretraining These data distributions can be expressed as the product of the probability of the input data p(x) and the conditional probability of the output labels given the input data p(y|x): p(x, y) = p(x) p(y|x) (4) This allows us to define four main types of relation between two data distributions, depending on whether the distributions differ in terms of p(x), p(y|x), both or none.
We include this body of work in our Analysis and denote this type of shift with the label 'assumed shift'. Multiple shifts. Note that some studies consider shifts between multiple distributions at the same time, for instance to investigate how different types of pretraining architecture generalize to o. o. d. splits in a finetuning stage 52 or which pretraining method performs better cross-domain generalization in a second training stage 53. In the GenBench evaluation cards, both these shifts can be marked (Supplementary section B), but for our analysis in this section, we aggregate those cases and mark any study that considers shifts in multiple different distributions as multiple shift. Shift source-how are the train and test data produced? We have discussed what types of shift may occur in generalization tests.
Formally, such a partitioning scheme is a rule f ∶ 𝒯𝒯 𝒯 𝒯true, false} that discriminates data points according to a property τ ∈ 𝒯𝒯. To investigate how a partitioning scheme impacts model behaviour, the pretraining, training and test distributions can be defined as p (x ptr , y ptr ) = p (x base , y base | f pretrain (τ) = true) (6) p (x tr , y tr ) = p (x base , y base | f train (τ) = true) (7) p (x tst , y tst ) = p (x base , y base | f test (τ) = true) (8) Using these data descriptions, we can now discuss four different sources of shifts. Naturally occurring shifts. The first type of shift we include comprises the naturally occurring shifts, which naturally occur between two corpora. In this case, both data partitions of interest are naturally occurring corpora, to which no systematic operations are applied.
For the purposes of a generalization test, experimenters have no direct control over the partitioning scheme f(τ). In other words, the variable τ refers to properties that naturally differ between collected datasets. Artificially partitioned natural data. A slightly less natural set-up is one in which a naturally occurring corpus is considered, but it is artificially split along specific dimensions. In our taxonomy, we refer to these with the term 'partitioned natural data'. The primary difference with the previous category is that the variable τ refers to data properties along which data would not naturally be split, such as the length or complexity of a sample. Experimenters thus have no control over the data itself, but they control the partitioning scheme f(τ).
Acknowledgements We thank  A. Williams ,  A. Joulin ,  E. Bruni ,  L. Weber ,  R. Kirk  and  S. Riedel  for providing feedback on the various stages of this paper, and  G. Marcus  for providing detailed feedback on the final draft. We also thank the reviewers of our work for providing useful comments. We thank  E. Hupkes  for making the app that allows searching through references, and we thank  D. Haziza  and  E. Takmaz  for other contributions to the website. M. G. was supported by the  European Research Council (ERC)  under the  European Union 's  Horizon 2020 research and innovation programme  (grant agreement no. 819455 ). V. D. was supported by the  UKRI Centre for Doctoral   Training in Natural Language Processing , funded by the  UKRI  (grant no. EP/S022481/1 ) and the  University of Edinburgh.
Data availability The full annotated list of articles included in our survey is available through the GenBench website (https://genbench. org/references), where articles can be filtered through a dedicated search tool. This is an evolving survey: we encourage authors to submit new work and to request annotation corrections through our contributions page (https://genbench. org/contribute). The exact list used at the time of writing can be retrieved from https://github. com/GenBench/ GenBench. github. io/blob/cea0bd6bd8af6f2d0f096c8f81185b1d-fc9303b5/taxonomy_clean. tsv. We also release interactive tools to visualize the results of our survey at https://genbench. org/visualisation. Source data are provided with this paper.
Extended Data Fig. 1 | 1 Different loci of splits, and the parts of the modelling pipeline for which they may investigate generalisation. The shifts that characterise generalisation experiments in NLP can occur in different places in the modelling pipeline. In this figure, we visualise the three stages of the contemporary modelling pipeline: the pretraining stage, consisting of pretraining data as well as a pretraining procedure; the training stage, which involves training data, a pretrained model, and a training procedure; and finally, the test stage, in which an already trained model is tested on a test dataset. As visualised in this figure, shifts can occur between all or multiple of those stages, which allows to investigate different parts of the modelling pipeline.
instance, which specifies the probability distribution over the target test labels 𝒴𝒴 tst , given the model's parameters θ* and a set of test inputs 𝒳𝒳 tst. Equation (10)  defines a training procedure, by specifying a probability distribution over model parameters θ * ∈ ℝ d given a training dataset 𝒳𝒳 tr , 𝒴𝒴 tr , a set of training hyperparameters ϕ tr and a (potentially pretrained) model initialization θ. Finally, equation (11)  defines a pretraining procedure, specifying a conditional probability over the set of parameters θ , given a pretraining dataset, a set of pretraining hyperparameters ϕ pr and a model initialization. Between which of these stages a shift occurs impacts which modelling distributions can be evaluated. We now discuss the different potential loci of shifts. The train-test locus.
A colored de Bruijn graph (also called a set of k-mer sets), is a set of k-mers with every k-mer assigned a set of colors. Colored de Bruijn graphs are used in a variety of applications, including variant calling, genome assembly, and database search. However, their size has posed a scalability challenge to algorithm developers and users. There have been numerous indexing data structures proposed that allow to store the graph compactly while supporting fast query operations. However, disk compression algorithms, which do not need to support queries on the compressed data and can thus be more spaceefficient, have received little attention. The dearth of specialized compression tools has been a detriment to tool developers, tool users, and reproducibility efforts. In this paper, we develop a new tool that compresses colored de Bruijn graphs to disk, building on previous ideas for compression of k-mer sets and indexing colored de Bruijn graphs.
Introduction Modern methods for analyzing biological sequences often reduce the input dataset to a set of short, fixed length strings called k-mers. When working with a collection of such datasets E = (E 0 ,. , E |E|-1 ), it is fruitful to represent them as one union set of k-mers and, for each k-mer, the indices of the datasets to which it belongs. The set of indices of each k-mer is referred to as its color class, and E is referred to as a colored de Bruijn graph  [1]. A colored de Bruijn graph (cdBG) is commonly used to represent a sequence database, such as a collection of sequencing experiments or a collection of assembled genomes. For example, it is used by tools for inferring phylogenies  [2] , quantification of RNA expression  [3] , and studying the evolution of antimicrobial resistance  [4].
As sequence database sizes grow to petabytes  [5] , the cost of storing or transferring the data (e. g. on Amazon Web Services or in-house compute infrastructure) has underscored the need for efficient disk compression algorithms. Such costs are often prohibitive for smaller labs and, even for larger labs, limit the scale of data that can be analyzed. Large file sizes also hamper tool development, which relies on iterative loading/copying/modifying data, and reproducibility efforts, which require downloading and storing the data. For example, storing the 31-mers from 450,000 microbial genomes in compressed form takes about 12 Tb  [4]. Unfortunately, there has not been a lot of work to develop methods for disk compression of colored de Bruijn graphs. In contrast to disk compression, indexing cdBGs have received much attention  [6].
A slew of data structures have been developed, optimizing various metrics such as index size, construction time, or query time (see the survey  [6]  and its follow up  [7] ). Indexing data structures exploit the structure of cdBGs and use clever tricks to compress the color classes of similar k-mers. But they also carry a space overhead to efficiently support queries; since this is not needed for disk compression, indexing data structures are usually not competitive with custom made cdBG compression methods. The simplest option for compressing a cdBG is to compress each color (i. e. dataset) independently, using a compression tool designed for a single set of k-mers (e. g. [8] ). This approach can work well when k-mers tend to not be shared among colors. However, most cdBGs have a large overlap between the k-mers of various colors.
We exploit the fact that consecutive k-mers in an ESS have similar color classes in order to efficiently compress the color vectors of each k-mer. We evaluate ESS-color on a variety of datasets, including bacterial, fungi, human, and including whole genome sequencing data, metagenome sequencing data, and whole assembled genomes. ESS-color achieves better compression than all evaluated tools and on all datasets, with all other tools using ≥ 44% more space on at least one of our datasets. On some datasets the improvement over all other tools is quite large, e. g. for a gut metagenome, all the other tools use at ≥ 27% more space than ESS-color. Compressing each color independently with ESS uses between 1. 2x and 6. 9x more space than ESS-color. The absolute compression ratio is more than 26x on datasets of assembled genomes and between 1.
Preliminaries In this section we give some important definitions. Please refer to Figure  1  for examples of the introduced concepts. Strings: A string of length k is called a k-mer. We assume k-mers are over the DNA alphabet. Let K be a set of k-mers. A spectrum-preserving string set (SPSS) of K is a set of strings S such that each string 1 s ∈ S is at least k characters long, every k-mer that appears in S appears exactly once, and the set of k-mers that appear in S is K  [11, 12, 13]. For example, if K = {ACG, CGT, CGA}, then {ACGT, CGA} would an SPSS of K. Note that K can have multiple spectrum-preserving string sets. There are several efficient tools for computing an SPSS so as to minimize the total number of characters  [14, 10]. In this paper, we rely on the implementation in  [8]. Each string in the resulting SPSS is referred to as a simplitig.
Compression of a k-mer set ESS is a disk-compression format to store a set of k-mers K. It can be computed using the ESS-basic tool  [8]. The details of the format are not relevant for this paper, though intuitively it can be thought of an extension of an SPSS that allows some simplitigs to be stored as "offshoots" of other simplitigs, thereby saving the need to store their first k -1 characters. An ESS representation cannot be queried efficiently but can be decompressed into an SPSS of K. Furthermore, the decompression algorithm is deterministic, i. e. the same SPSS is always output. Note that this output gives an ordering of the k-mers of K, and therefore the ESS compression of K induces an ordering on K. Colored k-mer sets: Let C > 0 be an integer indicating the number of colors. Let E = {E 0 ,.
Compression of the color matrix Given an SPSS S of E and a color matrix of E over the order induced by S, we now generate a compressed representation of the matrix. Our representation consists of a global class table and, for every simplitig of S, a few bits of metadata, a local class table and one bitvector m. The local class table is optional, as we describe below. Figure  2  gives a schematic representation. We now explain each of these in detail.
These three files are then sufficient to reconstruct the global IDs during decompression. 1 imultaneously, we need to be able to map a color vector to an ID during the compression process. To do this, we create a minimal perfect hash function h (CHM  [20] ) that maps from each distinct color vector to an integer between 0 and M -1. We then maintain an array A of size M , where for each color vector c, A(h(c)) holds the global ID of color vector c. After the global class table is created, we process the simplitigs of S one at time. For each simplitig, we dynamically set two parameters: a boolean variable U seLocalID, and an integer 0 ≤ maxDif ≤ 2. We postpone the discussion on how these are set until the end of the section. The values of maxDif and U seLocalID are stored using 3 bits of metadata per simplitig.
All bitvectors are additionally compressed with RRR, making it difficult to determine in advance which parameters result in the least space. We therefore try all possibilities of maxDif ∈ {0, 1, 2} and U seLocalID ∈ {T rue, F alse}, and, for each combination, compute the encoding. We then use the encoding that takes less space and disregard the rest. Though this step can likely be optimized, we found that the time taken to try all possibilities was not a large factor in the overall compression time. The decompression algorithm for the m vector is straightforward since our color matrix compression scheme is designed to be unambiguously decompressed. Simultaneously, we decompress E with ESSdecompress. The result is an SPSS S of E and a color matrix of E in the order of S. If the output is to be processed downstream in a streaming manner, our decompression algorithm can trivially stream out k-mer sequence and color vectors, one k-mer at a time.
Evaluated tools and datasets As far as we are aware, there are two other tools designed for compressing colored de Bruijn graphs: KS  [9]  and GGCAT  [10]. We refer to the first tool as KS after the authors' last names  [9]. KS is limited to support only three k values (15, 19, and 23), so we compare against it for k = 23 but also evaluate ESS-color on a more practical k value of k = 31. For GGCAT, we additionally compressed its Fasta output file with MFC and its binary color table with gzip to maximize its compression ratio. We also compare ESS-color against the naive approach of compressing each color independently using the ESS algorithm of  [8]. We refer to this as ESS-basic. Table  1  shows the datasets we use for evaluation and their properties. We chose five datasets so as to cover a broad range of input types.
Three of the datasets are from assembled genomes, one is from RNA-seq reads, and one is from metagenome reads. We used all k-mers from the three assemblies datasets and all k-mers that appear at least twice from the two read datasets. The datasets cover various species, from Bacteria to Fungi to Human. Concretely, we have 1) 100 arbitrarily selected E. coli strains from 2) an arbitrary subset of 10 of those, 3) 20 arbitrarily selected fungi sequences from RefSeq, 4) gut microbiome read sets from nine individuals sequenced in  [23] , and 5) 19 paired-end, human, bulk RNAseq short-read experiments previously used in  [19]. All accession numbers are listed in https://github. com/medvedevgroup/ESSColor/wiki/Experiments. Finally, all experiments were run on a server with an Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.
Comparison against other disk compression tools Table  2  shows the bits per k-mer achieved by ESS-color compared with KS, GGCAT ESS-basic, and a gzip compression of the original fasta file. ESS-color achieves better compression than all evaluated tools and on all datasets. No other tool able to consistently achieve less than 44% space overhead compared to ESS-color. On some datasets the improvement over all other tools is quite large, e. g. for Gut (k = 23), all the other tools use at least 27% more space than ESS-color. The compression ratio of ESS-color relative to the original Fasta. gz files varies (Table  2 ). For the read datasets, it is more than 26x, since high-coverage FASTA files are by their nature very redundant. For the assemblies datasets, the ratio is between 1. 4x and 8. 7x.