dataesr/scientific-paragraphs-categorization huggingface dataset:
For each dataset, the Piélou evenness index was calculated (y-axis). Aliquots are identified according the sampling size as previously. (D) Similarity between datasets before and after Shannon filtering. The Morisita-Horn similarity index between all pairs of datasets is color-coded according to the indicated scale before (lower half-triangle) and after (upper half-triangle) Shannon filtering. Aliquots are identified according to sampling size as previously. We then decomposed the clonotype collection by labeling the clonotypes as private (not shared between replicates) or shared by two or three replicates. For each dataset, clonotypes were sorted from the most to the least abundant and enrichment curves were built for each category according to the sharing status of each clonotype (Figure  3C ).
We applied this observation to build clonotype distributions of a fixed size and known diversity to simulate the sampling effect occurring during a RepSeq experiment. Seven Zipf distributions of 6•10 5 sequences each were simulated with a parameter A = 1/Zipf-α ranging from 2 to 100. These distributions were then assigned to a list of clonotypes randomly generated using the tcR package (65), leading to seven TRB clonotype repertoires of known diversity. As observed in Figure  4A , the distribution slope varies according to the depth of sequencing of the clonotypes. For example, for the distribution simulated with A = 2 (A2), the resulting distribution is skewed in a way that clonotype counts range from 1 to 31,109, whereas when A = 100 (A100), clonotype counts do not    The A3-datasets were subsampled at increasing depth (from 500 to 1•10 6 sequences as indicated in the legend from light to dark blue).
For each depth, 100 subsamples were produced. Within each subsample series, an increasing number of subsamples (x-axis) were randomly selected and their cumulative clonotype richness was calculated relative to the original dataset richness (clonotype richness coverage). exceed 9. These different distributions lead to datasets of varying richness, as summarized in Table  2. For each of our seven "known" repertoire distributions, we generated 100 subsamples at 6 sample sizes (from 500 to 1•10 6 sequences) reflecting several levels of sequencing depth. The clonotype richness observed within each subsample increased according to the depth, as expected (Figure  4B ). We used the MH similarity index to assess (i) representativeness (Figure  4C ) by comparing the diversity captured for each subsample with the original repertoire diversity and (ii) reproducibility (Figure  4D ) for the 100 subsamples for a given depth.
When comparing the seven distributions at a given sequencing depth (5•10 4 sequences, representing 8% of the original repertoire), the representativeness of the diversity between distributions was different (Figure  4C ), yet with similar relative richness values. For the "A2" condition, the similarity index between this subsample and the original repertoire was above 0. 8, while it varied from 0. 2 to 0. 5 for the other conditions (Figure  4C ). A dataset of 5•10 5 sequences (80% of the original repertoire size) is needed to reach a 0. 9 similarity for the latter. However, a suitable representativeness does not ensure good reproducibility of the observations. With 500 or 1,000 sequences, even if the diversity observed for the "A2" condition is quite representative (MH ~ 0. 8), the high variability between the subsamples implies a low reproducibility and thus an inability to observe exhaustively all the clonotypes (Figure  4D ).
We sought to identify which simulated distribution would be the most representative of our experimental datasets. To this end, we compared the slope at the steepest descent point of each simulated distribution with those of all the experimental data analyzed in this study. The experimental distribution slopes are most comparable with the "A3" and "A5" distributions, with the exception of that of the R500_2 sample (Table  S3  in Supplementary Material). Thus, we chose the "A3" distribution dataset as the most representative. In order to understand the low overlap observed between experimental replicates in Figure  3B , for each size we compared the "A3" simulated subsamples to determine the proportion of clonotypes shared by three independent subsamples, as performed experimentally in Figure  3.
As summarized in Table  3 , the proportion of private and shared clonotypes varies according to the coverage of the initial repertoire stretch. For subsamples with sizes representing less than 1% of that of the initial dataset, almost all the clonotypes observed are private (only captured in one subsample). For the "5•10 4 sequence" subsamples, the size of which represents 8% of the original repertoire size, 16% of the clonotypes observed are captured at least twice. These proportions correspond to the observations we made in Figure  3  between the three experimental replicates. Finally, using subsamples of size close (80%) to that of the original, 95% of the observed clonotypes are shared by at least two replicates. In addition, as represented in Figure  5 , at this depth, while one sample only captures about 12% of the overall existing clonotypes, three replicates cover a third of the overall richness.
Indeed, when sequencing samples of varying sizes at a commonly used depth, we found that small datasets contained 20 times more clonotypes than would be expected regarding the sample size. This figure decreases when the starting material is increased, demonstrating that over-sequencing small samples dramatically generates noise that cannot be corrected by removing only singletons. Although the relationship between sample size and sequencing depth that we used may appear extreme, it can commonly occur when studying small cell subsets involved in immunological processes. These observations demonstrate the drawbacks of discarding clonotypes based only on their counts and the need for objective approaches in order to assess the actual richness of a repertoire effectively. Single-cell sequencing technologies are an alternative to accurate study of the repertoire of small cell subsets and therefore will surely not require the use of Shannon filtering, because the number of expected unique TR sequences will be at most two per single cell.
While the distribution evenness was sample size-dependent when considering all the reads, filtering by the Shannon entropy index removed this variability between replicates (Figure  2C ). This proposed strategy therefore offers an accurate assessment of clonotype identification and representability, even in extreme situations. We applied our method to data produced following multiplex-PCR amplification on bulk polyclonal CD4 + T cells, for which the targeted genes and bias should be constant from one experiment to another. Although the number of uninformative clonotypes should be assessed when analyzing datasets prepared by different molecular methods, we believe that the Shannon index should reflect the true diversity by excluding uninformative clonotypes. Once single-cell sequencing becomes standardized and applicable to a range of very small to very large sample sizes, such correction metrics may not be necessary anymore.
Our results strongly suggest that sequencing depth must be adapted to the initial cell amount. We show that "50,000-cell" replicates are closer to each other than lower input pairs of samples (Figure  2D ). This observation emphasizes the need to adapt the sample size to the population of interest. All aliquots analyzed here were obtained from a rich and polyclonal cell population. In order to be reliable, a sample needs to be large enough to ensure that most of the clones are represented. Here, about 20% of the clonotypes observed in the two replicates (6,766 out of 30,422 and 35,020 clonotypes) are shared. Altogether these results show how complex defining a RepSeq strategy can be in guaranteeing the representativeness of the repertoire diversity. If sequencing depth is not adapted to the population size, it can negatively affect the resulting observed diversity, in particular if data are not properly analyzed.
FigUre1| Impact of sequencing depth on measured diversity. (a) Experimental design: 600,000 CD4 + Foxp3-GFP -cells were sorted from female Balb/C Foxp3 splenocytes. RNA was extracted (1) and split into aliquots equivalent to the amount of mRNA of 500, 1,000, 5,000, and 50,000 cells (2). Two aliquots were produced for each amount of RNA. The eight prepared aliquots were processed for TRB chain deep sequencing. (B) Dataset summaries. Histograms show, for each resulting dataset, the number of reads (black), productive TRB sequences (gray), observed clonotypes (blue), and clonotypes observed more than once (NoSingletons; light blue). (c) Diversity profiles. For each dataset, a Rényi diversity profile was computed: diversity metrics using clonotype frequencies were calculated for increasing values of Rényi order α until stabilization of the resulting diversity.
FigUre2| Clonotype distributions before and after data filtering. (a) TRB clonotype counts of the eight aliquots according to sampling size. Within each dataset, clonotypes were ranked according to their counts from the most to the least predominant (decreasing clonotype rank) and their abundance (clonotype count). Both axes are log-scaled. Plots were colored according to the sampling size: "500 cells" in red, "1,000 cells" in green, "5,000 cells" in cyan, and "50,000 cells" in purple. Clonotypes filtered out using the Shannon index (see main text) are colored in gray [scarce uninformative clonotypes (SUC)]. (B) TRB clonotype distributions of the eight aliquots before and after data filtering. Before (left) and after (right) filtering each dataset using the Shannon index as threshold, clonotypes were ranked from the most to the least predominant (decreasing clonotype rank) according to their relative frequencies (clonotype frequency).
The X-axis is log-scaled. Distributions were colored according to the sampling size as previously. (c) Impact of clonotype filtering on the clonotype distribution evenness. The ratio between the number of clonotypes and the number of cells (x-axis) was calculated for each aliquot before (circles) and after clonotype filtering either by removing only singletons (triangles) or using the Shannon index as a threshold (squares). For each dataset, the Piélou evenness index was calculated (y-axis). Aliquots are identified according the sampling size as previously. (D) Similarity between datasets before and after Shannon filtering. The Morisita-Horn similarity index between all pairs of datasets is color-coded according to the indicated scale before (lower half-triangle) and after (upper half-triangle) Shannon filtering.
FigUre4| Impact of sequencing depth on the observed diversity. (a) Clonotype distribution within the seven simulated datasets-within each A-dataset, clonotypes were ranked decreasingly according to their abundance and their frequency was plotted. Both axes are log-scaled. Distributions are colored according to the A parameter used to simulate it. (B) Impact of sequencing depth on the observed clonotype richness-for a given A-dataset, clonotype richness was measured within the 100 subsamples produced for each depth and divided by that of the original dataset. The median value by depth is represented for each condition. The 95% CI was calculated but cannot be seen since it merged with the median value. (c) Representativeness of the sequencing-the Morisita-Horn similarity index was calculated between each subsample and its original dataset.
FigUre5| Clonotype coverage of A3-dataset richness increases with multiple subsamples. The A3-datasets were subsampled at increasing depth (from 500 to 1•10 6 sequences as indicated in the legend from light to dark blue). For each depth, 100 subsamples were produced. Within each subsample series, an increasing number of subsamples (x-axis) were randomly selected and their cumulative clonotype richness was calculated relative to the original dataset richness (clonotype richness coverage).
Between 7 objects removed from the final list, 3 of them are defined as radio-loud quasars (QSO) or blazars, with catalogued L-band 1. 4 GHz radio emission larger than 1 mJy (5. 04 mJy, 799. 97 mJy and 6. 2 mJy respectively), and their redshifts span the range from 0. 6 to 0. 965. An example is the brightest radio source catalogued as HVS387 in our catalogue or J122222. 5+041316 in the FIRST database. It has IR excess and very strong UV and gamma-ray emission. Source has a spectroscopic redshift of z = 0. 964 and according to the classical PSF-fitting photometry performed with e. g SUSSEXtractor or DAOphot it could passes our red source criteria, but it cannot be fitted accurately using the MBB-fitter. Since they may have colours similar to those of high-z DSFGs, heterogeneously selected "500 µm-risers" that exist in the literature thus should be considered with caution, since they may be polluted by synchrotron-dominated sources whose IR energy is not star-formation related.
Acknowledgements. We are thankful to the anonymous referee for very constructive comments and points which significantly improved this paper. We would like to acknowledge  David Corre ,  Denis Burgarella ,  Alessandro Boselli ,  Tom Bakx  and  Eric Jullo  for useful discussions. The project leading to this publication has received funding from  Excellence Initiative of Aix-Marseille University  -A*MIDEX, a  French "Investissements d'Avenir" programme  ( ANR-11-IDEX-0001-02 ) and from the  OCEVU Labex  ( ANR-11-LABX-0060 ). d'Avenir"  French government program  managed by the  ANR. C. P. acknowledges support from the  Science and Technology Foundation (FCT, Portugal)  through the  Postdoctoral Fellowship   SFRH/BPD/90559/2012 ,  PEst-OE/FIS/UI2751/2014 ,  PTDC/FIS-AST/2194/2012 , and through the support to the IA activity via the  UID/FIS/04434/2013  fund.
This publication makes use of public data products from the  Two Micron All Sky Survey  ( 2MASS ), which is a joint project of the  University of Massachusetts  and the  Infrared Processing and Analysis Center California Institute of Technology , funded by the  National Aeronautics and Space Administration  and the  National Science Foundation. We express are gratitude to Daft https://github. com/dfm/daft,a probabilistic graphical model available under the MIT License.
Neural approaches to data-to-text generation generally handle rare input items using either delexicalisation or a copy mechanism. We investigate the relative impact of these two methods on two datasets (E2E and WebNLG) and using two evaluation settings. We show (i) that rare items strongly impact performance; (ii) that combining delexicalisation and copying yields the strongest improvement; (iii) that copying underperforms for rare and unseen items and (iv) that the impact of these two mechanisms greatly varies depending on how the dataset is constructed and on how it is split into train, dev and test 1.
Introduction The input to data-to-text generation often contains rare items, i. e. low frequency items such as names, locations and dates. This makes it difficult for neural models to predict their verbalisation. To address these issues, neural approaches to datato-text generation typically resort either to delexicalisation  (Wen et al. , 2015; Dušek and Jurcicek, 2015; Trisedya et al. , 2018; Chen et al. , 2018)  or to a copy mechanism  (Chen, 2018; Elder et al. , 2018; Gehrmann et al. , 2018). Character-based encodings  (Agarwal and Dymetman, 2017; Deriu and Cieliebak, 2018)  and byte pair encodings have also been used  (Elder, 2017; Zhang et al. , 2018). However, when applying a character-based approach within a standard sequence-to-sequence model to the WebNLG and E2E datasets, the results were low.
, 2017; Gu et al. , 2016; Cheng and Lapata, 2016) , paraphrasing  (Cao et al. , 2017) , answer generation  (He et al. , 2017)  and data-to-text generation  (Gehrmann et al. , 2018; Chen et al. , 2018)  is a generic technique which is easy to integrate in the encoder-decoder framework and can be used independently of the particular domain and application. In this paper, we investigate the impact of copying and delexicalisation on the quality of generated texts using two sequence-to-sequence models with attention: one using the copy and coverage mechanism of  See et al. (2017) , the other using delexicalisation. We evaluate their respective output on two data-to-text datasets, namely the E2E  (Novikova et al. , 2017)  and the WebNLG  (Gardent et al. , 2017a)  datasets. We also compare the two methods in two different settings: the original train/dev/test partition produced by the E2E and by the WebNLG challenge vs.
Datasets Two recently released corpora for data-to-text generation served as experimental datasets for our study: the E2E  (Novikova et al. , 2017)  and the WebNLG  (Gardent et al. , 2017a)  datasets. In the E2E dataset, the input to generation is a dialogue act consisting of three to eight slot-value pairs describing a restaurant, while the output is a restaurant recommendation verbalising this input. Table  1  shows an example with an input consisting of six slot-value pairs. In average, each input is associated with 8. 1 references. The number of possible values for each slot ranges from two (binary slots) to 34 (restaurant name). Tables  2  and 3  summarise the statistics of the E2E dataset. In WebNLG, the aim is to verbalise a set of RDF (Resource Description Framework) triples describing entities of different categories.
An RDF triple is of the form (subject, property, object) where subject and object denotes entities or values and property denotes a binary relation holding between subject and object. The inputs consist of sets of (one to seven) triples and the entities belong to fifteen distinct DBpedia categories 2. Both dataset releases gave rise to a shared task in NLG in 2017 3. Note though that for WebNLG, the present study relies on the final release data (version 2) 4  , which is a larger dataset than that used for the WebNLG Challenge 2017.
Delexicalising Datasets We derive delexicalised datasets from the original E2E and WebNLG datasets as follows. For each dataset, we replicated the delexicalisation procedure which was applied to the baseline systems developed for the E2E  (Novikova et al. , 2017)    they have a large number of distinct values). In WebNLG, delexicalisation was done on the subjects and objects of RDF triples. While delexicalisation was flawless in E2E, WebNLG data poses additional challenges as the subject and object values in the input do not necessarily match the corresponding text fragment in the output. As a result, not all subjects and objects were delexicalised. In the delexicalised E2E corpus, placeholders constitute 5. 7% of all tokens, while they reach 15. 7% in the WebNLG data.
Constraining Datasets The train/dev/test split is often constrained to ensure that there is no overlap in terms of input between the training, the development and the test set. As  Aharoni and Goldberg (2018)  recently showed however, this may result in a setup where certain input fragments (in that case, subject and object entities present in the input set of RDF triples) are present so often in the test set that models built on this standard split, overfit and memorise rather than learn. Thus, in the split-andrephrase application they studied,  Aharoni and Goldberg (2018)  observed that, given some input containing the entity e and some set of facts T (e) about this entity, the model will regularly output a text which mentions e but is unrelated to the set of facts T (e). That is, instead of learning to generate text from data, the model learns to associate a text with an entity.
To better assess the impact of delexicalisation and copying on the output of data-to-text generation models, we therefore consider two ways of partitioning the corpus into train, dev and test: the traditional way (Unconstrained) where the overlapping constraint applies to entire inputs (i. e. , sets of RDF triples in WebNLG and dialogue moves in E2E) and a more challenging split (Constrained) where the no-overlap constraint applies to input fragments (i. e. , RDF triples in WebNLG and slotvalues in E2E). Table  3  shows the statistics for both splits for each dataset. Unconstrained The unconstrained split is the original split provided by the challenge organisers. The E2E dataset was split into training, validation and test sets following a 76. 5/8. 5/15 ratio. It was ensured that the input were distinct for all three sets and that a similar distribution of input and reference text lengths was kept.
We found 1,430 identical (data, text) pairs in the original E2E data. They were deleted for the subsequent experiments. In WebNLG, the original split follows an 80/10/10 ratio. As with the E2E dataset, there is a null intersection in terms of input between train, dev and test. In addition, sets of triples of different sizes and sets of triples of different categories were proportionally distributed between training, dev and test sets. Constrained We consider a second partitioning where we aim to minimise the overlap between train, dev and test in terms of input fragments. As shown in Table  2 , in the E2E dataset, most of the slots have under eight possible values. As these few values appear with a large number of distinct slot-value combinations (49,966 input-text instances), they are unlikely to trigger fact memorisation.
We therefore focus on those slots which have a higher number of values and restrict the test set using restaurant names, a slot with 34 possible values. Four restaurant names were selected to occur only in the test data and to support a distribution of inputs types and text length similar to that of the original train/dev/test (cf. Table  3 ). Nonetheless, it is worth noting that the E2E dataset was constructed in such a way that a specific restaurant name may have mutually exclusive values in different inputs, such as low customer rating and high customer rating. This might result in weak association between restaurant names and specific inputs and therefore, in little risk of memorising facts related to a specific restaurant name. As we shall see in Section 3, this intuition is confirmed by the results which show little differences, for the E2E data, in terms of both automatic and human-based metrics between the Constrained and the Unconstrained setting.
Note also that since the E2E Constrained split is defined with respect to a slot value (restaurant names) which is delexicalised, the constrained vs. unconstrained split distinction loses its impact in the delexicalised setting. For the WebNLG dataset, the constraint on the train/dev/test partition is in terms of triples. In addition to the exclusion from the test set all inputs (set of RDF triples) which occur in either the dev or the train set, we require that no RDF triple occurs in two of these sets. Let t = (s, p, o) be an RDF-triple, with p a property and s, o subject and object RDF resources. In the constrained dataset, if, t is in the test set, then t may not be in either the dev or the training set but variants such as (s , p, o), (s, p, o ), (s , p, o ) or (s, p , o) may (with s = s , p = p and o = o ).
Model Parameters We trained two types of models: a standard sequence-to-sequence model and the same model augmented with a copy and coverage mechanism (denoted as C in the tables). For the standard sequence-to-sequence model, we made use  of an LSTM encoder-decoder model with attention  (Luong et al. , 2015)  from the OpenNMT-py toolkit 6 , a PyTorch port of OpenNMT  (Klein et al. , 2017). The default parameters of OpenNMT-py were used for training and decoding. The encoder and decoder both have two layers. Models were trained for 13 epochs, with a mini-batch size of 64, a dropout rate of 0. 3, and a word embedding size of 500. They were optimised with SGD with a starting learning rate of 1. 0. Data was not lowercased, nor was it truncated (the maximal sequence length was used in the source and target).