dataesr/scientific-paragraphs-categorization huggingface dataset:
Representation of low-entropy data As mentioned earlier, Compressed Static Functions (CSF) of  [11]  do not properly deal with datasets generated by low-entropy distributions, in particular with entropy smaller than 1. This case occurs when the dataset has a dominant value representing a large fraction (say, more than a half) of all values. This is typically the case with genomic k-mer count data, especially whole-genome data, where a very large fraction of k-mers occur just once. For example, in E. Coli genome (≈5. 5Mbp), about 97% of all distinct 15-mers occur once, and only the remaining 3% of 15-mers occur more than once. For such datasets, the method of  [11]  does not approximate well the empirical entropy, as it cannot achieve less than 1 bit per key. Here we propose a technique to circumvent this deficiency in order to achieve, in combination with CSFs of  [11] , a compression close to the empirical entropy.
We conclude that in order to decide if a Bloom filter enables space saving, we have to check the value ε 0. If ε 0 ≥ 1, we do not need a Bloom filter, otherwise we need one with ε = ε 0. This shows that a Bloom filter is needed whenever α > C BF log e C CSF + C BF log e (5) For C BF = C CSF , this gives α > 0. 59. In the rest of the paper we use the term Bloom-enhanced Compressed Static Function, BCSF for short, to speak about CSF possibly augmented by a prior Bloom filter, as described in this section. Algorithm 1 summarizes the computation of the BCSF data structure.
Experimental results We report experiments on three datasets, two with lower and one with higher empirical entropy. The first is the k-mer counts computed on the Sakai strain of E. Coli from  [36]  (NCBI accession number B000007). The second one is a full genome of C. Elegans, strain Bristol N2 downloaded from RefSeq (accession number GCF_000002985. 6). The last one is the whole dataset from the same paper  [36] , hereafter referred to as "df", of k-mer "document frequencies" across 29 E. Coli genomes made of approximately 25 million k-mers. Here the document frequency of a k-mer is the number of genomes containing this k-mer. Experiments were performed on a machine equipped with an Intel ® Core ™ i7-4770k (Haswell), 8 GB of RAM and Kubuntu 18. 04. All construction code is written in python, except for the CSF part which is handled by a simple Java program using Sux4J  [11].
Time measurements are performed by a program written in C using the code provided by Sux4J for reading and querying its CSFs. We use xxHash 1 as µ m (q) to define an ordering over the minimizers of a given k-mer q. All code is available at https://github. com/yhhshb/locom. git. We only report the best methods for each case with the following naming convention: CSF: baseline CSF implementation from Sux4J. BCSF: extended CSF with Bloom Filter from Section 3. It may get reduced to a simple CSF if the Bloom Filter is not useful. FIL m 1 k: our first implementation, saving into each bucket a majority-selected representative and saving corrections into its second layer. FIL m 1 m 2 k: same as before but with an additional layer. AMB m 1 k: our second implementation, selecting each representative by minimum and marking colliding buckets with a special value.
Compression of higher entropy data With very skewed data, collisions of k-mer counts may happen between unrelated k-mers simply because one counter value strongly dominates the spectrum. In order to demonstrate that minimizers are useful as well for less skewed distributions than whole genome count tables, we applied our methods to the df dataset, see Figure  3. The use of minimizers for larger k's, proves to be beneficial again, with AMB and FIL requiring much less space that the empirical entropy of the data. A similar scenario to the previous case represents itself for relatively small k (k = 13), for which both AMB and FIL do not have an advantage over a simpler (B)CSF. For even smaller k-mers (B)CSF remains the best option (results not shown). The seemingly erroneous exceptions (BCSF taking more space than simple CSF) are explained by the approximation carried by formula (2) (assumption of equal values of C CSF in both sides).
Conclusions In this work, we introduced three data structures to represent compressed k-mer count tables. Our BCSF algorithm combines Compressed Static Functions, as implemented in Sux4J software  [11] , with Bloom Filters. This allows for a much better compression for skewed distributions with empirical entropy smaller than 1. Note that to our knowledge, this is the first time that CSFs are used in bioinformatic applications. We also provide a method to dimension the Bloom filter in a BCSF in order to minimise the final space. Our two other algorithms, FIL and AMB, pair BCSF with a bucketing procedure where count values are mapped into buckets according to minimizer values of respective k-mers. This locality-sensitive hashing scheme allows us to efficiently factor the space of counts, which leads to breaking the empirical entropy lower bound for large enough k's.
FIL and AMB use slightly different strategies in decomposing the input table across minimizer layers. Our last contribution is an extension of AMB to the approximate case, gaining more space at the expense of a small and user-definable absolute error on the retrieved counts. We validated our algorithms on three different datasets, two fully assembled genomes (E. Coli and C. Elegans), and one document frequency example, for different k-mer lengths showing how BCSF, AMB and FIL behave in different situations. FIL and AMB have a clear advantage when minimizers are long enough to bucket k-mers in a meaningful way, for both skewed and high entropy data. When it is not possible to define a long-enough minimizer length, the advantage of using intermediate minimizer layers vanishes, and simple CSF and its BCSF provide a better solution.
2 0 2 1 8:10 Compressed k-Mer Count Tables 1. 0 1. 2 1. 4 1. 34 0. 87 0. 87 0. 87 0. 90 0. 91 1. 23 1. 22 1. 22 H0 CSF BCSF AMB 2 layers AMB 3 layers FIL 2 layers FIL 3 layers bits / k-mer 0. 6 0. 8 0. 61 0. 2 0. 4 0. 24 0. 34 0. 26 0. 26 0. 29 0. 29 0. 19 0. 26 0. 11 0. 10 0. 13 0. 13 0. 18 0. 25 0. 08 0. 07 0. 10 0. 10 0. 0 13 15 k 18 21 1 https://github. com/Cyan4973/xxHashWA B I.
0 2 1 8:12 Compressed k-Mer Count Tables Compressed space usage for the high entropy df dataset. bits / k-mer 2 3 4 3. 95 4. 43 4. 43 4. 43 4. 43 4. 43 4. 43 3. 58 4. 07 4. 13 3. 23 3. 15 3. 63 4. 13 2. 25 2. 11 2. 59 2. 61 4. 10 3. 97 3. 48 1. 94 1. 80 2. 29 2. 29 4. 06 3. 89 3. 41 1. 84 1. 71 2. 18 2. 20 4. 00 3. 82 3. 36 H0 CSF BCSF AMB 2 layers AMB 3 layers FIL 2 layers FIL 3 layers 1 0 13 15 18 k 21 24 0 1 2 3 Figure 3 10 bits / k-mer 4 3. 88 4. 38 4. 38 4. 38 4. 38 2. 27 2. 61 11 2. 68 2. 67 2. 67 k 1. 20 1. 62 12 1. 61 1. 61 1. 61 0. 61 1. 34 13 0. 87 0. 87 0. 87 H0 CSF BCSF AMB 2 layers AMB 3 layers.
Space usage across all values of k, for the Sakai dataset. Additional figures bits / k-mer 2 3 4 3. 88 4. 38 4. 38 4. 38 4. 38 4. 41 4. 47 2. 27 2. 61 2. 68 2. 67 2. 67 2. 68 2. 68 1. 62 1. 61 1. 61 1. 61 1. 65 1. 67 H0 CSF BCSF AMB 2 layers AMB 3 layers FIL 2 layers FIL 3 layers 1. 20 1. 34 1. 23 1. 22 1. 22 1 0. 61 0. 87 0. 87 0. 87 0. 90 0. 91 0 10 11 12 13 k 0. 24 15 0. 34 0. 26 0. 26 0. 29 0. 29 0. 19 18 0. 26 0. 11 0. 10 0. 13 0. 13 0. 18 21 0. 25 0. 08 0. 07 0. 10 0. 10 bits / k-mer 2 3 4 5 2. 68 3. 07 3. 37 3. 28 3. 28 3. 46 3. 51 4. 46 5. 01 5. 01 5. 01 5. 01 5. 01 5. 16 4. 40 4. 94 4. 94 4. 94 4. 94 4. 94 4. 94 3. 95 4. 43 4. 43 4. 43 4. 43 4. 43 4. 43 3. 58 4. 07 4. 13 3. 23 3. 15 3. 63 4. 13 2. 25 2. 11 2. 59 2. 61 4. 10 3. 97 3. 48 1. 80 1. 94 2. 29 2. 29 4. 06 3. 89 3. 41 1. 84 1.
, 2021). For this assessment, density functions of reference RH are derived from the brightness temperatures measured by SAPHIR (Sondeur Atmosphérique du Profil d'Humidité Intertropicale par Radiométrie), the microwave sounder on board the Megha-Tropiques satellite orbiting over the tropical belt  (Roca et al. , 2015). This paper is divided into five sections. The datasets and the matching procedure between SAPHIR probabilistic relative humidity (RH) estimates and ARPEGE simulations are presented in Sect. 2. The probabilistic method is introduced and confronted with the deterministic comparison in Sect. 3. Section 4 discusses the results of the two comparison methods and the added value of the probabilistic method. Concluding remarks are then drawn in Sect. 5.
, 2017)  while also representing the uncertainty in the retrieval scheme and the radiometric noise. The beta model is used as follows: PDF FS (RH; α, β) = RH α-1 (1 -RH) β-1 1 0 u α-1 (1 -u) β-1 du , (α; β) > 0, (1) with PDF FS (RH) being the probability density function of RH defined on the interval [0, 1], and α and β are the parameters of the distribution. The subscript "FS" stand for "footprint scale". As detailed in Table  1  of  Brogniez et al. (2016) , the bulk standard errors in the dataset lie in the range of 3. 6 % RH-14. 8 % RH, depending on the pressure range (3. 6 % RH for layer 250-350 hPa, 15. 8 % RH for layer 750-800 hPa). These have been estimated using oceanic and continental radiosoundings co-located with satellite overpasses. Stevens et al. (2017)  also highlighted the role of the vertical inhomo-geneities in the discrepancies, strong gradients of moisture being the most difficult to capture by the passive sensors.
Uncertainty related to the assumption of Gaussian distributions In deterministic comparison settings, the uncertainty may be defined based on a priori assumptions, instrumental biases and retrieval errors. This uncertainty is often assumed to be multiplicative with respect to the reference value, and it is assumed that the underlying density function is unimodal, symmetric and follows a Gaussian model so that the uncertainty is defined as a standard deviation. However, a Gaussian model would not have been adapted to the dataset. A Shapiro-Wilk test is run with each and every individual PDF of the dataset (with α SW = 0. 01). The Shapiro-Wilk test is a widely used test of normality in statistics  (Shapiro and Wilk, 1965). Finding a p value above α SW would mean that the null hypothesis ("the PDF fits a normal distribution") cannot be refuted.
4c ) confirms the spatial correlation between the IQR and the classical patterns of the humidity field. This is particularly visible around the South Pacific and South Atlantic highs, where the proportion of IQR under the 15 % RH threshold reaches 70 % and even 100 %. In these subsiding areas, the atmospheric RH is ruled by largescale processes that have little to no instantaneous variability at the scale of our grid. This results in more homogeneous conditions within the same grid box, which explains the narrower distributions of the retrievals (i. e. , smaller IQR). The Intertropical Convergence Zone (ITCZ) appears through areas of low proportion (0 % to 30 % of the retrievals' dataset) of under 15 % RH IQR. High dynamics characterizing this zone result in smaller-scale processes that impact the RH field and result in heterogeneous conditions within the same grid box and larger IQR.
Most importantly, IQR varies across space and time, and it can be partly linked to the RH field and explained by large-and fine-scale processes. These highlight the need for a comparison method that exploits and takes into consideration the variability in the dataset and adapts the comparison to each situation. One can note that while these results vary significantly depending on the atmospheric layer, they are coherent with the expected RH field patterns. For example, in the upper two atmospheric layers (100-200 and 250-350 hPa), the homogeneous dryer conditions result in almost all retrieval distributions having IQR under 15 % RH. The two lower layers (750-800 and 850-950 hPa), closer to the ground, show strong ocean-continent contrasts. This contrast shows the difference in processes that depend on the surface, with extremely low frequencies of IQR under 15 % RH above the continents.
Distributions The two methods are applied to the entire dataset over the period from April to June 2018. The distributions of each method's results are represented as a histogram (Fig. 6a ) and rank histogram diagram, also known as a Talagrand diagram  (Hamill, 2001; Wilks, 2011; Kirstetter et al. , 2015; Fig. 6b). Note that while the Talagrand diagram is often used to assess ensemble forecasts by comparing a single reference to a distribution of forecast ensembles, its interpretation is similar when comparing a single simulated value to a reference distribution. This graphical method illustrates where the ARPEGE's RH mod falls in the distribution function CDF of RH obs. In the case of perfect forecasts, each quantile represents an equally likely scenario for the ARPEGE model. The perfect case is a flat rank histogram indicating that the RH obs probability distribution is well represented by RH mod.
The improved assessment with the probabilistic approach is demonstrated by comparing the insights obtained on ARPEGE with those from a deterministic method involving the difference RH mod -RH obs and a ±15 % RH confidence interval. Initial results highlight the inherent inaccuracy of solely using averaged references due to the important variability in spread and shape of the reference estimates. By computing the inter-quartile range (IQR) for the whole reference dataset, it was found that the spread of the PDFs varies significantly and is linked to the RH magnitude, with wider distributions in moist areas and narrower distributions in drier conditions. A deterministically set confidence interval is relevant to the variability in the spread to some extent only. This promotes a comparison method that quantifies more precisely the deviation of the simulated value irrespective of the reference distribution variability, spread and shape.
the dataset can be found in  Sivira et al. (2015)  and  Brogniez et al. (2016). Both papers are already listed in the references section. ARPEGE data could be made available upon request. Supplement. The supplement related to this article is available online at: https://doi. org/10. 5194/acp-22-3811-2022-supplement. Author contributions. The present work is the result of an original idea of HB and PEK, developed by CR under the supervision of HB, PEK and PC. HB provided and added her expertise on the SAPHIR dataset, PC on the ARPEGE dataset and PEK on the statistical method itself. All authors discussed the results and contributed to the final paper. Competing interests. The contact author has declared that neither they nor their co-authors have any competing interests. Disclaimer. Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Acknowledgements. We thank the  CNES  for its financial support through the  Megha-Tropiques  project and the national AERIS data center, which hosts the satellite data. Christophe Dufour (LAT-MOS/IPSL) contributed to the data processing. Computing resources of the ESPRI IPSL mesocenter were greatly appreciated. Pierre Kirstetter  acknowledges support from the  NASA   Global Precipitation Measurement Ground Validation program  under grant  NNX16AL23G. We would also like to thank  Joern Ungermann  and the anonymous referee, whose insightful comments helped us convey our work in the clearest way. Financial support. Financial support was partly brought by the  CNES French Space Agency  through the  Megha-Tropiques  project. Review statement. This paper was edited by  Martina Krämer  and reviewed by  Joern Ungermann  and one anonymous referee.
Code availability. The underlying codes were developed for research and are not suited for direct implementation. Their specificities may be shared and/or discussed upon request from the author and acknowledged colleagues. Data availability. SAPHIR data are available through the AERIS/ICARE ground segment of Megha-Tropiques (https://www. icare. univ-lille. fr/product-documentation/?product= SAPHIR-L2B-RH,  Sivira et al. , 2022). More information on.
We ultimately followed these patients longitudinally for up to 18 months, collecting rich bio-samples and a range of clinical and omic data on the genome, transcriptome, proteome, metabolome and autoantibodies. The RA-MAP project has generated an unparalleled range of data and insights into the molecular heterogeneity of RA phenotypes in peripheral blood, which can serve as a fundamental reference for analysis of the blood immunological landscape in RA and other IMIDs.
Subjects received Hepatitis B vaccination (20 micrograms Engerix B by IM injection at 0, 1 and 2 months), and were followed up for 8 visits (day -7, 0 (baseline), 1, 3, 7, 56, 57, 63). Clinical, lifestyle, demographics, and biological samples were collected at each visit. Blood samples for RNA extraction were taken at all 8 visits, with transcriptomics performed to the TACERA transcriptomic protocol on day -7, 0, 3, 56 and 63. Clinical data is summarised in Table  2. Informed, written consent was obtained from all study participants (study protocol is available on the RA-MAP figshare  16  ). patient biosampling, extraction, and biobanking. Biosamples, including blood, serum, plasma and urine were obtained from patients every 3 months for 18 months in the TACERA and Vaccine studies. The SOPs required that blood was drawn and stored locally for up to 30 minutes before transport to the local processing hub.
Multi-omic analysis of samples was performed at baseline and 6 months. All samples (n = 34,540) were deposited with UK Biocentre (Milton Keynes, UK) and are available to researchers on request to the RA-MAP sample access panel (https:// research. ncl. ac. uk/ra-map/). Detailed SOPs are available in the RA-MAP figshare  16. Some of the logistical processing challenges faced by the study are described elsewhere  12. Microarray mRNA sample analysis. mRNA transcriptome data analysis. To perform QC and exploratory data analysis (EDA), all mRNA sample data was imported to the R Limma package  17  as a combined matrix to enable direct comparison of cell subsets. Where the goal is to isolate effects within a particular cell type, single cell type processing may be more optimal. Background correction and quantile normalization were performed using the Limma neqc function  18  , based on methods described by Shi et al.
19. All probes with detection p-values of at most 0. 05 in at least 100 samples were removed. This ensured that all remaining probes were expressed in a minimum of 100 samples, or just under 10% of all profiles. Probes were aggregated to gene level using the Limma avereps function to further reduce dimensionality and increase reproducibility  20  , using a simple mean aggregator as implemented in the limma package. After pre-processing, filters and exclusions, 18,562 genes were present in the analysis ready expression matrix. Dimensionality reduction by Principal component analysis (PCA) was used to check for outliers and unsupervised clustering effects. 12 samples were flagged as outliers in the dataset, we removed these from subsequent analysis (See full EDA and outlier markdown documents on GitHub).
Small RNA sample and transcriptome data analysis. For Small RNA sequencing analysis, amplified small RNA was sequenced on an Illumina HtSeq. 2500 unit with single read flow cells to a depth of at least 10 million 50-bp reads per sample. TACERA samples were randomized across the analysis plates, with samples from same RA subject (baseline and 6-month time points) assigned within same plate. Adaptors were clipped off the reads using Trimmomatic 21 (version 0. 33) and then aligned to the GRCh38 genome using Bowtie-2 (version 2. 3. 0)  22. Next, alignments to miRNA reference were counted with the htseq-count function from HTSeq (version 0. 6. 1p2)  23 and the miRBase annotation release 22. 1. Prior to normalization, transcripts in the resulting count table were filtered to a mean count per million (CPM) of at least 2, and normalised using the EdgeR CalcNormFactors function  24.
Relative fluorescence unit (RFU) data were sequentially normalized for hybridization controls (internal standards per sample) to remove inter-run hybridization artifacts, median signal across all samples to remove other potential assay biases (assumes same total protein concentration across sample set), and calibration controls (common sample standards across analysis plates). The normalized RFU values were log 2 -transformed and then each analyte was independently 0-centered to the mean of the healthy subject cohort by shifting. 2 samples failed the vendor's QC standards for median normalization scale factors within range of 0. 4 to 2. 5 and were excluded from further analysis (both 6-month samples from the active disease group). Auto-antibody sample analysis. 501 serum samples were analysed from the TACERA cohort, comprising 265 baseline samples and 235 6-month follow-up samples.
In parallel, 44 baseline and 38 follow-up samples from Vaccine recipients were measured. All samples were distributed on 96-well assay plates applying a randomised block design (timepoint, age, gender, healthy, RA). A Luminex bead-based antigen array was produced (Protagen AG, Switzerland) to measure the autoantibody response against 192 human protein antigens. Antigens were selected based on literature data and autoantibody reactivity data of previous high-content profiling studies in RA and other rheumatic diseases. A subset of protein antigens (n = 46) were citrullinated in vitro using peptidyl arginine deiminase (PAD) to compare the autoantibody reactivity towards citrullinated and corresponding uncitrullinated antigens in early RA patients. Briefly, proteins were produced in Escherichia coli as His-tagged fusion proteins and purified by immobilised metal affinity chromatography.
Coupling of antigens to magnetic carboxylated colour-coded beads (MagPlex microspheres, Luminex Corporation, Austin, Texas) was performed according to manufacturers' protocols. Beads coupled with BSA, human IgG (hIgG), E. coli lysate and the eluate of vector only transformed E. coli were used as internal quality controls to evaluate the background reactivity, the measurement range or patient anti-E. coli reactivity, respectively. Finally, beads were combined and stored at 4-8 °C until use. An aliquot of the bead mix was incubated with the 1:100 diluted patient serum sample. Bound antibodies were measured following incubation with a secondary PE-labelled anti-human-IgG antibody in a FlexMap3D instrument (Luminex Corporation, Austin, Texas). The IgG reactivity values are given as median fluorescence intensity (MFI) and data of antigens fulfilling the minimum bead count criterion (>10 beads measured per bead ID) was used for data analysis.
To monitor the inter-assay coefficient of variation, three in-process control samples were measured in triplicate on each 96-well assay plate using the autoantibody MFI values of all measured antigens. The overall median inter-plate CV was 7. 7%. Evaluation of the control beads showed that the MFI values of control beads was as expected: The background reactivity towards BSA was 4 MFI, the reactivity to the E. coli lysate was 8 MFI, to the vector only eluate 6 MFI and to hIgG 22,000 MFI. The hIgG coupled bead was used to confirm the reactivity of the PE-conjugated detection antibody. To obtain reliable MFI values for data analysis the bead count statistics of the autoantibody data were evaluated. The median bead count of all samples was 167, with 0. 01% of all samples having a bead count <10 and 0.