dataesr/scientific-paragraphs-categorization huggingface dataset:
2 TB of storage per ensemble member. Complete output variable lists and sampling frequencies for each model can be found at https://www2. cesm. ucar. edu/models/ experiments/LENS/data-sets. We restrict our attention in this work to data from the atmospheric model component of CESM, which is the CAM. CAM output data for the CESM-LE simulations consists of 159 distinct variables, many of which are output at multiple frequencies: 136 have monthly output, 51 have daily output, and 25 have 6-hourly output (212 total variable outputs). Note that due to storage constraints, the 6-hourly data are only available during three time  periods: 1990-2005, 2026-2035, and 2071-2080.
Approach To provide climate scientists with the opportunity to determine whether the effects of lossy compression are detectable and to solicit community feedback, we first designed a blind evaluation study in the context of the CESM-LE project. By utilizing the CESM-LE project, we were able to question whether the effects of compression could be distinguished from model internal variability. Three new simulation runs were set up identically to the original 30, differing only in the unique perturbation to the initial atmospheric temperature field. We then contributed these three new additional ensemble members (labeled 31-33) to the CESM-LE project, first compressing and reconstructing the atmospheric data output from two of the new ensemble runs (31 and 33). By not specifying which of the new ensemble members (or how many) had been subject to compression, we were able to gather feedback from scientists in the climate community detailing which ensemble member(s) they believed to have been compressed and why.
In addition, we supplied several scientists with both the original and reconstructed data for ensemble members 31 and 33, allowing direct comparison of the two. Participants were recruited in a number of ways, including announcements at conferences, advertisement on the CESM-LE project web page, and direct e-mail to scientists working with CESM data. Participants in both the blind and not blind studies were specialists in their fields, and while all participants were aware that multiple scientists were participating in the study, their analyses were conducted independently. Because we did not specify how the data should be analyzed, participants studied aspects of the data relevant to their interests, and the analyses described are a mixture of mathematical and visual approaches. Note that if we determined that a particular analysis technique would provide more insight in a not blind context, then that scientist was given both the original and reconstructed data (e.
g. , the results in Sect. 5). The analyses in Sects. 4 and 5 were presented to give the reader a flavor of the types of post-processing analysis that occur in practice with CESM data as well as the concerns that different scientists may have when using a dataset that has undergone lossy compression. For this study, we chose the publicly available fpzip algorithm  (Lindstrom and Isenburg, 2006)  for lossy data compression, based on its superior performance on the climate data in  Baker et al. (2014). The fpzip algorithm is particularly attractive because it is fast at both compression and reconstruction, freely available, grid independent, and can be applied in both lossless and lossy mode. The fpzip method uses predictive coding, and its lossy mode is invoked by discarding a specified number of least significant bits before losslessly encoding the result, which results in a bounded relative error.
Note that if a variable is output at more than one temporal frequency, we do not assume that the same precision will be used across all output frequencies. Recall that the CAM time series data in CESM-LE contain single-precision (32 bit) output. While one could specify that fpzip retains any number of bits (up to 32), we restrict our choices to 16, 20, 24, 28, and 32, the latter of which is lossless for single-precision data. In  Baker et al. (2014) , the appropriate level of compression was chosen for each of the CAM variables in the dataset by selecting the most aggressive (lowest CR) such that a suite of four quality metrics all passed. The quality metrics in  Baker et al. (2014)  are largely based on evaluating the error in the reconstructed dataset in the context of an ensemble of simulations and test the variables for Z score, maximum pointwise error, bias, and correlation.
The ensemble distribution is intended to represent acceptable internal variability in the model, and the goal is that the error due to lossy compression should not be distinguishable from the model variability as represented by the ensemble distribution. Note that for some variables, the lossless variant of a compression algorithm was required to pass the suite of metrics. (In the case of fpzip, the lossless variant was required for less than 5 % of the variables. ) While many of the variables present in the CAM dataset in  Baker et al. (2014)  are also present in the CESM-LE dataset studied here, we did not necessarily use the same fpzip parameter settings for the variables common to both for several reasons. First, the data in  Baker et al. (2014)  were output as annual averages, which we would expect to be smoother (and easier to compress) than the 6hourly, daily, and monthly data from CESM-LE.
Climate characteristics We now describe an analysis aimed at determining whether the effects of the lossy compression could be distinguished from the internal variability inherent in the climate model as illustrated by the CESM-LE project ensemble member spread. The CESM-LE historical simulation  data are examined for ensemble members 2-33 (member 1 is excluded due to a technicality related to its different starting date). Multiple characteristics of interest across the ensemble are examined: surface temperature, top-of-the-atmosphere (TOA) model radiation, surface energy balance, precipitation and evaporation, and differenced temperature fields. The effects of compression are discernable in several characteristics.
Application to ensemble data For this study, we utilize POD to investigate whether lossy compression introduced any detectable artifacts that could indicate which ensemble member(s) of the new set 31-33 had been compressed and to determine whether any such artifacts were acceptable or not (i. e. , in terms of impact on the physics of the problem). We examined the monthly averaged output of four variables: Z3 (geopotential height above sea level), CCN3 (cloud condensation nuclei concentration), U (zonal wind), and FSDSC (clear-sky downwelling solar flux at surface). For each variable and for each ensemble member (1-33), POD was applied to a period of 25 years (300 time slices beginning with January 2006) to obtain the modes and the energy associated with each mode. This methodology enables the identification of any perturbations introduced by the compression method into the dynamics of the field.
Causal signatures The goal of causal discovery in this context is to identify potential cause-effect relationships from a dataset to better understand or discover the dynamic processes at work in a system. Causal discovery tools have been developed from probabilistic graphical models (e. g. , detailed in  Pearl, 1988 and Spirtes et al. , 2000) , which are a graphical representation of probable dependencies between variables in highdimensional space. In particular, causal discovery methods reveal more than simply correlation, but rather the patterns of information flow and interactions. To determine the flow of information, the initial assumption is made that every variable (graph node) is causally connected to every other vari-able. Then conditional independence tests (e. g. , testing for vanishing partial correlations) are used to disprove causal connections, resulting in a remaining "interaction map" of causal connections (that may or may not be given direction through additional techniques).
Such tools were initially applied in the fields of social sciences and economics, but have more recently been applied successfully to climate science data (e. g. ,  Chu et al. , 2005; Ebert-Uphoff and Deng, 2012a, b; Zerenner et al. , 2014). For example, for atmospheric data, one could imagine using causal discovery methods to understand large-scale atmospheric processes in terms of information flow around the earth. Of interest here is determining whether compressing the climate data in the CESM-LE dataset affected the flow of information. Using causal discovery for this purpose is proposed in  Hammerling et al. (2015) , where interaction maps were generated for both the original and reconstructed data. We call these interaction maps causal signatures. This type of analysis is unique to this compression study as it is aimed at inter-variable relationships.
Recall that the number of daily variables contained in the CESM-LE datasets is 51. To simplify the analysis, we created a subset of 15 daily variables. The subset was chosen such that only one variable was kept from each like-variable group. For example, eight of the 51 total daily variables report temperature in some form: at several defined pressure surfaces, at the surface, and at a near-surface reference height (TREFHT); therefore, we only include the temperature variable TREFHT in the subset. We then developed temporal interaction maps for the 15 daily variables that show interactions across different lag times between variables. We performed this analysis for several different temporal scales, i. e. , we identified separate signatures considering lag times between variables that are multiples of 1, 5, 10, 20, 30, or 60 days, in order to capture interactions for example on a daily (1 day) or monthly (30 days) scale.
Recall that these interaction maps are highlighting potential cause-effect relationships. Figure  18  contains the interaction map for the daily timescale (lag times are multiples of 1 day) for the original data for CESM-LE member 31, and the 15 variables are indicated in the ovals. Note that only the weak connection between SHFLX (surface sensible heat flux) and FSNTOA (net solar flux at top of atmosphere), which is indicated by a dotted line, is missing in the map corresponding to the reconstructed data. In general, the maps for all of the lagged times only indicated tiny differences between the initial and reconstructed datasets. This result indicates that compressing and reconstructing the climate data has not negatively impacted the flow of information in terms of detectable cause-effect relationships in the data.
AMWG diagnostics package The publicly available and popular AMWG (Atmosphere Working Group) Diagnostics Package (AMWG-DP) computes climatological means of CESM simulation data from CAM and produces plots and tables of the mean climate in a variety of formats. The AMWG-DP uses monthly output to evaluate climate characteristics such as seasonal cycles, intraseasonal variability, Madden-Julian Oscillation (MJO), El Niño-Southern Oscillation (ENSO), and the diurnal cycle. The AMWG-DP can be used to compare model simulation output of observational and reanalysis data or to compare output from two simulations. Therefore, comparing the compressed and reconstructed CESM-LE ensemble members via the AMWG-DP is a natural choice. Note that the AMWG-DP is available at https://www2. cesm. ucar. edu/working-groups/ amwg/amwg-diagnostics-package.
Monthly variables FSNT and FLNT were both compressed by fpzip-24. The AMWG-DP results for RESTOM agree with the findings in Sect. 4. 2. 2 that indicate that the compression error cannot be distinguished from the ensemble variability. How-By providing climate scientists with access to data that had undergone lossy compression, we received valuable feedback and insights into the practicalities of applying data compression to a large climate dataset. Here we summarize the underlying themes or lessons that we learned from this lossy compression evaluation activity.
Concluding remarks In general, lossy data compression can effectively reduce climate simulation data volumes without negatively impacting scientific conclusions. However, by providing climate researchers with access to a large dataset that had undergone compression (and soliciting feedback), we now better appreciate the complexity of this task. All of the lessons detailed in the previous section highlight the importance of being data and science aware when applying data compression and performing data analysis. To reap the most benefit in terms of achieving low compression ratios without introducing statistically significant data effects requires an understanding of the characteristics of the data, their science use, and the properties (i. e. , strengths and weaknesses) of the compression algorithm.
Code and data availability All data for the CESM-LE project (including compressed and reconstructed members 31 an 33) are available via the Earth System Grid (http://www. earthsystemgrid. org; Earth System Grid, 2016). Also see http://www. cesm. ucar. edu/ projects/community-projects/LENS/ for more detailed information on the CESM-LE data (Large Ensemble Community Project, 2016). The CESM software is available from http: //www. cesm. ucar. edu/models/current. html  (CESM, 2016) , and CESM-LE data were generated with the CAM5 configuration described in  Kay et al. (2015). The fpzip compression utility is available from http://computation. llnl. gov/projects/ floating-point-compression  (zfp & fpzip, 2016).
Acknowledgements. We thank  Adam Phillips  ( NCAR ) for his input on CVDP. We thank  Reto Knutti  ( ETH Zurich ) for his suggestions and ideas. We also thank  William Kaufman  ( Fairview HS ) for his work on Fig. 17. This research used computing resources provided by the  Climate Simulation Laboratory at NCAR's Computational and Information Systems Laboratory (CISL) , sponsored by the  National Science Foundation  and other agencies. Part of this work has been supported by the  ANR-DADA ,  LEFE-INSU-Multirisk, AMERISKA ,  A2C2 , and Extremoscope projects. Part of the work was done during  P. Naveau 's visit to IMAGe- NCAR in Boulder, CO, USA. Part of this work was performed under the auspices of the  U. S. Department of Energy  by  Lawrence Livermore National Laboratory  under contract  DE-AC52-07NA27344 , and this material is based upon work supported by the  U.
This work aims to model and to simulate the polyurethane foaming process. Models taking into account the two main chemical reactions of the formation of polyurethane, the exothermic effect of these reactions as well as the thermo-rheo-kinetic coupling characterizing this process are proposed and implemented in the software NOGRIDpoints based on a meshless method (Finite Pointset Method). A prediction of some acoustic foam characteristics is also proposed based on the results of the numerical simulation of the foaming process and semi-phenomenological models.
, 2015). To cope with artifacts that do not depend on wavelength, a free wavelength-independent scaling parameter A is introduced: α 2-param (λ, θ ) = A r diff (λ, θ )α diff (λ) + 1 -r diff (λ, θ ) α dir (λ). (4) This two-parameter model is more flexible but to avoid overfitting, we discard spectra that would result in A out of the range 0. 9-1. 1. The fit is performed with a nonlinear least squares method (provided by the Python scipy. optimize. leastsq function) to minimize the squared difference between the model and observed spectrum. The choice of wavelength range to perform the fit depends on the available observation. In our case, spectrometers with silicon sensor are used so only data at wavelengths shorter than 1050 nm present a sufficient signal/noise ratio. In addition, we restrict the fit to wavelengths longer than 700 nm because the sensitivity of the albedo to SSA is stronger in the near-infrared than in the visible, and impurities are known to affect visible wavelengths  (France et al.
However, some data at very high SZA are not rejected although we believe they should have been. For this reason, we let the criteria be conservative and even discarded acquisitions at SZA higher than 75 • in the following. Valid data show small diurnal variations between 32 and 35 m 2 kg -1 for head 1 and slightly larger between 31 and www. the-cryosphere. net/10/1297/2016/ The Cryosphere, 10, 1297-1316, 2016 Figure  12. Temporal variations of downwelling irradiance measured by the two heads, averaged every day between 12:00 and 14:00 LT and averaged between 500 and 600 nm for three summer seasons (the year indicated in the legend refers to the month of December). 40 m 2 kg -1 for head 2. Root mean squared variations are, respectively, 0. 9 and 2. 6 m 2 kg -1 , which is well under the 15 % target accuracy.
Acknowledgements. This study was supported by the  ANR  program  1-JS56-005-01  MONISNOW. The authors acknowledge the  French Polar Institute (IPEV)  for the financial and logistic support at Concordia station in Antarctica through the  NIVO program. This work has also been supported by a grant from  OSUG@2020  (investissement d'avenir - ANR10 LABX56 ). We thank the IPEV winter-over staff at Concordia for monitoring our snow instruments, including Autosolexs. Edited by:  R. Brown.
Sequence clustering is a common early step in amplicon-based microbial community analysis, when raw sequencing reads are clustered into operational taxonomic units (OTUs) to reduce the run time of subsequent analysis steps. Here, we evaluated the performance of recently released state-of-the-art open-source clustering software products, namely, OTUCLUST, Swarm, SUMACLUST, and SortMeRNA, against current principal options (UCLUST and USEARCH) in QIIME, hierarchical clustering methods in mothur, and USEARCH's most recent clustering algorithm, UPARSE. All the latest open-source tools showed promising results, reporting up to 60% fewer spurious OTUs than UCLUST, indicating that the underlying clustering algorithm can vastly reduce the number of these derived OTUs. Furthermore, we observed that stringent quality filtering, such as is done in UPARSE, can cause a significant underestimation of species abundance and diversity, leading to incorrect biological results.
sequence reads into operational taxonomic units (OTUs) based on sequence similarity, a process frequently referred to as OTU clustering or delineating. Sequencing costs are dropping faster than Moore's law  (4) , increasing the need for efficient and accurate OTU clustering software. QIIME  (5)  has been using UCLUST  (6)  as the default clustering method since UCLUST's publication (corresponding to QIIME version 1. 0. 0), due to its increase in performance over other popular tools, such as BLAST  (7) , DOTUR  (8) , or CD-HIT  (9) (10) (11). However, UCLUST and USEARCH are closed-source software (the 64-bit versions, which are needed to handle large datasets, require an expensive license, even for academic use) and have limited documentation (http://www. drive5. com/usearch/). Moreover, based on UCLUST documentation (http://www.
drive5. com/uclust/uclust_ userguide_1_1_579. pdf), the allegedly serial implementation is impractical for massive high-throughput sequencing data. More accurate, faster, community-accessible tools are needed to overcome these challenges. Within the previous 2 years, four new sequence-clustering tools have emerged: OTUCLUST from the Micca package  (12) , Swarm  (13, 14) , SUMACLUST (C. Mercier, F. Boyer, E. Kopylova, P. Taberlet, A. Bonin, and E. Coissac, submitted for publication), and SortMeRNA  (15). These tools include open-source implementation, and the latter three implement multilevel parallelization, providing excellent potential alternatives to UCLUST. In this study, we evaluated these new open-source tools and compared them against UCLUST and USEARCH, two commonly used options available in QIIME, UPARSE  (16) , the latest USEARCH amplicon analysis pipeline, and the three hierarchical clustering algorithms available in mothur  (17).
RESULTS Software description. OTU clustering can be performed in three different ways  (18) : closed reference, de novo, and open reference. In the closed-reference approach, the input sequences are clustered against a reference sequence database. In de novo clustering, the input sequences are grouped based on pairwise similarity among all sequences in the data set. The open-reference approach  (19)  begins by running a closed-reference step, which is followed by a de novo step that clusters the sequences that fail closed-reference assignment. Swarm  (13, 14 ) is a de novo clustering algorithm based on an unsupervised singlelinkage-clustering method that reduces the impact of clustering parameters on the resulting OTUs by avoiding arbitrary global clustering thresholds and input sequence ordering dependence.
Query sequences (e. g. , rRNA amplicons) are searched against a reference database, and an E value threshold is applied to evaluate the quality of resulting alignments. In SortMeRNA 2. 0, the reference sequence achieving the lowest E value when aligned with a query sequence is chosen as the OTU centroid for that query. In addition to passing the E value threshold, the query must also have sufficient percent identity and coverage (both set to 97% by default). Contrary to UCLUST, the run time of SortMeRNA is not affected by reducing these thresholds (e. g. , clustering at 60% identity). UCLUST and USEARCH (versions 5. 2 and 6. 1) are supported in QIIME (v1. 8. 0). Both tools can perform de novo, closed-reference, and open-reference (except for USEARCH 5. 2) clustering. In QIIME's implementation, USEARCH 5.
2 is executed via a pipeline closely shadowing otupipe  (6, 21)  to cluster OTUs, and USEARCH 6. 1 performs chimera checking in an external script. UPARSE  (16)  is the latest de novo amplicon analysis pipeline from USEARCH; it applies stringent quality filtering and length trimming to remove erroneous reads and implements a novel greedy algorithm that performs OTU clustering and chimera removal concurrently. Experimental design. Swarm 1. 2. 19, SUMACLUST 1. 0. 00, and SortMeRNA 2. 0 have been integrated into QIIME 1. 9. 0 and can be used through QIIME's three different OTU clustering commands  (18) : pick_closed_reference_otus. py, pick_de_novo_otus. py, and pick_open_reference_otus. py. A variety of datasets were chosen to evaluate the performance of these open-source OTU clustering approaches relative to QIIME's UCLUST/USEARCH-based OTU clustering approaches as well as UPARSE (see Table  1  for details).
Two 16S rRNA gene simulated datasets were generated as FASTQ files. The first one (sim_even) represents an even distribution of 1,076 species, randomly subsampled from the Greengenes 97% database and computationally amplified at the same depth (100 reads/amplicon) and length (150 bp) using PrimerProspector  (23)  for extracting the V4 region and the ART  (24)  simulator for amplification and sequencing simulation. The second data set (sim_ staggered) represents the same 1,076 species as the sim_even data set but amplified at different (random) species abundance levels. We used four different previously published mock community data sets: three 16S rRNA gene mock community data sets (Bokulich_2, Bokulich_3, and Bokulich_6) from Bokulich et al. (25)  and an 18S gene (mock_nematodes) data set from Porazinska et al.
UPARSE was run using the recommended settings with truncation lengths of 150 bp and 250 bp; similarly to OTUCLUST, runs were performed with a default quality score cutoff of 16 (labeled as UPARSE_q16) and additionally with a quality score cutoff of 3 (labeled as UPARSE_q3). Biological observation matrix (BIOM) format  (30)  tables were used as input to post clustering analyses. Taxonomy for reported OTUs was assigned using the RDP Classifier (31) against the 97% representative databases for Greengenes (32, 33) (version 13. 8) and Silva (34) (version 111) for all methods. Performance was evaluated using a variety of metrics, including the accuracy of OTU and taxonomic assignments, alpha diversity (within-sample diversity), beta diversity (between-sample diversity), and taxonomic correlation.