dataesr/scientific-paragraphs-categorization huggingface dataset:
Results Set-CK. Our performance study compared the automatically vectorized C++ code generated by Autovesk with the automatic vectorization techniques of scalar kernels produced by the GCC compiler, the Clang LLVM-based compiler, and the Intel icpx compiler. Figure  11  shows the execution times of the 10 kernels with different problem sizes for both the INTEL AVX512 and ARM-SVE configurations. The comparison is made between the vectorized code generated by Autovesk and the automatic vectorization of the GCC compiler on scalar codes. The results show that vectorized kernels provide a noticeable speedup over scalar kernels, as indicated by the speedup shown at some points. In Figure  12 , we provide a detailed breakdown of the number and types of graph nodes for the different kernels and problem sizes in our study.
The evaluation showed that the performance of the vectorized kernels using Autovesk is better than the compiler versions for most kernels. For the cases "(N,N) → N" and "(N,1) → N" using Intel icpx, see Figure  13b , we obtain similar performance, which we expect since the data accesses are contiguous and linear. However, Clang did not deliver the same performance, see Figure  13a. The average time to generate vectorized code for all kernels and sizes is 4. 12 seconds, with a minimum of 0. 001 seconds and a maximum of 179. 258 seconds. However, this phase is not fully optimized, so there is room for improvement. We expect that strategies such as a branch and cut technique could improve the efficiency of the process by pruning the research of the best coefficients.
LIMITATIONS AND FUTURE WORKS Our research has focused on automatically vectorizing unstructured static kernels with irregular data access patterns, but there are several limitations and room for future improvement. One area we intend to address is extending the applicability of Autovesk to cases where non-static loops can be expressed as repetitions of static ones. Indeed, non-static kernels can be divided into those that can be described by the polyhedral model, for which our approach is not beneficial and those that cannot. For the latter, there are non-static kernels that require alternative optimization methods, and those that can be viewed as repetitions of static kernels, where we aim to apply our method. Although our tool automatically generates vectorized code, at this stage, it still requires the user to provide input code in a specific format, since we use C++ templates to extract the graph of instructions.
To eliminate user intervention and extend the range of code that can be automatically vectorized, we plan to integrate Autovesk's transformations into an existing compiler. Alignment is not handled in the current implementation of Autovesk. Integrating our techniques into a compiler offers a potential way to address alignment issues in future work. In addition, we can investigate incorporating a cost model for each strategy in the graph transformations. By considering the potential benefits and overheads of vectorization, informed decisions can be made to avoid unnecessary vectorization and focus on cases where it will provide significant performance gains. These directions improve the usability and effectiveness of Autovesk in various applications.
ACKNOWLEDGMENTS This work used the Isambard 2 UK National Tier-2 HPC Service (http://gw4. ac. uk/isambard/) operated by GW4 and the  UK Met Office , which is an  EPSRC  project ( EP/T022078/1 ). We also used the PlaFRIM experimental testbed, supported by  Inria ,  CNRS (LABRI  and  IMB) ,  Université de Bordeaux ,  Bordeaux INP  and  Conseil Régional d'Aquitaine  (https://www. plafrim. fr).
Our work takes an opposite stance. For each job, previously scheduled based on its compute requirements, we want to allocate an optimal set of storage nodes and disks that will fit the job's storage and I/O needs without needing to change the way the application does I/Os. Allocating storage resources involves partitioning and sizing these resources to meet the needs of submitted jobs. In that field, models such as burst buffers have been studied  [4, 34, 18, 24]. This research is complementary to StorAlloc. The proposed techniques are the basis of storage-aware job scheduling algorithms that could be evaluated in our simulator. For instance, the Harmonia dynamic I/O scheduler  [24]  considers I/O interference in the allocation process. It targets burst-buffer systems and offers multiple scheduling policies (such as maximizing buffer efficiency and improving application bandwidth) which achieve better performances than vendor software for these resources.
General design The objective of StorAlloc is to provide a simple way to develop and evaluate storage-aware job scheduling algorithms targeting heterogeneous storage resources (any kind of disk-based storage can be described). Therefore, StorAlloc has been designed following the basic principles of a job scheduler, i. e. a middleware allowing clients to request resources available on a supercomputer. Extending from this architecture, we then added the ability to run it as a simulator, using a single code base.
Simulation A longer-term goal of StorAlloc is to provide a single code base for a storageaware job scheduler and its simulator. Therefore, we have designed our simulation server with a "component in the middle" approach. The core components run as if they were actually deployed on a real system except that, if the simulation mode is enabled, the requests are rerouted to the simulation server which stacks them until a specific message triggers the actual execution of the simulation. This mode of operation allows us to seamlessly switch from the theoretical case to real allocations if StorAlloc needs to be tested on an actual platform. Then, the simulation is unrolled and go through the scheduler, using a discrete event simulation (DES) model  [16]. During that phase, we replay the allocation decisions that were made by the scheduler onto the simulated storage.
Implementation details The simulator presented in this paper is implemented using Python3. Our messaging protocol relies on ZeroMQ, while the DES model used for the simulation comes from the SimPy library 1. These choices allow us to limit the size of the source code by relying on robust on-shelf implementations. Our simulation is nevertheless constrained by the use of DES instead of more complex and accurate model: the potential impact of allocations choices, for instance on the runtime of a job, cannot be taken into account due to the lack of any form of feedback loop. However we believe this limitation, which will be addressed in future works, is acceptable when experimenting with static traces and focusing on the storage system usage. The source code of StorAlloc can be found at https://github.
Simulation setup To simulate storage requests from clients representative of real applications, we used a dataset composed of one year (2020) of Darshan 2  logs on Theta, a 11. 7 PFlops Cray XC40 supercomputer at Argonne National Laboratory 3. We extracted from these traces jobs spending at least 10% of their run time doing IO, and reading or writing at least 10 GB of data. It resulted in about 24 000 jobs out of approximately 624 000 jobs, each one translating into a storage request in StorAlloc: the requested capacity is conservatively based on the maximum of either read or write volume while the allocation time uses the initial job duration. Using Darshan traces as input data allows us to work with multiple HPC platforms which offer datasets of traces generated with this broadly available library.
In order to have a good overview of what can be observed with our simulator, we have run 256 different simulation setups based on the settings presented in Table  3. The average simulation time is around 5m9s per run, in a range of [4m50s; 6m51s] on a single core of a Intel Xeon E5-2630 processor. This variability is due to the difference in complexity of the algorithms and the activation or not of the requeuing and splitting strategies. It has to be noted that total capacity, individual disk size and layout parameters have been chosen based on the study of our dataset, and allow us to simulate platforms which bracket a wide range of practical scenarios, from under-provisioned to over-provisioned storage resources. 1N1D serves as baseline.
Analysis We present here results plotted based on StorAlloc simulation data. From these figures, we can conclude on a right sizing of the intermediate storage architecture while we can compare the efficiency of the tested scheduling algorithms. For this analysis, platforms and algorithms have been chosen to reflect a variety of behaviors. Platform sizing In our dataset, the sum of all the storage capacities requested by clients, called sum_cap, reaches 1. 6 PB. In Figure  4 , we plot the percentage of this value achieved by each of the 256 runs of our simulation according to storage layouts and algorithms, grouped by platform capacity and split strategy. On the top row (no request split), only the 1N1D layout at 32TB and 64TB capacities reaches 100% of sum_cap. However this layout is merely a baseline which shouldn't be used, as it leads to a high concurrency and consequently a very low I/O bandwidth.
The other two algorithms would instead make a 32TB platform suitable, although with a very small margin before entirely filling up disks. Eventually, it is possible to quantify a potential underutilization, as seen for the 64TB platform where the best algorithms now never use more than 50% of max_disk_use. The worst-fit algorithm is specifically intended for maximising the use of a single disk from a single node, which explains that it still reaches 100% of max_disk_use for one of its disks. This first analysis shows that a 32TB intermediate storage tier can handle at least 99% of all the I/O intensive jobs in our dataset, as long as the requests are split into 200GB blocks. In that case, some of the targeted disks are used at their full capacity at least once, leaving little flexibility in case of a sudden overload, while the average disk utilization rate is however still very low (1.
As a result, we show that it is possible to compare our algorithms not only in terms of fairness but also regarding their allocation capabilities. This lets us make sure that aiming for the best fairness, or other comparable goals, doesn't unnecessarily affect the basic capabilities of our algorithms. In this situation, best-bandwidth and round-robin are once again better choices than the other two algorithms. Behavior in the face of more heterogeneity The previous observations led us to conclude that a possible solution to allocate all the requests in our dataset was to use a 32TB platform, in conjunction with the best-bandwidth algorithm and a split request strategy. Requeuing doesn't seem necessary in this case. Using these parameters, we now run a final simulation with a different platform that presents more heterogeneity than our other simulated systems.
In particular, we have shown that a 32TB burst-buffer partition on Theta, a top-tier supercomputer at Argonne National Laboratory, would be sufficient to absorb the I/O burden of the I/O intensive jobs running on the system. Eventually, we proposed to assess the reliability of one of our algorithms, best-bandwidth, capturing its behaviour through multiple metrics and a statistical analysis. In future work, we plan to carry out additional experimental campaigns using other datasets, along with new metrics, infrastructures and storage-aware scheduling algorithms. We are also currently evaluating the benefits we can obtain from simulation frameworks such as Wrench  [9]  for the implementation of our simulation component. In that regard, we started to work on the necessary foundations for porting features from StorAlloc to Wrench.
sands of genetic markers introduces the challenge of obtaining large sets of shared loci with similar coverage across multiple individuals. Here, we present an approach to handling large-scale gNIS-based datasets using data from the spotted hyena population inhabiting the Ngorongoro Crater in Tanzania. We generated 3RADseq data for more than a thousand individuals, mostly from faecal mucus samples collected noninvasively and varying in DNA degradation and contamination level. Using small-scale sequencing, we screened samples for endogenous DNA content, removed highly contaminated samples, confirmed overlap fragment length between libraries, and balanced individual representation in a sequencing pool. We evaluated the impact of (1) DNA degradation and contamination of non-invasive samples, (2) PCR duplicates and (3) different SNP filters on genotype accuracy based on Mendelian error estimated for parent-offspring trio datasets.
, 2008). However, artefacts generated during library construction can introduce bias into downstream analyses if not carefully treated  (Mastretta-Yanes et al. , 2015). For example, PCR duplicates, which stem from a random allele at a given locus being amplified more than the other allele, result in a spurious inflation of homozygosity and a false increase in variant call confidence  (Andrews et al. , 2016; Flanagan & Jones, 2018). Discrepant fragment-size range among libraries and variable coverage across loci and individuals in a given dataset strongly influence the number of SNPs and genotyping confidence  (DaCosta & Sorenson, 2014; Driller et al. , 2021). Furthermore, different bioinformatic processing approaches for loci building and SNP calling, as well as SNP filtering strategies, can strongly impact results, potentially creating false patterns and leading to incorrect biological interpretations  (O'Leary et al.
, 2018). Moreover, any genomic method requiring individual pooling presents the challenge of obtaining an even read-depth distribution across individuals, which is not straightforward  (Maroso et al. , 2018) , especially in large datasets. Readdepth inconsistency can impact the missing rate among individuals and genotyping accuracy, ultimately impacting the estimation of population genomic parameters  (Davey et al. , 2013). All of these challenges in library construction and data analysis highlight the need to develop new strategies to overcome these issues. Here, we present a novel strategy for the improved handling of large-scale gNIS-based datasets using non-invasive samples collected from spotted hyenas (Crocuta crocuta) inhabiting the Ngorongoro Crater in Tanzania. We generated 3RADseq data  (Bayona-Vásquez et al.
| Library construction We used the method 3RADseq following the protocols described by  Bayona-Vásquez et al. , 2019  with a modification in one primer to include unique template molecule tags  (Hoffberg et al. , 2016). We chose the 3RADseq method because of two main features: (1) the use of the third enzyme increases the adapter ligation efficiency by cutting adapter dimers and making them available for further sample DNA ligation, which allows for the use of low quantity and quality DNA samples  (Bayona-Vásquez et al. , 2019) ; and (2) the inclusion of the iTru5-8N primer, which incorporates 8 degenerate bases in the P5 adapter and permits the identification and removal of PCR duplicates during preprocessing of the sequence data. We simulated the digestion of the Crocuta crocuta genome (GenBank accession number GCA_008692635.
1) with different restriction enzyme combinations in order to choose a set of enzymes that would provide us with approximately 25,000 loci. We used a dedicated Python script (RAD_digestion_v2. 0. py) that performs in silico digestion based on selected restriction enzymes (in ddRAD or 3RAD mode) and generates double-digested DNA sequences and their length distribution. We selected the combination of EcoRI, XbaI and NheI, which allowed us to sequence around 23,500 loci in a range between 380 and 460 bp. We started the library preparation with an initial DNA amount per sample of 20, 50, 70 and 100 ng, depending on the available material, in a volume of 10 μL. The three restriction enzymes (10 units of each enzyme), NEB 1× CutSmart Buffer and a unique combination of dual-internal barcodes (5 μM) were added to each sample separately for the DNA digestion reaction.
Importantly, all the pipetting steps for normalization of the DNA concentration, DNA digestion, adapter ligation and pooling were performed with the Beckman Coulter Biomek i7 automated Workstation. While not a requirement for implementing this method, a robot is useful to optimize these steps. Following sample pooling and cleaning, we performed fragment size selection for each library with the Blue Pippin using a 1. 5% cassette (Sage Science). After standardization, we set the size-selection range configured in the BluePippin software to 480-640 bp in order to obtain a fragment-size range of 390-450 bp. The difference in the configured and obtained size range refers to the adapter size (80 bp) and recurrent deviation error (tending to narrower and shorter size ranges) we often observe in our BluePippin equipment.
Note that the PCR reactions were done after the size selection. This is a strategy to overcome small fragment preferential amplification that can bias subsequent processing steps  (DaCosta & Sorenson, 2014). The final libraries were checked with an Agilent Tapestation using High Sensitivity DNA ScreenTape and sequenced using 300 bp paired-end reads on a MiSeq platform (Illumina), aiming to obtain 2000 reads per individual. This first small-scale sequencing aiming to obtain a low number of reads per individual is hereby referred to as a spike-in run. These data can be obtained cost-efficiently by running spike-ins in shared sequencing runs or using nano or microsequencing kits. The sequencing output is used to check the balance between the individuals, the length distribution of each library and the individual contamination level.
| Preprocessing analysis The 2 × 300 bp reads are submitted to a dedicated preprocessing pipeline automatized in a Snakemake workflow, as described in Figure  S3A. The first step involves phiX control library cleaning, as phiX is included in the sequencing run to increase sequence diversity and improve base reading quality. The raw data are mapped to the Enterobacteria phage phiX174 reference genome (NC_001422. 1) using Bowtie2  (Langmead & Salzberg, 2012)  with default parameters, and the unmapped reads are saved for the following processing steps. Next, we demultiplexed reads per individual using the software Flexbar  (Roehr et al. , 2017). PCR duplicates were then filtered from each individual, based on recognition of identical reads with the same iTru-8 N index sequence, using a custom Python script (filterPCRdups_CM.
py) that outputs only one copy of each unique DNA molecule. After this step, we concatenated the two replicates of each library and merged forward and reverse reads using PEAR v. 0. 9. 11  (Zhang et al. , 2014). At this point, the Snakemake pipeline generates a series of plots with the fragment length distribution of each individual. Then, we perform an in silico digestion, in order to recognize and filter out undigested and chimeric sequences. Next, we check restriction sites using a custom Python script (checkRe-strictionSites. py), keeping only reads with correct sequences at both ends in the output. Finally, quality filtering is performed using the software Trimmomatic  (Bolger et al. , 2014) , keeping only reads with a Phred score higher than 30.
Third, filtered spike-in reads allow us to check for contamination. We mapped reads to the C. crocuta genome (GenBank accession number GCA_008692635. 1) using Bowtie2 (default parameters). et al. , 2011). The percentage of reads mapping to the C. crocuta genome was thereafter considered to be the endogenous hyena DNA content. Individuals with less than 50% endogenous DNA content were discarded, due to the high effort (in terms of sequencing coverage and associated costs) needed to obtain good quality data from these samples. Once we define the samples that will continue in the analysis, we calculate the volumes for re-pooling considering the number of mapped reads from each sample. This accounts for the proportion of non-endogenous reads per sample, as well as varying sample proportions, in order to obtain a balanced library (i.
NovaSeq reads are preprocessed following the pipeline described before, except for the phiX filter (because no phiX was added to the high-output sequencing run) and the merging of reads (Figure  S3B ). As we now have shorter reads (150 bp), only short fragments (<270 bp) overlap (with minimum overlap length of 30 bp), thus merging allows us to filter out these short fragments and continue analysing the unassembled forward and reverse reads. Additionally, we used the software Cutadapt  (Martin, 2011)  to perform adapter trimming, aiming to filter out any remaining adapter sequences in the 3′-sequence ends. Preprocessed reads were mapped to the C. crocuta genome using Bowtie2 with default parameters and the options "-no-mixed" and "-no-discordant" to ensure unique paired-end alignments. We excluded the mitogenome and sex scaffolds using Samtools  (Danecek et al.
| SNP calling The Stacks reference-based pipeline v2. 61  (Catchen et al. , 2011; Rochette et al. , 2019)  was used to call variant sites and perform population filters, allowing only loci genotyped in at least 60% of individuals within at least one population to be retained in the final dataset (N = 761). This selection was achieved using the population program with parameters p = 1 and r =. 6. We subsampled the number of reads for individuals with greater than 2. 5 million reads (equivalent to roughly 60× coverage per site), in order to obtain even coverage among individuals. Further variant filtering was done using VCFtools  (Danecek et al. , 2011)  using constrained (minimum depth: 20, maximum depth: 100, missing rate: 0. 96, minimum allele frequency: 0. 05, unlinked SNPs) and permissive (minimum depth: 5 or 10, maximum depth: 100 or 110, missing rate: 0.
| Relatedness analysis The constrained set of SNPs was used to assign parentage among individuals. Pairwise relatedness values were calculated with the software Related  (Pew et al. , 2015) , and parent-offspring trio relationships were identified with the software Colony  (Jones & Wang, 2010). We identified 158 trios in our dataset, comprising 284 individuals, which were used to estimate genotype accuracy in subsequent analyses. We confirmed trio identification by comparing our results with trios previously identified based on microsatellite data  (Höner et al. , 2010).
| Genotype accuracy estimation based on parent-offspring trio datasets We estimated genotype accuracy using parent-offspring trio datasets, assuming correctly specified familial relationships. We identified genotype errors as any variant calls inconsistent with the rules of Mendelian inheritance, assuming that de novo mutation rates (~10 -8 per bp) are several orders of magnitude smaller than calling and genotyping error rates (often 10 -2 -10 -5 )  (Kómár & Kural, 2018). Using a dedicated python script (GenotypeAccu-racyEstimation. py), we classified each variable position in a VCF file as missing data, correct or incorrect genotype calling for each trio based on the offspring genotype assuming that the parents' genotypes were correct. Although genotype classification is defined based on the offspring genotype, the Mendelian error might have occurred in either the offspring or one or both parents.