dataesr/scientific-paragraphs-categorization huggingface dataset:
Before processing with shiver, short reads must be assembled into contigs. This important step, though difficult technically, is not onerous for the user: our chosen assembler IVA assembles contigs from reads with a single command from the command line, and can be run on a virtual machine provided by the Sanger pathogens group (http://sanger-pathogens. github. io/pathogensvm/). The user can use any assembler; others are available in the Genomic Virtual Laboratory, including SPAdes, Velvet and MIRA, though currently none designed specifically for viral data. shiver is run from the command line using three commands. Firstly, a one-off initialisation command: shiver_init. sh MyInitDir config. sh MyReferences. fasta \ MyAdapters. fasta MyPrimers. fasta (the slash indicating that one command is here being split over multiple lines), which sets up an initialisation directory of files for shiver based on the user's choice of existing references, and adapter and primer sequences to remove.
Example data and its processing by shiver We used two datasets as examples for processing with shiver. The first was sixty-eight publicly available Miseq samples: those sequenced and released with the IVA publication  (Hunt et al. 2015) , namely accession numbers ERR732065-ERR732132 on the European Nucleotide Archive. The samples have different origins; six are from a longitudinally sampled transmission pair studied by  Brener et al. 2015. ERR732065-ERR732072 were sequenced with 150 bp reads, ERR732073-ERR732132 with 250 bp reads. Only forty-two of these sixty-eight samples were assembled by  Hunt et al. 2015 : the rest failed quality control checks designed to pre-select robust whole-genome samples. After downloading the short reads from the European Nucleotide Archive, we reassembled all sixty-eight samples with IVA for processing with shiver, as by design our method can be run in exactly the same way for those samples devoid of genuine sequence, those with partial genomes and those with whole genomes.
The second dataset was fifty Hiseq samples newly generated for the BEEHIVE project, from confirmed seroconverters from Europe. RNA was extracted manually from blood samples following the procedure of  Cornelissen et al. 2016. This was amplified using universal primers that define four overlapping amplicons spanning the whole genome, following the procedure of  Gall et al. 2012. Specifically, 5 ll of Amplicon 1 (the shortest and most successfully amplified amplicon) was pooled with 10 ll each of Amplicons 2-4. Multiple samples were pooled during library preparation, using one of 192 multiplex adaptors for each sample. The library was sequenced in 'rapid run mode' on both lanes of a HiSeq2500 instrument with read lengths of 2 Â 250 bp, resulting in two lanes of short reads per sample. Automatic processing at the Wellcome Trust Sanger Institute used IVA to generate contigs for each lane, i.
e. two sets of contigs per sample. We combined the two sets to allow comparison of the assembly output resulting from two technical replicates of short reads. For the large majority of cases the contigs were nearly identical, but stochastic differences in the read populations between lanes mean the resulting contigs occasionally differ. The fifty Hiseq samples were chosen from a larger dataset currently being collected and sequenced for the BEEHIVE project's primary aim of investigating the viral-molecular basis of virulence. Selection criteria for inclusion in the project include a known date of infection, either by negative and positive tests separated by less than a year, or by clinical signs of acute infection at diagnosis; and a sample obtained for sequencing between 6 and 24 months after diagnosis, before beginning antiretroviral treatment and before progression to AIDS.
The fifty samples processed here were chosen as follows. (1) One sample chosen with a large difference in the fraction of the genome assembled between the two Hiseq lanes, as an example of the variability of assembly output. (2) Nine samples chosen with misassembled contigs for one or both Hiseq lanes, to illustrate the necessity of shiver's contig correction. (3) From each of the Dutch, French, German and Swiss cohorts, ten samples with contigs spanning the whole genome: five subtype B and five non-B samples (subtype was determined with the COMET software  (Struck et al. 2014) ). The existing reference set we used was the 2016 'compendium' group M genome alignment from the Los Alamos National Laboratory, with a small amount of sequence trimmed from both edges of the alignment to match the region of the genome amplified by the sequencing protocol used for all data here  (Gall et al.
2012) , which partially excludes the flanking long terminal repeat regions. For comparison with shiver's constructed mapping reference, for each sample we used kallisto  (Bray et al. 2016 ) to pseudo-align all the reads, using an index constructed from 3,249 whole genome references from the Los Alamos National Laboratory HIV database (those in the 2016 'all genome' alignment) together with the whole human genome (as an attractor for human contaminant reads). We defined the closest existing/ real reference sequence for that sample as the one with the highest transcript per million score. For this analysis, we set the minimum coverage threshold (the number of mapped reads required to call the base at each position) to be 10 throughout, since the assembler we used-IVA-requires at least ten reads to extend a contig, and we compare the consensus to the contigs.
To illustrate application of shiver outside of HIV, we used it to process Illumina paired reads from a whole-genome Hepatitis C Virus (HCV) sample: accession number DRR000928 on the European Nucleotide Archive. We assembled the reads into contigs using SPAdes  (Bankevich et al. 2012) , and for the existing reference set required as shiver input we used the 2008 'all genome' alignment of 471 references from the Los Alamos National Laboratory HCV database  (Kuiken et al. 2005). We also ran shiver on Illumina paired reads from a whole-genome Respiratory Syncytial Virus (RSV) sample: accession number ERR438932 on the European Nucleotide Archive. We assembled the reads into contigs using SPAdes, and for the existing reference set we used the sixty-three whole genomes sequenced by  Bose et al. 2015  from four continents to help capture global RSV diversity.
g. a median of thirty-one bases called differently with higher coverage in the shiver consensus, and 0 bases called differently with higher coverage mapping to the real reference. This illustrates that problematic contigs do not mean that mapping to an existing reference becomes preferable, thanks to shiver's contig correction. The IVA QC failures are mostly partial genomes; statistics for these samples are scaled down from their values for the whole data set due to these being shorter sequences. An exception is the increase in the consensus sequence length over the length of the contigs, whose median value is 0 for the whole dataset but thirty-two for the QC failures. It is not surprising that contigs should be shorter than mapping-derived consensuses for problematic samples previously excluded from consideration for assembly.
2. Though the coverage here drops due to the problem aligning the reads, it is still more than 4,000, showing that a large absolute number of reads is no guarantee of accuracy. Mapping to the shiver reference on the other hand, coverage remains locally smooth. Similar errors mapping to the real reference in this figure can be seen in gag and in five different places in gp120. obtain each sample's sequence as accurately as possible, so that small numbers of differences between similar samples can be meaningfully interpreted. The problems arising from mapping to a reference that differs from the sample in question do not arise simply from an inappropriate choice of mapper. To illustrate this, for the Miseq dataset we also mapped the reads to their closest real reference sequence using BWA and bowtie in both its 'local' and 'end-to-end' modes (for both mappers we used their default settings except for specifying a maximum insert size of 2,000 for bowtie, retaining only properly paired reads as we did with smalt).
In this methods article we do not present conclusions drawn from analyses of sequences generated with shiver. Two such analyses published so far are by  Blanquart et al. 2017 and Ratmann et al. 2017. Blanquart et al. 2017  determined that the fraction of variability in HIV set-point viral load that is explained by viral genetic factors was around one third, using 1,373 European whole genomes. Ratmann et al. 2017  found predictors of HIV sequencing success or failure for 3,985 African whole genomes, and studied the effect of the observed amplification failure patterns on phylogenetic inference. Analysis of both genomic datasets is ongoing.
Acknowledgements Thanks to  Martin Hunt  and  Dan Frampton  for helpful discussions and to  Simon Burbidge  and  Matt Harvey  for help with  Imperial College London High Performance Cluster computing. This work was funded by  ERC Advanced Grant  ( PBDR-339251 ). This work used the computing resources of the UK MEDical BIOinformatics partnership-aggregation, integration, visualization, and analysis of large, complex data (UK MED-BIO), which is supported by the  Medical Research Council  [grant number  MR/L01632X/1 ].
Agri-food research calls for changes in the practices dealing with data collection, collation, processing and analytics, and publishing thus to fully benefit from and contribute to the Open Science movement. One of the major issues faced by the agri-food researchers is the fragmentation of the "assets" that can be exploited when performing research tasks, e. g. data of interest are heterogeneous and scattered across several repositories, the tools exploited by modellers are diverse and often rely on local computing environments, the publishing practices are various and rarely aim at making available the "whole story" with datasets, processes, workflows. This paper presents the AGINFRA+ endeavour to overcome these limitations by providing researchers in three designated communities with Virtual Research Environments facilitating the access to and use of the "assets" of interest and promote collaboration.
The platform enacts tasks execution by a distributed and hybrid computing infrastructure including EGI resources; (v) a catalogue-based publishing platform to make the existence of a certain artefact "public" according to FAIR principles  [16] ; (vi) a Jupyter notebook based environment for documenting and recording analytics processes  [19] ; (vii) a scholarly publishing platform integrated with Pensoft infrastructure to enact the creation of innovative papers including datasets and methods hosted by AGINFRA; (viii) a platform for data visualisation enabling to create smart graphs and share them. An overall picture of the system architecture is given in Fig. 1.
This paper is organized as follows: In Section 2, we describe the HARMONI simulator HSIM, which is used to carry out the simulations of expected performance. Section 3 explains how the input data cube is created. The actual simulation runs and the extraction of the spectra are explained in Section 4. Section 5 provides details of the analysis, and Section 6 summarizes the key conclusions of this work. will have a wavelength coverage from 0. 5 to 2. 45 µm and will be able to take full advantage of both the E-ELT's large collecting area and angular resolution to achieve a wide range of scientific goals. Four spaxel scale configurations have been chosen: 4 × 4, 10 × 10, 20 × 20, and 30 × 60 mas, and three resolving powers R (=λ/ λ) of 3000, 7000, and 20 000. HARMONI will work in conjunction with two adaptive optics systems: LTAO [high-order correction using laser guide stars (LGS) over most of the sky], Single Conjugate Adaptive Optics [high-order correction using natural guide stars (NGS) in the vicinity of the bright reference star], or with no AO at all.
As the design phase is still ongoing, instrument simulations are vital to understanding the capabilities and limitations of the instrument design in achieving certain science goals. The HARMONI Simulator HSIM  (Zieleniewski et al. 2015 , source code: https://github. com/szieleniewski/HSIM) is a dedicated pipeline for simulating observations with HARMONI. HSIM is a powerful tool for quantifying the predicted HARMONI performance to assess the feasibility of science programmes. Specifically, HSIM takes an input 3D data cube (x, y, λ), which contains the source flux, and then simulates an observation by adding sky, telescope, instrument, and detector effects. In this process, several parameters can be chosen for the simulated observations according to the science program. The result is a mock-observed data cube and by comparing it with the input data cube, users can determine how well the information would be recovered when observing with E-ELT and HARMONI.
S U P E R N OVA I A S I M U L AT I O N S I N S I D E A G A L A X Y The main goal of this project is to simulate SNe Ia inside a galaxy at z = 3 for different luminosities, which approximates different points on the light curve of the SN, and to determine what the E-ELT would be able to detect. This will yield a limiting magnitude, i. e. the magnitude beyond which the SN cannot be detected. In this study, the input galaxy data cube is taken from the output of a large-scale cosmological simulation. We use a galaxy at z = 3 from the RAMSES simulations  (Teyssier 2002). The particular galaxy is extracted from the z = 3 output of the NutFB simulation, described in detail by  Powell, Slyz & Devriendt (2011)  and  Kimm et al. (2011). The Nut suite is a collection of high-resolution cosmological simulations of a Milky Way-like galaxy forming at the intersection of three filaments in a cold dark matter cosmology.
The  (1 + z) factor is due to the bandpass stretching. To derive the luminosity distance at z = 3, we use the Ned Wright's Javascript Cosmology Calculator  (Wright 2006 ) with cosmological parameters as follows: M = 0. 258, = 0. 742, and H 0 = 72 km s -1 Mpc -1 , yielding d L = 25817. 9 Mpc. At z = 3, the observed apparent magnitude of the SNe can be computed using the rest-frame B-band absolute magnitudes listed in Table  1  and then redshifting using equation (1). In detail, the  known B-band magnitude is used to determine the flux scaling for the spectrum. The observed, redshifted spectrum, computed using equation (  1 ), is then used to determine the observed H-band magnitude of the SN, which is compared with the integrated H-band light of the host galaxy. Thus, we get the ratio between luminosity of the SN and luminosity of the galaxy in H band (reported in Table  2 ).
Lastly, we use point spread functions (PSFs) simulating LTAO with six LGS to correct distortion from the atmosphere towards the galaxy. The PSF for each wavelength in this LTAO data cube is the result of a nine layers atmospheric  reconstruction and the seeing was fixed at 0. 64 arcsec, which means optimistic reconstruction and median seeing conditions (T. Fusco, private communication). For all simulations, HSIM returns a mock observed data cube and a background cube (among others). Fig. 5  shows the observed SN within the host galaxy at maximum light ∼10 spaxels to the right of the centre of the galaxy. The same view of resulting data cube at different times after maximum light is shown in Appendix A1. Output cubes have a size of (64, 64, 2805) with a spatial sampling of 20 × 20 mas and 3.
The SUPERFIT output is a list of matches depending on the goodness of fit to the observed SN spectrum. The SUPERFIT program has been tested on the simulated spectra of this study. In the same way than in the cross-correlation method, spectra of the SN and of the galaxy have been converted into energy units and divided by the transmission spectrum. Note that a galaxy spectrum has been provided separately to SUPERFIT, and together with the template, it is used to simultaneously fit the observed SN spectrum. The SUPERFIT best match is obtained for the spectrum template used in HSIM sim-ulations and the redshift z is estimated to be 3. 000 ± 0. 005. Fig. 9  displays the spectrum that corresponds to the best fit (black) to the observed spectrum (in blue) at maximum light. The true (noiseless) galaxy spectrum has been subtracted from the observed spectrum and the appropriate fraction from the SUPERFIT best fit.
was to determine whether (and for how long past maximum light) HARMONI could be used to reliably recover the SN spectral type and redshift. We find (using two different techniques) that HARMONI spectroscopic observations can reliably recover SN Ia classification for up to 80 d past maximum light in the observed frame (20 d in rest frame) at z ∼ 3, corresponding to a luminosity ratio (SN/galaxy) of 0. 024. The technique employing the SUPERFIT code also recovered the correct redshift of the SN, although spectra of the host galaxy (obtained simultaneously) could also be used to deduce the redshift with high accuracy. These simulations also imply that if the SN can be observed close to maximum light, spectroscopic characterization/confirmation of SN type could be performed out to redshifts of 7 or 8! A detailed study of the ability to robustly detect the SN type as a function of distance from the nucleus and host galaxy redshift is beyond the scope of this work.
What leads to classically recognized patterns of biodiversity remains an open and contested question. It remains unknown if observed patterns are generated by biological or non-biological mechanisms, or if we should expect the patterns to emerge in nonbiological systems. Here, we employ analogies between GNU/Linux operating systems (distros), a non-biological system, and biodiversity, and we look for a number of wellestablished ecological and evolutionary patterns in the Linux universe. We demonstrate that patterns of the Linux universe generally match macroecological patterns. Particularly, Linux distro commonness and rarity follow a skewed distribution with a clear excess of rare distros, we observed a power law mean-variance scaling of temporal fluctuation, but there is only a weak relationship between niche breadth (number of software packages) and commonness.
The diversity in the Linux universe also follows general macroevolutionary patterns: the number of phylogenetic lineages increases linearly through time, with clear per-species diversification and extinction slowdowns, something that has been indirectly estimated, but not directly observed in biology. Moreover, the composition of functional traits (software packages) exhibits significant phylogenetic signal. The emergence of macroecological patterns across Linux suggests that the patterns are produced independently of system identity, which points to the possibility of non-biological drivers of fundamental biodiversity patterns. At the same time, our study provides a step towards using Linux as a model system for exploring macroecological and macroevolutionary patterns.
Here, we propose structural analogies between biological diversity and the universe of open source GNU/Linux-based computer operating systems (hereafter Linux). We have chosen Linux, since its components are all openly available and well-documented, and because a large number of structural analogies can be drawn with the biological system of evolution, distribution and abundance of biological species, hereafter biodiversity (see Table  1  and Material and methods for details). Based on these analogies, we test the Linux universe for classical macroecological and macroevolutionary patterns and relationships, and we discuss the structural properties that likely generate common patterns in both biological and computer operating systems.
Biodiversity structures in Linux The development and distribution of GNU/Linux operating systems uses various open-source licenses (e. g. GNU GPL2, www. gnu. org), which allows the code to be freely copied and modified. This has led to the development of hundreds of operating systems, connected in a branching pattern of descent with modification. These different operating systems possess different applications, and are used with various degrees of popularity. We propose that each Linux-based operating system, commonly referred to as a distro (Table  1 ) can be viewed as a lineage or species  (Mens et al. 2014a, b). Our decision to give a Linux distro a distinct name, rather than calling it another version of an existing distro, was based on developers' subjective view that the distro is sufficiently different, which is similar to how many biological species have been defined to date.
The number of hardware devices on which a distro is installed can be interpreted as its commonness (e. g. population abundance or related, but not identical, range size). While popular distros can spread across devices, unpopular ones may go extinct. Thus, because distro abundances change through time, distros have population dynamics. Further, computer operating systems, and especially Linux distros, come with diverse functionalities, which are pre-installed in the form of software packages, and which can be made analogous to functional traits of biological species. Finally, new distros emerge through a process similar to biological speciation: When developers decide to come up with a new distro, part of the code of an ancestor distro is reused and combined with custom-tailored new code, and with existing open-source packages.
The evolution of distros is influenced by environmental factors such as hardware architecture and user requirements  (Yan et al. 2010) ; and constrained by user habits and the need for cost-effective development through reuse of code  (Myers 2003 , Fortuna et al. 2011) ; as a result, Linux distros have a genealogy which is potentially analogous to a phylogenetic tree. Once the qualitative structural analogies were set, we examined if the analogous structural elements of Linux follow the same quantitative patterns as biodiversity. During this process, we found that some biodiversity patterns were hard to imagine in the Linux realm; such as the latitudinal gradient of biodiversity, which would depend on a reasonable analogy to latitude. Hence, from the set of known biodiversity patterns, we chose those for which reasonable analogies can be made given available data on Linux.
Specifically, we looked for the following macroecological patterns: 1) the skewed species-abundance distributions, 2) power-law mean-variance scaling of population fluctuations, and 3) a positive relationship between niche breadth and range size. Between macroevolutionary patterns we looked for: 4) slowdowns of diversification rates over evolutionary time and 5) a phylogenetic signal in functional traits. We note that the analogies between Linux and biology (Table  1 ) are imperfect. For example, the evolution of Linux distros involves no genetic inheritance, there is little individual-level variation, and the inheritance is not from one user's computer to another, but is instead enforced centrally by developers. Thus, the micro-evolutionary mechanism is not Darwinian sensu stricto  (Lewontin 1970).
This is a problem that is widely discussed in the field of cultural evolution  (Mesoudi 2017) , and it is usually concluded that even imperfect analogies are useful. In our case the analogies work well at the macro-evolutionary level -there certainly is a selection by users, there is origination and extinction, and distros do change in time. As a result, a clear phylogeny emerges, with each tip of the phylogeny (distro) having a unique and inherited combination of traits. We can then argue that if biodiversity and Linux share similar categorization of objects, but they differ in their inner processes and mechanisms, and yet they display the same emerging patterns, then we have a case for a non-biological and non-mechanistic explanation of these patterns.