dataesr/scientific-paragraphs-categorization huggingface dataset:
Negative Apparatus In a negative apparatus, only witnesses with a reading diering from the lemma are explicitly mentioned, the <lem> element usually having no @wit. Witnesses whose siglum does not appear in the @wit of a <rdg> are assumed to bear the same text as the lemma. This type of apparatus is slightly quicker to encode, since there is no need to list all witnesses, but on the other end it gives fewer opportunities for automated verication: it is impossible to be sure that the editor added the reading of each witness, since unmentioned manuscripts are assumed by default to correspond to the lemma. In this case, the Critical Apparatus Toolbox can nevertheless highlight apparatus entries that mention a particular witness. Practically, this will highlight only apparatus entries where a witness's reading diers from the lemma.
Application Design The Critical Apparatus Toolbox is an online application built on a set of XSLT stylesheets served through PHP les, the output of which is made interactive thanks to Javascript and CSS. It makes use of some parts of TEI Boilerplate, most notably its web design. But despite the similar look and feel, the core functions are very dierent: all the parts of the TEI Boilerplate stylesheets pertaining to critical edition elements have been overridden. The XSLT stylesheets analyze the TEI XML edition, and determine its characteristics. 10 When they transform each <app> into <span> HTML elements, this information is used to assign one or more CSS classes to each <span>: all are assigned the "app" class; the ones not explicitly giving a text for each witness are assigned the class "incomplete," those giving more than one reading for the same witness are assigned the class "doubles," and so on.
A list of all the witnesses mentioned in the apparatus entry is also created by the XSLT and stored in the @title attribute of the HTML <span>. Those @class and @title values are used by the javascript functions to control the consistency of the encoding and highlight elements according to the user requests. The default TEI Boilerplate XSLT and CSS are used for all the TEI elements which are not directly related to critical editions. Within the Parallel Segmentation method, the latest evolutions of the TEI are implemented, like the possibility for <lem> or <rdg> to contain model. divLike and model. pLike elements. 11 In the future, keeping up with the developments and evolutions of the Critical Apparatus module will be a priority. We hope that the Critical Apparatus Toolbox will be able to adapt to these evolutions: since the functions of the interface are powered by Javascript, updating the XSLT should be enough to adapt to new rules or elements in the module.
Future Developments The beginning of the development of the Critical Apparatus Toolbox was a lonely endeavor, but the project has since beneted from the collaboration of Magdalena Turska 12 who wrote a prototype for the integration of the Toolbox into an oXygen framework. Decisive help was also found via a collaboration with the Erasmus SP+ DEMM program (Digital Edition of Medieval Manuscripts). 13 For the three years beginning in June 2015 DEMM is holding an annual hackathon event where the Critical Apparatus Toolbox is the base application that small, mixed teams of textual scholars and computer scientists try to enhance to meet their particular needs. 14 These events will play an important role in the future developments of the Toolbox, since they confront us directly with the real-life experience and needs of editors.
A Basic Web-publication Kit Another direction would be a feature similar to the "Web View" output system proposed in Martin Holmes's Image Markup Tool. 15 Even if the Critical Apparatus Toolbox is not a publication application, such an output would provide users with a ready-to-use static version of their edition, a set of les (HTML, CSS, Javascript, etc. ) that they could publish on their website or show in a demo session. While complex projects will always need a proper publication framework, this sort of lightweight publication output would provide a simple tool for basic self-publication. 16.
I am preparing a generic TEI-to-LaTeX and TEI-to-PDF conversion feature that will be implemented in the Critical Apparatus Toolbox. I chose LaTeX as an intermediary format because it oers all the desired options, thanks to the reledmac package especially designed for typesetting critical editions. 17 It is better suited to the specic needs of critical editions than XSL:FO. Another advantage of an intermediary le is that it leaves users the opportunity to edit the LaTeX code to obtain a better PDF result, which they might prefer over a modication of the XSLT templates, depending on their skillset. This feature, still a work-in-progress but well advanced, lets the user customize many parameters of the output through a graphical interface, without requiring any knowledge of LaTeX. When users need heavy customization of the default settings, they can easily override the templates transforming the TEI into LaTeX (although this requires some understanding of XSLT and LaTeX).
Y. Chen et al. :  Evaluating the performance of ORCHIDEE-CAN v1. 0 observations and long-term surface fluxes were used to calibrate the parameters in the sub-canopy radiation, turbulence, and resistance modules with an automatic tuning process. The multi-layer model was found to capture the dynamics of sub-canopy turbulence, temperature, and energy fluxes. The performance of the new multi-layer model was further compared against the existing single-layer model. Although the multi-layer model simulation results showed few or no improvements to both the nighttime energy balance and energy partitioning during winter compared with a single-layer model simulation, the increased model complexity does provide a more detailed description of the canopy micrometeorology of various forest types. The multi-layer model links to potential future environmental and ecological studies such as the assessment of in-canopy species vulnerability to climate change, the climate effects of disturbance intensities and frequencies, and the consequences of biogenic volatile organic compound (BVOC) emissions from the terrestrial ecosystem.
, 2016 ) that was integrated with vertically discrete reflectivity, photosynthesis, stomatal resistance, and carbon allocation schemes. This new design resulted in a new version of ORCHIDEE named ORCHIDEE-CAN (ORCHIDEE-CANopy, revision 2290)  (Naudts et al. , 2015). Despite its code including a multi-layer energy budget scheme  (Ryder et al. , 2016) , ORCHIDEE-CAN is currently applied using a single-layer energy budget, due to a lack of validated parameters for the multi-layer energy budget scheme. In  Ryder et al. (2016) , the model was developed and tested for a single site. In this study, we compiled a set of within-canopy and above-canopy measurements of energy, water, and CO 2 fluxes and used these data to parametrize and validate the new multi-layer energy budget scheme in the ORCHIDEE-CAN v1.
Observational data For this study forest sites were retained if the following data were available: (a) short but intensive campaigns taking flux and profile measurements within and/or below the tree canopy, and (b) multi-year monitoring of top-canopy fluxes. Through numerous regional projects such as CAR-BOEUROPE, AMERIFLUX, Fluxnet Canada, OZFLUX, ICOS, and NEON, and efforts such as FLUXNET  (Baldocchi and Wilson, 2001) , multiple year-long time series are now commonly available, especially for the temperate and boreal zones in Europe, Japan, Australia, and North America. Site selection was thus mostly limited by the availability of within-canopy and below-canopy measurements. Eight flux observation sites (Table  2 ) met the aforementioned criteria, and represented various climates from the Mediterranean to the boreal zone and different vegetation types including broad-leaved summer green, broad-leaved evergreen and needle-leaved evergreen.
Model optimization Optimization procedure Parametrizing the scaling coefficients and weighting factors enabled us to simultaneously improve the match between the simulated and observed sub-canopy micrometeorology, including temperature and specific humidity when available, and between the simulated and observed top-canopy heat fluxes (LE and H ). Within-canopy fluxes were also simulated but are not usually measured. The parametrization made use of an in-house optimization package called OR-CHIDAS (ORCHIDEE Data Assimilation Systems; http:// orchidas. lsce. ipsl. fr/). ORCHIDAS provides a range of numerical approaches for assimilating multiple data streams in ORCHIDEE. We used the maximum gradient approach to tune the parameters a 3 to a 10 , W br , and W sr for each study site independently.
-Experiment 1 (EXP1): single-layer scheme with a prescribed canopy The first experiment was run at the site level and made use of the default single-layer energy budget scheme. The energy budget scheme was driven by the observed climate forcing and the observed total LAI (Table  2 ). In this experiment, the vertical LAI profile was only used for the photosynthesis module in ORCHIDEE-CAN v1. 0. Note that vertical LAI profiles cannot be used by the single-layer scheme and the results are therefore limited to the top-canopy fluxes. This experiment was used as the reference simulation to document the performance of the single-layer approach. -Experiment 2 (EXP2): single-layer scheme with a simulated canopy The second experiment was identical to the first experiment, except that the LAI was now simulated by ORCHIDEE-CAN v1.
Code and data availability The code and the run environment are open source. Nevertheless, readers interested in running ORCHIDEE-CAN v1. 0 (revision 2754) are encouraged to contact the corresponding author for full details and the latest bug fixes. The ORCHIDEE-CAN branch is available via the follow web link: https://forge. ipsl. jussieu. fr/orchidee/browser/branches/ ORCHIDEE-DOFOCO/ORCHIDEE. The Supplement related to this article is available online at doi:10. 5194/gmd-9-2951-2016-supplement. Author contributions. Yiying Chen, James Ryder, and Sebastiaan Luyssaert developed the parametrization scheme. Yiying Chen, Sebastiaan Luyssaert, and Philippe Peylin designed the study and Yiying Chen wrote the manuscript with contributions from all co-authors. James Ryder, Matthew J. McGrath, Juliane Otto, Kim Naudts, Sebastiaan Luyssaert, and Aude Valade helped Yiying Chen with integrating the parametrization scheme for the multilayer energy budget in ORCHIDEE-CAN v1.
Acknowledgements. Yiying Chen ,  James Ryder ,  Matthew J. Mc-Grath ,  Juliane Otto ,  Kim Naudts  and  Sebastiaan Luyssaert  were funded through  ERC starting  grant  242564  (DOFOCO), and  Aude Valade  was funded through  ADEME (BiCaFF). We thank the two anonymous reviewers for their very helpful observations and suggestions. Edited by:  J. Kala  Reviewed by: two anonymous referees.
Introduction Estimation of haplotypes from genotypes, known as phasing, is a central part of the pipeline of many modern genetic analyses. Estimated haplotypes are important for many population genetics analyses1,2, but also form a central part of imputation algorithms that are routinely used in genome-wide association studies (GWAS) 3,4. The ability to phase large data sets is especially important in the context of biobanks that comprise hundreds of thousands of genotyped samples. For example, in May 2015 the UK Biobank (UKB) released genotypes from ~152,000 samples, and this will rise to ~500,000 in 2016. Other Biobanks have already collected large scale genetic datasets5 or are in the process of doing so6. The unprecedented scale of these datasets, and the depth of phenotype information, allows researchers studying many different phenotypes to make novel discoveries7.
HAPI-UR12 has been applied on datasets of up to ~60,000 samples with sub-quadratic scaling, but has reduced accuracy compared to SHAPEIT2. Other algorithms that compare all individuals to all others, will also have quadratic O(N 2 ) scaling11,13. Here we present SHAPEIT3, an extension to SHAPEIT2 that exhibits O(NlogN) scaling and results in very low switch error rates on large cohorts. In practice, when run on the ~152,000 UKB samples the method exhibits very good scaling in N (Table  1 ). To describe the context in which we have developed our method, we use the following notation; G i denotes a vector of genotypes for the i th of N unphased individuals at L markers; H denotes a set of estimated haplotypes from other individuals at the same set of markers; and H* is a subset of H of size K.
SHAPEIT2 introduced a novel strategy that splits G i into small segments of distinct haplotypes that are consistent with G i , and results in the HMM component of the method having complexity O(LK). HAPI-UR adopts a similar strategy but in such a way that the scaling with N depends on the diversity of the dataset and is typically super-linear. SHAPEIT2 also gains accuracy by combining linear complexity with the haplotype selection approach of IMPUTE216 instead of a compressed representation of H. In this approach, at each iteration, a search is carried out to find a good conditioning set of haplotypes H* that can be used to update each individual. This method can be thought of as a generalization of the LRP11 approach in which a search is carried out for just two other samples that can be used as surrogate parents when phasing each individual.
However, these selection approaches used by SHAPEIT2 and LRP involve comparison of all haplotype pairs, which has complexity of O(N 2 ). For N > 10,000 this begins to dominate. SHAPEIT3 enhances SHAPEIT2 in two ways that enable it to deal with very large datasets, such as the UK Biobank study. The first advance is based on the intuition that larger sample sizes are likely to result in increased local similarity between groups of haplotypes due to the higher probability of more recent shared ancestry. This idea is exploited by using a recursive clustering algorithm to partition the haplotypes into clusters of similar haplotypes of specified size M ≪ 2N. Distances are computed only between haplotypes within a cluster. This reduces the complexity of this step to O(M 2 ) so the complexity of the whole algorithm is dominated by the O(NlogN) scaling of the clustering routine (Online Methods).
In this way, the method adapts to the local patterns of haplotype sharing, and computation becomes increasingly focused on the challenging parts of the dataset. On the UK BiLEVE dataset of ≈50,000 samples we observed almost 20% of 2Mb windows will have a zero Hamming distances as the algorithm converges (Supplementary Figure  2 ). We have also updated the MCMC algorithm so that threading is now performed per window rather than per individual as was done in SHAPEIT2. That is, each iteration involves updating N × W haplotype windows (typically around 2Mbp in size), and we now process each window in a separate thread as opposed to entire individuals per thread (Online Methods).
Phasing the UK BiLEVE dataset We compared SHAPEIT3 to SHAPEIT2 (r768) and HAPI-UR (v1. 01) using a large Biobank scale dataset of N = 49,458 individuals from the UK-BiLEVE study7 each of whom were genotyped on an Affymetrix Biobank genotype microarray. We phased chromosome 20, which had 15,795 SNPs after QC filtering, for a range of sample sizes (N =1000,2000,5000,10000,20000,49074) and evaluated computation time and accuracy. We developed a novel method to leverage close relatives within the dataset to assess phasing accuracy. We identified 384 likely sibling pairs, and obtained partially phased, accurate haplotypes for each pair by detecting IBD1 segments using an HMM (Online Methods). After filtering, these individuals had an average of 28. 7% of their heterozygous sites phased and a total of 337,634 phased heterozygous sites on which to calculate switch error.
One member of each pair was removed from the test dataset and used to provide 'truth' haplotypes for the other sibling. Figure  1  shows the variation in computation time and switch error rate (SE) with sample size for each method. SHAPEIT3 was run using both one and four threads, with a cluster size of M = 4,000 (Online Methods). HAPI-UR 3X indicates that haplotypes were averaged across three runs of HAPI-UR, as recommended12. As expected, accuracy and computation time increase with sample size for all methods. While SHAPEIT2 is consistently the most accurate method for N ≤20,000, its quadratic distance calculations make it more computationally challenging for larger sample sizes (hence it was not run for N >20,000). For the largest sample size (N = 49,074), SHAPEIT3 had an SER of 1. 60% and took 121 hours to run (32 hours using four threads).
Phasing the UK Biobank dataset As a further validation, we used SHAPEIT3 to phase the first release of the UKB dataset, consisting of 152,256 individuals, and genotyped on a combination of the UK BiLEVE array and the UKB Axiom array (see URLs). The self-reported ethnicity of these individuals is primarily white-British (90. 5%) with the remaining 9. 5% comprising various ethnic groups. To evaluate performance we used 72 mother-father-child trios that were detected in this dataset. We used these trios to obtain a ground-truth set of haplotypes. We removed the trio parents from the dataset and phased the whole of chromosome 20 (16,265 genotyped sites) for the remaining 152,112 individuals. This run resulted in a median switch error (SE) of 0. 4% and took 38. 5 hours using 10 threads (Table  1 ).
Supplementary Figures  3 4 5 show visually the accuracy of each trio child and demonstrates how this error rate corresponds to many long stretches of accurately phased sequence. When SHAPEIT3 is run on the full 152,112 samples, it can be seen that many samples have just a handful of switch errors per chromosome. We find that 68. 5% of the inferred haplotypes consist of correctly inferred chunks of length 10Mb or greater (Supplementary Table  2 ). By increasing the number of conditioning states and the cluster size parameter we have obtained switch error rates as low as 0. 3%. To assess the advantages of phasing such a large dataset we also ran SHAPEIT3 and SHAPEIT2 on a subset of 10,072 samples that included the trio children, and obtained mean switch error rates of 1. 3% and 1. 1% respectively.
These runs took 2. 5 hours and 3. 3 hours respectively using 10 threads. On a subset of size 1,072 SHAPEIT3 had a switch error rate of 2. 6% and took 0. 25 hrs. We also ran SHAPEIT3 on the 10,072 subset without using the URLs SHAPEIT3: https://jmarchini. wordpress. com/software/ UK Biobank QC documentation: http://biobank. ctsu. ox. ac. uk/crystal/refer. cgi?id=155580 Haplotype Reference Consortium http://www. haplotype-reference-consortium. org/ Oxford Statistics Phasing Server https://phasingserver. stats. ox. ac. uk/ 100,000 Genomes Project http://www. genomicsengland. co. uk/the-100000-genomes-project/ new clustering routine, resulting in a mean switch error of 1. 1% and which took 4. 2 hours using 10 threads. These results show that the new MCMC scheme results in no loss in accuracy using ~80% of the run time.
Discussion Overall, we have demonstrated that SHAPEIT3 provides a highly accurate and scalable solution to phasing biobank scale datasets. The ultra low switch error rates that we have obtained represent strong validation of using SHAPEIT3 to phase the UKB dataset in early 2015. Switch errors that occur megabases apart (see Supplementary Figure  3 ) will have negligible impact on subsequent downstream imputation13. Due to the multi-ethnic nature of the UK population, many thousands of UKB samples will not have European ancestry (~10% of samples in the first release). While many novel loci will be uncovered using predominantly European samples, these non-European samples will be poorly phased (and imputed) using LRP approaches that search for IBD matches of a specific length. Such non-European samples are likely to be invaluable when deciphering the trans-ethnic nature of novel associations.
Given SHAPEIT2 was demonstrated to perform well (relative to other methods) on cohorts with heterogeneous ethnicity, we expect SHAPEIT3 to have similar performance8. Hence SHAPEIT3 represents a scalable method that can adapt to the multiethnic nature of large cohorts due to the custom selection of template haplotypes on a persubject basis. Reducing switch errors down to the levels produced by SHAPEIT3 can result in downstream imputation performance at low-frequency SNPs19. Phasing samples altogether also avoids having to phase in batches, which would be more likely to introduce artifacts to any downstream analysis. In addition, SHAPEIT3 has also been used successfully to re-phase the first release of the Haplotype Reference Consortium (HRC) dataset (see URLs), which consists of genotypes at ~40M SNPs called from low-coverage sequencing of ~33,000 samples20.
The boost in imputation performance due directly to this re-phasing, has in effect already led to a boost in power to detect associations at low-frequency variants, since a large number of samples have already been imputed using HRC. Phasing sequence derived genotypes in this many samples could not have been performed using other long-range phasing methods13. In other work21, we have shown that large and accurately phased haplotype reference panels can be used to help phase single sequenced samples. This approach uses rare variant sharing between the sequenced sample and the reference panel to efficiently select a set of template haplotypes to use within the HMM model. The Oxford Statistics Phasing Server allows users to phase their samples against the HRC haplotype panel20 (see URLs). Extending this approach further, we suggest that an accurately long-range phased and imputed version of the UK Biobank dataset, at a union of all SNPs on commonly used genome-wide SNP microarrays, could act as a highly accurate reference panel for phasing newly genotyped samples with predominantly European ancestry.
Then, a thread performs the window-sample pairs in its queue independently of the other queue and therefore does not need any particular synchronization, which results in better usage of all CPU cores. Once all the jobs are done, the haplotype for each sample are updated and we move on the next MCMC iteration. Of note, this scheme implicitly assumes a MCMC scheme in which the phase of all samples is updated in parallel at the same time. In other words, we move from a Markov chain state to the next by updating all samples simultaneously which differs to the standard MCMC scheme in which we move from as state to the other by updating only one sample. We refer to this as completely parallel updating. We assessed the performance of this approach by running SHAPEIT3 without the new clustering routine on the 10,072 sample subset of the UK Biobank dataset and compared it to SHAPEIT2 run on the same dataset.
Creating a validation data set for assessing accuracy SHAPEIT3 was initially developed and tested on the UK-BiLEVE dataset, which contained a large number of apparent sibling pairs but no identifiable mother-father-child trios. Regions in which siblings inherit exactly one chromosome from the same parent, (IBD1) can be phased using simple Mendelian rules. So we developed a novel method for inferring the identity by descent (IBD) status of siblings, with the aim of constructing a set of 'known' haplotypes against which we could compare accuracy of different methods. We identified 384 pairs of individuals with a kinship coefficient >0. 35, indicating they were likely first order relatives (sibling-pairs or parent-child duos). Parents will share 100% of the chromosomes IBD1 with their children and siblings will share (on average) their genomes 25% IBD0, 50% IBD1 and 25% IBD2.