dataesr/scientific-paragraphs-categorization huggingface dataset:
In the past, some models have used a date in the far future, like "2500:01:01," in order to indicate that data is "still valid now. " However, such an approach is both inelegant and potentially invalid if the database survives more than four centuries (we hope so. ). In PostgreSQL, it is possible to use the keyword "infinity" in order to express an unspecified end-date. But using a keyword specific to any DBMS will prevent the database from being fully compliant and interoperable. This issue is reminiscent of the past discussions on the semantics of "now" in databases  [6]. Similarly, in our solution, we associate the notion of "still valid now" with a Boolean attribute (isForever) set to true. Thus, to avoid potential implementation incompatibilities we explicitly store information about unbounded end dates as an element within our data model.
www. josis. org 4 Implementation The spatiotemporal model was implemented using an open-source DBMS, PostgreSQL, with its spatial extension PostGIS. The focus here is on two major difficulties: the temporal aspects of the model, and the rebuilding of historical boundaries of communes. This temporal model addresses historians' needs in term of content, but also in terms of computing performance. A set of SQL procedures have been stored inside the database in order to accelerate the processing of temporal queries. For example: • Get the system identifier of a date equal to 1793 with year-level precision (e. g. , returns "13" as an integer identifier). • Get the system identifier of a date interval between to 1800 and 1805, with year-level precision (e. g. , returns "189" as an integer identifier).
• Get the oldest date of a historical date having the identifier 189 (e. g. , returns "01-01-1800" as an SQL Date-type instance). • Is the historical date with identifier 13 (strictly) before the date with 189 as its system identifier? (Returns "true" as a Boolean). Instead of using a full temporal algebra (e. g. , Allen  [1] ) this set of procedures allows for the computing of precedence and equality between dates (specified as intervals or instants). Precedence and equality are necessary and sufficient for the application of qualitative reasoning tools, such as the one developed by  [7]  based on theory of S-languages  [32]. Our method for computing historical boundaries is based on that developed by Hervé Le Bras, from the LaDéHis laboratory, who has programmed FORTRAN code in order to automate this task.
2). The process might stop there. However, in addition our system offers the option of manually processing difficult cases in order to further enhance the results. Thus, an operator can use an interface to superimpose historical maps or photographs. This interface also provides the operator with constraints such as the locations of historical places and the limits of recent boundaries. The interface allows the operator to interactively compute the relative proportions of populations expected to be distributed as a result of redrawing of historical boundaries, indicating if putative boundaries conform to known demographic distributions. In this way, 18,152 boundaries have been inferred over the whole France, many cases of which are shown in Figure  5. The FORTRAN code used is currently being ported to Python, in order to build an open-source plugin for QGIS that could be reused by anyone.
Architecture for open dissemination of historical boundary data The architecture design is driven by the desire to provide free access to downloadable historical data, within the current open-data context. A conventional download page, or even a web interface to query the database, would be one mechanism to satisfy this desire, as used by the Minnesota Population Center (see the Figure  1 ). However, using desktop software the data would require special handling, for example in order to recreate manually the dynamic links originally defined in the DBMS. Thus, an alternative solution is to develop a website dedicated to the geovisualization of this data. This alternative could provide user-friendly access to historical data, and thus appears more flexible and usable than a simple data download site.
However, it also presents two major drawbacks. First, such a website would not necessarily allow full access to the database, being limited to the original use for which the website was designed. Second, the results of queries displayed using HTML cannot be easily handled by users. Creating a mash-up of HTML results and external data is expected to be a complex undertaking for novice users. Consequently, it was decided to provide a modular solution, by developing a web API that allows users to easily query the database; serves several formats handled directly on the Web; and supports the construction of the GeoPeuple website on top of these API capabilities. The overall architecture is summarized in Figure  6. The web API is based on the RESTful API initially, but future adaptations could use other technologies.
Our solution uses HTML 5, Javascript, JQuery, and three specialized libraries for charts and map-making: HighCharts 10  to create the demographic evolution charts (Highcharts is free for non-commercial use); Raphaël JS  11  to create the systemic diagram; and OpenLayers 12  to display maps. Building on the fact that the RESTful API serves JSON, which is fully compliant and easy to use with JavaScript technologies, we embed all the commune's history and relationships (with related history) into one unique exchange format, expressed in GeoJSON. We call this the acquaintance graph.
Constructing acquaintance graphs An acquaintance graph contains all communes having had in common some parcels with the commune searched for by the user in the database. Each commune is described by its various toponyms, codes, and boundaries, as well as by all the territorial events in which it was involved, and by its census values since 1791. There are two reasons to serve data in such a way: (1) For performance, since it avoids many exchanges between server and client. There is no need to make multiple requests of the server when one wishes to present data concerning neighboring communes to that first searched. (2) For pedagogic purposes, because it helps the user to understand that rapid changes of population in one commune may be due to changes in boundaries, such as merging between communes.
Further, when census data is missing (such as for Trancault between 1973 and 1999) the user can verify that this is due to a merging event and not to missing data. Acquaintance graphs are pre-computed on the server by an iterative call to the database, searching for territorial units having events in common. An index connecting a commune's identifier to its corresponding acquaintance graph is built, in order to respond rapidly to queries from the client, since many communes share the same graph. The answer is both human-readable, and can be easily processed by a JavaScript interpreter (see the example in Annex 1).
Conclusion This paper has presented the architecture developed in order to share the geo-historical database produced in the GeoPeuple project through an online geovisualization interface. Such historical content generally requires consideration of uncertainty, inaccuracies, and incompleteness. Our approach does not aim to unravel all the conflicts arising from this imperfect knowledge. Rather, the contribution of historians has been to conduct this investigation, and provide us with a corpus of expert knowledge about this uncertain past. Instead, this paper demonstrates that such historical content requires special handling: support for imperfect dating of events, and the substantial task of boundary reconstruction, based partly on demographic hypotheses.
www. josis. org We have addressed these challenges by proposing a model of imperfect data that satisfies historians' needs in terms of metadata capture and maintenance. Further, based on an event-oriented spatiotemporal model, the database has been constructed in order to capture and store historical boundaries, using procedures for semi-automatic acquisition of historical boundaries. The resulting architecture supports free sharing of this data through a web API. Users can easily view and analyze the data (demographic figures, historical and contemporary boundaries) through a web browser. On the geovisualization side, the concept of the acquaintance graph, represented in a GeoJSON format, has been introduced in order to improve client/server exchanges and to offer a more complete and pedagogical view on demographic phenomena.
This format has been used as the basis of four synchronized views of the evolving boundaries, the census data and associated metadata, and a systemic graph showing the evolution of communes in relation to each other. Finally, the design also makes use of the GeoPortal API in order to help people to situate any event inside the present (or past) topographic context. All this has been achieved using standard opensource web technologies, such as HTML5, JavaScript, and SVG, in order to ease maintenance and deployment. There are still improvements to be carried out before the deployment of this architecture on the "TGIR Huma-num"  13  site, a French national platform for humanities data. In connection with the other activities in the GeoPeuple project, one such improvement is to visualize in dedicated thematic layers historical topographic data (mills, farms, churches, castles, etc.
) that have been digitized on Cassini and Etat-Major maps  [29] , although only in a small number of few areas (around Grenoble, Saint-Malo and Reims). In the future, we also envisage citizens digitizing historical topographic maps through the use of an online crowd-sourced tool, much as can already be done with OpenStreetMap for contemporary topographical maps. Such historical data would be most useful for our understanding of the past of the humanity  [13, 18]. Our works brings this crowd-sourcing tool a step closer, since our database has been designed to support fine-grained lineage information including transaction metadata. Finally, we hope that the interest in historical data can be enhanced by our proposal for the interactive exploration of territorial entities' history, knowing that our model can handle any kind of territorial nomenclature, not only French ones.
Acknowledgments We would like to thank the  French National Agency for the Research (ANR)  who funded the GeoPeuple project, as well as the four laboratories who supported this research, namely the  French Mapping Agency (IGN) ,  LIP6 ,  IFSTTAR , and  EHESS. In particular the authors would like to thank  Claude Motte  and  Marie-Christine Vouloir  without whose support and collaboration this project would not have been possible. Finally, we thank the Coriolys company, http: www. coriolys. frfor helping us to integrate this research within a website.
g. suffix trees or suffix arrays) can correct substrings of different lengths  (Salmela, 2010; Schr€ oder et al. , 2009). Spectral alignment-based approaches are more efficient and scalable than DBG of the SR data, and then corrects an erroneous region within an LR by searching for an optimal path within the DBG. The sequence of the overlapping k-mers along the path provides a corrected sequence for that region. Taking advantage of recent developments in compact representation of  DBG [Fig. 1, Chikhi and Rizk (2012) ;  Salikhov et al. (2013) ], we develop a program, called LoRDEC (Long Read DBG Error Correction), that allows correcting a dataset of typical size on common computing hardware. We compare our program with state-of-the-art methods and find that it provides an equal accuracy with low memory usage and reasonable running times.
g. 0. 03% in the Escherichia coli dataset) of the reported solid k-mers are false positives. Consider the k-mers of a long read starting at position 1,2,3,. : some k-mers belong to the graph and are solid, while others do not and are weak. Basically, solid k-mers are expected to be correct, while weak ones suspectedly include sequencing errors and require a correction. Solid k-mers are entry points in the DBG, and LoRDEC corrects a region made of weak k-mers by finding the best path in the DBG between the solid k-mers bordering this region. Sometimes, an LR has no solid k-mer, in which case, LoRDEC marks it as such in the output and skips it. Our results show that only short erroneous reads (51500 nucleotides) completely lack solid k-mers (data not shown). As the goal of PacBio sequencing is to get long reads and thus our goal is to yield long and correct sequences, we decided to filter those reads in the present version of LoRDEC.
Data and computing environment We used three datasets of increasing size: one from E. coli, two eukaryotic ones from yeast and from the parrot. They include, respectively, 98 Mb, 1. 5 and 6. 8 Gb of PacBio reads, with 231, 451 Mb, and 35 Gb of Illumina reads. All details are given in Supplementary Table  S1. All experiments were run on servers with 16 cores operating at 2. 53 GHz and 32 GB of memory. The runtimes were recorded with the Linux/Unix time command, and the memory and disk usage was recorded by polling the operating system periodically. Because all the correction tools support parallel execution on several cores, we report both total CPU time and elapsed (wall clock) time.
Evaluation approach We used two approaches to evaluate the accuracy of correction. The first approach measures how well the reads align against the reference genome. The second approach compares the differences in the alignments of the original and corrected reads against the reference to evaluate the accuracy of correction. For the E. coli and yeast datasets, we used BLASR  (Chaisson and Tesler, 2012)  to align the original and corrected reads to the genome and for the parrot dataset, we used BWA-MEM  (Li, 2013). For the smaller datasets, BLASR was used because it tends to bridge long indels better and thus reports longer alignments. For the parrot dataset, we preferred BWA-MEM because it is faster. For each read, we kept its best alignment against the genome. We then counted the size of the aligned region of the reads, the size of the aligned regions in the genome and the number of identical positions in the alignments.
Effect of parameters on our approach We investigated the effect of parameters on our method on the E. coli dataset. We varied one parameter at a time and recorded the runtime and evaluated the accuracy of the method by computing gain. Figure  3  shows the results for this experiment when varying five parameters: the k, the threshold for a k-mer to be solid in the Illumina dataset, the maximum error rate of corrected regions, the branching limit and the number of target k-mers for path finding from a source k-mer. We see that k = 19 gives the best results for this dataset, and further experiments with the yeast data confirmed k = 19 to be optimal also for that dataset (data not shown). The solid k-mer threshold had only a modest effect on the accuracy of correction, a smaller threshold giving slightly better results.
The accuracy of correction was improved by setting a higher maximum error rate of the corrected region with a slight increase in the runtime. Increasing the number of explored branches or the number of target k-mers had only a small effect on the gain, whereas runtime was increased considerably. Based on these observations, we ran our method on the E. coli and yeast data with the following parameters: k, 19; threshold for solid k-mers, 3; maximum error rate, 0. 4; branching limit, 200; and number of target k-mers, 5. For the parrot data, we found k = 23 to give better results both in terms of runtime and accuracy. Supplementary Table  S2  provides an explanation for each parameter and its default value.
Comparison against LSC and PacBioToCA We compared LoRDEC against LSC  (Au et al. , 2012)  and PacBioToCA  (Koren et al. , 2012). LSC was run with default parameters except that we set short read covered depth to the estimated coverage of the dataset, i. e. 50 for the E. coli dataset and 38 for the yeast dataset. PacBioToCA was run with default parameters except for tuning the parameters for parallelization to be suitable for our platform. The parameters for LoRDEC were set as explained above.
Escherichia Coli The runtime and memory and disk usage of the error correction tools are shown in Table  1 (top). LoRDEC is 17 times faster and requires 88% less memory and 95% less memory than LSC, which is the more resource efficient of the two previous tools on this dataset. The right side of Table  1  shows the correction statistics as reported by Error Correction Toolkit for LSC and LoRDEC, and we see that LoRDEC outperforms LSC. The statistics of aligning the reads against the reference genome are shown in Table  2   (top). For LSC, we report the statistics both for the full corrected read set as reported by the tool and for the trimmed set. We note that LSC leaves out from the full read set any reads that it was not able to correct at all. Similarly, we report for LoRDEC statistics for full reads, reads trimmed at the ends and trimmed and split reads (see Section 2.
Yeast Both LSC and PacBioToCA failed to complete the correction of this dataset on a single server. We split the PacBio data and run LSC on three servers and PacBioToCA on six servers. Whereas PacBioToCA is designed to run distributed on several servers, LSC does not support distributed execution. Therefore, we chose to use as few servers as possible with LSC to minimize the effect of the distributed execution on the correction accuracy. Table  1  (middle) shows the runtime, memory and disk usage statistics for the yeast dataset. Also for this dataset, LoRDEC uses at least one order of magnitude less time or memory and two orders less disk than PacBioToCA and LSC. For instance, LoRDEC is six times faster and uses 93% less memory and 99% less disk space than PacBioToCA. The gain and sensitivity of LSC remain 532%, while they stay 480% for LoRDEC.
Experiments on the parrot data We investigated the scalability of LoRDEC on a much larger eukaryotic dataset: the parrot data. As the parrot genome is a vertebrate, hence complex, genome, that is about one-third of the Human genome in length (Supplementary Table  S1 ), it represents a real test for both scalability and issues regarding the impact of genome organization. Given the running times of LSC and PacBioToCA on the smaller yeast data, these were not included in this experiment. The data contain three PacBio libraries, and we ran the correction of each on its own server. Fig. 3. Effect of parameters on the runtime and gain of our method. We varied k, solid k-mer threshold, branching limit, maximum error rate and number of target k-mers one at a time, while other parameters were kept constant  Note.
Impact of the genome organization The evaluation of the correction delivered by LoRDEC indicates that it is accurate globally on all datasets. However, the genome organization, and especially the presence of repeats, could impact the quality of the correction. One could argue that in repeated regions, the solid k-mers and paths found in the DBG of SR may come from a different or from several copies of the repeat and mislead the correction process. In other words, the accuracy of the correction may vary along the genome. If this is the case, the distribution of reads with respect to the observed genome coverage should differ between raw and corrected reads. To assess this possibility, we computed the expected and observed genome coverages of the aligned raw and corrected reads (Last columns of Table  2 ).
Runtime, memory, disk usage and accuracy statistics as reported by Error Correction Toolkit for the error correction tools on the E. coli (top), Note. Memory and disk usage are in gigabytes. The statistics could not be computed for reads corrected by PacBioToCA because PacBioToCA only reports trimmed and split reads. Run parallel on three servers. Memory usage is for one server. sufficientlytocorrectreads of large eukaryotic genomes on common computing hardware. The Error Correction Toolkit in Table1(bottom) and alignment statistics in Table2(bottom) show that the correction accuracy is comparable with the yeast dataset, although the reference is a draft genome containing more errors, and the alignment statistics also suffer from reads aligning to the end of scaffolds having only partial alignments.
yeast (middle) and parrot (bottom) datasets Data Method CPU time Elapsed time Memory Disk FP TP FN Sensitivity Gain E. coli PacBioToCA 45 h 18 min 3 h 12 min 9. 91 13. 59 NA NA NA NA NA LSC 39 h 48 min 2 h 56 min 8. 21 8. 51 695773 3149629 7845597 0. 2865 0. 2232 LoRDEC 2 h 16 min 10 min 0. 96 0. 41 102427 9994561 1000665 0. 9090 0. 8997 Yeast PacBioToCA a 792 h 41 min 21 h 57 min 13. 88 214 NA NA NA NA NA LSC b 1200 h 46 min 130 h 16 min 24. 04 517 7766700 38741658 80597251 0. 3246 0. 2596 LoRDEC 56 h 8 min 3 h 37 min 0. 97 1. 63 2784685 100568850 18770059 0. 8427 0. 8194 Parrot LoRDEC b 568 h 48 min 29 h 7 min 4. 61 74. 85 10591097 226996640 26296446 0. 8962 0. 8544 a Run parallel on six servers. Memory usage is for one server. b.
The species-thinning experiment The above model chain was applied to pine, beech and oak single-species forests. For each species we simulated four different thinning strategies, going from unmanaged to intensively thinned forest (see Table  1 ). All simulations were based on the growing conditions that approximate to the highest site class of Dutch yield tables  (Jansen et al. , 1996) , using observed climate over the period 1975-2010 for the central Netherlands from the Monitoring Agricultural Resources (MARS) database (accessed online 21 January 2014: www. marsop. info/marsop3/). The simulations start with about 5000 trees per hectare, aged between 10 and 15 years and forest regeneration was considered absent within a rotation. All simulations were performed for 1 ha of forest. For each species and thinning type, canopy albedo was calculated for a transect across Europe along the 9 • E line of longitude stretching from 40 • to 60 • N in discrete steps of 5 •.
Validation The radiation transfer scheme was validated against observational data of forest properties from the five forest sites under study (two pine, two beech, one oak forest; see Table  3  and Appendix). We compared the observed top-of-thecanopy albedo data and mean MODIS-collection-5 albedo for June 2001 to 2010 to the modelled white-sky canopy albedo values for each site. For the validation, the radiation transfer model was forced by observed diameter and height distribution and single-sided LAI and, in the case of needles, hemispherical LAI. In contrast to the model chain used in the remainder of the study, the validation thus did not make use of ForGEM simulations to describe the canopy structure in the radiation transfer model. Foliage in tree crowns often forms clumps, and typically clumping is more important in needleleaf compared to broadleaf plants.
Thinning does not need to be explicitly implemented in Earth system models but it needs be expressed as, at least, changes in leaf area index and crown volume. These findings imply that changes to the structure of Earth system models are required for more accurate and computationally efficient canopy radiation transfer calculations. The 1-D modelling approach presented here could be implemented in Earth system models. The simulations of the forest properties provided by the forest gap model ForGEM could be replaced by the process-based global vegetation models that are already included in several Earth system models. Most likely, the existing vegetation model will need to be extended by a semi-explicit or statistical forest management module that simulates forest growth and management  (Bellassen et al.