dataesr/scientific-paragraphs-categorization huggingface dataset:
Parameterisation The TERRA algorithm was tested for the six test datasets with the parameters given in Table  2. Aspect aggregation factor η and number of iterations M were determined from the physical dimensions of area-specific objects. Both parameters should be at least the downslope length of non-terrain objects: for η to correctly determine terrain aspect, and for M to allow enough "scrapping" iterations to remove the object (as described in Section 2. 1). Parameter values could be translated into metric equivalents by considering raster resolutions (see Table  1 ), e. g. if the pergola canopy found in VER-D has a maximum downhill length of 8 m, the η and M parameters could be set to 100 (10 m equivalent * 0. 1 m resolution). Finally, kernel size (λ) was set to 7 for all datasets, based on an arbitrary assumption of allowed semi-circular (downhill) smoothing.
Reference DTMs and filtering performance assessment Reference DTMs were available from different sources varying among datasets (Table  1 ). For the SfM-photogrammetric datasets (i. e. TRE, VER-O, VER-D, KAM), a reference terrain model was obtained by manual filtering based on the original point cloud, which is an established approach for relatively small datasets  (Meng et al. , 2010; Sithole and Vosselman, 2004). The manual filtering was done using CloudCompare v2. 9. 1 software, based on the vertical distance of any point to surrounding points while paying close attention to the preservation of complex landscape features (examples in Figure  4 ). An exceptional case was the VER-O dataset, in which ground points were insufficient to provide a complete reference DTM, hence the VER-D reference DTM was used (thus implying an uncertainty due to shifts).
For the LiDAR-based dataset ROU-L, a reference DTM was readily available. The same reference DTM was also used for comparison with the filtered ROU-P DTM, after a vertical shift was performed corresponding to the average distance of their DSMs (3. 64 m, which relatively homogeneous throughout the study site, with a standard deviation of 0. 27 m). Such comparison can further be justified as no changes in terrace morphology were detected between the timing of the ROU-L (  2002 ) and ROU-P (2013) datasets  (Sofia et al. , 2016) , and erosion rates in Roujan are relatively low with a reported 0. 695 mm/year  (Paroissien et al. , 2010). The filtering performance of TERRA was done in several steps. Firstly, the filtered DTMs were compared to dense DGPS transect available from the field surveys of TRE and VER-O, based on Root Mean Square Error (RSME) values (Section 3.
1). Secondly, the filtered DTMs were compared to the original DSMs, based on a visual interpretation of the difference maps (Section 3. 2). Lastly, the filtered DTMs were compared to the reference DTMs based on difference maps and transect elevation profiles (Section 3. 3). The maps were classified according to Type I and II errors in order to allow a quantitative comparison with literature. For this particular purpose, a threshold was introduced to distinguish between appropriate filtering and terrain underestimation (i. e. false ground removal or Type I error) or overestimation (i. e. false non-ground preservation or Type II error). This threshold was set variable for the different datasets, as twice the raster resolution in case of SfM-derived datasets as empirically estimated in many previous studies (e.
e. nonground is falsely preserved). On average, Type I errors are relatively sparse with 5. 1%, which indicates an improved performance compared to the 8 filtering approaches evaluated by  Sithole and Vosselman (2004) , of which 7 show Type I errors in more than 50% of cases. Type II errors here are generally more frequent with an average of 19. 9%, but performance according to  Sithole and Vosselman (2004)  would be classified as Fair (10-50%) or Good (<10%) for the individual datasets (Table  3 , third column), which is comparable with the 8 considered filters under vegetated slopes. Two datasets show notably high errors: (i) ROU-P in Type I errors occurring on the horizontal terrace banks (Figure  7   Furthermore, the derived DTMs have mean elevations relatively close to the reference DTMs (Table  3 ), i.
e. <10 cm for high-resolution datasets TRE, VER-D and KAM, and <70 cm for the LiDAR-based and Pleiades-based datasets ROU-L and ROU-P. The mean distance between the filtered and reference DTMs of VER-O is slightly higher, which is biased by the different origin of the reference (i. e. deriving from the VER-D survey). Pearson's correlation coefficients of reference and filtered DTMs range between 0. 995 and 0. 999 for all datasets, emphasising the expected overall agreement of derived elevation values with the reference on steep slope test sites. Only in the case of KAM the reference DTM is generally higher than the filtered DTM (Table  3 ), which is related to the slight "scrapping" of the top of the contour bunds (Figure  7 , dashed circles and arrows).
Performance and novelty of TERRA algorithm This study presented TERRA, a novel filtering algorithm for deriving a DTM from a DSM. Its development was motivated by the general difficulties that existing filters commonly have with the preservation of sharp terrain features, due to their similarities with non-ground features  (Meng et al. , 2010 (Meng et al. , , 2009;; Pingel et al. , 2013; Shan and Aparajithan, 2005). To the best knowledge of the authors, TERRA is the first filter to make use of anisotropy in terrain aspect to guide the filtering direction, and as such, minimise the loss of valuable terrain information (Type I errors). The studied topographic datasets represent a typical challenge for existing methods, i. e. steep, vegetated and discontinuous slopes  (Sithole and Vosselman, 2004) , which are all common features of the agricultural terraces considered here.
Limitations of methodology Limitations to the presented methodologic set-up affect the results due to various reasons, including the algorithm structure and the data origin and comparison. Tests of the TERRA algorithm in various conditions give a good initial confidence of wider application and further testing of the method. The major source of errors in this study is the presence of vegetation around the terrace front, covering the edge either from above or from the side (Section 3. 3). Additionally, certain specific topographic conditions could hypothetically to be more difficult in terms of vegetation filtering and terrain preservation: (i) The presence of contour-structures with risers such as contour bunds or stone walls, that are actually considered part of the terrain but are likely to be "scrapped" (as suggested by the results from the KAM dataset here); (ii) Highly sinuous hillslopes with strong contour curvature, e.
g. in a strongly concave or convex hill segment, will impede the determination of hillside aspect, while the aspect aggregation factor η has a lower limit dictated by the dimensions of non-terrain objects. Uncertainties in the presented material also result from topographic data sources. Examples include photogrammetric errors or the inherent noise related to satellite stereography, creating artefacts in the DSM. On the one hand, the TERRA filter can cope with this issue, or even improve it by filtering it out. On the other hand, final results will be affected negatively when artefacts are in the dimensional order of magnitude of features of interest (e. g. terrace wall height). Other data-related uncertainties derive from the set-up of this study, where reference DTMs of VER-D and ROU-L were used for VER-O and ROU-P as well (in order to provide a more precise or complete reference).
CONCLUSIONS The proposed TERRA filtering algorithm is shown to have a convincing performance in first tests. Filtered DTMs produced from DSMs are relatively close to the reference DTMs in the six datasets, under various conditions of topography, presence of aboveground features, and data source and resolution. Sharp terrain features such as terrace edges are very well preserved (low Type I errors), which distinguishes the TERRA algorithm from most existing filters. Minor Type II errors occur where terrace edges are covered by vegetation on top of the terrace, or form a continuous surface with downslope vegetation located at the foot of the terrace wall. Presented results create confidence for further application of the algorithm, based on its filtering skill and supported by minimal parameterisation requirements and computational efficiency due to the raster-based approach (as compared to a 3D-cloud-based approach).
Further applications and analyses are encouraged for DTM creation and testing purposes. The algorithm may also play a key role in (semi-)automatic mapping of terrace structures, allowing a rapid DTM preparation step while maintaining typical terrain features (e. g. sharp edges) often critical for such analyses. Testing for DSM filtering in other environments such as anthropogenic landscapes with sloping terrain (e. g. hydraulic engineered lowlands, openpit mines) is also encouraged in future exploration. The TERRA algorithm is freely available as R script under GNU GPL licence at: https://www. umr-lisah. fr/?q=fr/scriptsr/terra-script-r (see Supplementary Material A). ########################################################################## ## available at https://www. umr-lisah. fr/?q=fr/scriptsr/terra-script-r ########################################################################## ########################################################################## # ######################################################################## ######################################################################### ########################################################################## # Output : # -filtered dtm in raster R format ########################################################################## ########################################################################## #.
, ROU-P, purple colours), which can be related to general quality of the dataset combined with the usage of an external reference DTM (from ROU-L); (ii) ROU-L in Type II errors, which are mostly related to a large flat vineyard canopy covering the eastern segment of the study site (Figure 7, ROU-L, cyan colours). Additionally, Type II errors are indeed relatively high in the zones where terrace edges are covered in vegetation (situation -iiin the previous paragraph), i. e. VER-O (41. 2%), ROU-L (18. 1%) and ROU-P (50. 9%, combined with the effect of the flat vineyard).
Treviso and Verona, UAV surveys were carried out during October 2017 (TRE and VER-O datasets) using a DJI Phantom 4 Pro (20 MP optical camera with 8. 8 mm focal length). In the latter site, a repeated UAV survey was carried out during December 2017 (VER-D dataset), when no leaves were present on the grape vines, using a DJI Mavic Pro (12. 3 MP optical camera with 4. 7 mm focal length). Nadir images were taken from a 50-m altitude with >75 % front-and side-overlap, and oblique images were sparsely captured to better cover hidden parts (e. g. terrace fronts or vegetation blind spots), with a total number of 316, 146 and 254 images for TRE, VER-O and VER-D, respectively. Reference terrain elevation points were measured using a TopCon HyperV DGPS device for calibration of the photogrammetric analysis (resp.
18, 19 and 17 ground control points) and as an independent validation dataset for the vegetation filtering process (dense transects of 60 and 200 points for the two locations, resp. ). UAV imagery was processed using Structure-from-Motion (SfM) photogrammetry software Agisoft Photoscan, in order to derive a 3D model from the overlapping 2D images and additional ground control points. The resulting dense point clouds had a point density of 694, 1545 and 2023 pt/m² and a recommended DEM resolution of 0. 03, 0. 02 and 0. 02 m/pix, In Roujan, Aerial Laser Scanning (ALS) was carried out during June 2002. A helicopter mounted with a Falcon II Toposys.
ACKNOWLEDGEMENTS This work is part of the  HighLandDEM  project funded by the  MISTRALS EnviMed IV program. It was partly supported by project  ViTE  " Vineyard Terraced  landscapes: understanding the Environmental constraints to improve sustainable managements", funded by the  Linda Scattolin research program  at  TESAF department of the University of Padova (Italy ). The authors would further like to thank data providers  French Space Agency CNES  and  Airbus Defense and Space for Pleiades images , the Tunisian office of topography and cadastre for aerial images, the  French National Program for Remote Sensing PTNS  and the  CNES R&T program  for LiDAR data acquisition, and UAV-services Zenith Aerial Solutions s. r. l (October 2017 mission) and Cambisol B. V. (December 2017 mission) for drone-imagery.
18, 19 and 17 ground control points) and as an independent validation dataset for the vegetation filtering process (dense transects of 60 and 200 points for the two locations, resp. ). UAV imagery was processed using Structure-from-Motion (SfM) photogrammetry software Agisoft Photoscan, in order to derive a 3D model from the overlapping 2D images and additional ground control points. The resulting dense point clouds had a point density of 694, 1545 and 2023 pt/m² and a recommended DEM resolution of 0. 03, 0. 02 and 0. 02 m/pix, In Roujan, Aerial Laser Scanning (ALS) was carried out during June 2002. A helicopter mounted with a Falcon II Toposys DATASET LOCATION TERRACE TYPE VEGETATION DATA SOURCE REFERENCE RESOLUTION ACRONYM TYPE DTM (m) TRE Treviso (IT) earth banks, vineyards (rows) UAV SfM manual filtering + 0.
1 relatively steep DGPS VER-O Verona (IT) earth banks & dry- vineyards (pergola UAV SfM manual filtering 0. 1 stone walls cultivation), (VER-D) + DGPS leaves-on VER-D " " vineyards (pergola UAV SfM manual filtering + 0. 1 cultivation), no DGPS leaves ROU-L Roujan (FR) dry-stone walls, vineyards (rows), LiDAR LiDAR DTM 1. 0 partly vegetated lines of trees and bushes ROU-P " " " Pleiades LiDAR DTM 1. 0 (stereoscopic) (ROU-L) KAM Kamech (TN) contour bunds sparse trees and aircraft SfM manual filtering 0. 3 low bush which was harmonised to 0. 1 m for the TRE, VER-O and VER-D datasets. In addition to the DSMs, reference DTMs were derived by manual point cloud filtering (further elaborated in Section 2. 5), with vertical errors to DGPS points of 0. 09±0. 06m,0. 02±0. 06mand0. 02±0. 10m,respectivelyfor the three datasets.
Moreover, an important aspect of exploring complex multivariate data--here GWAS summary statistics of mul-2 NAR Genomics and Bioinformatics, 2020, Vol. 2, No. 1 tiple phenotypes--is the visualization and management of results and data. This is a common issue in human genetics, and a number of software have been published to help investigators coping with these issues. There are now many userfriendly tools allowing for the annotation of results based on existing functional database [e. g. Dalliance  (20)  and Toppar  (21) ], the secure storage and extraction of results from database (e. g. GWAS ATLAS  (7) ] or the plotting of results integrating specific features (e. g. LocusZoom  (22)  and Assocplots  (23) ]. Similarly, the dissemination of the methodology for the joint analysis of multiple GWAS summary statistics requires the development of robust and computationally efficient tools.
With thousands of GWAS studies now publicly available, this requirement has become even more crucial. Here, we present JASS (Joint Analysis of Summary Statistics), an integrated package for the joint analysis of multiple GWAS summary statistics, including at the same time analytical tools, visualization functions and an embedded web interface. JASS provides the following features: (i) functions for the fast and efficient computation of multiple joint statistics from dozens to hundreds of GWAS results; (ii) an interactive web server for the visualization of a selection of GWAS, along with the result of their joint analysis; (iii) the possibility to install the software locally, so that interested users might apply the analysis to their own data in the safety of their own computation facility; (iv) a command line interface for advanced users; and (v) a nextflow  (24)  pipeline integrating GWAS cleaning, imputation and analyses into one tool.
Overview of JASS JASS is a Python package that handles the computation of joint statistics over sets of selected GWAS results through a command line interface and/or a web interface. The derivation of joint statistics, and the generation of static plots to display the results, as well as more advanced features such as the implementation of user-defined statistics, can be easily performed using the command line interface. Many of these functionalities can also be accessed through a web application embedded in the Python package, which also enables the exploration of the results through a dynamic JavaScript interface. The lists of available functions and features are provided in Supplementary Tables  S1  and S2 , respectively. Figure  1  shows a standard analysis workflow, including the steps performed through companion packages.
Data pre-and postprocessing A critical issue to ensure a valid multi-GWAS analysis is the harmonization and cleaning of the input data. Raw GWAS data are usually heterogeneous in their content and format. To avoid the user series of time-consuming operations, we automatized all of these steps into a preprocessing Python package tool (jass preprocessing) provided on behalf of JASS. This companion package addresses the following: (i) it maps heterogeneous column names to standard names (rsID, pos, A0, A1, Z); (ii) it derives Z-scores from P-values and signed statistics (log odds ratio or linear regression coefficients); (iii) when missing, it infers per-SNP sample size from the effect estimate standard deviation and the minor allele frequency; (iv) it filters SNPs with sample size <75% of the maximum; and (v) it aligns coded genetic variants with a reference panel.
This reference panel should be filtered out for strand-ambiguous variants and variants with low frequency. In our example, we used 6 978 319 SNPs selected using the European ancestry samples from the 1KG project  (28). This reference panel is available in the preprocessing package. Importantly, joint statistics commonly require complete data to perform a statistical test. To increase the number of SNPs with complete data, we chose to perform the systematic imputation of GWAS included in our real data examples. Because of the complexity of the task, we developed a tool for the imputation of missing statistics in an independent study implemented in the RAISS package  (29). The input and output formats of RAISS are the same as input format for the JASS package, so the imputation step can easily be integrated in a JASS pipeline.
Technical details Architecture. JASS is a Python package that can be run both as a command line tool and as an embedded web application, either on a local or on a public web server. An overview of its architecture is illustrated in Figure  2 , which describes its primary components. Data storage and computations. The analysis module of JASS uses the pandas library for data processing, allowing for convenient and fast computations over large datasets. Furthermore, pandas includes a native support for HDF5 files, which JASS uses to store both the initial data and the results of joint analyses. This format presents multiple advantages, including (i) generating cross-platform and compressed files, (ii) enabling indexing (which later reduces the access time when accessing the results during the interactive exploration) and (iii) storing multiple data frames in a single file.
Interactive web interface. The JASS web interface is implemented in JavaScript, HTML and SVG. To browse the results efficiently, we use a plotly library. This library enables the simultaneous display of very large numbers of points, as encountered in JASS, where the size of GWAS matrices to be explored can potentially reach hundred phenotypes per several million SNPs. It is already used in other GWAS analysis R packages, for example Manhattanly. The tables The JASS Python package is composed of a central 'analysis module' that defines computational features, enables reading input data and writing results to HDF files, and generating static plots as PNG files. These features can be accessed from a command line interface or from an embedded web application powered by a Flask server that uses the connexion library and a JavaScript user interface using the plotly library that enables the execution of joint analysis and the exploration of the results.
In the latter case, the management of the analysis jobs requires the setup of a Celery server. we use for selecting phenotypes and exporting results use the jquery datatable plugin. Web server. The web server provides all functionalities (except for the initial data import), through a RESTful API. This API is described with OpenAPI, and connected to the Python JASS analysis module through the connexion library. It is used primarily by the web user interface, but could also be used to run JASS analyses remotely, or with other client frameworks. Contrarily to the command line API, the statistics computation and plot generation operation is launched asynchronously, to avoid the potential timeouts that can occur during its execution. The server uses the Celery task queue to run these operations.
Labels attached to data points give the number of genomic regions processed at once. For example the two boxes filled with a "2000" in the top left of the plot correspond to the case were the entire dataset is loaded in memory (as there is a total of 1703 regions in this example). Color represents the group of traits. (C) The computation time optimization. We computed the omnibus statistics for 3 673 285 SNPs for 11 traits; the y-axis is the time in seconds (logarithmic scale) and x-axis is the number of missing data in a pattern. The straight green line shows the result from a naïve computation loop approach over all SNPs, and the red line shows the results from our optimized algorithm. more regions would increase dramatically memory usage. The maximum memory usage being reached when all regions are analysed in a single iteration.
The resulting performance makes it possible to compute a joint analysis in less than a minute, while preserving the stability of the server. Note that computation time was not a major issue in our implementation, and this strategy addresses primarily memory usage burden due to the loading of the complete GWAS results. This process is particularly critical when using the JASS web server. Additionally, note that the Celery server is by default configured to queue and run sequentially the analyses launched potentially simultaneously on different datasets, to avoid excessive loads on the memory or CPU usage of the server. Finally, when analyzing jointly a large number of phenotypes (e. g. >100), command line on a local instance of JASS should be preferred.
Optimized algorithm for fast computation It is well established that computation time of genome-wide screening can be decreased dramatically by using matrix product rather than through an iterative loop (e. g. (32). For illustration purposes, we considered an example of a real GWAS dataset including 11 traits and 3 673 285 SNPs after filtering out SNPs with missing data. The naïve loop-based implementation (computing the statistic for each SNP one by one and inverting the matrix each time) took nearly 4 h. On the other hand, processing the same data with a matrix product took 127. 1 s (Figure  3C ). However, matrixbased derivation cannot be readily implemented in the presence of missing data. Missing values imply that the statistics will differ with rows according to which traits are missing.