dataesr/scientific-paragraphs-categorization huggingface dataset:
A number of authors have used remote sensing to analyze forest degradation using different sensors  (Matricardi et al. , 2010 (Matricardi et al. , , 2005;; Monteiro, Souza, & Barreto, 2003; Souza et al. , 2013; Souza, Firestone, Silva & Roberts, 2003). The high frequency of MODIS imagery enables efficient mapping of burned forest areas, and the MODIS multi-temporal dataset enables differentiation between deforestation and degradation processes when they affect large areas  (Shimabukuro, Beuchle, Grecchi, & Achard, 2014). DETER with MODIS images (250 m) and DETER-B with AWIFS data (56 m) are tools developed by INPE for low-resolution sensors to detect deforestation and large clearings caused by forest cutting or burning in the Brazilian Amazon at near-real time  (Diniz et al. , 2015). The INPE conceived another method to specifically monitor forest degradation, called DEGRAD (obt.
, 2003). CLASlite makes it possible to distinguish the fractions of photosynthetic vegetation (PV), non-photosynthetic vegetation (NPV) and bare substrate (S). CLASlite-based analysis is an approach to monitor forest degradation that can be used to detect selective logging  (Chicas, Omine, & Saqui, 2016). However, CLASlite has rarely been used to assess the cumulative impacts of degradation processes, and hence their mediumand long-term effects on forest degradation. In a region of Mato Grosso heavily impacted by deforestation and selective logging,  Grecchi et al. (2017)  investigated the annual changes due to forest disturbance caused by selective logging using a Landsat multitemporal dataset over a 15-year period processed with spectral mixture analysis. These authors reported on the areas affected by selective logging using a regular grid with 300 m × 300 m cells.
| Field observations and identification of reference plots Two field missions in May and August 2015 made it possible to build a typology of degraded forests using GPS data acquisition and qualitative descriptions (see further details in  Bourgoin et al. , 2018). Based on the spatial distribution of the data, Google Earth images and Esri World Imagery (July 2016), the plot reference database was completed by visual interpretation of disturbance impacts. We identified five reference plots for each type of forest degradation (Figure  1 ). Including all the area affected by similar management or by fire meant the plots differ in size. Plots of undisturbed forests and logged forests with a management plan were selected in the western part of the municipality, in the fazenda Rio Capim owned by CIKEL-Brazil Verde.
| Processing of remote-sensing images CLASlite is a software package designed to automatically identify deforestation and forest disturbance on satellite imagery. It is used to convert raw satellite images into fractional cover maps of live vegetation canopy, dead vegetation, and bare surface. These fractional covers are core determinants of forest composition, structure, biomass, physiology and biogeochemical processes  (Asner et al. , 2005; Oliveira et al. , 2007). The method requires "libraries" of spectral end-members for each of three relevant surface cover types: PV (fractional cover PV), NPV (fractional cover NPV), and bare substrate (fractional cover S). Spectral properties of the PV band are specific to leaf photosynthetic pigments, canopy water content, and the amount of foliage in the canopy.
Spectral properties of the NPV band are associated with dried carbon compounds in dead leaves and exposed wood. The S band represents exposed mineral soil, rock outcrops, buildings and roads. The fractional cover of each band for each pixel is provided by a Monte Carlo method (Automated Monte Carlo Unmixing Process), where the possible combinations of the endmember spectra are pre-computed with libraries derived from extensive field databases and satellite imagery provided with CLASlite. Each image pixel is decomposed using the following linear equation  (Asner, Knapp, Balaji, & Paez-Acosta, 2009) : where P(λ) e is the reflectance value of the observed pixel at wavelength λ and C PV , C NPV and C S are the percentages of fractional covers for the aforementioned principal constituents or endmembers, P(λ) PV , P(λ) NPV and P(λ) S being the corresponding reflectance at wavelength λ for every one of the three endmembers considered, and (ε) is an error term.
The dynamics of forest disturbances at the municipal scale also revealed marked differences in deforestation dynamics. In Paragominas, deforestation decreased from 334 km² in 2005 to an average annual deforestation of 33 km² over the period 2010 to 2015 (source: PRODES database, INPE), due to the stricter application of the Forest Code and the Green Municipality commitment  (Laurent et al. , 2017; Piketty et al. , 2015). However, the remaining primary forests are undergoing degradation that is affecting a larger area than deforestation. In a region in Mato Grosso, Grecchi et al. NPV variance (2017) also identified areas affected by greater disturbances caused by selective logging than by deforestation and an increasing ratio between annual rates of disturbance and annual rates of deforestation during the period 2000-2015.
Herein, at the end of each calculation, the residual stresses and plastic strains are calculated by FEM as internal variables. After a crack propagation event, these tensors are transferred to the new mesh and treated as initial fields in the next DIC-FEM coupling computation (Fig. 7 ). Finally, the DIC-FEM coupling process is led with a minimum crack length increment of 10 pixels (corresponding approximatively to 50 µm) allowing to deal with a great number of cases within a reasonable time. This strategy was preferred to a temporal increment, i. e. in terms of cycles, as large scale yielding occurs in a few cycles at the end of the test. The procedure was fully automated using a Python script coupled with ABAQUS and requires a few minutes to treat one boundary conditions case.
Background: The study of genetic variations is the basis of many research domains in biology. From genome structure to population dynamics, many applications involve the use of genetic variants. The advent of next-generation sequencing technologies led to such a flood of data that the daily work of scientists is often more focused on data management than data analysis. This mass of genotyping data poses several computational challenges in terms of storage, search, sharing, analysis, and visualization. While existing tools try to solve these challenges, few of them offer a comprehensive and scalable solution. Results: Gigwa v2 is an easy-to-use, species-agnostic web application for managing and exploring high-density genotyping data. It can handle multiple databases and may be installed on a local computer or deployed as an online data portal.
Background Nowadays, next-generation sequencing technologies have become a standard tool for many applications in basic biology as well as for medicine and agronomic research. With the decreasing cost of genome sequencing, many laboratories are increasingly adopting genotyping technologies as routine components in their workflows, generating large datasets of genotyping and genome sequence information. Additionally, scientists are also interested in re-using data produced by large international consortia that have performed re-sequencing or high-density genotyping on material from representative, publicly available diversity collections. For instance, the 3000 Rice Genome Project  [1]  and the 1000 Plants Project  [2]  provide huge amounts of sequence variation data to search and download through, respectively, their SNP-SEEK  [3]  or 1001genomes.
org portals. Such information is not easy to handle because of its size and its complex structure, both unsupported by standard software such as spreadsheet processors. This kind of data is indeed mostly made available as variant call format (VCF)  [4]  and often needs to be converted into specific software formats for subsequent analyses (e. g. , PLINK  [5] , Darwin  [6] , Flapjack  [7] ). In addition, the tools available to filter data or perform more complex operations are mainly available in the command line. Because these results may contain tens of millions of variants for thousands of samples, scalable and user-friendly solutions need to be offered to the biological community. We thus developed Gigwa  [8]  with the aim of providing a system that helps relieve biologists from the burden of technical aspects of variation data manipulation.
Gigwa is a web application designed to store large volumes of genotypes (up to tens of billions), initially imported from VCF or other file formats, in a NoSQL database  (MongoDB [9] ), and to provide a straightforward interface for filtering these data. It makes it possible to navigate within search results, to visualize them in different ways, and to re-export subsets of data into various common formats. In the first version published in 2016, we focused our work on the following important aspects: (i) filtering features that include genotype pattern search, e. g. , minor allele frequency (MAF) and missing data ratio to name a few; (ii) storage performance by choosing a NoSQL engine and designing data structure in order to scale with growing dataset sizes and support incremental addition of data into projects; (iii) sharing capabilities, i.
e. , enabling multiple users to efficiently work on the same datasets without the need to replicate them; and (iv) graphical visualization, which allows either a summarized or detailed view of the dataset contents. Our experience with biologists operating in various research fields and studying different species helped us improve the application with regard to many aspects. In version 2, we overhauled the graphical interface to improve user experience and visualization features. This new release also integrates a data and user management section to facilitate system administrators' work. We took the evolution of next-generation sequencing and analysis software into account by adding new import and export formats. Gigwa's scaling capacities along with its speed performance were also improved, thus making it able to deal with much larger datasets.
Finally, we enabled interoperability with other applications, in particular by implementing standard representational state transfer (REST) application programming interfaces (APIs). Since the release of Gigwa version 1  [8] , the application has been adopted by several institutes, in some cases embedded within information systems like the Musa Germplasm Information System  [10] , in others deployed as a self-sufficient portal providing convenient access to public data  [11]. Feedback was thus collected, suggesting ideas for significant improvement. In this article, we describe the list of newly added features, provide details about software improvements, discuss the benchmarking work done to assess performance progress, and finally present a concrete use case showing the usefulness and efficiency of the application.
New import functionalities The first version of Gigwa only supported importing data via specification of an absolute path on the webserver's filesystem. While this method is still supported because it is useful to administrators, new ways of feeding genotyping data into the system have been added: r By uploading files from the client computer (either using drag and drop or by browsing the filesystem); r By providing http(s) URLs to online files; r By specifying the base-URL of a BrAPI  [12]  v1. 1 compliant ser- vice that supports genotyping data calls. Indeed, this version embeds a client implementation of BrAPI, which allows users to select a genome map and a study in order to feed a Gigwa project with corresponding genotypes pulled from the BrAPI datasource. Additionally, the application now allows anonymous users to import genotyping data as temporary databases for filtering purposes.
Such datasets are only guaranteed to be maintained online for a limited period. An adjustable size limit can be set for files uploaded by any users, including anonymous ones. As for import formats, the PLINK (PLINK, RRID:SCR 001757)  [5]  flat-file standard format is now also supported as input for genotyping data. Finally, version 2 supports enriching permanent databases by importing metadata as tabulated files for the individuals they refer to. Those metadata facilitate individual selection in the interface based on complementary information beyond the individual identifier (e. g. , passport data, traits).
New export functionalities The export features have also been extended as follows: r The ability to refine the individual list at export time has been added. It is therefore possible to selectively export data relating to a chosen subset of individuals, independently from the one used for filtering variants; r A new export format was added (. fjzip) for compatibility with the Flapjack  [7]  software; r In the case of data files being exported to server, Gigwa v1 provided the means to push this output to a running instance of the IGV (Integrative Genomics Viewer, RRID:SCR 011793)  [15]  stand-alone software. Version 2 additionally supports pushing it to online tools such as Galaxy (Galaxy, RRID:SCR 0 06281)  [16, 17]  or SNiPlay  [18]. The list of connected tools can be managed by administrators, and a custom tool can be configured by each end-user.
APIs and data interoperability Much effort has been put into making Gigwa data interoperable: External, online genome browsers can now be configured for viewing each variant in its genomic context. Administrators have the ability to specify the URL of a default genome browser (e. g. , GBrowse [GBrowse, RRID:SCR 006829]  [20] , JBrowse [JBrowse, RRID:SCR 001004]  [21] ) per database. End-users may override this default configuration by specifying another tool, thus only affecting their own interface. When such a configuration exists for a database, each variant line in the main browsing interface table features a clickable icon leading to opening the genome browser at the position of the variant so that it can be checked against available tracks. Moreover, 2 REST APIs have been implemented to automatically provide access to any data imported into the system: r The GA4GH  [22]  v0.
6. 0a5 API. The new graphical user inter- face (GUI) mentioned above is implemented as a client for this API; i. e. , most interaction between Gigwa's client-side and server-side code is performed in compliance with the standards defined by the GA4GH API; r The BrAPI  [12]  v1. 1 API. Flapjack  [7]  and BeegMac  [23]  are ex- amples of clients that are compatible with the data Gigwa serves via BrAPI. The Musa Germplasm Information System  [10]  also interacts with Gigwa through BrAPI by serving Gigwa-hosted data using a proxy approach. Both APIs have different purposes and, respectively, work with health-related data and crop-breeding data. One clear overlap between them being the support for sharing genotyping data, we thought it relevant to implement for each API the calls that rely on the type of data held in our system.
This feature is particularly convenient when working with large amounts of data because of the time taken by imports and exports. (ii) Enhanced performance in terms of query speed. As a reminder, each search operation is performed via multiple MongoDB aggregation queries targeting evenly sized variant chunks, thus improving response times while allowing progress monitoring. Run on configuration 1 using dataset 1 (along with sub-sampled versions, so as to obtain 6 different databases), all with the same number of individuals (i. e. , 3,000) but with various numbers of markers. Query was a MAF range between 10% and 30% applied to the first 2,000 individuals 2 (i) Assess performance of latest versions of tools (Gigwa v2 and VCFtools v0. Run on configuration 2 using dataset 2, sub-sampling being performed on the fly by restricting the search to a varying list of chromosomes.
Although this method worked satisfactorily, it did not scale efficiently with dataset size. Indeed, in the case of a lenient variant-level query, the system would spend much time writing into the temporary collection (copying most of "variants"' contents) and also updating it afterwards (especially when the genotype-level query was stringent). In version 2, all searchable contents in the variants collection are duplicated into variantRunData, thus allowing all filters to be applied simultaneously by querying a single collection. This data duplication is small and leads to a negligible volume increase that is advantageously compensated by other structure modifications (e. g. , removal of empty genotype fields for missing data). This improvement is illustrated in Additional File 1. In addition, the use of temporary collections has been reduced to a minimum.
Previously, any filtering resulted in the creation of temporary data. Thus, only the browsing and exporting of an entire (i. e. , unfiltered) database were performed on the main variants collection. With version 2, a temporary collection is created only if a genotype-level query has been submitted. Thus, when the query applies solely to variant-level fields, it is remembered and re-applied to the main variants collection when browsing or exporting data. These indexed queries being extremely fast to execute, responsiveness is not affected even on very large datasets. Additionally, the JavaScript Object Notation syntax of search queries has been optimized to reduce both the number of operations involved in applying filters and the amount of data processed at each stage of MongoDB's aggregation framework.
Finally, a "multithreading regulation" mechanism was implemented, which adjusts the number of concurrent threads at run time when executing queries. It is based on the database server's live responsiveness and therefore automatically adapts to the current load without taking hardware considerations into account. More detail can be found in Additional File 2. (iii) Enhanced filtering workflow, improving responsiveness. When the search button is clicked, depending on the status of the "Enable browse and export" checkbox, the system either builds-and keeps track of-the list of matching variants (find procedure) or simply returns a count value telling how many matching variants were found (count procedure). Each query count is cached as an array of sub-values (1 for each genome chunk), the sum of which equals the query's total result count.
When creating export files, instead of synchronously reading data chunks from the database and writing them to the output stream, we implemented 2 separate processes, one dedicated to reading, the other dedicated to writing, both designed to run concurrently. The reading process was optimized using a multithreading regulation routine as described above. As for the density visualization functionality, it was improved by making chart zooming dynamic: a new query is now sent to the server each time the zoom level changes, thus always ensuring optimal data resolution.
Benchmarking We performed benchmarking tests to (i) assess how tools tested in our previous article evolved in terms of speed, (ii) demonstrate the benefit of targeting a genome region when applying a genotype-level filter in Gigwa, and (iii) evaluate our system's capacity to work with very large datasets. Two hardware configurations were used in this benchmark: Configuration 1: comparable to the one tested in the original benchmark  [8] , and essentially used for assessing the progress made since then. It is a Hewlett Packard EliteBook 850 G3 laptop computer with an Intel Core i7-6500U central processing unit (CPU) at 2. 50 GHz, 16 GB of random access memory (RAM), and a Samsung PM871 512 GB (6Gbit/s) TLC SSD 850. Configuration 2: high-performance machine typically suitable to serve as a production environment for MongoDB and thus for Gigwa.
We used it to evaluate the performance of the latest software versions running on production hardware, including on large datasets. It is a Dell PowerEdge R640 server based on an Intel Xeon Gold 5122 CPU at 3. 60 GHz, 384 GB of RAM, and a 1. 92To SAS (12Gbit/s) Toshiba PX05SV SSD. Two datasets were used in this benchmark: Dataset 1: dataset tested in the original benchmark, the Old Subset SNP Dataset v0. 2. 1 (formerly named CoreSNP v2. 1) from the 3000 Rice Genomes Project  [26] , containing genotypes for 3,000 individuals on 365,710 single-nucleotide polymorphisms (SNPs). Its reasonable size (4. 4 GB in VCF format) was suitable for experimenting with Configuration 1. Dataset 2: filtered SNP v1. 0 Dataset from the 3000 Rice Genomes Project, containing genotypes for 3,024 individuals on 4,817,964 SNPs (VCF file of 60.
4 GB, preliminarily annotated with SnpEff v4. 3T). Because it was demonstrated in the original article that relational database management system-based implementations were suitable only for querying on indexed fields (which Gigwa v1 could do nearly as efficiently) but not on genotype-level information, such solutions were left out in the present work. Therefore, we mostly concentrated on executing queries at the genotype level, especially using the MAF range query, which is among the most CPU-intensive. All Gigwa instances were set up with MongoDB's WiredTiger storage engine, using the zlib compression level, which had seemed to be the best option in the original tests. Three different comparison tests were run in this benchmark, which are described in Table  1  and whose results are reported in Fig.
5% and 36. 5% was observed in moving from Gigwa v1 to Gigwa v2. However, rather oddly, a consistent speed loss ranging between 40% and 48. 5% was observed in moving from VCFtools v0. 1. 13 to v0. 1. 16. From Test 2 results, a first observation is that using production hardware in which much RAM is available for MongoDB and Tomcat, the difference in speed between tools is far smaller. If we take the full dataset (365,710 variants) as a comparison reference, the Gigwa v2 query takes 3. 8 times longer to execute than with VCFtools v0. 1. 16 for Test 1, whereas for Test 2 it is only 1. 36 times slower. Besides, when targeting a region of the genome, Gigwa takes advantage of its indexing strategy and even responds faster than VCFtools as the given region becomes narrow enough. Test 3 results demonstrate that Gigwa v2 is able to efficiently handle and search very large datasets (here, >14 billion genotypes) when running on suitable hardware.
Also, the trend observed in Test 2 is confirmed here, i. e. , targeting a precise genome region for applying a genotype-oriented filter is of great benefit in terms of speed. Benchmark discussion. The benchmarking work previously performed by Semp ér é et al. [8]  had shown that, and provided reasons why VCFtools excels in executing genotype-level queries on an entire large dataset. Although equaling its performance in a 3-tier application like the one presented here does not seem feasible for such queries, we thought it relevant to assess the progress made since version 1, still in comparison with VCFtools. This work led to several conclusions: (i) Gigwa v2 performs substantially better than v1 in applying genotype-level queries; (ii) setting up Gigwa on production hardware (with large amounts of RAM) greatly improves its performance; (iii) combining variant-level and genotype-level filters whenever possible is a good way to make the most of Gigwa's indexed fields and can lead it to outperform VCFtools.