dataesr/scientific-paragraphs-categorization huggingface dataset:
IV. VELASSCO: DATA VISUALIZATION QUERIES This section addresses the optimization of VELaSSCo  [21] , a Big Data visualization architecture that relies on the MapReduce model to extract information from simulation datasets. More details of this project are provided in Section IV-A, while the main characteristics of the MapReduce workloads are described in Section IV-B. Section IV-C explains the main challenges of running these workloads with Flame-MR to improve their performance. The experimental configuration and performance results are then presented in Sections IV-D and IV-E, respectively. Finally, some concluding remarks are provided in Section IV-F.
A. Overview VELaSSCo is a query-based visualization framework that aims at providing users with a tool to manipulate and visualize large simulation datasets. These datasets are generated by large parallel simulations relying on Finite Element Methods (FEM) or Discrete Element Methods (DEM). For both methods the simulation updates the properties of the nodes (FEM) or particles (DEM) at each time step. The user runs a 3D visualization client to request the execution of specific visualization algorithms on given parts of the data. The query is sent to the VELaSSCo cluster and translated into a Hadoop job that queries the input data and performs the expected transformation. The result is sent back to the client for the final 3D rendering and display. The VELaSSCo architecture can be decomposed into three subsystems: client, analytics and storage, described next.
The client subsystem provides data visualization to the user, generating a new query when the user performs an action. Each query has a certain type depending on the action performed by the user. Analytical queries are the ones that require to extract some information from the dataset by means of a MapReduce workload (e. g. calculating the bounding box of a model). The analytics subsystem is in charge of receiving these queries and determining the computation needed to complete each one. That computation is performed by a MapReduce workload on a Hadoop cluster. The workloads employed in VELaSSCo consist of a single MapReduce job that typically operates over a subset of the dataset (a few simulation time steps for instance). Finally, the data persistence is performed by the storage subsystem.
This subsystem employs HDFS to distribute the data among the computing nodes of the cluster. It also relies on HBase  [22] , a database system on top of HDFS, to allow the extraction of parts of the dataset without reading it entirely. This optimizes the amount of I/O operations needed to perform the computations. Instead of searching the relevant data through the entire dataset, the MapReduce workload uses the key-value format provided by HBase to fetch the required elements. The indexed system used in HBase accelerates the retrieving operation by avoiding the reading of unnecessary data. As VELaSSCo is a real-time visualization platform, the performance of the actions executed by the user is crucial to ensure an appropriate user experience. However, Hadoop is not able to achieve this goal when dealing with largescale datasets.
B. MapReduce implementation We list below the main analytical queries included in VELaSSCo: • GetBoundingBoxOfAModel (BB): Computes the spatial bounding box for the selected dataset, i. e the min and max coordinate of the enclosed elements in the x, y and z dimensions. • GetBoundaryOfAMesh (BM): Computes the set of elements that are at the boundary of the selected mesh, i. e. the surface given by the triangles belonging to only one mesh cell. • GetListOfVerticesFromMesh (LVM): Obtains a list of identifiers (IDs) of the elements contained in a mesh. • GetMissingIDsOfVerticesWithoutResults (MIV): Obtains the IDs of those mesh elements that do not contain any simulation result. • GetSimplifiedMesh (SM): Obtains a simplified version of the mesh model, reducing the total dataset size by combining nearby elements.
C. Challenges The use of Flame-MR to optimize the VELaSSCo queries must take into account the characteristics that differ from standard Hadoop jobs. In particular, reading input data from HBase and using custom data types must be handled correctly to avoid incompatibility problems. This section describes how they are supported in Flame-MR. Flame-MR is oriented to processing large textual datasets stored in HDFS, which is a common use case in MapReduce applications. Therefore, the reading of input data has been designed to make the common case fast. When launching a map operation, Flame-MR connects to HDFS and reads a full input split (e. g. 256 MB) to memory by copying the data to a set of medium-sized buffers (e. g. 1 MB) allocated in the data pool (see Figure  2 ). Once the input split is read, the connection to HDFS is closed and the data buffers are parsed in memory to obtain the input pairs and feed the mappers.
The in-memory parsing mechanism of Flame-MR is only possible when the input dataset is stored in HDFS in textual format. For other formats, the data source is unknown, and the software interface defined by Hadoop only allows reading the input pairs one by one. Therefore, copying an input split entirely to memory is not allowed and the reading of input data needs to be addressed differently. That is the case of VELaSSCo, which reads the data from HBase. When a map operation is launched in Flame-MR, the input formatter connects to the HBase server to read the data contained in the input split. Then, the map operation uses the interfaces provided by the input formatter class to read the input pairs, passing them to the user-defined map function. By doing this, the correct functioning of the queries is ensured.
number of CPU cores, memory size). The most important parameters of the resulting configuration are shown in Table  II , including a brief explanation of each one. The VELaSSCo queries used in the evaluation are the ones described in Section IV-B. The input dataset has been extracted from a FEM simulation that represents the wind flow in the city of Barcelona with an eight-meter resolution. This dataset has 12,089,137 vertices and occupies 367 GB. For each query, the graphs show the median elapsed time of 10 executions, although the standard deviations observed were not significant.
A. Overview The datasets generated by Next Generation Sequencing (NGS) platforms are composed of a large number of DNA sequence fragments, which are small pieces of genomic information contained in a string of characters (called reads). Each character of a read represents a DNA base, namely Adenine (A), Cytosine (C), Guanine (G), and Thymine (T). The analysis of these datasets is performed by processing the sequences and identifying relationships between them. During the generation of genomic datasets, NGS sequencers often introduce errors by placing incorrect bases in the reads. This can affect the quality of the results obtained by downstream analysis, and so it is usually minimized by introducing an error correction phase in the preprocessing stage of the NGS pipeline. In fact, this is a critical step in NGS workflows like de novo genome assembly or DNA resequencing.
CloudRS is a popular tool for performing this preprocessing task, being based on the ReadStack (RS) algorithm  [25]. This algorithm makes use of the characteristics of NGS datasets to identify common patterns in the sequences and correct the mismatching ones. The DNA sequences that compose a dataset are not necessarily disjoint, as they can share information due to the overlap of reads performed by the sequencer. CloudRS takes advantage of this characteristic to identify redundant information in the sequences and correct errors in the bases. First, it splits each sequence into several subsequences. Second, it compares the different candidates for each subsequence, choosing the one that appears most of the times. CloudRS is implemented with the MapReduce model by operating over datasets stored in HDFS.
B. MapReduce implementation CloudRS is an iterative workload that follows several phases to process the input dataset, explained below: 1) LoadReads: This phase prepares the input dataset to be processed, discarding noisy information and converting the sequences into a more suitable format for Hadoop. In order to avoid the comparison of very repetitive sequences, it also builds a list of the most frequent subsequences. Later, this list is used to filter them out and avoid workload imbalance. 2) PreCorrection: Each sequence is split into different subsequences that are candidates in the next phases. The candidates for each subsequence are aligned to allow their comparison, using a wildcard pattern. 3) ErrorCorrection: The set of subsequence candidates is iterated through by using the information obtained in the previous phases.
First, the most frequent subsequences are filtered out. Then, the candidates are compared by emitting a vote for each position. When all the votes have been emitted, the correct alternative is chosen by majority. This calculation repeats several times until the obtained subsequences remain invariant. 4) Screening: Once the correct subsequences have been calculated, the input dataset is reprocessed to fix the errors, replacing each subsequence with its corresponding correct alternative. 5) Conversion: The output dataset is converted to a standard format in order to be processed by subsequent NGS applications (e. g. sequence alignment). Using these five phases, the execution of CloudRS involves a total of 11 MapReduce jobs, some of them being repeated during the ErrorCorrection phase. Their implementation uses an old version of the Hadoop API, although this only affects the interfaces used by the source code of the workload.
Regarding data input and output, Flame-MR is oriented to the processing of large textual datasets, as mentioned in Section IV-C. Therefore, it uses optimized input and output formatters that minimize the amount of connections to HDFS. Similarly, the implementation of textual data objects used in Flame-MR reduces the amount of memory copies and object creations when performing sort and copy operations. CloudRS makes use of both characteristics, and so it is especially well suited to be optimized with Flame-MR. However, the inefficient data formatting explained in the previous section is intrinsic to CloudRS, as it is performed inside the map/reduce functions. The goal of Flame-MR is to improve applications' performance without modifying their source code, and so we cannot modify those user-defined functions.
D. Experimental configuration This section describes the experimental configuration used to evaluate CloudRS with Hadoop and Flame-MR. As genomic applications are executed in many kinds of systems, the evaluation has considered two different scenarios: a private cluster with 9 nodes, Pluton, and a public cloud platform, Microsoft Azure  [26] , using 17 and 25 instances. As in the case of VELaSSCo, each cluster size n corresponds to 1 master and n-1 slaves. The hardware and software characteristics of Pluton and Azure are shown in Tables III and IV, respectively. Pluton nodes are equipped with 16 cores each, 64 GB of memory and one local disk of 1 TB, being interconnected via InfiniBand FDR and Gigabit Ethernet (GbE). The experiments in Azure The experiments have used Hadoop 2. 7. 4 and Flame-MR 1.
1. The configuration has been adapted to the characteristics of the systems, resulting in the parameters shown in Tables V and VI for Pluton and Azure, respectively. These parameters have been described in Table  II. Furthermore, the frameworks have used the IP over InfiniBand (IPoIB) interface available in Pluton, which allows taking advantage of the In-finiBand network via the IP protocol. Finally, some parameters such as the HDFS block size have been experimentally tuned to obtain the best performance on each system. The input dataset used in the experiments is SRR921890, which has been obtained from the DDBJ Sequence Read Archive (DRA)  [27]. It is composed of 16 million sequences of 100 bases each (5. 2 GB in total). The results shown in the following section correspond to the median elapsed time of 10 executions.
A. Overview As explained in Section V-A, genomic datasets generated by NGS sequencers contain redundant information due to the existence of overlapped reads. This characteristic causes the appearance of duplicate or near-duplicate sequences in large datasets, which neither provide new information nor improve the results of analytical processes. However, processing them consumes system resources and wastes execution time. Therefore, they are often removed to decrease the overall runtime of the downstream analysis. MarDRe is a MapReduce application that is used to detect and remove duplicate sequences in genomic datasets stored in HDFS. It is based on a prefix-clustering mechanism that groups the sequences by similarity. Then, the sequences within a group are compared by using an optimized algorithm that discards the sequences that do not provide new information.
B. MapReduce implementation The MapReduce workload used in MarDRe performs a single Hadoop job to process the data stored in HDFS. In contrast to CloudRS, MarDRe supports input datasets stored in FASTQ/FASTA, which are standard formats commonly employed in genomic datasets. The map phase is used to cluster the DNA sequences into groups. Each mapper reads the data belonging to its input split by using a custom formatter that reads the sequences in FASTQ/FASTA format. The input sequences are then divided into prefix and suffix to group the ones that share the same prefix. During the shuffle phase, the prefix is used as key to partition and sort the map output pairs. The value of the pair contains the sequence information by using a custom data type defined in MarDRe. Next, the map output pairs are sent to the reducer nodes where they are processed.
Furthermore, its implementation leverages the use of custom formatters and data objects to avoid inefficient parsing of the input dataset. Although MarDRe shows good performance with balanced workloads, real-world datasets are highly skewed, with lots of sequences that share a common prefix. This situation introduces important load balancing problems in the reduce phase due to the comparison of large sets of sequences. This causes some reducers to have excessive execution times. As a MapReduce job has to wait for all reducers to finish, the load balancing problem in the reduce phase affects the.
C. Challenges Flame-MR must adapt to the characteristics of MarDRe when optimizing its performance. As in the VELaSSCo use case, the use of custom formatters and data objects in MarDRe requires the utilization of the standard API provided by Hadoop, while keeping the in-memory optimizations, as explained in Section IV-C. Regarding the load balancing problem explained before, the standard behavior of Flame-MR emulates the way Hadoop processes the data without modifying the operation of the map and reduce functions. Therefore, load imbalance also affects Flame-MR. To alleviate it without changing the source code of the application, a new load balancing mode has been developed in Flame-MR version 1. 1. During the reduce phase, large partitions are detected and split into several chunks. In doing so, the computation is parallelized and the execution time of heavy-loaded reducers is decreased.
D. Experimental configuration MarDRe and CloudRS are both executed as preprocessing steps of an NGS analysis on Big Data infrastructures. Therefore, the evaluation of MarDRe has employed the same experimental configuration as CloudRS using Pluton and Azure as testbeds (see Tables III-VI in Section V-D), and the BDEv tool to conduct the evaluation. However, the computational requirements of CloudRS are significantly higher than those of MarDRe, and so a larger dataset was used in these experiments: SRR377645. This dataset is composed of 214 million reads of 100 bases each (67 GB). The evaluation includes the results of Hadoop, Flame-MR and Flame-MR with the load balancing mode activated (labeled as Flame-MR-LB in the graphs). In the experiments, the number of chunks of the load balancing mode has been tuned for improved performance, splitting each partition in 13 and 9 chunks for Pluton and Azure, respectively.
VII. CONCLUSION The MapReduce computing model and Hadoop are commonly used by many applications to extract valuable information from datasets stored in HDFS. Although other alternatives such as Spark can provide improved performance over Hadoop, the effort of adapting existing MapReduce applications to new APIs can be significant (provided that the source code is available). Flame-MR solves this problem by providing huge performance improvements in a transparent way without needing to change the applications. This paper has shown three different real-world use cases from two application domains: visualization queries (VE-LaSSCo) and preprocessing of genomic datasets (CloudRS and MarDRe). On the one hand, Flame-MR improves the execution time of the analytical queries of VELaSSCo by adapting its behavior to the custom input formats and data objects defined in the workload.
On the other hand, the iterative algorithm performed by CloudRS is significantly accelerated, overcoming some of the inefficiencies of its underlying implementation. Finally, the use of Flame-MR in MarDRe has not only optimized the underlying Hadoop data engine but also alleviated its load balancing problems. The execution of several use cases with distinct characteristics, together with the assessment of real-world datasets on different systems, have proved the significant performance benefits provided by Flame-MR over Hadoop.
ACKNOWLEDGMENT This work was supported by the  Ministry of Economy, Industry and Competitiveness of Spain  (Project  TIN2016-75845-P, AEI/FEDER/EU ) and by the  FPU Program  of the  Ministry of Education  (grant  FPU14/02805 ). The Grid'5000 testbed used in this paper is supported by  Inria ,  CNRS ,  RENATER and several French Universities. The authors would like to thank  Iván Cores  for his contribution to the deployment of VELaSSCo, and also  Pierre Neyron  and  Michael Mercier  for their help in the use of the Grid'5000 platform. Also thanks to  Microsoft Research  for awarding this work with a sponsored Azure account (ref. MS-AZR-0036P ).
Catchment set and data To compare the performance and behaviour of the reference and the discrete and continuous state-space GR4 model versions, a large data set of 240 catchments across France was set up (Fig. 3 ). Testing the models on many catchments will help obtain general conclusions  (Andréassian et al. , 2006; Gupta et al. , 2012). The data set was built by  Ficchí et al. (2016)  to test GR4 at different time steps. In this article, we only used daily and hourly data. The climate data of the SAFRAN daily reanalysis  (Quintana Seguì et al. , 2008; Vidal et al. , 2010)  are used as input data (precipitation and temperature). Precipitation and temperature are spatially aggregated on each catchment since the GR4 models are lumped. The hourly precipitation data were obtained by disaggregating the daily SAFRAN precipitation using the subdaily distribution of rain gauge measurements.
Acknowledgements. The first author's PhD grant was provided by  Irstea. We thank  Météo France  for providing the SAFRAN climatic data used in this work. We also would like to thank  Martyn Clark  for his advice in setting-up the differential equations,  Nicolas Le Moine  for sharing his ideas to replace the unit hydrographs and on the numerical integration,  Fabrizio Fenicia  for his advice on numerical integration and  Paul-Henry Cournède  for his analysis of the mathematical adequacy of the model. Finally, we give special thanks to  Andrea Ficchí  for his work on the database and for the discussions on the temporal stability of the GR4 model. We thank the topical editor,  Jeffrey Neal , for his monitoring of the review process and his relevant reviewers choice. We also acknowledge the two reviewers,  Barry Croke , and the anonymous reviewer for their very interesting and complementary remarks.
Code and data availability. The Fortran code used in this article can be freely downloaded from GitHub at https://github. com/ HYDRO-group-Irstea-Antony/GR4-State-space-version-1. 0 (last access: 18 April 2018). The state-space model can be tested on an example catchment data set with already calibrated model parameters. The full reference for this code can be found in the references (Santos, 2017); it is referenced with the following DOI: https://doi. org/10. 5281/zenodo. 1118183.
g. ,  Haberl et al. 2006; Mirtl et al. 2008 Mirtl et al. , 2013;; Anon 2009; Grove et al. 2013; Singh et al. 2013; Gingrich et al. 2016; Bretagnolle et al. 2018). Currently there are * 80 LTSER platform initiatives globally  (Mirtl et al. 2018). The LTSER network emerged as a bottom up process, where existing local and national initiatives became part of a network and recognized at the European level  (Singh et al. 2013). Conditions for joining the LTSER network (usually) include: support by the platform's local, regional and national authorities, the existence of long-term datasets (especially biodiversity indicators but also abiotic variables) and the inclusion and integration of socio-economic data  (Haberl et al. 2006 (Haberl et al. , 2009;; Mirtl et al. 2013). Dick et al. (2018)  showed that this vision has become reality in terms of a rapid increase of research publications.
Number of LTSER platforms and their spatial distribution At the core of our effort towards learning through evaluation is the Dynamic Ecological Information Management System-Site and Dataset Registry (DEIMS-SDR) database (https://data. lter-europe. net/ deims/). The aim of DEIMS-SDR is to be the most comprehensive catalogue of global environmental research facilities, featuring foremost but not exclusively information about all Long-Term Ecosystem Research (LTER) sites  (Mollenhauer et al. 2018) , and to provide data accessibility to researchers, policymakers, land managers and the general public. Thus, it is crucial to analyze both (1) LTSER platform data and (2) the LTER sites contained within each LTSER platform using the DEIMS-SDR system (see Fig. 2 ). We used this database to define the initial population of LTSER platforms to be analyzed.
To create a complete list of all LTSER platforms, we downloaded all the data for all European countries and Israel. Due to uncertainties in the design and terminology (see metadata https://data. lter-europe. net/deims/ documentation/site) used within the DEIMS-SDR system, we consulted the managers of the system. For the audit process we compiled a complete database of the 67 LTSER platforms listed as of December 2016, and gradually populated it with verifier variable data for the 16 indicators (Table  1 ). Platforms managed by European countries but not located in Europe were removed from the analysis (e. g. , LTSER Zone Atelier Hwange, Zimbabwe). Using this list, we first interrogated the DEIMS-SDR data base to create e-mailing lists for contacts with and surveys of the 67 LTSER platforms, extract data about the spatial location of the platforms and other information including the accreditation status of the platforms (formally approved or not by national organizations).