dataesr/scientific-paragraphs-categorization huggingface dataset:
Dominant SNR period analysis during the wheat growth cycle Figure  1  shows an example of the multipath SNR data from the ascending track of GPS01 on 21 January 2015. Its average power spectrum (Fig. 1b ) derived from a wavelet analysis is also shown, together with the power spectrum (Fig. 1c ) for periods ranging from 128 to 1024 s. The average power spectrum presents a single peak and the corresponding peak period is 362 s. The SNR data is reconstructed well (red line in Fig. 1a ), using this peak period. Both phases and amplitudes match very well. This shows that the peak period from the average power spectrum can be used to represent the multipath SNR data. Limiting elevation angle values from 5 to 20 • (Sect. 2. 1) ensures a relatively stable value of the peak period. The peak period is considered as the dominant period (T d ) of the multipath SNR data.
Additionally, the major part of the signal power is concentrated on elevation angles ranging from 7 to 11 • (see Fig. 1 ). A preliminary analysis for the entire wheat growing cycle showed that, more often than not, the best elevation angle corresponding to the peak power was around 9 •. In this study, elevation and its change rate at 9 • are used to represent the SNR data for all available satellite tracks (37 per day). It must be noted that this reference elevation angle is specific to the gain pattern and height of the antenna encountered in this experiment. It could present different values in other antenna configurations. During the wheat growth cycle, preliminary tests showed that the average power spectrum could present multiple peaks together with a reduced maximum average power. This made T d unsuitable for the representation of the multipath SNR data.
The area sampled by GNSS retrievals is much larger than what can be achieved using individual soil moisture probes and much smaller than pixel size of satellitederived products. Longer continuous time periods of GNSS retrievals should be envisaged to serve as independent validation data sources in statistical methods such as Triple Collocation  (Dorigo et al. , 2010). We successfully assessed the surface soil moisture retrieval technique for a wheat crop field, during the start of the growing period. However, the rather narrow range of surface soil moisture values during the corresponding experimental time period limited the representativeness of the obtained retrieval accuracy. Furthermore, our data set did not include GNSS data and in situ VSM measurements for periods of bare soil. Longer periods presenting a bare soil surface should be investigated in future studies.
At the same time, more in situ vegetation measurements should be carried out. The retrieved vegetation height was based on the dominant period of the average power spectrum. The latter was derived from GPS multipath SNR data for elevation angles between 5 and 20 •. We only considered the dominant period variations, without accounting for instantaneous phase changes. The accuracy of the retrieved vegetation height could probably be improved considering changes in both period and phase of the multipath SNR oscillations. In this study, only the SNR data of the L1 C/A signal was used, SNR data from different wavelengths (e. g. , L1 C/A, L2C and L5) should also be compared or combined to survey canopy characteristics. A linear relationship between wheat height and dry biomass was observed during the period from wheat tillering to ripening.
Conclusions GNSS SNR data were obtained using the SNR GNSS-IR technique over an intensively cultivated wheat field in southwestern France. The data were used to retrieve either soil moisture or relative vegetation height during the growing period of wheat. Vegetation growth tended to decrease the relative antenna height and broke up the constant height assumption used in soil moisture retrieval algorithms. Soil moisture could not be retrieved after wheat tillering. A new algorithm based on a wavelet analysis was implemented and used to extract the dominant period of the SNR and to retrieve vegetation height. The dominant period was derived from the peak period of the average power spectrum derived from a wavelet analysis of SNR. The method proposed by CH16 was used to retrieve soil moisture under sparse vegetation conditions, before wheat tillering.
Acknowledgements. The work of  Sibo Zhang  and  Karen Boniface  was supported by the  STAE (Sciences et Technologies pour l'Aéronautique et l'Espace) foundation , in the framework of the  PRISM (Potentialités de la Réflectométrie GNSS In-Situ et Mobile ) project. Authors would also like to thank the farmer,  A. Blanquet , for his time and the person who helped for collecting ground data. Edited by: Alberto Guadagnini Reviewed by: two anonymous referees.
The increase of data collection in various domains calls for an adaptation of methods of visualization to tackle magnitudes exceeding the number of available pixels on screens and challenging interactivity. This growth of datasets size has been supported by the advent of accessible and scalable storage and computing infrastructure. Similarly, visualization systems need perceptual and interactive scalability. We present a complete system, complying with the constraints of aforesaid environment, for visual exploration of large multidimensional data with parallel coordinates. Perceptual scalability is addressed with data abstraction while interactions rely on server-side data-intensive computation and hardware-accelerated rendering on the client-side. The system employs a hybrid computing method to accommodate pre-computing time or space constraints and achieves responsiveness for main parallel coordinates plot interaction tools on billions of records.
Introduction Recent years have seen a striking increase in the amount of collected and generated data. For instance, in web analytics, visitors' behavior is analyzed to the event-level to improve services and marketing strategies. Naturally, such amounts bring about new challenges for storing, querying and analyzing these ever-growing datasets. Among other approaches, information visualization enables the exploration and understanding of complex data without prior knowledge of the patterns and trends to identify. However, traditional information visualization systems are not fitted for large-scale data exploration and have to be adapted to tackle the new challenges that arise from the surge of dataset sizes. The first consequence of this growth is the increase of the time needed to process the data.
However, some may conceal patterns and most techniques are not adapted for remote visualization constraints. Adapting parallel coordinates for the interactive visualization of large and remote data requires schemes to bound the size of the transferred data from the storing location to the visualization point. One solution lies in using a form of data reduction and representing an abstract version of the data, of a bounded size, regardless of the original dataset size. A second requirement is having a similar adaptability (or scalability) in the processing capabilities of the system, relative to the workload induced by interactive manipulation like brushing. This can be achieved using horizontal scalable systems which seamlessly parallelize processing over a network of computing and storage units where data is replicated and partitioned.
Related Work On line-based parallel coordinate representations, each multidimensional data item is depicted with a polyline and the multiplication of such polylines produces a rapid increase in the usage of pixels as the dataset gets larger. Such consumption of pixels is not practical for large datasets: the plot rapidly becomes overcrowded as polylines cross and overlap. Moreover, rendering and interaction complexity generally depend on the dataset size which requires fast data querying techniques to ensure responsiveness for large datasets. In this context, we discuss two topics of related work. First, existing methods for effectively avoiding clutter and over-plot in parallel coordinates and secondly, previous works on scalable visualization systems for parallel-coordinate representations and others.
For instance, Rübel et al. [32]  and Perrot et al. [33]  propose systems associating several of these solutions to scale interactive representations (respectively parallel coordinate and heatmaps) to extremely large datasets. Rübel et al. [32]  present a system combining the multi-resolution binning technique of Novotny and Hauser  [28]  and FastBit  [37] , an index/query tool, on a supercomputer. Perrot et al. [33]  use Canopy clustering for multilevel heatmaps computed over a distributed infrastructure, most similar to our setting. Ideally, a visual exploration system should support interaction following the user's flow of thoughts  [38]  i. e. , should operate in less than one second. Generally, bringing interactivity on modest systems operating on large data is a trade-off between pre-computation, approximate computation  [39]  and the cost of the hardware used.
For instance, Rübel et al. [32]  use a supercomputer system to achieve 0. 15 s response time for tracing 500 items, over a dataset of almost 200 million items, with 100 computing units. In this paper, we target common-hardware and use an infrastructure where the distributed storage is leveraged for computation. Compared to supercomputers, these infrastructures do not offer as efficient communication between units and generally less computing power. Nevertheless, they are much more accessible and affordable and quite common nowadays in various domains (e. g. , web analytics). To address data transfer, we use data reduction in the form of aggregation, with per-dimension clustering, as described in the following section.
Distributed Processing Work-Flow The platform is used for hosting the raw data and computing abstract representations for different states of the visualization: corresponding to the initial view or resulting from user interactions. The different types of queries performed by the client are: display of a dataset for a given axis ordering and brushing data given a set of selected aggregates. The back-end computes a per-dimension clustering of the raw dataset and the aggregates (clusters and meta-links) corresponding to those two types of query. While the aggregates formed from the full data are always computed in a preparatory step, the one formed from subsets (used in brushing), result from both beforehand and on-demand computed data depending on the query type. The pre-computing step (see Figure  2 ) encompasses computing an abstraction of the raw data and the results of several queries which are inserted into the distributed database.
Tasks & Interactions Common tasks for parallel coordinates include gaining an overview, evaluating correlation between dimensions, identifying subsets of items presenting similar features, and searching for item with a particular multidimensional profile. Gaining a complete overview of multidimensional data implies being able to see every pairs of dimensions. However, without interaction, the ordering of axes only represents a fraction of all the information the dataset contains (non-contiguous dimension relationships are not represented). Bringing axes side-by-side allows an analysis of the meta-links and eases clusters distribution comparison. Using the interval encoding, it also allows the identification of correlation, recognizable by parallel meta-links between the two considered axes.
Perceptual Scalability This section presents the effectiveness of the technique described in this article. Our technique is based on abstract visualization and as such, functions identically regardless of the dataset size, large or not. Indeed, only the chosen resolution parameter influences the representation (see Section 5). On the contrary, traditional line-based technique do not scale visually to large data. Thus, we perform two case-studies: a comparison with traditional parallel coordinate plot using a state-of-the-art dataset (the cars dataset provided in  [44]  with 400 records) and an exploratory analysis of large data (with 1. 6 billions of records).
Large Dataset Visual Analysis The C2C dataset contains 1. 6 billions elements of web traffic data on a C2C (Consumer to Consumer) website (see Figure  8 ). This dataset contains 9 dimensions where each item is a sequence of pages visited by the same user in a given time. The three first axes represent the user's browser, OS and device type; the following three are its screen properties (resolution height, width and total resolution). The last ones correspond to the number of visited pages during the session, the session duration and the average time spent per page. One can easily identify on Figure  9  two major devices corresponding to Desktops and Smartphones and a smaller one, the Tablets (respectively labeled D, S and T). We can notice that users can be distinguished in two equal size categories according to the device they use: either a mobile device (Smartphones or tablets) or desktop device.
While analyses with classical parallel coordinates are element-oriented, using abstracted parallel coordinates, they are aggregate-oriented. This makes sense as abstracted parallel coordinates are dedicated to big data analysis and rather focus on major trends rather than single element analysis. As for any abstract method, the aggregation used, in our case the clustering algorithm and resolution parameters chosen, strongly affect the representation and thus, the possible analyses. Pre-computing the same dataset at various resolution parameter or/and using various algorithms makes possible to refine the analysis but at the expense of storage. Furthermore, using a higher resolution parameter has direct impact on the data transfer size and thus, can affect responsiveness negatively.
Implementation Details On the client side we use WebGL, a web variation of OpenGL, to perform GPU-based computation (using GLSL vertex and fragment shaders) and render visualizations in browsers. More precisely, the client is compiled from C++ to Javascript using emscripten which also binds OpenGL calls to WebGL instructions. We implemented our back-end system in an Hadoop environment, a data-intensive infrastructure providing distributed storage (HDFS), computing (Spark-MapReduce) and database system (HBase) with horizontal scalability. For on-demand computing, we use Elasticsearch  [46] , a search system which runs along with the Hadoop components (see Figure  2 ). Pre-computation operations (abstraction and prepared selections) are implemented with Spark. The basic of the method is to produce RDDs (for Resilient Distributed Dataset, the main data structure in Spark), where rows represent records with appropriate (key, values) and to retrieve the desired metrics (minimum, count, etc) by reducing values by keys.
The outcomes of these experiments depend on different factors. In addition to the number of computing units and the communication overheads, they depend on the chosen dataset and abstraction properties (itself dependent on the clustering and the resolution parameter k). We considered three types of generated datasets presenting different inter-dimension correlation factors  [47] , and with size ranging from 10 6 to 10 9 records for 15 dimensions and 15 clusters per dimension. The first type is independent: every pair of dimensions has a close to null correlation factor and close to the maximum number of meta-links between each couple of dimensions (about k 2 ). This type represents the worst case for our system. The second type is correlated, generated to obtain a correlation factor of at least 0.
Pre-Computing Performance Preparing a dataset consists of computing an abstraction and all single-aggregate (clusters and meta-links) selections. The result of this one-time operation populates HBase with abstract data and prepared queries. When using on-demand computation, an Elasticsearch index is also populated with cluster identifiers for each dimensional components of each record. Pre-computing time measurements (excluding Elasticsearch indexing) are shown in Figure  11a  for different data sizes (in number of records) of the three types of dataset. The pre-computation processing is dominated by the computation of all single-aggregate selections. Although we consider the clustering method as parameter that should be chosen in regard of the studied data, we included this step in the measurement.
However, it only accounts for about 0. 10% of the total processing time on average. This processing effectively occurs over pre-aggregated data composed of identical records which are records falling into the exact same cluster for each dimensions (as well as the same cluster histogram bin). These records are considered identical because they are not distinguishable given the interaction tools provided by our system. Due to its properties, the best case dataset is reduced to about 10 • k such identical records (where 10 is the number of inner-cluster histogram bins). Additionally, the best case dataset presents only about k meta-links between each dimensions while correlated and independent have about k 2 , that is k times more. Therefore, about k times more single-aggregate selections are pre-computed for those two later datasets.
The difference in number of pre-computation and the size of their input data explains the diverging trend observed on Figure  11a  between, on the one side the correlated and independent datasets and on the other side the best case dataset. Indeed, this optimization is not efficient for the correlated and independent datasets (almost no records are identical). On our platform, the pre-computation step takes up to 32 h for datasets of a billion of records. For the largest independent and correlated datasets, the execution time raises up to 24 h. This computation step runs distributively, hence can be accelerated by increasing the number of computing units participating the task. Figure  11b  presents the mean execution time and the corresponding speedup of this step run on the correlated dataset with 10 7 records.
Prepared Selections Query Performance To evaluate the performance of single-aggregate selection retrieval, we perform a comparative benchmark between fetching prepared results from HBase and on-demand computation with Elasticsearch (both using 16 computing units). For different dataset types and sizes, we measure response time for selection queries corresponding to each visible elements on an initial display, that is each of its clusters and meta-links. We average the results of the meta-links and clusters independently and examine results for the same operations using Elasticsearch. This way, we can characterize the gain of pre-computing compared to (distributed) on-demand computation. Figure  12  compiles these results for correlated datasets (independent datasets shows similar results).
Overall, prepared queries can be retrieved in less than 0. 1 s from HBase and the response time does not seem to increase with the dataset size while for Elasticsearch, the response time increases until exceeding the second for the largest datasets. This shows that past a certain size of dataset, there is a significant gain in using pre-computation. Here response time was measured for all possible single-aggregate selection o(clusters and meta-links) on an initial view of correlated datasets with varying number of records.
On-Demand Query Performance Compound selection queries rely on on-demand processing and operate as a conditional filter followed by an aggregation. As such, they are dependent on the size of the targeted subset of records. Therefore, we are particularly interested in evaluating a cost bound for this operation to ensure good performance. To limit the cost of the aggregation, we query Elasticsearch for at most a subset of half the size of all the dataset by choosing to filter the dataset complement of the original query when it is preferable. In these cases, the server uses the pre-computed full abstraction to compose the original query's result. We choose to evaluate a higher upper bound for this type of query: the selection of all aggregates at once which means aggregating all clusters and meta-links.
This case is presented on Figure  13a. The response times increase with the dataset size, which is expected. For datasets of up to 10 8 items, the response time is lower than one second, however, for upper sizes, the responses times are getting very important. It indicates that the number of computing units is not sufficient to keep up with the aggregating costs for correlated and independent datasets. To ensure sub-second compound selections for this type of datasets and configuration, more resources have to be allocated. The surprisingly low latencies observed for the best case dataset result from the same pre-aggregation treatment of identical records mention above: when populating Elasticsearch indices, only distinct records are inserted. Consequently, all aggregation are applied on smaller data.
As shown, the testing platform shows its limits for the largest datasets used. We measure the speedup gained by using different number of instances in Elasticsearch cluster relative to using only two. Figure  13b , presents the result of the execution time of all possible cluster selection queries on the 2 × 10 8 version of the independent dataset. Here, the speedup appears better than the optimum as it is relative to non-sequential execution. The results indicates that the computing capabilities of the platform are linked to the number of executor with no major loss of performances. (a) Upper bound for all on-demand queries, corresponding to computing the total initial view for varying dataset types and number of records; (b) Scalability of on-demand queries, tested on cluster-selection queries for an independent dataset with 2 × 10 8 records.
In our case, as the data abstraction requires pre-processing, we leveraged that first necessary step by also preparing query results for several interactions. As demonstrated, using pre-processing, we ensure lower response time compared to on-demand computation for single-aggregate selection queries on the largest datasets of our experimentation set. Choosing one strategy or the other for a given query type generally depends on three factors: the available storage, the time allowed for pre-processing and the needs of interactivity i. e. , how fast the response-time should be. For the particular case of compound aggregate selections, there is a combinatorial explosion in the number of possible queries which motivates the choice of on-demand computation. Overall, the experiments shows that on our testing platform, we achieve the targeted sub-second performance for all pre-computed queries: single-aggregate selections and axis reordering queries.
Among theses interactions, two of them (axis reordering and subset highlighting) imply data transfers between the infrastructure and the rendering client. Our system guarantees upper bounds for these transfers, ensuring interactivity and responsiveness of the rendering client. We present and evaluate an implementation of the back-end using using the Hadoop ecosystem (HDFS and HBase), Spark and Elasticsearch. The results indicate a good scalability and reasonable responsiveness of our system on up to a billion of items. A first interesting work to pursue is thoroughly studying real-time distributed computing scalability of different technologies to ameliorate the pre-computing step in our solution. Lower preparation times would let users change dataset and tune clustering parameters in a more responsive way.
( a )Figure 5. a5 Figure 5. Two selection views on a C2C dataset described in the next section. Here, items are visiting session on a website. Selecting an aggregate (cluster or meta-link) triggers the highlight of the selected subset through all the displayed axes. The inner-cluster histograms are also updated according to the selection. (a) Selection of the smartphone category (on DeviceType); (b) Selection of the higher range of values (between 37 and 300) on the NbPages axis which relates to the number of visited pages.