dataesr/scientific-paragraphs-categorization huggingface dataset:
Cost Model Join computation cost is the total cost of several stages, including pre-processing, data reading, map processing, communication between nodes, reduction processing, and data storage. The parameters used in the cost model are shown in Table  1. The general cost model for the join computation of two datasets is described in Equation (4). C(J) = C pre + C read + C map + C tran + C reduce + C write (4) where: • C pre = 0 (there is no preprocessing task of the three algorithms) • C read = (|R| + |L|) • c r • C map = (|R| + |L|) • c m • C tran = |D| • c t • C reduce = MAX(C reduce(1) , C reduce(2) ,. , C reduce(n) ) • C write = C write(1) + C write(2) + • • • + C write(n) Some values (|D|, C reduce , and C write ) are different in the three algorithms. Cost of reading/writing data in reducer i.
Range-Based Partition A range-based partition processes skewed data by randomly distributing skew records to the reducers and duplicating records with the same key in the remaining dataset. If α = 50%, the intermediate data generated are significant, but the hash-based partition algorithm cannot be performed because the nodes processing the skewed data will be overloaded or even stop working. That is, in this case, a range-based partition is more efficient than a hash-based partition. Therefore, we only need to consider the cases of moderately skewed ratios and to compare the costs of the algorithms based on the performance of the reducers. • |D| = |R| + |L| -Σ k i=1 (|L| + α i ) + Σ k i=1 (|L| • α 2 i • n) • C reduce(1) = C reduce(2) = • • • = C reduce(n) • ⇒ C reduce ≈ ( |R |+|L |-Σ k i=1 (|L|+α i )+Σ k i=1 (|L|•α 2 i •n) n ) • c d (**) • ⇒ C write = n • ( |O| n ) • c r = |O| • c.
• We compared the three algorithms based on the performance of the reducers as follows: |D| ≈ |R| + |L| • C reduce(1) = C reduce(2) = • • • = C reduce(n) • ⇒ C reduce ≈ ( |R |+|L | n ) • c d (***) • C write = n • ( |O| n ) • c r = |O| • c r 4. 1. 4. Analysis Since skew-join records occur often in reducer 1 and other skew keys are insignificant, reducer 1 determines the execution time of the algorithms. We compared the costs of join computation between the three algorithms. Therefore, we have Equation  (5). C(J) HBP > C(J) RBP > C(J) MDRP (5) A range-based partition performs join operations with skewed data more efficiently than a hash-based partition thanks to the fragment-replicate mechanism. A multi-dimensional range partition is more efficient than a range-based partition thanks to the thorough skew processing of both datasets.
Data Description We generated experimental datasets with scalar skew distribution. The term "scalar skew" was introduced by Christopher Walton and his colleagues  [19]  when developing a taxonomy of skew effects. Scalar skew distribution was later used by other researchers for skew handling  [6, [23] [24] [25]. In this work, we generated six datasets, in which each had 100,000,000 records. The idea of scalar skew distribution is that, in 100,000,000 records, the skew-join key with a value of 1 appears in some fixed number of records. The remaining records contain randomly appearing join key values from 2 to 100,000,000. This will help us to easily understand which experiments are performed while keeping the output size constant over varying amounts of skew. The format of the datasets was plain text consisting of three fields separated by commas, the primary key, the join key, and other text (pk, jk, others).
The six created datasets had different skew ratios for the experiments. The details of the datasets are presented in Table  2. We used three algorithms-hash-based partition, range-based partition, and multidimensional range partition-in the three test cases. In each case of running the algorithms, we ran them three times to get the average execution time and evaluated the three algorithms based on their execution times. The join selectivity was the number of output records divided by the number of records in the cross product of the input relations. In this work, we kept the output size of 10 9 for all three test cases.
The multi-dimensional range partition was the worst in this case, as it was 1. 7 times slower than the hash-based partition. The range-based partition appeared to have an average performance in comparison with the other two algorithms in the three test cases. Regarding Formula (5), the experimental results were appropriate for the cost models presented above. Considering the advantages and disadvantages of the algorithms, with the hash-based partition, the advantage of this algorithm is that it is easy to implement and commonly used in join computation. However, it is not ideal for handling join operations with a high skew ratio. Tuples with the same join keys are hashed to the same reducers, leading to seriously imbalanced tasks. As can be seen, datasets with more skew-join keys had a worse performance; on the contrary, datasets with fewer skew-join keys had a better performance.
In the case of the range-based partition, the tuples with more skew data were selected for range determination. It is quite easy to determine sub-ranges with a small cost with this method; thus, it has been widely used to deal with data-skew problems. This method is more efficient than the hash-based partition algorithm, but it is not a good choice in the case of datasets with a low skew ratio. A limitation of this algorithm is that the determination of the sub-ranges does not take into account the size of the join results; thus, a joining product skew can arise. The multi-dimensional range partition overcomes the limitation of the range-based partition by using a partitioning matrix instead of a split vector. The cross products of the sub-ranges represented by cells in the partitioning matrix can be estimated to avoid the oversizing of the join results.
Conclusions Implementations of MapReduce are used to perform many operations on very large datasets, including join operations. It has become a prominent parallel and distributed programming model for efficiently handling massive datasets. The major obstacle of the model for join processing is data skew. If the data are significantly skewed, with some common join keys appearing to have a greater frequency than the others, the reduction tasks of these join keys will finish later than those of the others. Thus, any benefits from parallelism become meaningless. There are some algorithms that have been proposed to solve the problem of skew joins. Several surveys on solutions for skew joins have been made, but an adequate and systematic comparison in the Spark environment has still not been presented.
Range-based partition algorithm for dataset R-a build relation Input: An input record r ∈ R Input: Split vector Input: A list of partitions P Begin 1: Create Sub-range 2: for each r ∈ R do Range-based partition algorithm for dataset L-a probing relation Input: An input record l ∈ L Input: Split vector Input: A list of partitions P Begin 1: Create Sub-range 2: for each l ∈ L do.
Algorithm 1 2 12 Range-based partition algorithm for dataset R-a build relation Input: An input record r ∈ R Input: Split vector Input: A list of partitions P Begin 1: Create Sub-range 2: for each r ∈ R do Range-based partition algorithm for dataset L-a probing relation Input: An input record l ∈ L Input: Split vector Input: A list of partitions P Begin 1: Create Sub-range 2: for each l ∈ L do.
MVIC data processing Details of the MVIC instrument design and operation are given in Reuter et al. (  2008 )  33. Science operation of MVIC is under time delay integration (TDI) mode with two panchromatic and four color arrays. We select the observations obtained by the color arrays as they contain Pluto haze's spectral characteristics. Level 2 data is used for analysis in this work, whose identifier on NASA PDS is New Horizons MVIC Pluto Encounter Calibrated Data v3. 0  49. The data contains bias-subtracted, flattened images, but it does not include corrections for scattered light, cosmic rays, and geometric and motion distortion. We follow the New Horizons SOC to Instrument Pipeline ICD to convert the calibrated data number (DN) to I/F, then use the SPICE system  50  to compute the geometry with navigation data (Supplementary Figure  1 ).
Retrieving algorithm We use the Markov-chain Monte Carlo (MCMC) method  36  as the parameter searching tool. It derives the posterior probability density function (PDF) of each parameter by comparing the posterior probabilities of proposed parameter sets. The cost function with posterior probability (p) is defined as ln(𝑝) = - 1 2 ∑ (𝑋 𝑖 -𝜇 𝑖 ) 2 𝜎 𝑖 2 𝑖 ( 20 ) where μi and σi are the value and uncertainty of the i-th observation, respectively, and Xi is the modeled i-th observation computed using a given proposed parameter set during one MCMC attempt. The Python package emcee  37  is used to implement the parameter searching algorithm. We initiate the MCMC process with 40 chains and flat priors for all parameters, and run the parameter search for 1000 steps. The last 500 steps are selected for the result analysis, which are considered to be in the equilibrium state, as most of the chains converged after only 200 steps.
However, they were unable to differentiate between bimodal and power-law haze particle size distributions. In addition to observations in the UV and visible, scattering spectra of Pluto's haze was also observed in the IR by LEISA, which was not included in the aforementioned works. Here we discuss how consideration of LEISA data addresses the degeneracy in haze size distributions. Supplementary Figure  8  shows the scattering intensities of twodimensional aggregates and three-dimensional spheres in the visible (LORRI) and IR (LEISA) with unit UV extinction (Alice), assuming tholin refractive indices. The monomer radius in the aggregates is assumed to be our retrieved value of 20nm. The ratio of scattering intensity in the visible to the UV extinction for aggregates (Supplementary Figure  8a ) shows that large aggregates (>0.
Acknowledgements We thank  William M. Grundy  for sharing the LEISA data,  Michael L. Wong  and  Xue Feng  for improving figure representations, and  Yan Wu  for comments. P. G. is supported by  NASA   Hubble Fellowship  grant  HST-HF2-51456. 001-A  awarded by the  Space Telescope Science Institute , which is operated by the  Association of Universities for Research in Astronomy, Inc. , for  NASA , under contract  NAS5-26555. X. Z. is supported by  NASA Solar System Workings  Grant  80NSSC19K0791. A. F. C. is supported by  NASA  under the New Horizons Project.
Despite Ammonia sp. and H. germanica having similar cogwheel interface densities, the average bulk  18  O-enrichment in H. germanica was 45% higher than in Ammonia sp. (Fig. 3 , Supplementary Table  2 ). All three species belong to the same order, Rotaliida, but are differentiated at the Superfamily level, with Ammonia sp. and H. germanica belonging to Rotalioidea and A. lessonii to Asterigerinoidea. Consistent with this classification Ammonia sp. and H. germanica have similar test geometry (Supplementary Fig. 8 , Supplementary Software), but Ammonia sp. test chambers appear thicker than in H. germanica. To quantify this, tests from both species were embedded in epoxy, ground down to the widest portion of the test and polished, imaged in SEM, and the images analysed using the image processing software ImageJ (Supplementary Software).
The total cogwheel interface length is the total length of all cogwheel interfaces excluding pore-calcite interfaces and pixels along the image frame. The cogwheel interface density is calculated as the ratio of the total cogwheel interface length to the net test surface area (i. e. , in μm per μm 2 ). Supplementary Fig. 11  provides an overview of this process. The diagenesis resistance ratio was calculated using a modified version of the ImageJ macro from van Dijk et al. 43  (Supplementary Software). Foraminifera tests were embedded in epoxy and polished down to the widest cross-section of the test (using diamond polishing pastes with a final grain size of 0. 25 µm) and imaged using backscattered electrons (BSE) SEM. Grey value thresholding was used to select the test area to create a binary image.
In addition to quantitative values that are displayed via a color gradient, the dot size is also used to represent another quantitative information, e. g. the fraction of cells expressing a given gene within a single-cell RNA-sequencing (scRNA-seq) dataset  (Habib et al. , 2017; Lukassen et al. , 2018; Ordovas-Montanes et al. , 2018; Wu et al. , 2018). Although simple dot plots can be easily generated with basic graphics commands (with ggplot2 for example), it becomes more complex and time-consuming when more elaborated plots are needed (with multiple and customized layers/legends such as dendrograms). In recent years, multiple tools have been developed to make dot plots like the rain plot method  (Henglin et al. , 2019)  or the corrplot function of the feature-expression heatmap method  (Benno Haarman et al.
Implementation FlexDotPlot is implemented in R and takes advantage of several publicly available R packages for data visualization, manipulation and analysis (Supplementary Table  S2 ). It requires data in standardized input format (data frame) as input: the first two columns contain the two factors to spread along the x and y axes (e. g. genes and cell populations for scRNA-seq datasets), followed by the corresponding quantitative and/or qualitative data to be displayed (Fig. 1A ). From this input, dot plots can be produced with a single command line or interactively with a Shiny application.
S1 ). Several minor parameters are implemented in the function to easily custom the plot legends, shapes, colors and/or labels. Moreover, a ggplot2 object can be returned so that users can customize even further the output by adding custom layers if needed. FlexDotPlot also provides parameters to enhance the resulting figure with dendrograms, which is rarely available in existing methods (Supplementary Table  S1 )  (Wilk et al. , 2020). Depending on the variable types, a principal component analysis (quantitative variables), a multiple correspondence analysis (qualitative variables) or a factor analysis for mixed data (quantitative and qualitative variables) is performed prior to clustering analysis. To further illustrate the versatility and usefulness of FlexDotPlot in representing multimodal data, we used FlexDotPlot on several scRNA-seq datasets combined with complementary results including CITE-seq, copy number variation (CNV) or cell communication data  (Bi et al.
, 2021; Stoeckius et al. , 2017; Tirosh et al. , 2016). Particularly, we reproduced dot plot figures from published articles or tutorials and proposed enhanced versions using FlexDotPlot (Supplementary Figs. S1-S3 ). The first example (PBMC dataset) illustrates how dot plots are commonly used with scRNA-seq datasets (Supplementary Fig. S1A ). Instead of using dot size to represent the percentage of cells expressing a given gene, one can illustrate this parameter by using the shape feature (Supplementary Fig. S1B ). Row and column dendrograms are also added to this plot. The second example is also based on a classical dot plot representation (Supplementary Fig. S2 ). Here, we keep the common features (size to represent the percentage of cells expressing a gene and color to represent the average scaled expression) and take advantage of the shape feature to represent the CNV status of each gene in each patient.
Up to three additional factors can be optionally provided and used for the dot plot. (B) Description of how a factor is represented in a dot plot according to its type (qualitive or quantitative) and assigned dot feature. Size feature requires a quantitative factor while other features are compatible with any type with adapted rendering FlexDotPlot is an easy-to-use tool for generating highly customizable dot plot representations. It uses standardized input and output formats (data frame and ggplot2 object, respectively), which also makes the combination of FlexDotPlot and other R pipelines possible. A set of vignettes is provided with the package to illustrate the dot plot construction procedures and their associated parameters. Each dot represents multiple features for one gene (vertical axis) in a given cell cluster (horizontal axis).
Funding This work was supported by the  Swiss National Science Foundation  [SNF n CRS115_171007], the  French National Institute of Health and Medical Research (Inserm)  [ HuDeCA  project to F. C. ], the  Research Institute for Environmental and Occupational Health (IRSET) , the  University of Rennes 1  and the  French School of Public Health (EHESP). S. L. was supported by the chair " Cancer & Innovation" of Rennes 1 Foundation  (https://fondation. univ-rennes1. fr/)and a specific grant from the  LabEx IGO program  (n  ANR-11-LABX-0016 ) funded by the  «Investment into the Future» French Government program , managed by the  National Research Agency (ANR).
This information must have been previously computed, for example, by performing a multiple-genome alignment. Some of these methods, such as SplitMEM  [8]  and TwoPaCo  [9] , use graphs or combine graphs with a generalized compressed suffix array (GCSA)  [10, 11, 12]. Other methods use a custom data structure based on sequence alignment methods  [13, 14]. MuGI  [15]  stores the reference in compact form (4 bits to encode a single char), a variant database (one bit vector for each variant), and an array retaining information about each k-mer. Few methods aim to store a pan-genome without prior knowledge, and even fewer methods allow direct query on pan-genomes. CHICO  [16]  program uses a hybrid index that combines Lempel-Ziv compression techniques with the Burrows-Wheeler transform but fails to index our data (see  Section 3).
This extra payload in memory enables faster processing than directly merging the k-mer suffixes and their bit vectors. Indeed, this auxiliary vector (array (B) of Figure  1(b) ) uses 16-bit words (instead of 32 or 64 bits for pointers) to store the indices of the suffixes stored in K - pref and the merging of the two orders. Because the data structure partitions the set of indexed k-mers according to their common prefix of size k 1 , it is easy to parallelize the algorithm presented in Listing 1. Thanks to the Open-MPI specification  [21] , each instance of the RedOak program only processes a portion of the k-mers. This allows us to run RedOak on a cluster, on a multi-core architecture or on a combination of them. This feature has two major advantages: the required memory is split across the running instances, allowing scaling of the method to a very large collection of genomes, and the wall-clock time is drastically reduced (see Section 3).
Discussion Implementation. RedOak is implemented in C/C++ and its construction relies on parallelized data processing. A preliminary step, before indexing genomes, is performing an analysis of the composition in k-mers of the different genomes. During this step, k-mer counting tools could be involved and their performance is crucial in the whole process  [22]. We looked for a library allowing us to handle a large collection of genomes or reads, zipped or not, working in RAM memory, and providing a sorted output. Indeed, RedOak uses libGkArrays-MPI [1] which is based on the Gk Arrays library  [23]. The Gk array library and libGkArrays-MPI are available under CeCILL licence (GPL compliant). The libGkArrays-MPI library is highly parallelized with both Open MPI and OpenMP. To manipulate k-mers, the closest method is Jellyfish  [24].
Setting a higher value is not necessary; otherwise, it may allocate unused memory. Benchmark. The experiments were performed on a SGE computer cluster running Debian. The cluster (SGE 8. 1. 8) has two queues. The "normal" queue has 23 nodes, having 196 GiB of RAM and 48 cores [3] each. This queue represents 4. 4 TiB of memory and 1104 cores. The "bigmem" queue possesses 1 node, having 2 TiB of RAM and 96 cores [4]. The benchmark was performed on both "bigmem" and "normal" queues. Our dataset of 67 uncompressed rice genomes is equal to 25 Gib. which represent 26194967769 nucleotides. Comparison of RedOak, Jellyfish and BFT for the index build step. We compared RedOak to two other methods, namely Jellyfish  [24]  and Bloom Filter Trie (BFT)  [17]. The comparison was performed on the 67 de novo assembled rice genomes from Zhao et al.
[26]  by comparing the time used for the index build phase and the maximum memory consumption. The size of the data set was successively set to 10, 20, 30, 40, 50, 60 and 67 genomes out of the original data set. Jellyfish builds an index for each genome, and then these indexes were merged to produce a matrix where the counts for each k-mer in each genome are stored (small modification of the merge tool implementation of Jellyfish). For JellyFish, we also created a program that simulates a parallelization of jobs. BFT needs ASCII dumps to build its index. These dumps were produced using Jellyfish. For Jellyfish and BFT the reported values are the total time taken for both the counting and merging steps. For all experiments, the k-mer size was set to k = 27, since BFT requires k to be a multiple of 9.
For RedOak, the prefix length was set to 12 (default setting), which gives a table of prefixes of very reasonable total size i. e. , 4 12 = 128 MiB. Each prefix index of each running instance represents 3. 2 MiB i. e. , 32 MiB by node. This drastically reduces the risk of saturation during the experiments. For each subset, we used RedOak in parallel on 10 "normal" nodes of the cluster and on each node we reserved 4 cores. For each subset, we also used JellyFish (jellyfish count -m 27 -s 500 M -t 10) on 40 genomes in parallel using 40 nodes. BFT does not allow merging the indexes created and does not propose parallelization. Therefore, we ran each instance of BFT in parallel using one "bigmem" node for each subset. The results are summarized in Table  1  and in Figure  3. BFT was not able to index datasets in the runs with 40 or more genomes.
To evaluate the query time of JellyFish, we had to request each file individually. The Table  2  shows, for JellyFish, the max RAM memory (including the writing time of all k-mers), the time of the longest query, and the total time (sum of all times which gives us the average time: 4403. 7 sec per file). The results showed that RedOak has better performance than JellyFish for querying this dataset. Example of PAV analysis. Analysis of presence-absence variation (PAV) of genes among different genomes is a classical output of pan-genomic approaches  [1] ,  [27] ,  [26]. RedOak has a nucleotide sequence query function (including reverse complements) that can be used to quickly analyze the PAV of a specific gene among a large collection of genomes. Indeed, we can query, using all k-mers contained in a given gene sequence, the index of genomes.
7min). Automatic post-analysis of the residual water resonance in each individual scan resulted in its relative amplitude (A), linewidth (B), chemical shift (C) and phase (D) time courses. Peak linewidth thresholds were automatically adjusted (red, B) and corresponding scans were discarded from the dataset (E). Most of the scans acquired after subject nodding (scan index c. a. 64) were discarded demonstrating the efficiency of the correction algorithm in detecting motion. The data discard algorithm automatically adjusts thresholds to improve SNR and/or spectral resolution. These thresholds can be set in terms of peak amplitude, linewidth, chemical shift or phase. In the case of this particular experiment, the algorithm found that discarding data based on the peak linewidth estimation was the optimal method to improve the final SNR and/or decrease the final FWHM.