dataesr/scientific-paragraphs-categorization huggingface dataset:
Data To test various 3D architecture inference methods, we conducted experiments on both simulated datasets and publicly available genome-wide Hi-C datasets. For the simulation, we generated 170 datasets using the yeast genome architecture proposed by  Duan et al. (2010). Because the repetitive rDNA on yeast chromosome XII cannot be observed in practice, we discard all contacts involving these loci, and we do not infer the position of the corresponding rDNA. We generate these 170 datasets using the following model: c ij =Pðd ij Þ ð 8Þ where = 3 (corresponding to the theoretical exponent discussed in Section 2. 4) and varies between 0. 01 and 0. 7 (0. 01, 0. 01, 0. 02, 0. 03, 0. 04, 0. 05, 0. 06, 0. 07, 0. 08, 0. 09, 0. 1, 0. 15, 0. 2, 0. 3, 0. 4, 0. 5, 0. 6, 0. 7) with 10 different random generator seeds, thus obtaining 10 different datasets per parameter.
The parameter controls the number of contact counts in the datasets. A low will yield a dataset with few counts; hence, the corresponding wish distance matrix will be less likely to be close to the true distance matrix. To estimate how noisy the generated data are, we compute the following measure of signal-to-noise ratio (SNR): SNR= X c ij ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi X ðd ij À c ij Þ 2 q ð9Þ The numerator (the signal) corresponds to the number of counts, and the denominator (the noise) corresponds to the sum of deviation between each count and its expected value. We use this first ensemble of simulated datasets to assess the robustness to noise of the different methods. Note that in actual data, the SNR gets smaller when we sequence fewer reads or when we infer a structure at a higher resolution.
We simulated another ensemble of datasets to compare non-metric and metric methods when the parameters provided to the different algorithms are not the correct ones. We generate 20 datasets according to Equation (8), with between 4 and 2 (4, -3. 5, 3, 2. 5, 2) and between 0. 4 and 0. 7 (0. 4, 0. 5, 0. 6, 0. 7). We also applied our methods to publicly available Hi-C data from mouse embryonic stem cells  (Dixon et al. , 2012). We started with the data at 20 kb resolution and considered only chromosomes 1-19, with both available restriction enzymes (HindIII and NcoI). We then subsampled the data at resolutions of 100 kb, 200 kb, 500 kb and 1 Mb. Note that the methods studied here infer a single copy per chromosomes, thus yielding a consensus model for both homologous chromosomes.
Performance as a function of SNR We ran all six methods-MDS1, MDS2, NMDS, PM1, PM2 and ChromSDE-on the 170 simulated datasets with varying SNR levels. Our goal here is to assess how well the different methods manage to reconstruct a known 3D structure from simulated data at different SNR levels. Remember that SNR estimates how far the empirical counts differ from their expectations; in real Hi-C data, SNR typically decreases when we have fewer reads in total, or when we want to increase the resolution of the structure. In this first series of experiments, we provide the correct count-to-distance or distance-to-count transfer functions to the methods that need them (MDS1, MDS2, PM1). In this setting, for infinite SNR, all methods should consistently estimate the correct structure. Figure  1  shows the performance of the different methods in terms of RMSD (top) and distance error (middle) as a function of the parameter, which controls the SNR (bottom).
ChromSDE performs better than other MDS-based methods on datasets with a low SNR, corresponding to datasets with low coverage and, consequently, many non-interacting pairs of beads. This may be due to the way ChromSDE explicitly handles such pairs. On the other hand, in a more favorable setting (SNR42), ChromSDE does not perform as well as other MDS-based methods; we hypothesize that when the coverage is high enough, taking into account non-interacting pairs of beads does not add any additional information. Because ChromSDE is not better than other MDS-based methods, and requires much longer to run, we do not report its performance on the next experiments and instead focus on the differences between the other MDS-based methods and the PM methods. 3. 1. 2 Metric versus non-metric methods: robustness to incorrect parameter estimation Three of the methods tested, which we collectively refer to as metric methods, require as input a count-to-distance or distance-to-count transfer function: MDS1, MDS2 and PM1.
In reality, however, the DNA may not follow the ideal physical laws underlying the default transfer function discussed in Section 2. 4, and the structures inferred from these methods may diverge from the correct one because of miss-specification of the transfer function. To assess this phenomenon, and evaluate the robustness of the different methods (including NMDS and PM2, which automatically infer a transfer function), we now study the performance of the methods on datasets generated with varying parameters. We therefore run the MDS1, MDS2, NMDS, PM1 and PM2 methods on the second ensemble of simulated datasets. We provide the default transfer function to all metric methods, thus inducing a miss-specification for all simulated datasets with 6 ¼ 3. Figure  2  shows the RMSD of each method, averaged over the datasets with different , as a function of.
Note that, before computing our two error measures, we filter out from the pair of structures any beads for which the inference has not been done on either dataset, i. e. beads that have zero contact counts in either dataset. To give a sense of how similar the two replicate datasets are, we also compute the Spearman correlation directly on the data, rather than on the inferred structures. As expected (Table  1 ), the higher the resolution is, the lower the correlation between the pairs of datasets is and the more different the inferred structures are. Across different enzyme replicates, the PM2 method yielded significantly higher correlation than all of the other methods (P50. 05, signed-rank test adjusted for multiple tests with a Bonferroni correction). Zhang et al. (2012)  show that the mapping from contact counts to physical distance differs from one resolution to another, underscoring the importance of good parameter estimation.
We also showed that misspecification in the countto-distance transfer function can harm the performance of metric methods, while our model can adapt to unknown distributions within a parametric family. Finally, we also demonstrated, on real Hi-C data, the robustness of our methods to resolution change and enzyme duplicated datasets. Our probabilistic model of reads is similar to the model proposed by  Hu et al. (2013) ; however, instead of generating a family of structures by MCMC we use the model for direct maximum likelihood estimation of a consensus structure. Although the consensus structure might not be a definitive structure in vivo, it provides us with a rich model for further analysis, conserving hallmarks of genome organization such as the water lily form of the budding yeast  (Duan et al.
Funding: This work was supported by the  European Research Council  ( SMAC-ERC-280032  to J. -P. V. , N. V. ); the  European Commission  ( HEALTH-F5-2012-305626  to J. -P. V. , N. V. ); the  French National Research Agency  ( ANR-11-BINF-0001  to J. -P. V. , N. V. ) and by  National Institutes of Health awards  ( R01 AI106775 ,  P41 GM103533  and  U41 HG007000 ).
Implementation The GAPT framework is implemented in the programming language Scala  [19]. PROOFTOOL makes heavy use of Scala's Swing wrapper library. Details about PROOFTOOL and how it displays formulas, sequents and proofs can be found in  [7]. In this section we concentrate on the integration of the Sunburst view in PROOFTOOL. Our implementation is based on TREEVIZ, an open source library 3  written in Java. Since PROOFTOOL, and GAPT in general, are implemented in Scala, we have several wrappers around the classes from the TREEVIZ library. They also add functionality in the form of events which expose a newly selected node. We created a SUNBURSTTREEDIALOG that displays the structural information as a Sunburst tree as well as the information about the selected inference. The full architecture of SUNBURSTTREEDIALOG is shown in Figure  7  SUNBURSTTREEDIALOG consists of a SPLITPANE, containing our Sunburst wrapper REACTIVE-SUNBURSTVIEW on the left (upper) side.
Related Work Among the standard TPC benchmarks, TPC-DS, a decision support benchmark that models several generally applicable aspects of a decision support system, including queries and data maintenance  [5] , is closely related to data analytics we target at. However, although it can generate high volumes of data, its underlying business model is a classical retail product supplier, thence its dataset could not fully qualify as big data-oriented because of a lack in structural variety. MalStone is benchmark for data intensive computing and analysis  [6]. It features MalGen, a synthetic data generator that produces large datasets to perform benchmarking. Data is designed to assess systems from the parallel processing point of view. Data is generated probabilistically following specified distributions.
Cloud Harmony measures the performance of cloud providers as black boxes  [7]. The tests performed are mainly focused on assessing hardware performance or specific technologies. Cloud Harmony actually aggregates the results of benchmarks that existed before. The Yahoo! Cloud Serving Benchmark (YCSB) is a framework to facilitate performance comparisons among cloud database systems  [8]  that mainly focuses The first property we compare is whether the data processed by a benchmark is produced artificially or extracted from a real environment. Only PRIMEBALL offers the possibility to work with wholly real data, with the aim to better simulate real applications including facets of the problem difficult to emulate when using random distributions to produce data. Although some of the benchmarks used in Cloud Harmony do use real data, most of them are actually processing artificial data.
There are also in between positions such TPC-DS which produces data artificially but trying to follow the structure of a real environment. Proposing a real workload is the second property we selected. A benchmark system bearing this property is executing real-world operations to better simulate a production environment. PRIMEBALL, SWIM and TPC-DS execute only tasks that are closely related to real-world workloads. Moreover, some benchmarks in Cloud Harmony also execute tasks that are common in real environments but not all, for that reason has been marked with a tilde. The next property is satisfied by all analyzed benchmarks, i. e. , they are all aimed at assessing parallel processing. By contrast, Cloud Harmony is the only benchmark that assesses the performance of specific pieces of hardware.
g. , to measure the scale up of a distributed SQL database but it is not its purpose. On the other hand, PRIMEBALL has been designed as well to be able to assess this kind of properties. Complex data properties describe the benchmarks that are oriented to execute procedures using complex data structures to assess the system under test (SUT). TPC-DS implements a classical data warehouse with numerical and textual values. MalStone only aims to generate a big dataset as a log file and measures system response while processing it, thus the data processed is not complex. The same is true for YCSB, which assesses the performance of keyvalue stores. Values can be complex, but they are not processed, only stored. The other benchmarks, including PRIMEBALL, include complex data in some of their procedures.
Big data properties describe the benchmark systems that involve analytical aspects over large amounts of data. All benchmarks have analytical situations involving large amounts of data. PRIMEBALL has specifically been designed to satisfy this property. TPC-DS has been marked with a tilde because it can be used for this purpose but it depends mainly on the size of data used. Finally, technology independence describes the systems the are designed to work with several kinds of technologies. YCSB and SWIM do not fulfill this property, because YCSB is oriented to analyze the performance of key-value stores only; and SWIM only assesses MapReduce procedures, and thus only makes sense when the SUT is able to execute them. The other benchmarks, including PRIMEBALL, can be used in environments that are not constrained by a given technology.
Application Model PRIMEBALL's contextual application is set around New Pork Times: a fictitious on-line information service including international news, current affairs, documentaries, science, health and lifestyle sections. It is constantly updated and available 24-hours all around the world. New Pork Times hosts articles and multimedia documents about the latest news, as well as a large archive of past information. All of these data are stored in a system called New Pork Times' News Hub (NPT-NH), which resides in a cluster hosted by some cloud service provider. This cluster is managed by a framework for parallel data processing and provides a remote storage that allows the user to access the files in the cluster without having to worry about their distribution in nodes. This storage system allows the user to insert and update data, and also to execute batch processes for analyzing/processing the data.
The criteria to assess each metric are also defined; -does define data relationships. We provide a description of the information stored in the SUT to be processed during the benchmark run. On the other hand, PRIMEBALL does not: define technical execution details. It defines guidelines, but given that SUTs can be very different, the relevance of results is tightly related to implementation details; -define expected performance results. No absolute value is provided as a comparison point, given that they are subject to implementation details; -compare data retrieval or processing algorithms; -define a storage schema. However, we define how data are physically stored. Thence, PRIMEBALL is the first cloud-oriented unified benchmark aiming to assess all the elements involved in cloud-based big data application systems.
Types of Files to be Hosted The system's database shall hold only three types of files. However, there can be many files of the same type. The three types of files follow. -General information (XML): This set of files comprise the many XML documents that describe the standard information stored by NPT-NH, i. e. , information about authors, the actual news, and so on. Section 4. 2 describes the conceptual schema of this information. -Media files (binary): Some articles make references to these files, which can be either audio or video documents. -Metadata (XML): Several metadata for information retrieval and other tasks are extracted from the other two types of files by internal algorithms, for further querying. These metadata must be persisted as XML files in the system.
PRIMEBALL Schema The system must hold as XML files data about the following entities: articles: the actual news articles; -topics: the topics an article may belong to; -keywords: sets of words that roughly describe the content of an article; -languages: marks for indicating what language/dialect an article is written in; -authors: people who write the articles; -journalists: authors who work in journals and make interviews; -professionals: specialists in some topic who write special analyses; -countries: information about countries authors might be citizens of or work in; -dates: information about the day of the year when an article was written; -media: reference to a media file with some internal comments. The conceptual schema of this dataset is featured in Figure  1. Its actual implementation depends on the framework for parallel data processing to be benchmarked and its capabilities for storing data.
Initial Data To create an initial corpus for populating NPT-NH, PRIMEBALL shall come bundled with a crawler that extracts, transforms and loads information from one or more real world news hub akin to New Pork Times (a famous news site may come to mind). The crawler fetches information about news published during a requested period of time, which is recommended to be set up as the last 40 years. Moreover, it must also extract information about authors, media files available, and metadata about the articles relevant to the system's architecture. Due the big data properties of the benchmark, it is recommended to fetch at least Once the corpus has been fetched, it can be sliced at will for selecting any scale factor for the initial data population. However, it is important to bear in mind that if the whole corpus is used in the initialization phase, then it will not be possible to perform updates or scale operations, since there will not be extra data available.
Data Scaling and Maintenance This section describes the processes performed for updating and scaling data in NPT-NH. Basically, these processes are executed integrally as batch tasks. New Pork Times' authors write new articles every day. Articles have to be placed in NPT-NH for allowing access to users. New authors may also come in. The following scaling tasks have to be performed by the system. New data has to be obtained as progressive slices of PRIMEBALL's corpus, making sure that the data was not already inserted. Once new data are inserted, the system has to extract the necessary metadata from them so as to ensure that subsequent queries can be performed. More specifically, when new information is inserted, it is necessary to recalculate some structures used for information retrieval.
Query Set This section contains the set of queries that are typically performed by NPT-NH users and that must be used for testing the performance of any given parallel data processing framework. These queries must be performed over the dataset defined in Section 4 and involve the most important aspects of performance. Although the following set of queries is not finite, it covers a wide range of classes from Figure  1 , the relationships between them; it is applicable and enables to successfully measure performance. One first subset of queries concerns the most common and hottest topics published in the system. The user might be interested in this information in order to know what kind of issues are the most frequently described in a certain interval of time, and therefore attract attention.
Operations 1. Execute queries 4, 7 and 14 from the generic query set (Section 5. 1) choosing as date, e. g. , September 12, 2001. 2. Double the volume of the dataset according to scenario 7. 3. Repeat the queries executed in step 1, for the same date and another one, e. g. ,  November 5, 2008. Scenario 2 This scenario simulates an extreme situation: a very famous article has been published with many mistakes and publishers are correcting it constantly. Their main concern is to deliver a consistent view of the article to people. Here, we are interested in measuring how many times the article is read in the older versions after being read once in the new version.
Maximum number of versions of an article. Scenario 6 The aim of this scenario is to initialize the system and make it ready for handling the information for New Pork Times. Initial state Empty storage system. Operations Execute the steps described in Section 4. 3 to initialize the environment. Scenario 7 This scenario is used to increase the volume of the dataset to simulate the fact that new articles are inserted over time. Initial state NPT-NH has a consistent state. Operations Execute the steps described in Section 4. 5 for data scaling and maintenance.
If it takes too much time, users are going to look for a different website, thus a low latency is required. -Measurement : Latency of the SUT can be measured as the throughput when executing scenarios 4, 5 and 6 (Section 5. 2). 5. Durability: ability of the system to retain information for a long period of time. -Importance: In the case of a news website, it is very important to ensure that no information is lost. Users have to be able to check and find information they have read previously. -Measurement : Scenario 1 (Section 5. 2) is intended to measure data durability. We define the durability ratio as a metric for this purpose. Durability ratio = Correct reads Total reads 6. Consistency and version handling: two different readings of the same data at the same time should return the same value.
-Importance: It is important for a website to give a consistent view of data to all users at the same time around the world. In the proposed model, There may be several revisions of an article, which has to be consistent for all readers. -Measurement : Using scenario 2 (Section 5. 2), the performance of the system for this property can be measured using the consistency ratio as a metric. Consistency ratio = Consistent reads Total reads 7. Availability: data is accessible even when there are some inaccessible nodes. -Importance: It is very relevant for New Pork Times to guarantee the access to all the news stored in the system. -Measurement : Scenarios 3 and 6 (Section 5. 2) aim to measure this property, thus the throughput of the SUT can be taken as a metric for this property. 8. Concurrency: the system has to be able to offer a service to different clients at the same time.
Pricing In Section 6. 1, we defined sytem performance w. r. t. time and cost. The main pricing factors involved in processing data in the cloud follow. -Cloud provider: different cloud service providers may have different pricing policies. -Number of instances and type: infrastructure used to execute PRIMEBALL. -Required storage space: it is directly related to the scale factor used (Section 5. 2). -Platform inherent costs: operation, administration and maintenance. -Execution time: time spent to run the tests. The specific cloud provider model has to be applied to compute real cost.
Conclusions We propose in this paper the specifications for PRIMEBALL, a complete and unified benchmark for measuring the characteristics of parallel cloud processing frameworks for big data applications. In front of the already existing benchmarking options, PRIMEBALL can be used as a guideline to build an integral solution for benchmarking could platforms. The real-life model adopted in PRIMEBALL is that of a fictitious news hub called New Pork Times, which is basically a fair approximation of a popular real-life news site. The general architecture and inner processes of the system are well-defined so as to allow an unambiguous implementation of the benchmark. The workload applied on this news dataset is not only made of queries, but also of data-intensive batch processes. Moreover, we propose several use-case scenarios together with relevant metrics for assessing the framework's performance from different points of view, such as data availability or horizontal scalability.
The novelty of our work lies in the fact that existing, related benchmarks measure parallelization capabilities, cloud features, big data analysis ability, but none of them combines all these properties while exploiting real-life data. Future work on PRIMEBALL will be mainly focused on implementing a crawler for fetching and transforming real data from the Web to feed the benchmark's dataset. Distributing the built dataset online will improve the repeatability of the experiments. Moreover, the actual feasibility and relevance of PRIME-BALL shall be validated by actually implementing the benchmark in several cloud environments to obtain experimental results and by publishing performance comparison results. For instance, implementation in popular data processing frameworks such as Hadoop should be achieved.