dataesr/scientific-paragraphs-categorization huggingface dataset:
For a given patient several context-specific pseudonyms may be assigned to a virtual and inaccessible patient identifier, namely the EUPID. The identity management enables data aggregation using a hidden reference table. Phonetic hashing is proposed to prevent duplicated patient registration. Re-identification of patients is possible via a trusted third party. The architecture allows an implementation in a distributed computing environment, including cloud-based elements. Of note, an application programming interface to query the EUPID metadata is not yet available. We deliberately chose to use specific nominative input data. A study on nominative data availability  [20]  showed that the most available patient information is, in the following order, the month of birth, the first name, the year of birth, the day of birth, and the last name, representing 99.
g. changes in the first name. This is very rare, but we recommend that source system managers notify the BNDMR team of these changes and to send the old and new IdMRs to avoid creating a new patient file in the national database. Second, errors may occur during data entry. We recommend integrating consistency and quality control measures during the data entry process. To be implemented in different European countries, if some characters do not have a corresponding uppercase in our referential, a complementary referential is necessary. For characters from other alphabets (e. g. , ñ, Å, Ø, letters which contain diacritical marks, Cyrillic or Greek characters,. ) a table of correspondence with our referential will have to be implemented. In effect, if these characters were eliminated it would increase the risk of collisions.
In this case, the issue of accuracy must be addressed in the data quality procedures at the national database level. The risk of re-identification is an important security issue  [23]. Hackers with access to the de-identified patient files could try to trace patient identities. The reconstruction of a mapping table linking all possible values of the nominative input data to their calculated IdMRs could be a way to re-identify patients. However, the necessary resources to build this table would be considerable, with a computation time on the order of several decades. Another approach would be to use the remaining existing information in the patient file to infer the patient's nominative identification, even though the IdMR is an anonymised, non-significant string of characters. Indeed, it may be possible to re-identify a patient from his age, place of birth, and diagnosis, especially since the number of patients per rare disease is by definition low.
Acknowledgements We acknowledge all the  BNDMR  project team members and  S Kabani  for her expert help in the manuscript editing procedure. The authors thank the reviewers for their fruitful and constructive comments. Banque Nationale de Données Maladies Rares, Hôpital Necker Enfants Malades, Assistance Publique des Hôpitaux de Paris, Paris, France. 2  INSERM ,  U1142 , and  UMR_S 1142 ,  LIMICS, Sorbonne University, Paris, France. 3 Pierre and Marie Curie University, Paris, France. 4 Paris 13 University, F-93430 Villetaneuse, France. 5 UPRES EA2415, Clinical Research University Institute, Montpellier University , 641 avenue du  Doyen Gaston Giraud , 34093 Montpellier, France. 6INSERM UMRS 933 ,  Rare Disease Cohorts (RaDiCo) ,  Sorbonne University , and  Hôpital Trousseau ,  Assistance Publique Hôpitaux de Paris, Paris, France.
ACKNOWLEDGMENTS This work was partly funded by the  Laboratoire d'Excellence Empirical Foundations of Linguistics  ( ANR-10-LABX-0083 ). We thank  Yair Haendler  for his helpful advice for the bayesian analysis and  Olaf Mikkelsen  for his contribution in running the experiments. We also thank  Eunice Fernandes ,  Holly Branigan , and  Moreno Coco  for kindly letting us use their visual materials for the eye-tracking experiment. The authors apologize for this error and state that this does not change the scientific conclusions of the article in any way. The original article has been updated.
Survey questionnaire and administration We sent the Web-based survey by email via Sphinx© software (SphinxOnline 3. 1. 5) to IRB members for whom an email address was provided. One reminder was sent to non-respondents at 6 weeks. The survey included six clinical vignettes (see Additional file 1) to illustrate one of the justifications typically evoked when reporting the summary results of a published unbalanced RCT. The justifications considered were (1) obtaining more safety data, (2) cost, (3) methodological reasons, (4) strengthening patients' acceptability to participate, (5) high expected drop-out rate, or (6) "bad deal trials" (e. g. , when a trial proposes a known inferior treatment option or a treatment with significant harm and no prospect of benefit). IRB members were asked if using an unbalanced randomization was justified and ethically sound in each of these different situations.
The aim of this chapter is to present an overview of the major past and present state-of-the-art data-centric benchmarks. Our review includes the TPC standard benchmarks, but also alternative or more specialized benchmarks. We survey benchmarks from three families: transaction benchmarks aimed at On-Line Transaction Processing (OLTP), decision-support benchmarks aimed at On-Line Analysis Processing (OLAP) and big data benchmarks. Eventually, we discuss the issues, tradeoffs and future trends in data-centric benchmarking.
INTRODUCTION In data management, both system designers and users casually resort to performance evaluation. On one hand, designers need to test architectural features and hypotheses regarding the actual (vs. theoretical) behavior of a system, especially in terms of response and scalability. Performance tuning also necessitates accurate performance evaluation. On the other hand, users are also keen on comparing the efficiency of different technologies before selecting a software solution. Thence, performance measurement tools are of premium importance in the data management domain. Performance evaluation by experimentation on a real system is generally referred to as benchmarking. It consists in performing a series of tests on a given system to estimate its performance in a given setting. Typically, a data-centric benchmark is constituted of two main elements: a data model (conceptual schema and extension) and a workload model (set of read and write operations) to apply on this dataset, with respect to a predefined protocol.
Both models may be parameterized. Most benchmarks also include a set of simple or composite performance metrics such as response time, throughput, number of input/output operations, disk or memory usage, etc. The Transaction Processing Performance Council (TPC), a non-profit organization founded in 1988, plays a preponderant role in data-centric benchmarking. Its mission is to issue standard benchmarks, to verify their correct application by the industry, and to publish performance test results. TPC members include all the major industrial actors from the database field.
BACKGROUND Transaction Processing Benchmarks The first TPC benchmark for relational, transactional databases, TPC-C (TPC, 2010), has been in use since 1992. TPC-C features a complex business database (a classical customer-order-product-supplier model with nine types of tables bearing various structures and sizes) and a workload of diversely complex transactions that are executed concurrently. The performance metric in TPC-C is transaction throughput. As all TPC benchmarks, TPC-C's only parameter is a scale factor SF that determines data size. TPC-C was complemented in 2007 by TPC-E (TPC, 2015a), which simulates a brokerage firm with the aim of being representative of more modern OLTP systems. In its principles and features, TPC-E is otherwise very similar to TPC-C. There are few alternatives to TPC-C and TPC-E for relational applications.
Yet, some benchmarks fit niches where there is no standard benchmark. For instance, OO7  (Carey et al. , 1993)  and OCB  (Darmont & Schneider, 2000)  are object-oriented database benchmarks modeling engineering applications, e. g. , computer-aided design or software engineering. However, their complexity makes both these benchmarks hard to understand and implement. Moreover, with objects in databases being more commonly managed in object-relational systems nowadays, object-relational benchmarks such as BUCKY  (Carey et al. , 1997)  and  BORD (Lee et al. , 2000)  now seem more relevant. Such benchmarks focus on queries implying object identifiers, inheritance, joins, class and object references, mul-tivalued attributes, query unnesting, object methods, and abstract data types. However, typical object navigation is considered already addressed by object-oriented benchmarks and is not taken into account.
Moreover, object-relational database benchmarks have not evolved since the early 2000's, whereas object-relational database systems have. Similarly, XML benchmarks aim at comparing the various XML storage and querying solutions developed since the late nineties. From the early so-called XML application benchmarks that implement a mixed XML database that is either data-oriented (structured data) or document-oriented (in general, random texts built from a dictionary), XBench  (Yao et al. , 2004)  stands out. XBench is indeed the only benchmark proposing a true mixed dataset (i. e. , data and document-oriented) and helping evaluate all the functionalities offered by  XQuery. FlexBench (Vranec & Mlýnková, 2009 ) also tests a large set of data characteristics and proposes query templates that allow modeling multiple types of applications.
Decision-Support Benchmarks TPC-H (TPC, 2014a) has long been the only standard decision-support benchmark. It exploits a classical product-order-supplier database schema, as well as a workload that is constituted of twenty-two SQL-92, parameterized, decision-support queries and two refreshing functions that insert tuples into and delete tuples from the database. Query parameters are randomly instantiated following a uniform law. Three primary metrics describe performance in terms of power, throughput, and a combination of power and throughput. However, TPC-H's database schema is not a star-like multidimensional schema that is typical in data warehouses. Furthermore, its workload does not include any true OLAP query. TPC-DS (TPC, 2015b) now fills in this gap. Its schema represents the decision-support functions of a retailer under the form of a constellation schema with several fact tables and shared dimensions.
TPC-DS' workload is constituted of four classes of queries: reporting queries, ad-hoc decision-support queries, interactive OLAP queries, and extraction queries. SQL-99 query templates help randomly generate a set of about five hundred queries, following non-uniform distributions. TPC-DS features one primary throughput metric that takes both query execution and data warehouse maintenance into account. Given the primordial importance of data integration in many data-centric (including data warehousing) scenarios, TPC-H was recently complemented by TPC-DI (TPC, 2014b). TPC-DI focuses on Extract, Load and Transform (ETL) processes. Data are first generated in a staging area as if they were extracted from a virtual retail brokerage firm's operational databases. Then, data are transformed through, e.
However, APB-1 is very simple and rapidly proved limited to evaluate the specificities of various activities and functions. Thus, more elaborate alternatives were proposed, such as DWEB  (Darmont et al. , 2007) , which can be parameterized to generate various ad-hoc synthetic data warehouses and workloads that include typical OLAP queries, and SSB  (O'Neil et al. , 2009) , which is based on TPC-H's database remodeled as a star schema and features a query workload that provides both functional and selectivity coverage. It is also worth noting that TPC-DS is a complex benchmark. Thence, simpler benchmarks are still in use, especially for testing OLAP scenarios in the cloud. For instance, TPC-H was used to benchmark Hadoop and Pig  (Moussa, 2012)  and SSB for testing the efficiency of view materialization in the cloud  (Perriot et al.
Big Data Benchmarks In the timely trend of big data analytics, benchmarking needs are as high as ever to compare alternative systems, including the many NoSQL database systems. In this context of data variety, volume and velocity, adaptability and scalability are premium features that must be evaluated. The TPC promotes two benchmarks for big data. TPC-VMS (TPC, 2013) is actually a benchmarking environment for virtualized databases that allows running TPC-C, TPC-E, TPC-H or TPC-DS on three virtual machines (VMs). Its metric is the minimum value of the selected benchmark's metric on all VMs. TPCx-HS (TPC, 2015c) focuses on Hadoop and MapReduce-based applications. It models a simple application (with no data model directly available) and features, in addition to classical metrics, availability and energy metrics.
Before the TPC could issue its big data benchmarks, and still in parallel, there are many industrial and academic initiatives. MalStone (Open Cloud Consortium, 2009) is a benchmark for assessing data intensive parallel processing. It features MalGen, a synthetic data generator that produces large datasets generated probabilistically following specified distributions. In the same line, HiBench (Huang et al. , 2010) is a set of Hadoop programs, ranging from data sorting to clustering, aimed at measuring metrics such as response time, HDFS bandwidth consumption and data access patterns. SWIM (Chen et al. , 2013) also measures the performance of Hadoop/MapReduce systems. SWIM contains suites of workloads of thousands of jobs, with complex data, arrival, and computation patterns, and therefore provides workload-specific optimizations.
Finally, HcBench (Saletore et al. , 2013) models real Hadoop usages in a datacenter. HcBench features various job types and data sizes. By contrast, YCSB  (Cooper et al. , 2010)  is a framework that focuses on data, and more specifically on performance evaluation of key-value stores. YCSB defines several metrics and workloads to measure system behavior in different situations, or the same system when using different configurations. OLTP-Bench (Curino et al. , 2012)  is the first true benchmarking framework designed for cloud transactional database systems as a service. OLTP-Bench actually features a set of existing microbenchmarks (i. e. , designed to test one very specific aspect of performance, e. g. , ResourceStresser), popular benchmarks (e. g. , TPC-C) and real-world applications (e. g.
, Wikipedia). Regarding big data analytics, PRIMEBALL  (Ferrarons et al. , 2013)  aims at providing a reallife context to cloud data warehouse benchmarking. Its authors provide the specifications of a fictitious news site hosted in the cloud that is to be managed by the framework under analysis, together with several objective use cases and measures for evaluating system performance. The Big Data Benchmark (Amplab, 2014) goes one step further by actually implementing existing analytical workload models by  Pavlo et al. (2009). Its only metric is the response time of such relational queries as scans, aggregations and joins. It can be used for both MapReduce-based systems (such as Shark and Hive) and classical parallel database systems. BigBench (Rabl et al. , 2015)  is a so-called a specification-based benchmark that is independent from technology.
BigBench relies a lot on TPC-DS, borrowing its data model and part of its workload model. The remainder of the workload is adapted from big data use cases issued by the McKinsey Global Institute. Finally, yet other benchmarks, i. e. , CloudSuite  (Yasin et al. , 2014)  and  DCBench (Jia et al. , 2013) , feature machine learning and data mining-oriented workload models that mostly run on Hadoop and exploit the Mahout library. Eventually, BigDataBench  (Wang et al. , 2014)  aims at providing the widest possible scope of big data models and workloads. It includes nineteen benchmarks representing a large variety of data models, workload models and application scenarios from search engines, social networks, ecommerce, multimedia analytics and bioinformatics. Workload models cover OLTP, "cloud OLTP" and OLAP.
ISSUES AND TRADEOFFS IN DATA PROCESSING BENCHMARKS 1. Relevance: The benchmark must deal with aspects of performance that appeal to the largest number of potential users. 2. Portability: The benchmark must be reusable to test the performances of different DBMSs. 3. Simplicity: The benchmark must be feasible and must not require too many resources. 4. Scalability: The benchmark must adapt to small or large computer architectures. In their majority, existing benchmarks aim at comparing the performances of different systems in given experimental conditions. This helps vendors position their products relatively to their competitors', and users achieve strategic and costly software choices based on objective information. These benchmarks invariably present fixed data and workload models. Gray's scalability factor is achieved through a reduced number of parameters that mainly allows varying database size in predetermined proportions.
All TPC benchmarks notably feature a single scale factor parameter. This solution is simple (still according to Gray's criteria), but the relevance of such benchmarks is inevitably reduced to the test cases that are explicitly modeled. For instance, the typical customer-order-product-supplier data model from TPC benchmarks is unsuitable to many application domains. This leads benchmark users to design more or less elaborate variants of standard tools, when they feel these are not generic enough to fulfill particular needs. Such users are generally not confronted to software choices, but to architectural choices or performance optimization tradeoffs within a given system or family of systems. In this context, it is essential to multiply experiments and test cases, and a monolithic benchmark is of reduced relevance.
To enhance the relevance of benchmarks aimed at system designers, three solutions are possible. The first one is to design an ad-hoc benchmark for a particular application, e. g. , RTW-bench, Spadawan and BenchDW, for real-time, spatial and biological data warehouses, respectively. However, the benchmark's application span is necessarily quite narrow. One alternative is to resort to benchmark generators, also called tunable or generic benchmarks, such as OCB, DWEB or FlexBench, which help generate various data or workload models, and thus allow experiments to run in various conditions. The other alternative, which is preferred in recent benchmarks such as YSCB, OLTP-Bench and BigDataBench, is to offer a unifying framework that includes a comprehensive set of stateof-the-art benchmarks. However, the two latter approaches are mechanically detrimental to simplicity, which is a primordial criterion.
FUTURE RESEARCH DIRECTIONS The previous section showed that classical transaction and decision-oriented benchmarks are well established. However, big data benchmarking, which predominantly uses cloud technologies, faces a new paradigm and must measure new features. Thus, in addition to  Gray's (1993)  criteria for building a good benchmark,  Folkerts et al. (2012)  propose that the quality criteria that are commonly accepted by the benchmarking community must be revisited. Although the cloud inherits from a long legacy of distributed systems, important issues are unique to the cloud. For instance, the concept of elasticity applied to data management may translate in the ability to bring in new data sources dynamically to meet emerging needs  (Pedersen, 2010). Thus, cloud benchmark data models should be dynamic.
Moreover, the three or four Vs (if we include veracity) of big data are unequally addressed in current benchmarks, which mostly focus on scaling (volume) and, to a lesser extent, on variety with multi-benchmark suites such as BigDataBench. Among recent benchmarks, only HcBench features inter-job arrival rates that can simulate data streams. Yet, it is a quite low-level benchmark. Thus, RTDW-Bench and Bär and  Golab's (2012)  stream warehouse benchmark could be welcome additions in multi-benchmark suites. With respect to veracity, specific security issues appeared in the new framework of the cloud, e. g. , cloud provider or subcontractor espionage, cost-effective defense of availability or uncontrolled mashups  (Chow et al. , 2009). Such features are important to assess, for security is one of the top concern of cloud users and would-be users.
Data consistency is also a concern, e. g. , PRIMEBALL's metrics do not only target transaction performance and storage costs, but also data consistency. Bermbach et al. (2013)  further advocate for a standard comprehensive benchmark for quantifying the consistency guarantees of eventually consistent storage systems. Moreover, for web application-based benchmarks, data quality assessment is also of premium importance. To the best of our knowledge, this intricate task has not been included yet in any big data benchmark. Finally, the economic model of the cloud is fundamentally new. Instead of a costly initial investment, pay-as-you-go models allow users to pay a small amount per use, e. g. , of a dataset, in return for a one-time advantage  (Pedersen, 2010). Thus, cost is also a key criterion when benchmarking cloud/big data solutions.
CONCLUSION Benchmarking is a small field, but it is nonetheless essential to data-centric research and industry. It serves both engineering and research purposes, when designing systems or validating solutions; as well as marketing purposes, when monitoring competition and comparing commercial products. We subdivide benchmarks in three classes. First, standard, general-purpose benchmarks such as the TPC's do an excellent job in evaluating the global performance of systems. They are well suited to software selection by users and marketing battles by vendors, who try to demonstrate the superiority of their product at one moment in time. However, their relevance drops for some particular applications that exploit data or workloads models that are radically different from those they implement.
Adhoc benchmarks are a solution. They either are adaptations of general-purpose benchmarks, or specifically designed benchmarks. Designing myriads of narrow-band benchmarks is not time-efficient, though, and trust in yet another new benchmark might prove limited in the community. Hence, the last alternative is to use generic or multi-benchmarks that feature a common framework for generating various experimental possibilities. The drawback of this approach is that benchmark complexity must be mastered. In conclusion, before starting a benchmarking experiment, users' needs must be carefully assessed so that the right benchmark or benchmark class is selected, and test results are meaningful. It is nonetheless clear that the TPC plays a primordial role in the data benchmarking community, not only by issuing standards, but also by structuring and leading the community, e.
Vincent Chau is supported by the  national key research and development program of China  under grant No. 2019YFB2102200 , by the  Fundamental Research Funds for the Central Universities , and by the  CAS President's International Fellowship Initiative  n o  2020FYT0002 ,  2018PT0004. Nguyen Kim Thang  is supported by the  ANR  project  OATA  n o  ANR-15-CE40-0015-01 ,  Hadamard PGMO  and DIM RFSI.