dataesr/scientific-paragraphs-categorization huggingface dataset:
Historical range of the urban data bases. Former Soviet Union United States Europe * China India Brazil South Africa 1300 1800 1850 1900 1950 2000 © GeoDiverCity -UMR Géographie-cités CNRS * European continent, Turkey and the former Soviet Union excepted FIG. 1: Database N0 (initial date) NF (final date) D Method Brazil (a) 531 (1872) 2615 (2010) 11 Administrative (municipios + metropolitan areas) Former Soviet Union (DARIUS) (b) 91 (1840) 1929 (2010) 11 Morphological (municipalities at three administrative levels) India (IndiaCities) (c) 503 (1901) 5841 (2011) 12 Combined morphological, and functional China (ChinaCities) (d) 605 (1964) 9294 (2000) 4 Combined administrative (xian, qu, xianjishi (district level) and zhen, xiang (subdistrict level)) morphological and functional South Africa (DYSTURB) (e) 14 (1911) 220 (2001) 10 Combined morphological and functional (white cities + black townships) Europe (PARIS- Bairoch-Geopolis) 3619 (1950) 4413 (2010) 14 Morphological (municipalities then agglomerations) (f) United States (Harmonie-cités) (g) 5 (1790) 909 (2010) 23 Combined administrative then morphological and functional (places, cities, retropolated micropolitan and metropolitan areas).
With 8,559 patients and 825 participating physicians, the EPI3 La-Ser study has been the largest study carried out to date in France to describe and compare, in a large representative sample of patients consulting their GPs, factors associated with different types of medical practice  [13] [14] [15] [16] [17] [18]. We used data from this study to examine the economic perspectives. This current-economic analysis was carried out on the population from the cross-sectional sample of the EPI3 study. In France, the gate keeper coordinates all of the patients' care. He/she orients it if necessary towards a specialist clinician or a hospital service. The GP provides the first access to healthcare and has an overall view of the health status of the patient. As patients presenting with chronic musculoskeletal disorders and who consult with a homeopathic physicians state that their GP is their regular GP in 54.
Data collection The data collected in the cross-sectional EPI3 study are described elsewhere  [13] [14] [15]. Briefly, the descriptive variables of the patients were: age, gender, professional group (when the patient was <18-years the professional status of one of his/her parents was recorded), supplementary health insurance coverage, governmental universal health insurance coverage, type of GP consulted, length of follow-up by the GP, the main reason for consultation, hospital admissions during the previous 12 months, presence of a DLD (because additional governmental coverage was available for these individuals), a physical score on the SF12 scale and a mental score on the SF12 scale. The variables concerning the physicians' prescriptions were as follows: the medicines, phytotherapy (plant derived therapy) and/or oligotherapy (trace mineral derived) products, absences from work (duration of sickleave, if there was a prescription), other medical examinations (radiological, CT scans, MRI, biological, non-hospital nursing care, and physiotherapy) and if patients were referred to a specialist.
Criteria used for the economic evaluation We evaluated the economic impact of medical consultations and medical prescriptions, according to the calculation used to estimate healthcare costs for long-term and non-hospital care, as described by Ehreth  [21]. Hence, costs such as hospital admissions, which are relatively rare and were not the focus of this study, were excluded. Three types of costs were analysed: (i) the cost of the consultation, using the tariffs for medical consultations for the 2008 period; (ii) the cost of medical prescriptions, involving the total amount of medicines prescribed using the prices listed in the Gers Officine January 2008 database  [22] ; and (iii) the total cost of management which combined the costs of consultations plus prescriptions. According to the system of healthcare management in France, each of these costs was evaluated as the: (i) cost to Social Security; (ii) the remaining cost (to the patient and/or to the supplementary health insurance); and (iii) the health expenditure (a combination of the two previous costs).
Analyses were conducted using the unit price of the drug, the number of prescribed boxes, and the Social Security reimbursement rate of the drug in 2008 (100 %, 65 %, 35 % or 0 %), with these data having been previously validated by experts. The costs of the prescriptions were calculated by taking into account a medical exemption of 0. 25 € because, according to the Commission des Comptes (in June 2009  [25] ), one box of medicines out of two involves a medical exemption of 0. 50 €. The cost to the patient and to the supplementary health insurance was the cost remaining after reimbursement by Social Security. Disassociation of the two costs was not possible because the conditions of the supplementary health insurance contracts differed from one patient to another.
and this strategy has been shown to reflect real-life clinical practice. Although these definitions potentially limit the generalisation of our results as they represent the practice in France, we believe they can be compared to other practices in Europe and the US. Non-economic data and data on medical prescriptions (particularly for NSAIDs, antibiotics and psychotropic drugs) can be compared with data from other countries. The costs of non-drug prescriptions such as for sickleave were only measured in physical units and showed that sick-leave was significantly lower in patients treated by Ho-GPs compared to CM-GPs (8 % vs 11 %, respectively; p = 0. 0051). These costs, not measured in terms of economic value, may be significant to Social Security, supplementary health insurance and the patient's out-ofpocket expenses, and may also have an impact on the employer or business in general.
Participants and design Patients were enrolled in two separate trials: the MIG-HD trial (N = 54, Ref NCT00190450, PI AC Bachoud-Lévi)  [8] , which is currently underway, and the placebo group of the cognitive ancillary study of the RIL-HD trial conducted only in France (N = 39, Ref NCT00277602, study coordinator Sanofi)  [9]. Both trials were approved by the institutional review board (Comité Consultatif de Protection des Personnes dans la Recherche Biomédicale) of Henri-Mondor Hospital at Créteil (MIG-HD the September 25, 2001, and RIL-HD the  December 18, 2002). Patients had signed an inform consent. The data were analyzed anonymously. The MIG-HD trial is a phase II randomized trial assessing the efficacy of cell transplantation in HD patients at early stages of the disease. Patients were assessed at inclusion (A 1 ), then 35 days (SD = 15) later (A 2 ).
For quantitative variables, β variable was multiplied by the value of the variable. For qualitative variables (sex, inheritance and first symptom), "woman", "maternal inheritance" and "motor symptom" constituted the reference factors, such that β woman = β maternal = β motor = 0. Calculation of the associated 95% predictive interval (95% PI) is explained in the supplemental data (S1 Text). External validation on the RIL-HD cohort. We used models constructed from data for the MIG-HD cohort to predict performances at A 3 for each patient in the RIL-HD cohort. Then, for each task, we measured the concordance between observed (O) and predicted (P) values, using the intraclass correlation coefficient (ICC) and the coefficient of determination (R 2 e ). The ICC was calculated with a two-way mixed effect model  [21]  and evaluates agreement between observed (O) and predicted (P) performances at A 3 in the RIL-HD cohort.
The identification of such patients is helpful for trial design, in two ways. First, the exclusion of such patients would probably decrease intersubject variability, making it possible to decrease sample size. Second, rapid decliners could be uniformly allocated to the different arms of the study by stratified randomization, to ensure the constitution of comparable groups, in terms of both baseline data and disease progression. Our findings suggest that the retest effect is detrimental, if uncontrolled, in clinical trials for neurodegenerative diseases, such as Huntington's disease. We show here that if two assessments are performed a short time apart, use of the second assessment as the baseline increases the chances of detecting an effect of treatment, if there is one. In addition, including the retest effect in models renders the resulting models more predictive, making it possible to refine the design of future trials.
Acknowledgments The authors thank  Julie Sappa  from  Alex Edelman & Associates  for her language corrections. We thank the neurologists and the neuropsychologists from the MIGHD group trial who collected the data:  A-C. Bachoud-Lévi  ( Henri Mondor  hospital, Créteil, Principal investigator),  M-F Boissé  ( Henri Mondor hospital, Créteil, Neuropsychologist ),  L. Lemoine  ( Henri Mondor hospital, Créteil, Neuropsychologist ),  C. Verny  (Angers hospital, Site coordinator),  G. Aubin  ( Angers hospital, Neuropsychologist ),  J-F Demonet  ( CHU Rangueil, Toulouse, Site coordinator ),  F. Calvas  ( CHU Rangueil, Toulouse, Investigator ),  P. Krystkowiak  ( Roger Salengro  hospital, Lille and  CHU d'Amiens, Amiens , Sites coordinator),  C. Simonin  ( Roger Salengro  hospital, Lille, Investigator),  M.
This study was supported by investment for the future  NeurATRIS: Infrastructure de recherche translationnelle pour les biothérapies en Neurosciences  ( ANR-11-INBS-0011 , http://www. agence-nationale-recherche. fr/), European Community Seventh Framework Program Neurostemcell  (Grant Agreement no. 222943 , http:// ec. europa. eu/research/fp7/), European Community Seventh Framework Program Repair-HD  (Grant Agreement no 602245, http://ec. europa. eu/research/fp7/). The  Département d'Etudes Cognitives of the Ecole Normale Supérieure  is supported by two  ANR  grants from the  French Research Agency  ( ANR-10-LABX-0087 IEC  and  ANR-10-IDEX-0001-02 PSL , http://www. agence-nationale-recherche. fr/). Assistance Publique-Hôpitaux de Paris  is the sponsor for the  MIG-HD study (Ref NCT00190450 ) and  Sanofi Aventis  for the Riluzole study (Ref  NCT00277602 ).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: CS was successively supported by the  NeuroStemcell Consortium (European Community Seventh Framework Program  grant agreement no. 222943 ) and by " Investments for the future" (ANR11INBS0011 NeurATRIS: Infrastructure de recherche translationnelle pour les biothérapies en Neurosciences ). SK: no financial disclosures. KY: no financial disclosures. JFD has received financial support from  Eli Lilly ,  Lundbeck ,  Novartis ,  Schwabe  and  Vifor Pharma  over the Rebuttal letter past 2 years as a member of scientific boards and speaker at sponsored sessions. disclosures. CBL acted as a consultant for  Teva , once, in 2014. She received grants from the  Ministry of Health supporting the National Reference Center for Huntington's Disease  and several grants for academic trials provided by the  Direction de la Recherche Clinique (APHP).
The problem of aggregating multiple rankings into one consensus ranking is an active research topic especially in the database community. Various studies have implemented methods for rank aggregation and may have come up with contradicting conclusions upon which algorithms work best. Comparing such results is cumbersome, as the original studies mixed different approaches and used very different evaluation datasets and metrics. Additionally, in real applications, the rankings to be aggregated may not be permutations where elements are strictly ordered, but they may have ties where some elements are placed at the same position. However, most of the studies have not considered ties. This paper introduces the first large scale study of algorithms for rank aggregation with ties. More precisely, (i) we review rank aggregation algorithms and determine whether or not they can handle ties; (ii) we propose the first implementation to compute the exact solution of the Rank Aggregation with ties problem; (iii) we evaluate algorithms for rank aggregation with ties on a very large panel of both real and carefully generated synthetic datasets; (iv) we provide guidance on the algorithms to be favored depending on dataset features.
INTRODUCTION The problem of aggregating multiple rankings into one consensus ranking has started to be investigated two centuries ago and has been actively studied again in the last decades. Direct applications are numerous and include aggregating answers returned by several web engines  [20] , computing a global rating based on numerous user ratings  [4, 33] , determining the winner in a sport competition  [5] , or combining biomedical orderings  [12, 18, 32]. This topic has been of particular interest in the information retrieval and database communities (  [20]  and  [21, 22, 23, 27, 31, 34] ) while several other communities have also deeply looked into it, including algorithmics (  [1, 2, 31] ), artificial intelligence (  [5] ), and social sciences (  [3] ). Various studies have implemented methods for rank aggregation and may have come up with contradicting conclusions upon which algorithms work best.
Comparing such results is cumbersome, as the original studies mixed different kinds of approaches and used very different evaluation datasets and metrics. Additionally, in real applications, the rankings to be aggregated may not be permutations where elements are strictly ordered, but they may have ties where some elements are placed at the same position (e. g. , ex-aequo competitors, or ratings involving the same grade to be possibly associated with several elements). While the first efficient solution to rank aggregation considering input rankings with ties has been introduced in 2004  [21] , most of the approaches and studies introduced since then have continued to focus on permutations, leaving several open questions in the context of ranking with ties. The purpose of this paper is thus twofold: (i) it provides a clear overview of the approaches able to aggregate rankings with ties and (ii) it introduces the first complete study on ranking with ties, including possible (or not) adaptation of existing algorithms to deal with ties, and experimentation of the approaches both on real and carefully generated synthetic datasets.
More precisely, this paper makes four contributions. First, we provide a comprehensive review of the existing rank aggregation approaches and a clear overview of the results provided in the literature. Second, we carefully study the impact of considering ties in all approaches, both on exact and approximation (or heuristics) solutions, and provide a new translation into integer linear programming for finding an optimal consensus ranking in the context of ties. Third, we present the results we obtained on real and synthetic datasets by using available rank aggregation algorithms. Finally, a careful analysis of the results allows us to provide a comprehensive view of the algorithms to be used depending on the kind of application and carefully identified datasets features (e. g. , number of elements, similarity between input rankings.
). The paper is organized as follows. After introducing the formal background in Section 2, we review the major rank aggregation algorithms available in the literature (Section 3). In Section 4, we study the consequence of considering ties both on exact and approximation (or heuristics) solutions. Previous results on the performance and relative quality of existing approaches are summarized in Section 5. Section 6 describes the experimental setting of the study, the datasets we gathered and/or generated, and the methodology used for comparing the available approaches. We analyze our results and provide guidance on the algorithm to be favored based on dataset features in Section 7 while we discuss perspectives in Section 8.
Here, we consider a cost of one in both cases. Let Rn be the set of all possible rankings with ties over [n]. Given any subset R ⊆ Rn and a ranking r, the generalized Kemeny score denoted K is : K(r, R) = s∈R G(r, s) An optimal consensus ranking of a set of rankings with ties R ⊆ Rn under the generalized Kemeny score is a ranking with ties r * such that ∀r ∈ Rn : K(r * , R) ≤ K(r, R). A consensus ranking (or consensus for short) denotes a not necessarily optimal solution of the problem. When a solution is optimal it is explicitly denoted as an optimal consensus. Example: Let us consider the set of input rankings R = {r1, r2, r3} where r1 K(r * , R) is made of inversions A- D in r3, D-B in r2, D-C in r2, tying B-C in r3 and untying A-C in r3 thus K(r * , R) = 5. In this paper, a dataset systematically denotes a set of input rankings R.
Given any solution, Constraint (1) ensures that for any pair of elements a and b, either a is ranked before b if x a<b = 1, or a is ranked after b if x b<a = 1, or a and b belong to the same bucket if x a=b = 1. Moreover, Constraint (2) ensures the transitivity of the ranking. Finally, Constraint (3) ensures that the resulting ranking is a ranking with ties. x a<b = x b<a = x b<c = x c<b = 0, The objective function fully corresponds to the computation of the generalized Kendall-τ distance, thus an optimal solution to our LPB is an optimal consensus ranking. Due to the intrinsic complexity of this linear problem, optimal solutions can be computed for moderately large datasets only. This allows us evaluating the quality of nonoptimal algorithms.
Normalization Process Studies have designed approaches to convert any dataset (i. e, a set of rankings) over different elements into dataset over the same elements. Normalization strategies are described here and illustrated in Table  3. Projection consists in removing from a dataset all the elements which are absent in at least one ranking  [5] , resulting in a projected dataset. In Table  3  dp is the projection of dr. It always produces permutations when the input rankings are permutations. Its major drawback is to possibly remove (large sets of) relevant elements. Unification consists in adding at the end of each ranking of a dataset a unification bucket with elements appearing in other rankings (and absent in the current ranking). This last unifying bucket can be considered as is (as in  [12, 31] ), we say that such datasets are unified, du is the unification of dr.
Others  [3]  prefer to arbitrarily break this bucket to consider only permutations in the input rankings, such datasets are said to be unif[ied] broken, as dataset d b in Table  3   [5] projected no WebCommunities  [13, 31]  yes no WebSearch  [20, 31, 3, 5]  [5] unified  [31, 3]  yes projected  [5]  no Mallows model  [3, 5]  yes no Placket-Luce model  [3, 5]  yes no RandomGraph  [17, 13]  yes no Random  [3]  yes no Random  [12]  yes yes Table  2 : Datasets properties and availability.
Previous results In this subsection, we provide a view of the results obtained by the previous studies found in the literature. The only experimental study which has evaluated approaches both taking in and producing rankings with ties is  [12]. They concluded their experimentations by stating that BioConsert outperformed all other considered approaches. However, results have been obtained on small real (biomedical) datasets and on very small generated datasets (4 to 8 elements). Both the algorithms and the datasets considered in  [12]  totally differ from those used in other studies  [3, 5, 13, 31]. Other studies (  [3]  and  [31] ) have compared approaches taking in rankings with ties but based on the classical Kendall-τ distance, thus necessarily producing permutations and unable to consider the cost of (un)tying.
[3]  work on WebSearch and GiantSlalom datasets while  [31]  work on WebSearch and WebCommunities datasets. From their respective experimentations they conclude that (i) Chanas produces good quality results, (ii) positional approaches (such as BordaCount and CopelandMethod) are approaches able to quickly provide results of good quality, (iii) KwikSort provides a good trade-off between the previous recommendations. Additionally,  [3]  recommends to use BnB with beam search techniques as an intermediate solution between KwikSort and ChanasBoth. While in  [3]  CopelandMethod returns better results than BordaCount, in  [31]  they appear to be equivalent. As for CopelandMethod and MC4, in  [31]  they present comparable results in term of quality, MC4 being much more time consuming. Another study,  [13] , focuses on permutations only, using WebCommunities and EachMovie datasets.
Compared to  [3, 31] , they confirm that Chanas and ChanasBoth produce quality results but they fully disagree on the use of KwikSort (which obtains bad performance). Positional algorithms are not considered at all in this study. Optimal solutions (ILP-based) are intensively studied by  [5]  which introduces pre-processes to reduce the search space of the problem. In addition to real datasets, a few generated datasets have been considered. In particular and interestingly,  [3]   daCount appears to be the best choice while BnB should be preferred when the similarity is low. However, as authors do not provide any means to determine the similarity between given input rankings, such recommendations remain difficult to apply in concrete settings. As for the normalisation process,  [3, 31, 12]  normalize datasets with the unification process, while  [5]  project them.
Whether or not the normalization process may have an impact on their results is an open question. To summarize, each study has considered a given (restricted) set of algorithms and has performed experimentations in very different datasets, curated with diverse methods, and producing mostly permutations. Given the current results that can be extracted from the literature, it is thus very difficult to determine in which context one approach should be preferred over the others. The next two sections introduce for the first time the results obtained on a large-scale study conducted on rank aggregation algorithms considering ties.
Datasets We have first considered real-world datasets which have already been used in previous works (the four groups of datasets in bold in Table  2 ) while extending the experiments to a very large set of algorithms. We have then carefully generated datasets to better understand the possible impact of three dataset features on rank aggregation algorithms: the size of the dataset (number of rankings, number of elements in each ranking), the (level of) similarity between the rankings taken as input, the normalization process which has been applied to the dataset (unification or projection). The generation of synthetic datasets is described in the next two subsections.
Uniformly generated synthetic datasets The datasets introduced here are made of rankings with ties, where all rankings have the same probability to be present. This has been carefully ensured by using the MuPAD-Combinat package  [26]  based on the work of  [25]. More precisely, we produced datasets of m ∈ [3; 10] rankings for different lengths: n ∈ [5; 95] with a step of 5, and n ∈ [100; 500] with a step of 100. We produced 100 datasets for each pair < m, n >. Such numbers have been chosen to mimic real-world settings.    [{A}, {B, C}, {F }, {D}, {E}] [{D}, {A, E}, {F }, {B}, {C}] [{A}, {C}, {D}, {B}, {E, F }]    original dataset retaining -→ top-2    [{A}, {B, C}] [{D}, {A, E}] [{A}, {C}]    sub-dataset unification -→ process    [{A}, {B, C}, {D, E}] [{D}, {A, E}{B, C}] [{A}, {C}, {B, D, E}]    unif ied dataset Figure  1 : Generation of a unified synthetic dataset: a dataset over 6 elements is generated (100 in the experiment), then only the top-k=2 elements are retained for each ranking (k∈  [1;35]  in the experiment).