dataesr/scientific-paragraphs-categorization huggingface dataset:
Increasingly dissimilar synthetic datasets To study the impact of input rankings similarity on the results obtained, we modeled the data generation process by a Markov chain where states are rankings with ties and transition between states represents a possible modification of one ranking into another. Modifications are done using four operators: move an element of a ranking in the previous or the following bucket, put it in a new bucket right before or right after its current position. Such operators ensure (with restrictions when buckets contain one or two elements, details omitted) that the Markov chain converges to the uniform stationary distribution. Considering a seed ranking rs and a number of steps t, a dataset over m rankings consists in starting m times from rs in the Markov chain and adding the state currently visited after t steps.
The modeling allows us to generate rankings with ties biased by the starting state, and with different levels of similarity to the seed ranking (depending of the number of steps allowed in the Markov chain), and thus different levels of similarity. We generated 1000 datasets of m = 7 rankings over n = 35 elements with a number of steps to walk in the Markov chain process t ∈ {50, 100, 250, 500, 1000, 2500, 5000, 10000, 25000, 50000 1 }.
Unified synthetic datasets with similarities Here, we study the impact of the unification process (introduced in Section 5. 1) used either when datasets are not over the same elements (BioMedical  [12] ), or when the dataset is made of only the top-k elements of raw rankings (WebSearch  [20, 31] ). We mimicked this second use case and generated datasets with different levels of similarity (cf. Section 6. 1. 2), retained only the top-k elements and then applied the unification process (cf. Figure  1 ). We generated 1000 datasets with m = 7 rankings over n = 100 elements with a number of steps t to walk in the Markov chain modeling, t ∈ {10 3 , 2. 5 * 10 3 , 5 * 10 3 , 10 4 , 2. 5 * 10 4 , 5 * 10 4 , 10 5 , 2. 5 * 10 5 , 5 * 10 5 , 10 6 }. The topk elements are retained with k ∈  [1; 35]  in order to have datasets of n = 35 elements.
Measuring similarity The Kendall-τ rank correlation coefficient  [29]  is a wellknown measure to quantify the correlation of two permutations. Its extension to rankings with ties is straightforward. Formally, considering r1, r2 two rankings with ties, it is defined as: τ (r1, r2) = 1 2 n(n -1) -2G(r1, r2) 1 2 n(n -1) (4) where G is the generalized Kendall-τ distance (cf. Section 2. 2). To measure the intrinsic correlation of a dataset R = {r1,. rm}, we average the correlation coefficient of each pair of rankings in this dataset: s(R) = 2 m(m -1) m i=1 m j=i+1 τ (ri, rj) (5).
Comparing over time We paid particular attention to the configuration we set up to ensure fair time comparison. Experiments were conducted on a four dual-core processor Intel Xeon 3GHz with 16GB memory using Java 1. 6. 0 37, LPSolve 5. 5. 2. 0, CPLex 12. 4 and Python 2. 4. 4. To measure the execution time of an algorithm, we ran it numerous times in a row such that the time needed to do all executions was greater than two seconds (the execution time is then the overall execution time divided by the number of executions). Each measure was preceded by a warm-up time to ensure that all classes were already loaded in the JVM memory. Implementations were single-threaded. LPSolve (used by Ailon 3  2 ) and CPlex (used by the ExactAlgorithm) were used in their default configuration. For every algorithm we limited the computing time to two hours: after that limit, we considered that the algorithm was not able to provide a solution.
RESULTS In this section, we present the results obtained by a large panel of rank aggregation algorithms using various kinds of datasets. Firstly, we evaluate the quality of results produced by the algorithms and their time consumption while considering only the size of the datasets, to this end we use uniformly generated datasets. Secondly, we take into account different levels of similarity with the help of synthetic datasets with progressive dissimilarity. Thirdly we analyze the impact of the normalization processes (unification, projection) on real datasets. Last, we analyze the impact of the unification process on the quality of the results produced. In each analysis, we also study the impact of results obtained on real world datasets with the same knowledge (size, similarity, normalization) allowing us to highlight the information needed to fully understand the behaviour of algorithms in real settings.
Measuring quality We have conducted a first series of experiments on uniformly generated datasets and observed that the number of rankings considered has no significant impact in the results. We have systematically considered datasets with m ∈ [3; 10] rankings and n ≤ 60 elements (60 is the highest number of elements for which the optimal consensus ranking can be computed in a reasonable amount of time). Results are presented in Table  5. Four points deserve attention. First, for both synthetic and real datasets (cf. Table  4  and 5 ), BioConsert provides the best results in the very large majority of the cases (88. 56%) and in 33. 94% it strictly outperforms the other algorithms. In 68. 01% of the cases, the solutions produced are optimal. When focused on real datasets only, BioConsert provides the best results in 91.
8% of the datasets. Second, Ailon is provided. On synthetic datasets, Ailon 3 2 and BioConsert provide results with a similar quality: Ailon 3  2 strictly outperforms BioConsert in 14. 39% of the datasets, while it is outperformed in 18, 67% of them. Third, positional algorithms BordaCount and Copeland-Method have an interesting behavior on synthetic datasets. When being compared to the other algorithms, the average disagreements of their solution surprisingly decreases as the number of elements grows: BordaCount (resp. Copeland-Method) is ranked 8 th (resp. 9 th ) when considering datasets of 20 elements, and 3 rd (resp. 4 th ) with datasets of 500 elements. Fourth, we have evaluated the behavior of MEDRank when varying its threshold. General observations of the gap for various values of the threshold have shown that MEDRank is very sensitive to its threshold value.
More precisely, values higher than the default one (threshold of 0. 5) do not lead to any improvement in the quality of the consensus provided, neither in real nor in synthetic datasets. In 76. 37% of the synthetic datasets a threshold of 0. 5 provides the best results. It is thus the threshold value  to be preferred when using MEDRank. While some algorithms show coherent behavior between synthetic and real datasets (cf. Table  4  and 5 ), others do not. Two observations highlight the need of a finer grained information on the datasets in order to understand the situation where algorithms perform well or do not. First, when considering FaginDyn variants, FaginSmall performs better on 99. 59% of the synthetic datasets. This observation is not repeated on real world datasets, as the two versions are even: FaginSmall performs better than Fagin-Large in 49.
Experimental computation time on uniformly generated datasets We now consider the time consumption for different values of n the number of elements in the datasets and fixed m = 7. First of all, the average computing time needed by positional algorithms MEDRank, CopelandMethod, Re-peatChoice and BordaCount to return a solution is very small: for datasets of n = 400 they take in average respectively 436µs, 463µs, 468µs and 574µs. With the synthetic data, we observed that when n grows, MEDRank, CopelandMethod, BordaCount still return consensus of good quality compared to the other algorithms while being must faster. They are thus very good candidates for large datasets. Second, considering only the two algorithms returning quality results, BioConsert strictly outperforms Ailon 3  2 e. g. for n = 40 elements they take resp.
Considering the similarity of datasets Over the 10 5 datasets generated and used in this subsection, the exact solution is found in 99. 91% of the datasets by the ExactAlgorithm with the computation time allocated. It ensures that no bias is introduced by datasets where it was not possible to compute the exact solution. The Markov chain modeling designed in Section 6. 1. 2 converges to the uniform distribution when an infinite number of steps is considered. Knowing the number of steps needed to reach the uniform distribution with a relative error less than a given ε is out of the scope of this paper. Instead, we focus on two indicators to highlight that datasets generated with 50 000 steps are equivalent to uniformly generated datasets with the same numbers of elements and rankings: (1) the average similarity of uniformly generated datasets is s = -0.
0388 while the Markov based datasets have a similarity of s = -0. 0384 (note that with 50 steps s = 0. 88, with 1000 steps s = 0. 55 and with 5000 steps s = 0. 17),  (2)  the results obtained on uniformly generated datasets with the same numbers of rankings and elements are equivalent to the results obtained when considering datasets generated with the Markov chain modeling. Time consumption. Experiments conducted on synthetic datasets with different levels of similarities regarding the time consumption reveal two groups of algorithms: the algorithms from the first group take significant advantage of datasets with intrinsic similarities, while the second group is not impacted by the similarity. The local search algorithms are expected to be in the first group and they are: compared to not similar datasets (50 000 steps, s = 0.
04), BioConsert proposes a consensus up to 57% faster with similar datasets (50 steps, s = 0. 88). Similarly, ExactAlgorithm and Ailon 3  2 also propose results faster on similar datasets: respectively 85% and 11% faster. The second group is made of BordaCount, CopelandMethod, FaginLarge, FaginSmall, KwikSort, Pick-a-Perm and RepeatChoice. Considering the gap. Three interesting behaviors have been spotted (cf. Figure  4 ). First, KwikSort is positively influenced by the similarity, the average gap is 24 times smaller with very similar datasets (50 steps) compared to not similar datasets (50 000 steps). BioConsert has the same behavior: it always finds the optimal solution of similar datasets (≤ 500 steps), and has an average gap = 0. 02% with not similar datasets (50 000 steps). Observations on KwikSort are completed when using real-world dataset: KwikSort is indeed more efficient with similar datasets, but it is also negatively impacted by a negative similarity of datasets "Web-Search Unif" and "SkiCross Unif" (cf.
Observations done on BordaCount or FaginDyn with synthetic datasets (cf. Figure  4 ) and which cannot be repeated on real world data (cf. Table  4  and Figure  3 ) highlight clearly that for some algorithms, knowing the size of a dataset and its similarity is still not enough to make an informed choice on whether using or not these algorithms. For this reason, the next section focuses on the influence of the standardization process. Two approaches particularly benefit from the similarity in term of quality (Figure  4 ), namely BioConsert and Kwik-Sort, while BioConsert and ExactAlgorithm benefit from similarity in terms of time consumption.
Unifying and projecting real datasets When considering a raw dataset with m rankings where the length of the longest ranking is l, it can be noticed that the projection process produces a dataset over 0 to l elements while the unification process produces a dataset over l to l × m elements. The F1 datasets are seasons of F1 championship where each ranking is the order of arrival of pilots finishing the race. As done by  [5] , the projection removes 53. 42%±25. 03% of the pilots. Among the removed pilots we find the 1961 vice-champion and the 1970 champion. Those two pilots can clearly be qualified as relevant elements, not to be removed. In average, projected datasets are over 15. 81 ± 8. 53 elements while unified datasets are over 38. 73 ± 11. 39 elements. When considering the WebSearch datasets, the projection removes in average 98.
42% ± 0. 89% of the elements (procedure followed by  [5] ). Each ranking contains the top 1000 results for a search engine; the projection produces, in average, datasets over 40 ± 20 elements while unified datasets are over 2 586 ± 388 elements (used in  [3, 31] ). Using a process that takes into account the elements not present in all rankings appears to be a crucial need in real life use cases. Indeed the usually used projection process can remove relevant elements, such as champion in the F1 datasets. Nevertheless the increase of size between projected and unified datasets (there can be from 2. 4 to 65 times more elements with unification) should be taken into account. Having a large difference of size between projected and unified datasets is an indicator of the presence of large unifying buckets.
Unification impact on similar data The smallest the number of elements input rankings have in common before applying the unification process, the largest the size of unification buckets is expected to be. On unified synthetic datasets with similarities (cf. Section 6. 1. 3), the average size of buckets of similar datasets (generated with 10 3 steps) is 1. 52 while it is 6. 52 for not similar datasets (generated with 10 6 steps). Algorithms are expected to be cleaved into two categories depending on whether or not they consider the cost of untying elements. We have on the one hand, BioConsert, KwikSort, and MEDRank which consider the importance of untying elements and are expected to be stable in quality, an assumption confirmed in this experimentation (cf. Figure  5 ). On the other hand, BordaCount, CopelandMethod, and RepeatChoice are not able to consider the cost of untying elements.
They are expected to be dramatically impacted by the unification process which can create a large ending tie. Experimentally they induce 15 times more disagreements with not similar and unified datasets than with similar unified datasets. The variation of quality of results for Borda-Count and CopelandMethod with real world datasets which was not reproducible with synthetic datasets, whether they were similar or not, is now fully reproduced (cf. Figure  5  vs Figure  4 ). FaginDyn variants had behaviors which were not consistent when only taking into account the size and similarity of the datasets (cf. Section 7. 2). In this experiment, we clearly observe that the unification process, leading to the creation of a big ending bucket for datasets with no similarities, has a negative impact on FaginSmall.
In the present experiment, favoring small buckets when there are large buckets in the input rankings is clearly a disadvantageous choice as shown in Figure  5. On both unified synthetic and unified real world datasets CopelandMethod outperforms BordaCount, and they have the same performance on projected datasets. The unification process as a standardization causes large ending buckets which are now understood as being the cause of bad performance of BordaCount and FaginSmall. Results obtained on real and synthetic datasets are consistent to each others now that the impact of the standardization process is understood.
Guidance based on known properties Based on our experiments, we are now able to provide recommendations according to the known properties of the data to deal with, and the desired trade-off between time efficiency and expected quality of the results. Figure  6  illustrates the choice that could be proposed to the user, it has been generated with 100 uniformly generated datasets of m = 7 rankings over n = 35 elements. Results are representative of other uniformly generated datasets. As a general outcome, BioConsert appears to be the best approach to follow in a very large number of cases. In extreme situations, some alternatives may be considered. First, when highest quality results are mandatory and beside the ExactAlgorithm which provides optimal consensus, BioConsert should be favoured. While being reasonably more time consuming than other algorithms, it takes advantage of the similarity of datasets and it is independent of the standardization process applied.
Second, when extremely large datasets are considered (number of elements n>30 000), the implementation of BioConsert with a O n 2 memory complexity can face physical limitation and may additionally take time to propose a consensus. KwikSort is then the best alternative for good quality results, and it is positively influenced by the similarity of datasets (cf. Figure  4 ). If time is highly important then BordaCount is to be considered if only a few ties are involved whereas with large ties (possibly obtained by unification process) MEDRank is an excellent candidate.
CONCLUSION We have conducted the first large-scale study of algorithms for rank aggregation with ties. First, we have proposed an Integer Linear Programming algorithm to compute the optimal consensus ranking for the rank aggregation problem in the presence of ties. We have used the results of such exact solution to evaluate other approaches. Second, we have reviewed the algorithms available in the literature and described which changes were needed in such algorithms to handle ties. We have considered a set of very diverse categories of algorithms which have all been reimplemented in a general framework and experimented both on available real-world and new synthetic datasets. Very importantly, we have been able to identify the data features to be considered to understand the behavior of algorithms (to then be able to guide users), namely, the size, level of similarity and normalization process.
A total of 19 000 datasets of 10 to 500 elements are involved in our experiments and are all available at http://rank-aggregation-with-ties. lri. fr/datasets/. We now discuss future work. First, the surprising improvement shown by BordaCount and CopelandMethod when increasing the number of element for a fixed amount of ranking is being investigated under a theoretical point of view. Second, unification and projection processes can be seen as two extreme variants of the same standardization process where the elements belonging to less than k rankings are removed, and the others are appended into a unification bucket when they are missing. Studying intermediate values of k would allow keeping a reasonable amount of data while ensuring the presence of relevant elements. Third, considering strategies chaining several algorithms is another line of research to be explored (in the same spirit of  [10] ).
generated some datasets with different levels of similarities. Bor-{D}, {B}, {C, E}] [{A}, {D}, {B}, {C}, {E}] [{B}, {E, A}, {C, D}] [{B}, {A}, {E}, {C}, {D}] [{D}, {A, B}, {C}, {E}] [{D}, {A}, {B}, {C}, {E}] Resulting datasets after applying the various normalization processes to the raw dataset dr.
A further 6 services with the second best score were also explicitly targeted by the PNMI Management Plan, and it was decided that these should also be assessed if sufficient time, resources and data were available. These were: commercial finfisheries, supporting services provided by kelp for biodiversity, commercial fish habitats, foraging bottlenose dolphins and seabirds (that in turn provide recreational and existence val- Based on discussions regarding the 3 questions of Stage 3, the expert panel chose the following metrics to measure: the biomass and spatial distribution of kelp, the size of the habitats available for protected or charismatic species, kelp and fisheries harvest, the economic indicators associated with kelp and fish harvest (added value, employment and contribution to traditional activities), and recreational activities (visitor numbers, especially changes in visitor numbers caused by new rules).
Acknowledgements. The  VALMER  project was selected under the European cross-border cooperation programme  INTERREG IV A France (Channel) -England , co-funded by the  European Regional Development Fund. The participation of L. P. was supported by the  'Laboratoire d'Excellence '  LabexMER  ( ANR-10-LABX-19 ) and co-funded by a grant from the  French government  under the  program 'Investissements d'Avenir'.
The power output measurements from the single turbine, are collected continuously with a sampling rate of T s = 1 s during more than six months: this corresponds to 12, 257, 600 data points. This wind turbine is located at Risø Campus, Roskilde, Denmark and is a three-bladed stall regulated Nordtank, NTK 500/41 wind turbine. Figure  1  gives an example of power output delivered by this wind turbine during two days. Table  1  gives a description of following characteristics, sampling frequency, number of continuously data points, implementation site, installed capacity, for each dataset. The study of complex systems in many fields such as ecology, physics, life sciences and engineering sciences  [12] , has highlighted the universality of the Taylor power law established by L. R. Taylor in 1961  [1].
Taylor Power Law in Wind Energy Data To verify the existence of Taylor power law for our datasets, the mean value P τ and the standard deviation σ τ are computed for time scales (or window sizes) τ ranging between approximately 4 h and 7 days for data sampled at 5 min, and τ ranging between approximately 16 min and 1 day for data sampled at 1 s. The choice of the lower limit of these time intervals is justified by the value of a Pearson correlation r > 0. 8 between the map (P r ,σ τ /C 0 ) and a non parametric kernel smoothing regression fit. Meanwhile, the choice of the upper limit is defined by a time window having a significant sample number for the estimation of the Taylor exponent λ τ. Hence, each dataset is splitted over a time window of length τ. In Figure  2a , to compare the datasets considered here with the straight line of 1/2 slope, σ τ /C 0 versus P r (P r = P / P max with P max the maximum of the mean value for the dataset considered) is illustrated for τ = 5 h, in log-log representation.
From the Taylor relationship defined in Equation (  4 ), a scaling relationship between the turbulent production intensity I p = σ/ P and the mean value P , is established by replacing P by P r : I p = σ P r = C 0 P λ r P r (5) This leads I p = C 0 (P r ) α (6) where the exponent α = λ -1 is here negative. Figure  4  illustrates the evolution of the turbulent production intensity I P divided by C 0 versus the adimensioned mean value P r , in log-log scale. As expected, the turbulent production intensity I P decreases following a -1/2 power law with the mean value P. Hence, taking into account the value obtained for the Taylor exponent λ τ ≈ 1/2, α ≈ -1/2 for time scales 1 h < τ < 7 days. Table  3  draws up the values of the exponent α estimated for each dataset with τ = 5 h. It can be seen that the values are generally corresponding to I P > 10% which is a very high level of turbulence and that larger values of P r correspond to smaller values of I p.
Conclusions and Discussions In this study, we have investigated the existence of a Taylor power law or temporal fluctuation scaling of power output delivered by five wind farms and a single wind turbine, with different installed capacity. The analyzed data are sampled at 1 s and five minutes, and recorded over different periods. A Taylor power law has been highlighted for all the datasets. Furthermore, a universal scaling exponent λ τ close to 1/2, is observed for time scales 1 h < τ < 7 days for the data sampled at 5 min, and 5 min < τ < 6 h for the data sampled at 1 s. The existence of such Taylor law has been shown in multiple disciplinary fields. This universality has conducted many authors to suggest the existence of a universal mechanism for its emergence. Various approaches and theoretical investigations have been dedicated to possible explanations of the origin of Taylor law.