Package: ranky
RankyCompute rankings in Python.Get startedpipinstallrankyimportrankyasrkRead thedocumentation.Main functionsThe main functionalities includescoring metrics(e.g. accuracy, roc auc),rank metrics(e.g. Kendall Tau, Spearman correlation),ranking systems(e.g. Majority judgement, Kemeny-Young method) and somemeasurements(e.g. Kendall's W coefficient of concordance).Most functions takes as input 2-dimensionalnumpy.arrayorpandas.DataFrameobjects. DataFrame are the best to keep track of the names of each data point.Let's consider the following preference matrix:Each row is a candidate and each column is a judge. Here is the results ofrk.borda(matrix), computing the mean rank of each candidate:We can see that candidate2 has the best average rank among the four judges.Let's display it usingrk.show(rk.borda(matrix)):Ranking systemsThe rank aggregation methods available include:Random Dictator:rk.dictator(m)Score Voting (mean):rk.score(m)Borda Count (average rank):rk.borda(m)Majority Judgement (median):rk.majority(m)Pairwise methods. Copeland's method:rk.pairwise(m), Success rate:rk.pairwise(m, wins=rk.success_rate)and more. You can specify your own "wins" function or select one from therk.duelmodule.Optimal rank aggregationusing any rank metric:rk.center(m),rk.center(m, method='kendalltau'). Solver used [1].(Kemeny-Young method is optimal rank aggregation using Kendall's tau as metric.)(Optimal rank aggregation using Spearman correlation as metric is equivalent to Borda count.)MetricsYou can useany_metric(a, b, method)to call a metric fromanyof the three categories below.Scoring metrics:rk.metric(y_true, y_pred, method='accuracy'). Methods include:['accuracy', 'balanced_accuracy', 'precision', 'average_precision', 'brier', 'f1_score', 'mxe', 'recall', 'jaccard', 'roc_auc', 'mse', 'rmse', 'sar']Rank correlation coefficients:rk.corr(r1, r2, method='spearman'). Methods include:['kendalltau', 'spearman', 'pearson']Rank distances:rk.dist(r1, r2, method='levenshtein'). Methods include:['hamming', 'levenshtein', 'kendall', 'winner', 'euclidean']To add: general edit distances, kemeny distance, regression metrics...VisualizationsUserk.showto visualize preference matrix (2D) or ranking ballots (1D).>>> rk.show(m)>>> rk.show(m['judge1'])Userk.mds, to visualize (in 2D or 3D) the points in a given metric space.Seerk.scatterplotdocumentation for display arguments.>>> rk.mds(m, method='euclidean')>>> rk.mds(m, method='spearman', axis=1)You can userk.tsnesimilarly tork.mds.Userk.critical_differenceto plot acritical difference diagram, comparing candidates' performance and grouping them by statistical equivalence. Such diagrams can be seen in [2, 3].>>> rk.critical_difference(m, comparison_func=rk.bayes_wins)OtherRank,rk.rank, convert a 1D score ballot into a ranking.Bootstrap,rk.bootstrap, sample a given axis.Consensus,rk.consensus, check if ranking exactly agree.Concordance, ,rk.concordance, mean rank distance between all judges of a preference matrix.Centrality,rk.centrality, mean rank distance between a ranking and a preference matrix.Kendall's W,rk.kendall_w, coefficient of concordance.Utility:read_codalab_csvto parse a CSV generated by Codalab representing a leaderboard into apandas.DataFrame.ReferencesPlease cite ranky in your publications if this is useful for your research. Here is an example BibTeX entry:@misc{pavao2020ranky,
  title={ranky},
  author={Adrien Pavao},
  year={2020},
  howpublished={\url{https://github.com/didayolo/ranky}},
}[1] Storn R. and Price K., Differential Evolution - a Simple and Efficient Heuristic for Global Optimization over Continuous Spaces, Journal of Global Optimization, 1997, 11, 341 - 359.[2] Janez Demsar, Statistical Comparisons of Classifiers over Multiple Data Sets, 7(Jan):1--30, 2006.[3] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, P. Muller, Deep learning for time series classification: a review, Data Mining and Knowledge Discovery, 2018.LicenseCopyright (c) 2020-2021, Adrien PAVAO. This software is released under the Apache License 2.0 (the "License"); you may not use the software except in compliance with the License.The text of the Apache License 2.0 can be found online at:http://www.opensource.org/licenses/apache2.0.php