Package: rankeval
RankEval: An Evaluation and Analysis Framework for Learning-to-Rank SolutionsRankEval is an open-source tool for the analysis and evaluation of
Learning-to-Rank models based on ensembles of regression trees. The
success of ensembles of regression trees fostered the development of
several open-source libraries targeting efficiency of the learning phase
and effectiveness of the resulting models. However, these libraries offer
only very limited help for the tuning and evaluation of the trained models.RankEval aims at providing a common ground for several Learning to Rank
libraries by providing useful and interoperable tools for a comprehensive
comparison and in-depth analysis of ranking models. Target audience is themachine learning(ML) andinformation retrieval(IR) communities.RankEval is available under Mozilla Public License 2.0.The official GitHub repository is:here.For questions/suggestions on how to improve RankEval, send us an email:rankeval@isti.cnr.itFeaturesRankeval provides a common ground between several pre-existing tools and offers
services which support the interpretation of differently generated models in a
unified environment, allowing an easy, comprehensive comparison and in-depth
analysis.The main functionalities of RankEval can be summarized along five dimensions:effectiveness analysisfeature analysisstructural analysistopological analysisinteroperability among GBRT librariessupport the model format of the most popular learning tools such as
QuickRank, RankLib, XGBoost, LightGBM, Scikit-Learn, etcThese functionalities can be applied to several models at the same time, so to
have a direct comparison of the analysis performed. The tool has been written
to ensureflexibility,extensibility, andefficiency.DocumentationThe official API documentation is available at:here.
Soon on ReadTheDocs!InstallationThe library works with OpenMP so you need a compiler supporting it.
If your machine uses a default compiler different from GNU GCC, change it
appropriately before proceeding with the installation:export CC=gcc-5
export CXX=g++-5Moreover, RankEval needs the following libraries to be installed before the
installation process begin:numpy >= 1.13scipy >= 0.14cython >= 0.25matplotlib >= 2.0.2RankEval can be easily installed from Python Package Index (PyPI).
You may download and install it by running:pip install rankevalAlternatively, you can build the library from source.
Below an example of installation.python setup.py installorpip install -e .DevelopmentInstallation of libraries required for development (documentation generation and unittests):pip install -e .[develop]Local installation of compiled libraries:python setup.py build_ext -iExecution of unit tests:python setup.py testor (if you have nose already installed):nosetests -vCite RankEvalIf you use RankEval, please cite us!@inproceedings{rankeval-sigir17,
  author = {Claudio Lucchese and Cristina Ioana Muntean and Franco Maria Nardini and
            Raffaele Perego and Salvatore Trani},
  title = {RankEval: An Evaluation and Analysis Framework for Learning-to-Rank Solutions},
  booktitle = {SIGIR 2017: Proceedings of the 40th International {ACM} {SIGIR}
               Conference on Research and Development in Information Retrieval},
  year = {2017},
  location = {Tokyo, Japan}
}Credits- Dataset loader: https://github.com/deronnek/svmlight-loader
- Query id implementation: https://github.com/mblondel/svmlight-loader/pull/6