Package: jury
JuryA comprehensive toolkit for evaluating NLP experiments offering various automated metrics. Jury offers a smooth and easy-to-use interface. It uses a more advanced version ofevaluatedesign for underlying metric computation, so that adding custom metric is easy as extending proper class.Main advantages that Jury offers are:Easy to use for any NLP project.Unified structure for computation input across all metrics.Calculate many metrics at once.Metrics calculations can be handled concurrently to save processing time.It seamlessly supports evaluation for multiple predictions/multiple references.To see more, check theofficial Jury blog post.🔥 News(2023.10.03) Jury paper is out currently is onarxiv. Please cite this paper if your work use Jury, and if your publication material will be submitted to the venues after this date.(2023.07.30)Public notice:You can reach our officialPublic Noticedocument that poses a claim about plagiarism of the work,jury, presented in this codebase.Available MetricsThe table below shows the current support status for available metrics.MetricJury SupportHF/evaluate SupportAccuracy-Numeric:heavy_check_mark::white_check_mark:Accuracy-Text:heavy_check_mark::x:Bartscore:heavy_check_mark::x:Bertscore:heavy_check_mark::white_check_mark:Bleu:heavy_check_mark::white_check_mark:Bleurt:heavy_check_mark::white_check_mark:CER:heavy_check_mark::white_check_mark:CHRF:heavy_check_mark::white_check_mark:COMET:heavy_check_mark::white_check_mark:F1-Numeric:heavy_check_mark::white_check_mark:F1-Text:heavy_check_mark::x:METEOR:heavy_check_mark::white_check_mark:Precision-Numeric:heavy_check_mark::white_check_mark:Precision-Text:heavy_check_mark::x:Prism:heavy_check_mark::x:Recall-Numeric:heavy_check_mark::white_check_mark:Recall-Text:heavy_check_mark::x:ROUGE:heavy_check_mark::white_check_mark:SacreBleu:heavy_check_mark::white_check_mark:Seqeval:heavy_check_mark::white_check_mark:Squad:heavy_check_mark::white_check_mark:TER:heavy_check_mark::white_check_mark:WER:heavy_check_mark::white_check_mark:Other metrics*:white_check_mark::white_check_mark:*Placeholder for the rest of the metrics available inevaluatepackage apart from those which are present in the
table.NotesThe entry :heavy_check_mark: represents that full Jury support is available meaning that all combinations of input
types (single prediction & single reference, single prediction & multiple references, multiple predictions & multiple
references) are supportedThe entry :white_check_mark: means that this metric is supported (for Jury through theevaluate), so that it
can (and should) be used just like theevaluatemetric as instructed inevaluateimplementation although
unfortunately full Jury support for those metrics are not yet available.Request for a New MetricFor the request of a new metric pleaseopen an issueproviding the minimum information. Also, PRs addressing new metric
supports are welcomed :).InstallationThrough pip,pip install juryor build from source,git clone https://github.com/obss/jury.git
cd jury
python setup.py installNOTE:There may be malfunctions of some metrics depending onsacrebleupackage on Windows machines which is
mainly due to the packagepywin32. For this, we fixed pywin32 version on our setup config for Windows platforms.
However, if pywin32 causes trouble in your environment we strongly recommend usingcondamanager install the package
asconda install pywin32.UsageAPI UsageIt is only two lines of code to evaluate generated outputs.fromjuryimportJuryscorer=Jury()predictions=[["the cat is on the mat","There is cat playing on the mat"],["Look!    a wonderful day."]]references=[["the cat is playing on the mat.","The cat plays on the mat."],["Today is a wonderful day","The weather outside is wonderful."]]scores=scorer(predictions=predictions,references=references)Specify metrics you want to use on instantiation.scorer=Jury(metrics=["bleu","meteor"])scores=scorer(predictions,references)Use of Metrics standaloneYou can directly import metrics fromjury.metricsas classes, and then instantiate and use as desired.fromjury.metricsimportBleubleu=Bleu.construct()score=bleu.compute(predictions=predictions,references=references)The additional parameters can either be specified oncompute()fromjury.metricsimportBleubleu=Bleu.construct()score=bleu.compute(predictions=predictions,references=references,max_order=4), or alternatively on instantiationfromjury.metricsimportBleubleu=Bleu.construct(compute_kwargs={"max_order":1})score=bleu.compute(predictions=predictions,references=references)Note that you can seemlessly access bothjuryandevaluatemetrics throughjury.load_metric.importjurybleu=jury.load_metric("bleu")bleu_1=jury.load_metric("bleu",resulting_name="bleu_1",compute_kwargs={"max_order":1})# metrics not available in `jury` but in `evaluate`wer=jury.load_metric("competition_math")# It falls back to `evaluate` package with a warningCLI UsageYou can specify predictions file and references file paths and get the resulting scores. Each line should be paired in both files. You can optionally provide reduce function and an export path for results to be written.jury eval --predictions /path/to/predictions.txt --references /path/to/references.txt --reduce_fn max --export /path/to/export.txtYou can also provide prediction folders and reference folders to evaluate multiple experiments. In this set up, however, it is required that the prediction and references files you need to evaluate as a pair have the same file name. These common names are paired together for prediction and reference.jury eval --predictions /path/to/predictions_folder --references /path/to/references_folder --reduce_fn max --export /path/to/export.txtIf you want to specify metrics, and do not want to use default, specify it in config file (json) inmetricskey.{"predictions":"/path/to/predictions.txt","references":"/path/to/references.txt","reduce_fn":"max","metrics":["bleu","meteor"]}Then, you can call jury eval withconfigargument.jury eval --config path/to/config.jsonCustom MetricsYou can use custom metrics with inheritingjury.metrics.Metric, you can see current metrics implemented on Jury fromjury/metrics. Jury falls back toevaluateimplementation of metrics for the ones that are currently not supported by Jury, you can see the metrics available forevaluateonevaluate/metrics.Jury itself usesevaluate.Metricas a base class to drive its own base class asjury.metrics.Metric. The interface is similar; however, Jury makes the metrics to take a unified input type by handling the inputs for each metrics, and allows supporting several input types as;single prediction & single referencesingle prediction & multiple referencemultiple prediction & multiple referenceAs a custom metric both base classes can be used; however, we strongly recommend usingjury.metrics.Metricas it has several advantages such as supporting computations for the input types above or unifying the type of the input.fromjury.metricsimportMetricForTaskclassCustomMetric(MetricForTask):def_compute_single_pred_single_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrordef_compute_single_pred_multi_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrordef_compute_multi_pred_multi_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrorFor more details, have a look at base metric implementationjury.metrics.MetricContributingPRs are welcomed as always :)Installationgit clone https://github.com/obss/jury.git
cd jury
pip install -e .[dev]Also, you need to install the packages which are available through a git source separately with the following command.
For the folks who are curious about "why?"; a short explaination is that PYPI does not allow indexing a package which
are directly dependent on non-pypi packages due to security reasons. The filerequirements-dev.txtincludes packages
which are currently only available through a git source, or they are PYPI packages with no recent release or
incompatible with Jury, so that they are added as git sources or pointing to specific commits.pip install -r requirements-dev.txtTestsTo tests simply run.python tests/run_tests.pyCode StyleTo check code style,python tests/run_code_style.py checkTo format codebase,python tests/run_code_style.py formatCitationIf you use this package in your work, please cite it as:@misc{cavusoglu2023jury,
  title={Jury: A Comprehensive Evaluation Toolkit}, 
  author={Devrim Cavusoglu and Ulas Sert and Secil Sen and Sinan Altinuc},
  year={2023},
  eprint={2310.02040},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  doi={10.48550/arXiv.2310.02040}
}LicenseLicensed under theMITLicense.