Package: nlg-metricverse
nlg-metricverse 🌌🚀 Spaceship👨‍🚀 Astronauts🛰️ Training Program📕 Operating ManualCOLING22 Long PaperOne NLG evaluation library to rule them allExplore the universe of Natural Language Generation (NLG) evaluation metrics.NLG Metricverse is an end-to-end Python library for NLG evaluation, devised to provide a living unified codebase for fast application, analysis, comparison, visualization, and prototyping of automatic metrics.Spures the adoption of newly proposed metrics, unleashing their potentialReduces the implementational burden, allowing users to easily move from papers to practical applications.Increases comparability and replicability of NLG research.Provides content-rich metric cards and static/interactive visualization tools to improve metric understanding and scoring interpretation.Tables Of ContentsMotivationsAvailable MetricsInstallationExplore on Hugging Face SpacesQuickstartMetric SelectionMetric DocumentationMetric FilteringMetric UsagePrediction-Reference CardinalityScorer ApplicationMetric-specific ParametersOutputsCode StyleCustom MetricsContributingContactLicense💡 Motivations📖 As Natural Language Generation (NLG) models are getting better over time, accurately evaluating them is becoming an increasingly pressing priority, asking researchers to deal with semantics, different plausible targets, and multiple intrinsic quality dimensions (e.g., informativeness, fluency, factuality).🤖 Task examples: machine translation, abstractive question answering, single/multi-document summarization, data-to-text, chatbots, image/video captioning, etc.📌 Human evaluation is often the best indicator of the quality of a system. However, designing crowd sourcing experiments is an expensive and high-latency process, which does not easily fit in a daily model development pipeline. Therefore, NLG researchers commonly use automatic evaluation metrics, which provide an acceptable proxy for quality and are very cheap to compute.📌 NLG metrics automatically compute a holistic or dimension-specific score, an acceptable proxy for effectiveness and efficiency. However, they are becoming an important bottleneck for research in the field. As we know, areas can stagnate due to poor metrics, and we believe that you shouldn't feel confined to the most traditional overlap-based techniques like ROUGE.💡 If you're working on an established problem, you'll feel pressure from readers to be conservative and use the metrics that have already been tested for the same task. However, this might be a compelling pressure. Our view is that NLP engineers should enrich their evaluation toolkits with multiple metrics capturing different textual properties, being free to argue against cultural norms and motivate new ones, also exploring the latest contributions focused on semantics.☠ New NLG metrics are constantly being proposed to top-tier venue conferences, but their implementation remains disrupted, with distinct environments, properties, settings, benchmarks, and features—making them difficult to compare or apply.☠ The absence of a collective and continuously updated repository discourages the use of modern solutions and slows their understanding.🎯 NLG Metricverse implements a large number of prominent evaluation metrics in NLG, seeking to articulate the textual properties they encode (e.g., fluency, grammatical correctness, informativeness), tasks, and limits. Understanding, using, and examining a metric has never been easier.🪐 Available Metrics and Supported FeaturesNLG Metricverse supports 38 diverse evaluation metrics overall (last update: October 12, 2022). The code for these metrics will be progressively released in the coming weeks.Some libraries have already tried to make an integrated environment. To our best knowledge,NLGEval,HugginFace Datasets,Evaluate,Torch-Metrics, andJuryare the only resources available.
However, none of them possess all the properties listed below: (i) large number of heterogeneous NLG metrics, (ii) concurrent computation of more metrics at once, (iii) support for multiple references and/or predictions, (iv) meta-evaluation, and (v) visualization.The following table summarizes the discrepancies between NLG Metricverse and related work.NLG-MetricverseNLGEvalDatasetsEvaluateTorchMetricsJury#NLG-specific metrics43 + Datasets822531319 + DatasetsMore metrics at once:white_check_mark::x::x::white_check_mark::x::white_check_mark:Multiple refs/preds:white_check_mark::white_check_mark::x::x::x::white_check_mark:Meta-evaluation:white_check_mark::x::x::x::x::x:Visualization:white_check_mark::x::x::x::x::x:🔍Complete comparison and supported metrics🔌 InstallationInstall from PyPI repositorypip install nlg-metricverseor build from sourcegit clone https://github.com/disi-unibo-nlp/nlg-metricverse.git
cd nlg-metricverse
pip install -v .Explore on Hugging Face SpacesTheSpacesedition of NLG Metricverse will be launched soon. Check it out here:🚀 QuickstartPrepare your environmentFor NLGmetricverse we recommend using a virtual environment. If you are not familiar with virtual environments, you can read more about themhere.
Using virtual environments within a library that encompasses numerous metrics proves invaluable for seamless development and efficient management. By encapsulating each metric within its isolated environment, potential conflicts between dependencies are mitigated, ensuring consistent and reliable behavior. This approach streamlines dependency management, enabling precise specification of version requirements for each metric. Moreover, venv facilitates rigorous testing and reproducibility, safeguarding the library's integrity across various metric-driven scenarios. As metrics expand, venv simplifies collaboration among team members, reduces the risk of global environment contamination, and eases deployment processes.Before running any code, you need to create and activate a virtual environment for the desidered metric and install the required dependencies.python-mvenvnlgmetricverse\env\rouge#activate the virtual environment on Command Promptnlgmetricverse\env\rouge\Scripts\activate.bat#or else on powershellnlgmetricverse\env\rouge\Scripts\activate.ps1!pipinstall-v.--quiet"""Also, you need to install the packages which are available through a git source separately with the following command.For the folks who are curious about "why?"; a short explaination is that PYPI does not allow indexing a package whichare directly dependent on non-pypi packages due to security reasons. The file `requirements-dev.txt` includes packageswhich are currently only available through a git source, or they are PYPI packages with no recent release orincompatible with NLGmetricverse, so that they are added as git sources or pointing to specific commits."""!pipinstall-rrequirements-dev.txt#if present, install the specific requirements for the metric!pipinstall-rnlgmetricverse\metrics\rouge\requirements.txtAfter that, you can run the code for the metric you want to use. After you are done, you can deactivate the virtual environment.deactivateThen it is only withtwo lines of codeto evaluate generated outputs:(i)instantiate your scorer by selecting the desired metric(s) and(ii)apply it!Metric SelectionSpecify the metrics you want to use on instantiation,# If you specify more metrics, each of them will be applyied on your data (allowing for a fast prediction/efficiency comparison)scorer=NLGMetricverse(metrics=["bleu","rouge"])or directly import metrics fromnlgmetricverse.metricsas classes, then instantiate and use them as desired.fromnlgmetricverse.metricsimportBertScorescorer=BertScore.construct()You can seemlessly access bothnlgmetricverseand HuggingFacedatasetsmetrics throughnlgmetricverse.load_metric.
NLG Metricverse falls back todatasetsimplementation of metrics for the ones that are currently not supported; you can see the metrics available fordatasetsondatasets/metrics.bleu=NLGMetricverse.load_metric("bleu")# metrics not available in `nlgmetricverse` but in `datasets`wer=NLGMetricverse.load_metric("competition_math")# It falls back to `datasets` package with a warningMetric UsagePrediction-Reference Cardinality☠ NLG evaluation is very challenging also because the relationships between candidate and reference texts tend to be one-to-many or many-to-many. An artificial text predicted by a model might have multiple human references (i.e., there is more than one effective way to say most things), as well as a model can generate multiple distinct outputs. Such cardinality is crucial, but official implementations tend to neglect it. We do not.1:1. One prediction, one reference ([p1, ..., pn] and [r1, ..., rn] syntax).predictions=["Evaluating artificial text has never been so simple","the cat is on the mat"]references=["Evaluating artificial text is not difficult","The cat is playing on the mat."]1:M. One prediction, many references ([p1, ..., pn] and [[r11, ..., r1m], ..., [rn1, ..., rnm]] syntax)predictions=["Evaluating artificial text has never been so simple","the cat is on the mat"]references=[["Evaluating artificial text is not difficult","Evaluating artificial text is simple"],["The cat is playing on the mat.","The cat plays on the mat."]]K:M. Many predictions, many references ([[p11, ..., p1k], ..., [pn1, ..., pnk]] and [[r11, ..., r1m], ..., [rn1, ..., rnm]] syntax). This is helpful for language models with a decoding strategy focused on diversity (e.g., beam search, temperature sampling).predictions=[["Evaluating artificial text has never been so simple","The evaluation of automatically generated text is simple."],["the cat is on the mat","the cat likes playing on the mat"]]references=[["Evaluating artificial text is not difficult","Evaluating artificial text is simple"],["The cat is playing on the mat.","The cat plays on the mat."]]Scorer Applicationscores=scorer(predictions,references)Thescorerautomatically selects the proper strategy for applying the selected metric(s) depending on the input format. In any case, if a prediction needs to be compared against multiple references, you can customize the reduction function to use (e.g.,reduce_fn=maxchooses the prediction-reference pair with the highest score for each of the N items in the dataset).scores=scorer.compute(predictions,references,reduce_fn="max")Metric-specific ParametersAdditional metric-specific parameters can be specified on instantiation.metrics=[load_metric("bleu",resulting_name="bleu_1",compute_kwargs={"max_order":1}),load_metric("bleu",resulting_name="bleu_2",compute_kwargs={"max_order":2}),load_metric("bertscore",resulting_name="bertscore_1",compute_kwargs={"model_type":"microsoft/deberta-large-mnli","idf":True}),load_metric("rouge")]scorer=NLGMetricverse(metrics=metrics)Code StyleTo check the code style,python tests/run_code_style.py checkTo format the codebase,python tests/run_code_style.py format🎨 Custom MetricsYou can use custom metrics by inheritingnlgmetricverse.metrics.Metric.
You can see current metrics implemented on NLG Metricverse fromnlgmetricverse/metrics.
NLG Metricverse itself usesdatasets.Metricas a base class to drive its own base class asnlgmetricverse.metrics.Metric. The interface is similar; however, NLG Metricverse makes the metrics to take a unified input type by handling metric-specific inputs and allowing multiple cardinalities (1:1, 1:M, K:M).
For implementing custom metrics, both base classes can be used but we strongly recommend usingnlgmetricverse.metrics.Metricfor its advantages.
When using a custom metric, you need to:Create a folder insidenlgmetricverse/metricswith the name of your metric.Create inside the folder__init__.py,*metric*.pyand*metric*_planet.py.Inside__init__.py, add the following code:fromnlgmetricverse.metrics.*metric*.*metric*import*Metric*Inside*metric*.py, add the following code:"""*Metric* metric super class."""fromnlgmetricverse.metrics._coreimportMetricAliasfromnlgmetricverse.metrics.*metric*.*metric*import*CustomMetric*__main_class__="*Metric*"class*Metric*(MetricAlias):"""*Metric* metric superclass."""_SUBCLASS=*CustomMetric*Inside*metric*_planet.py, add the following code:fromnlgmetricverse.metricsimportMetricForLanguageGenerationclassCustomMetric(MetricForLanguageGeneration):def_compute_single_pred_single_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrordef_compute_single_pred_multi_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrordef_compute_multi_pred_multi_ref(self,predictions,references,reduce_fn=None,**kwargs):raiseNotImplementedErrorFor more details, have a look at base metric implementationnlgmetricverse.metrics.MetricInside your metric folder add a README.md file, following themetric card guidelines.Add your metric to thecomparison tableand to theREADME.mdfile.Add your metric tonlgmetricverse/metrics/__init__.pyfile.Add your metric to metrics_list insidenlgmetricverse/metrics/_core/utils.pyfile.Add test cases for your metric insidetests/nlgmetricverse/metricsfolder, with its respective expected outputs, insidetests/test_data/expected_outputs/metricsfolder.🙌 ContributingThanks go to all these wonderful collaborations for their contribution towards the NLG Metricverse library:Giacomo FrisoniAndrea ZammarchiValentina PieriMarco AvagnanoWe are hoping that the open-source community will help us edit the code and make it better!
Don't hesitate to open issues and contribute the fix/improvement! We can guide you if you're not sure where to start but want to help us out 🥇.
In order to contribute a change to our code base, please submit a pull request (PR) via GitHub and someone from our team will go over it and accept it.If you have troubles, suggestions, or ideas, theDiscussionboard might have some relevant information. If not, you can post your questions there 💬🗨.✉ ContactContact person: Giacomo Frisoni,giacomo.frisoni@unibo.it.
This research work has been conducted within the Department of Computer Science and Engineering, University of Bologna, Italy.LicenseThe code is released under theMIT License. It should not be used to promote or profit from violence, hate, and division, environmental destruction, abuse of human rights, or the destruction of people's physical and mental health.Star History