Package: unbabel-comet
NEWS:AfriCOMETreleased, a new model to embrace under-resourced African Languages.We released our new eXplainable COMET models (XCOMET-XLand-XXL) which along with quality scores detects which errors in the translation are minor, major or critical according to MQM typologyWe releaseCometKiwi -XL (3.5B)and-XXL (10.7B)QE models. These models were the best performing QE models on the WMT23 QE shared task.Please check all available modelshereQuick InstallationCOMET requires python 3.8 or above. Simple installation from PyPIpipinstall--upgradepip# ensures that pip is currentpipinstallunbabel-cometNote:To use some COMET models such asUnbabel/wmt22-cometkiwi-dayou must acknowledge it's license on Hugging Face Hub andlog-in into hugging face hub.To develop locally install run the following commands:gitclonehttps://github.com/Unbabel/COMETcdCOMET
pipinstallpoetry
poetryinstallFor development, you can run the CLI tools directly, e.g.,PYTHONPATH=../comet/cli/score.pyScoring MT outputs:CLI Usage:Test examples:echo-e"10 到 15 分钟可以送到吗\nPode ser entregue dentro de 10 a 15 minutos?">>src.txtecho-e"Can I receive my food in 10 to 15 minutes?\nCan it be delivered in 10 to 15 minutes?">>hyp1.txtecho-e"Can it be delivered within 10 to 15 minutes?\nCan you send it for 10 to 15 minutes?">>hyp2.txtecho-e"Can it be delivered between 10 to 15 minutes?\nCan it be delivered between 10 to 15 minutes?">>ref.txtBasic scoring command:comet-score-ssrc.txt-thyp1.txt-rref.txtyou can set the number of gpus using--gpus(0 to test on CPU).For better error analysis, you can use XCOMET models such asUnbabel/XCOMET-XL, you can export the identified errors using the--to_jsonflag:comet-score-ssrc.txt-thyp1.txt-rref.txt--modelUnbabel/XCOMET-XL--to_jsonoutput.jsonScoring multiple systems:comet-score-ssrc.txt-thyp1.txthyp2.txt-rref.txtWMT test sets viaSacreBLEU:comet-score-dwmt22:en-de-tPATH/TO/TRANSLATIONSIf you are only interested in a system-level score use the following command:comet-score-ssrc.txt-thyp1.txt-rref.txt--quiet--only_systemReference-free evaluation:comet-score-ssrc.txt-thyp1.txt--modelUnbabel/wmt22-cometkiwi-daNote:To use theUnbabel/wmt22-cometkiwi-da-xlyou first have to acknowledge its license onHugging Face Hub.Comparing multiple systems:When comparing multiple MT systems we encourage you to run thecomet-comparecommand to getstatistical significancewith Paired T-Test and bootstrap resampling(Koehn, et al 2004).comet-compare-ssrc.de-thyp1.enhyp2.enhyp3.en-rref.enMinimum Bayes Risk Decoding:The MBR command allows you to rank translations and select the best one according to COMET metrics. For more details you can read our paper onQuality-Aware Decoding for Neural Machine Translation.comet-mbr-s[SOURCE].txt-t[MT_SAMPLES].txt--num_sample[X]-o[OUTPUT_FILE].txtIf working with a very large candidate list you can use--rerank_top_kflag to prune the topK most promissing candidates according to a reference-free metric.Example for a candidate list of 1000 samples:comet-mbr-s[SOURCE].txt-t[MT_SAMPLES].txt-o[OUTPUT_FILE].txt--num_sample1000--rerank_top_k100--gpus4--qe_modelUnbabel/wmt23-cometkiwi-da-xlYour source and samples file should beformatted in this way.COMET ModelsWithin COMET, there are several evaluation models available. You can refer to theMODELSpage for a comprehensive list of all available models. Here is a concise list of the main reference-based and reference-free models:Default Model:Unbabel/wmt22-comet-da- This model employs a reference-based regression approach and is built upon the XLM-R architecture. It has been trained on direct assessments from WMT17 to WMT20 and provides scores ranging from 0 to 1, where 1 signifies a perfect translation.Reference-free Model:Unbabel/wmt22-cometkiwi-da- This reference-free model employs a regression approach and is built on top of InfoXLM. It has been trained using direct assessments from WMT17 to WMT20, as well as direct assessments from the MLQE-PE corpus. Similar to other models, it generates scores ranging from 0 to 1. For those interested, we also offer larger versions of this model:Unbabel/wmt23-cometkiwi-da-xlwith 3.5 billion parameters andUnbabel/wmt23-cometkiwi-da-xxlwith 10.7 billion parameters.eXplainable COMET (XCOMET):Unbabel/XCOMET-XXL- Our latest model is trained to identify error spans and assign a final quality score, resulting in an explainable neural metric. We offer this version in XXL with 10.7 billion parameters, as well as the XL variant with 3.5 billion parameters (Unbabel/XCOMET-XL). These models have demonstrated the highest correlation with MQM and are our best performing evaluation models.Please be aware that different models may be subject to varying licenses. To learn more, kindly refer to theLICENSES.modelsand model licenses sections.If you intend to compare your results with papers published before 2022, it's likely that they used older evaluation models. In such cases, please refer toUnbabel/wmt20-comet-daandUnbabel/wmt20-comet-qe-da, which were the primary checkpoints used in previous versions (<2.0) of COMET.Also,UniTE Metricdeveloped by the NLP2CT Lab at the University of Macau and Alibaba Group can be used directly through COMET checkhere for more details.Interpreting Scores:When using COMET to evaluate machine translation, it's important to understand how to interpret the scores it produces.In general, COMET models are trained to predict quality scores for translations. These scores are typically normalized using az-score transformationto account for individual differences among annotators. While the raw score itself does not have a direct interpretation, it is useful for ranking translations and systems according to their quality.However, since 2022 we have introduced a new training approach that scales the scores between 0 and 1. This makes it easier to interpret the scores: a score close to 1 indicates a high-quality translation, while a score close to 0 indicates a translation that is no better than random chance. Also, with the introduction of XCOMET models we can now analyse which text spans are part of minor, major or critical errors according to the MQM typology.It's worth noting that when using COMET to compare the performance of two different translation systems, it's important to run thecomet-comparecommand to obtain statistical significance measures. This command compares the output of two systems using a statistical hypothesis test, providing an estimate of the probability that the observed difference in scores between the systems is due to chance. This is an important step to ensure that any differences in scores between systems are statistically significant.Overall, the added interpretability of scores in the latest COMET models, combined with the ability to assess statistical significance between systems usingcomet-compare, make COMET a valuable tool for evaluating machine translation.Languages Covered:All the above mentioned models are build on top of XLM-R (variants) which cover the following languages:Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Basque, Belarusian, Bengali, Bengali Romanized, Bosnian, Breton, Bulgarian, Burmese, Burmese, Catalan, Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Hausa, Hebrew, Hindi, Hindi Romanized, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish (Kurmanji), Kyrgyz, Lao, Latin, Latvian, Lithuanian, Macedonian, Malagasy, Malay, Malayalam, Marathi, Mongolian, Nepali, Norwegian, Oriya, Oromo, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Sanskri, Scottish, Gaelic, Serbian, Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Sundanese, Swahili, Swedish, Tamil, Tamil Romanized, Telugu, Telugu Romanized, Thai, Turkish, Ukrainian, Urdu, Urdu Romanized, Uyghur, Uzbek, Vietnamese, Welsh, Western, Frisian, Xhosa, Yiddish.Thus, results for language pairs containing uncovered languages are unreliable!COMET for African Languages:If you are interested in COMET metrics for african languages please visitafriCOMET.Scoring within Python:fromcometimportdownload_model,load_from_checkpoint# Choose your model from Hugging Face Hubmodel_path=download_model("Unbabel/XCOMET-XL")# or for example:# model_path = download_model("Unbabel/wmt22-comet-da")# Load the model checkpoint:model=load_from_checkpoint(model_path)# Data must be in the following format:data=[{"src":"10 到 15 分钟可以送到吗","mt":"Can I receive my food in 10 to 15 minutes?","ref":"Can it be delivered between 10 to 15 minutes?"},{"src":"Pode ser entregue dentro de 10 a 15 minutos?","mt":"Can you send it for 10 to 15 minutes?","ref":"Can it be delivered between 10 to 15 minutes?"}]# Call predict method:model_output=model.predict(data,batch_size=8,gpus=1)print(model_output)print(model_output.scores)# sentence-level scoresprint(model_output.system_score)# system-level score# Not all COMET models return metadata with detected errors.print(model_output.metadata.error_spans)# detected error spansTrain your own Metric:Instead of using pretrained models your can train your own model with the following command:comet-train--cfgconfigs/models/{your_model_config}.yamlYou can then use your own metric to score:comet-score-ssrc.de-thyp1.en-rref.en--modelPATH/TO/CHECKPOINTYou can also upload your model toHugging Face Hub. UseUnbabel/wmt22-comet-daas example. Then you can use your model directly from the hub.unittest:In order to run the toolkit tests you must run the following command:poetryruncoveragerun--source=comet-munittestdiscover
poetryruncoveragereport-m# Expected coverage 76%Note:Testing on CPU takes a long timePublicationsIf you use COMET please cite our workand don't forget to say which model you used!xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error DetectionScaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared TaskCometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared TaskCOMET-22: Unbabel-IST 2022 Submission for the Metrics Shared TaskSearching for Cometinho: The Little Metric That CouldAre References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared TaskUncertainty-Aware Machine Translation EvaluationCOMET - Deploying a New State-of-the-art MT Evaluation Metric in ProductionUnbabel's Participation in the WMT20 Metrics Shared TaskCOMET: A Neural Framework for MT Evaluation