Package: bert-score
BERTScoreAutomatic Evaluation Metric described in the paperBERTScore: Evaluating Text Generation with BERT(ICLR 2020). We now support about 130 models (see thisspreadsheetfor their correlations with human evaluation). Currently, the best model ismicrosoft/deberta-xlarge-mnli, please consider using it instead of the defaultroberta-largein order to have the best correlation with human evaluation.News:Updated to version 0.3.13Fix bug with transformers version > 4.17.0 (#148)Updated to version 0.3.12Havingget_idf_dictcompatible with DDP (#140)Fix setup bug (#138)Updated to version 0.3.11Support 6 DeBERTa v3 modelsSupport 3 ByT5 modelsUpdated to version 0.3.10Support 8 SimCSE modelsFix the support of scibert (to be compatible with transformers >= 4.0.0)Add scripts for reproducing some results in our paper (See thisfolder)Support fast tokenizers in huggingface transformers with--use_fast_tokenizer. Notably, you will get different scores because of the difference in the tokenizer implementations (#106).Fix non-zero recall problem for empty candidate strings (#107).Add Turkish BERT Supoort (#108).Updated to version 0.3.9Support 3 BigBird modelsFix bugs for mBART and T5Support 4 mT5 models as requested (#93)Updated to version 0.3.8Support 53 new pretrained models including BART, mBART, BORT, DeBERTa, T5, BERTweet, MPNet, ConvBERT, SqueezeBERT, SpanBERT, PEGASUS, Longformer, LED, Blendbot, etc. Among them, DeBERTa achives higher correlation with human scores than RoBERTa (our default) on WMT16 dataset. The correlations are presented in thisGoogle sheet.Please consider using--model_type microsoft/deberta-xlarge-mnlior--model_type microsoft/deberta-large-mnli(faster) if you want the scores to correlate better with human scores.Add baseline files for DeBERTa models.Add example code to generate baseline files (please see thedetails).Updated to version 0.3.7Being compatible with Huggingface's transformers version >=4.0.0. Thanks to public contributers (#84,#85,#86).See#22if you want to replicate our experiments on the COCO Captioning dataset.For people in China, downloading pre-trained weights can be very slow. We provide copies of a few models on Baidu Pan.roberta-largepassword: dhe5bert-base-chinesepassword: jvk7bert-base-multilingual-casedpassword: yx3qHuggingface's datasetslibrary includes BERTScore in their metric collection.Previous updatesUpdated to version 0.3.6Support custom baseline files#74The option--rescale-with-baselineis changed to--rescale_with_baselineso that it is consistent with other options.Updated to version 0.3.5Being compatible with Huggingface's transformers >=v3.0.0 and minor fixes (#58,#66,#68)Several improvements related to efficency (#67,#69)Updated to version 0.3.4Compatible with transformers v2.11.0 now (#58)Updated to version 0.3.3Fixing the bug with empty stringsissue #47.Supporting 6ELECTRAmodels and 24 smallerBERTmodels.A newGoogle sheetfor keeping the performance (i.e., pearson correlation with human judgment) of different models on WMT16 to-English.Including the script for tuning the best number of layers of an English pre-trained model on WMT16 to-English data (See thedetails).Updated to version 0.3.2Bug fixed: fixing the bug in v0.3.1 when having multiple reference sentences.Supporting multiple reference sentences with our command line tool.Updated to version 0.3.1A newBERTScorerobject that caches the model to avoid re-loading it multiple times. Please see ourjupyter notebook examplefor the usage.Supporting multiple reference sentences for each example. Thescorefunction now can take a list of lists of strings as the references and return the score between the candidate sentence and its closest reference sentence.Please seerelease logsfor older updates.Authors:Tianyi Zhang*Varsha Kishore*Felix Wu*Kilian Q. WeinbergerYoav Artzi*: Equal ContributionOverviewBERTScore leverages the pre-trained contextual embeddings from BERT and matches
words in candidate and reference sentences by cosine similarity.
It has been shown to correlate with human judgment on sentence-level and
system-level evaluation.
Moreover, BERTScore computes precision, recall, and F1 measure, which can be
useful for evaluating different language generation tasks.For an illustration, BERTScore recall can be computed asIf you find this repo useful, please cite:@inproceedings{bert-score,
  title={BERTScore: Evaluating Text Generation with BERT},
  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
  booktitle={International Conference on Learning Representations},
  year={2020},
  url={https://openreview.net/forum?id=SkeHuCVFDr}
}InstallationPython version >= 3.6PyTorch version >= 1.0.0Install from pypi with pip bypipinstallbert-scoreInstall latest unstable version from the master branch on Github by:pip install git+https://github.com/Tiiiger/bert_scoreInstall it from the source by:gitclonehttps://github.com/Tiiiger/bert_scorecdbert_score
pipinstall.and you may test your installation by:python -m unittest discoverUsagePython FunctionOn a high level, we provide a python functionbert_score.scoreand a python objectbert_score.BERTScorer.
The function provides all the supported features while the scorer object caches the BERT model to faciliate multiple evaluations.
Check ourdemoto see how to use these two interfaces.
Please refer tobert_score/score.pyfor implementation details.Running BERTScore can be computationally intensive (because it uses BERT :p).
Therefore, a GPU is usually necessary. If you don't have access to a GPU, you
can try ourdemo on Google ColabCommand Line Interface (CLI)We provide a command line interface (CLI) of BERTScore as well as a python module.
For the CLI, you can use it as follows:To evaluate English text files:We provide example inputs under./example.bert-score-rexample/refs.txt-cexample/hyps.txt--langenYou will get the following output at the end:roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0) P: 0.957378 R: 0.961325 F1: 0.959333where "roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)" is the hash code.Starting from version 0.3.0, we support rescaling the scores with baseline scoresbert-score-rexample/refs.txt-cexample/hyps.txt--langen--rescale_with_baselineYou will get:roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)-rescaled P: 0.747044 R: 0.770484 F1: 0.759045This makes the range of the scores larger and more human-readable. Please see thispostfor details.When having multiple reference sentences, please usebert-score-rexample/refs.txtexample/refs2.txt-cexample/hyps.txt--langenwhere the-rargument supports an arbitrary number of reference files. Each reference file should have the same number of lines as your candidate/hypothesis file. The i-th line in each reference file corresponds to the i-th line in the candidate file.To evaluate text files in other languages:We currently support the 104 languages in multilingual BERT (full list).Please specify the two-letter abbreviation of the language. For instance, using--lang zhfor Chinese text.See more options bybert-score -h.To load your own custom model:
Please specify the path to the model and the number of layers to use by--modeland--num_layers.bert-score-rexample/refs.txt-cexample/hyps.txt--modelpath_to_my_bert--num_layers9To visualize matching scores:bert-score-show--langen-r"There are two bananas on the table."-c"On the table are two apples."-fout.pngThe figure will be saved to out.png.If you see the following message while using BERTScore, please ignore it. This is expected.Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.decoder.weight', 'lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).Practical TipsReport the hash code (e.g.,roberta-large_L17_no-idf_version=0.3.0(hug_trans=2.3.0)-rescaled) in your paper so that people know what setting you use. This is inspired bysacreBLEU. Changes in huggingface's transformers version may also affect the score (Seeissue #46).Unlike BERT, RoBERTa uses GPT2-style tokenizer which creates addition " " tokens when there are multiple spaces appearing together. It is recommended to remove addition spaces bysent = re.sub(r' +', ' ', sent)orsent = re.sub(r'\s+', ' ', sent).Using inverse document frequency (idf) on the reference
sentences to weigh word importance  may correlate better with human judgment.
However, when the set of reference sentences become too small, the idf score
would become inaccurate/invalid.
We now make it optional. To use idf,
please set--idfwhen using the CLI tool oridf=Truewhen callingbert_score.scorefunction.When you are low on GPU memory, consider settingbatch_sizewhen callingbert_score.scorefunction.To use a particular model please set-m MODEL_TYPEwhen using the CLI tool
ormodel_type=MODEL_TYPEwhen callingbert_score.scorefunction.We tune layer to use based on WMT16 metric evaluation dataset. You may use a
different layer by setting-l LAYERornum_layers=LAYER. To tune the best layer for your custom model, please follow the instructions intune_layersfolder.Limitation: Because BERT, RoBERTa, and XLM with learned positional embeddings are pre-trained on sentences with max length 512, BERTScore is undefined between sentences longer than 510 (512 after adding [CLS] and [SEP] tokens). The sentences longer than this will be truncated. Please consider using XLNet which can support much longer inputs.Default BehaviorDefault ModelLanguageModelenroberta-largeen-sciallenai/scibert_scivocab_uncasedzhbert-base-chinesetrdbmdz/bert-base-turkish-casedothersbert-base-multilingual-casedDefault LayersPlease see thisGoogle sheetfor the supported models and their performance.AcknowledgementThis repo wouldn't be possible without the awesomebert,fairseq, andtransformers.