Package: eaas
Evaluation-as-a-Service for NLPUsageBefore using EaaS, please see theterms of use.
Detailed documentation can be foundhere. To install the EaaS, simply runpipinstalleaasRun your "Hello, world"A minimal EaaS application looks something like this:fromeaasimportConfig,Clientclient=Client(Config())inputs=[{"source":"Hello, my world","references":["Hello, world","Hello my world"],"hypothesis":"Hi, my world"}]metrics=["rouge1","bleu","chrf"]score_dic=client.score(inputs,metrics=metrics)Ifeaashas been installed successfully, you should get the results
below by printingscore_dic. Each entry corresponds to the metrics passed
tometrics(in the same order). Thecorpusentry indicates the corpus-level
score,sampleentry is a list of sample-level scores:score_dic={'scores':[{'corpus':0.6666666666666666,'sample':[0.6666666666666666]},{'corpus':0.35355339059327373,'sample':[0.35355339059327373]},{'corpus':0.4900623006253688,'sample':[0.4900623006253688]}]}Notably:To use this API for scoring, you need to format your input as list of dictionary.Each dictionary consists ofsource(string, optional),references(list of string, optional) andhypothesis(string, required).sourceandreferencesare optional based on the metrics you want to use.Please do not conduct any preprocessing onsource,referencesorhypothesis.We expect normal-cased detokenized texts. All the preprocessing steps are taken by the metrics.Supported MetricsCurrently, EaaS supports the following metrics:bart_score_en_ref:BARTScoreis a sequence to sequence framework based on pre-trained language model BART.bart_score_cnn_hypo_refuses the CNNDM finetuned BART. It calculates the average generation score ofScore(hypothesis|reference)andScore(reference|hypothesis).bart_score_en_src:BARTScoreusing the CNNDM finetuned BART. It calculatesScore(hypothesis|source).bert_score_p:BERTScoreis a metric designed for evaluating translated text using BERT-based matching framework.bert_score_pcalculates the BERTScore precision.bert_score_r:BERTScorerecall.bert_score_f:BERTScoref score.bleu:BLEUmeasures modified ngram matches between each candidate translation and the reference translations.chrf:CHRFmeasures the character-level ngram matches between hypothesis and reference.comet:COMETis a neural framework for training multilingual machine translation evaluation models.cometuses thewmt20-comet-dacheckpoint which utilizes source, hypothesis and reference.comet_qe:COMETfor quality estimation.comet_qeuses thewmt20-comet-qe-dacheckpoint which utilizes only source and hypothesis.mover_score:MoverScoreis a metric similar to BERTScore. Different from BERTScore, it uses the Earth Mover’s Distance instead of the Euclidean Distance.prism:PRISMis a sequence to sequence framework trained from scratch.prismcalculates the average generation score ofScore(hypothesis|reference)andScore(reference|hypothesis).prism_qe:PRISMfor quality estimation. It calculatesScore(hypothesis| source).rouge1:ROUGE-1refers to the overlap of unigram (each word) between the system and reference summaries.rouge2:ROUGE-2refers to the overlap of bigrams between the system and reference summaries.rougeL:ROUGE-Lrefers to the longest common subsequence between the system and reference summaries.The default configurations for each metric can refer to thisdocAsynchronous RequestsIf you want to make a call to the EaaS server to calculate some metrics and continue
local computation while waiting for the result, you can do so as follows:fromeaasimportConfigfromeaas.async_clientimportAsyncClientconfig=Config()client=AsyncClient(config)inputs=...req=client.async_score(inputs,metrics=["bleu"])# do some other computationresult=req.get_result()