Package: bleu
bleu (Python Package)A Python Wrapper for the standard BLEU evaluation for Natural Language Generation (NLG).GitHub project:https://github.com/zhijing-jin/bleu.PyPI package:pip installbleuInstallationRequirement: Python 3Option 1: Install pip packagepipinstall--upgradebleuOption 2: Build from sourcepipinstall--upgradegit+git://github.com/zhijing-jin/bleu.gitHow to RunThe most standard way to calculate BLEU is byMoses' script for detokenized BLEU. This package provides easy calls to it.Function 1: Calculate the BLEU for listsIf you want to check only one hypothesis (a list of sentences):>>>frombleuimportlist_bleu>>>ref=['it is a white cat .','wow , this dog is huge .']>>>ref1=['This cat is white .','wow , this is a huge dog .']>>>hyp=['it is a white kitten .','wowww , the dog is huge !']>>>hyp1=["it 's a white kitten .",'wow , this dog is huge !']>>>list_bleu([ref],hyp)34.99>>>list_bleu([ref,ref1],hyp1)57.91If you want to check multiple hypothesis (several lists of sentences):>>>frombleuimportmulti_list_bleu>>>multi_list_bleu([ref,ref1],[hyp,hyp1])[34.99,57.91]detok=False: It is not advisable to use tokenized bleu (bymulti-bleu.perl), but if you want to call it, just usedetok=False:>>>list_bleu([ref],hyp,detok=False)39.76# or if you want to test multiple hypotheses>>>multi_list_bleu([ref,ref1],[hyp,hyp1],detok=False)[39.76,47.47]verbose=True: If there are unexpected errors, you might want to check the intermediate steps byverbose=True.Function 2: Calculate the BLEU for filesIf you want to check only one hypothesis file:# if you already have the following files>>>frombleuimportfile_bleu>>>hyp_file='data/hyp0.txt'>>>ref_files=['data/ref0.txt','data/ref1.txt']>>>file_bleu(ref_files,hyp_file)34.99If you want to check multiple hypothesis files:>>>frombleuimportmulti_file_bleu>>>hyp_file1='data/hyp1.txt'>>>bleus=multi_file_bleu(ref_files,[hyp_file,hyp_file1])[34.99,57.91]detok=True: Set it if you want to calculate the (not recommended) tokenized bleu.verbose=True: Set it if you want to inspect how the bleu calculations are made:>>>bleu=file_bleu(ref_files,hyp_file,verbose=True)[Info]ValidReferenceFiles:['data/ref0.txt','data/ref1.txt'][Info]ValidHypothesisFiles:['data/hyp0.txt'][Info]#lines in each file: 2[cmd]perldetokenizer.perl-len<data/ref0.txt>data/ref0.detok.txt2>/dev/null[cmd]perldetokenizer.perl-len<data/ref1.txt>data/ref1.detok.txt2>/dev/null[cmd]perldetokenizer.perl-len<data/hyp0.txt>data/hyp0.detok.txt2>/dev/null[cmd]perlmulti-bleu-detok.perldata/ref0.detok.txtdata/ref1.detok.txt<data/hyp0.detok.txt2-refbleufordata/hyp0.detok.txt:34.99>>>bleu34.99Option 3: Detokenize files>>>frombleuimportdetok_files>>>detok_ref_files=detok_files(ref_files,tmp_dir='./data',file_prefix='ref_dtk',verbose=True)[cmd]perl./TMP_DIR/detokenizer.perl-len<data/ref0.txt>data/ref_dtk0.txt2>/dev/null[cmd]perl./TMP_DIR/detokenizer.perl-len<data/ref1.txt>data/ref_dtk1.txt2>/dev/null>>>detok_ref_files['data/ref_dtk0.txt','data/ref_dtk1.txt']In Case of Unexpected OutputsCheck the python filebleu.pyand adapt it.ContactIf you have more questions, feel free to check out the commonQ&A, or raise a new GitHub issue.In case of really urgent needs, contact the authorZhijing Jin (Miss).