Package: awesome-align
AWESOME: Aligning Word Embedding Spaces of Multilingual Encodersawesome-alignis a tool that can extract word alignments from multilingual BERT (mBERT)[Demo]and allows you to fine-tune mBERT on parallel corpora for better alignment quality (seeour paperfor more details).DependenciesFirst, you need to install the dependencies:pipinstall-rrequirements.txt
pythonsetup.pyinstallInput formatInputs should betokenizedand each line is a source language sentence and its target language translation, separated by (|||). You can see some examples in theexamplesfolder.Extracting alignmentsHere is an example of extracting word alignments from multilingual BERT:DATA_FILE=/path/to/data/fileMODEL_NAME_OR_PATH=bert-base-multilingual-casedOUTPUT_FILE=/path/to/output/fileCUDA_VISIBLE_DEVICES=0awesome-align\--output_file=$OUTPUT_FILE\--model_name_or_path=$MODEL_NAME_OR_PATH\--data_file=$DATA_FILE\--extraction'softmax'\--batch_size32This produces outputs in thei-jPharaoh format. A pairi-jindicates that theith word (zero-indexed) of the source sentence is aligned to thejth word of the target sentence.You can set--output_prob_fileif you want to obtain the alignment probability and set--output_word_fileif you want to obtain the aligned word pairs (in thesrc_word<sep>tgt_wordformat). You can also set--cache_dirto specify where you want to cache multilingual BERT.You can also setMODEL_NAME_OR_PATHto the path of your fine-tuned model as shown below.Fine-tuning on parallel dataIf there is parallel data available, you can fine-tune embedding models on that data.Here is an example of fine-tuning mBERT that balances well between efficiency and effectiveness:TRAIN_FILE=/path/to/train/fileEVAL_FILE=/path/to/eval/fileOUTPUT_DIR=/path/to/output/directoryCUDA_VISIBLE_DEVICES=0awesome-train\--output_dir=$OUTPUT_DIR\--model_name_or_path=bert-base-multilingual-cased\--extraction'softmax'\--do_train\--train_tlm\--train_so\--train_data_file=$TRAIN_FILE\--per_gpu_train_batch_size2\--gradient_accumulation_steps4\--num_train_epochs1\--learning_rate2e-5\--save_steps4000\--max_steps20000\--do_eval\--eval_data_file=$EVAL_FILEYou can also fine-tune the model a bit longer with more training objectives for better quality:TRAIN_FILE=/path/to/train/fileEVAL_FILE=/path/to/eval/fileOUTPUT_DIR=/path/to/output/directoryCUDA_VISIBLE_DEVICES=0awesome-train\--output_dir=$OUTPUT_DIR\--model_name_or_path=bert-base-multilingual-cased\--extraction'softmax'\--do_train\--train_mlm\--train_tlm\--train_tlm_full\--train_so\--train_psi\--train_data_file=$TRAIN_FILE\--per_gpu_train_batch_size2\--gradient_accumulation_steps4\--num_train_epochs1\--learning_rate2e-5\--save_steps10000\--max_steps40000\--do_eval\--eval_data_file=$EVAL_FILEIf you want high alignment recalls, you can turn on the--train_cooption, but note that the alignment precisions may drop. You can set--cache_dirto specify where you want to cache multilingual BERT.Supervised settingsIn supervised settings where gold word alignments are available for your training data, you can incorporate the supervised signals into our self-training objective (--train_so) and here is an example command:TRAIN_FILE=/path/to/train/fileTRAIN_GOLD_FILE=/path/to/train/gold/fileOUTPUT_DIR=/path/to/output/directoryCUDA_VISIBLE_DEVICES=0awesome-train\--output_dir=$OUTPUT_DIR\--model_name_or_path=bert-base-multilingual-cased\--extraction'softmax'\--do_train\--train_so\--train_data_file=$TRAIN_FILE\--train_gold_file=$TRAIN_GOLD_FILE\--per_gpu_train_batch_size2\--gradient_accumulation_steps4\--num_train_epochs5\--learning_rate1e-4\--save_steps200Seeexamples/*.goldfor the example format of the gold alignments. You need to turn on the--gold_one_indexoption if the gold alignments are 1-indexed and you can turn on the--ignore_possible_alignmentsoption if you want to ignore possible alignments.Model performanceThe following table shows the alignment error rates (AERs) of our models and popular statistical word aligners on five language pairs. The De-En, Fr-En, Ro-En datasets can be obtained followingthis repo, the Ja-En data is fromthis linkand the Zh-En data is available atthis link. The best scores are inbold.De-EnFr-EnRo-EnJa-EnZh-Enfast_align27.010.532.151.138.1eflomal22.68.225.147.528.7Mgiza20.65.926.448.035.1Ours (w/o fine-tuning, softmax)17.45.627.945.618.1Ours (multilingually fine-tunedw/o--train_co, softmax)[Download]15.24.122.637.413.4Ours (multilingually fine-tunedw/--train_co, softmax)[Download]15.14.520.738.414.5CitationIf you use our tool, we'd appreciate if you cite the following paper:@inproceedings{dou2021word,
  title={Word Alignment by Fine-tuning Embeddings on Parallel Corpora},
  author={Dou, Zi-Yi and Neubig, Graham},
  booktitle={Conference of the European Chapter of the Association for Computational Linguistics (EACL)},
  year={2021}
}AcknowledgementsSome of the code is borrowed fromHuggingFace Transformerslicensed underApache 2.0and the entmax implementation is fromthis repo.