Package: bicleaner-ai
Bicleaner AIBicleaner AI (bicleaner-ai-classify) is a tool in Python that aims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0).
Sentence pairs considered very noisy are scored with 0.Although a training tool (bicleaner-ai-train) is provided, you may want to use the available ready-to-use language packages.
Please, usebicleaner-ai-downloadto download the latest language packages or visit theGithub releasesfor lite models andHugging Face Hubfor full models since v2.0.
Visit ourWikifor a detailed example on Bicleaner training.CitationIf you find Bicleaner AI useful, please consider citing the following paper:J. Zaragoza-Bernabeu, M. Bañón, G. Ramírez-Sánchez, S. Ortiz-Rojas,"Bicleaner AI: Bicleaner Goes Neural",inProceedings of the 13th Language Resources and Evaluation Conference.Marseille, France: Language Resources and Evaluation Conference, June 2022@inproceedings{zaragoza-bernabeu-etal-2022-bicleaner,
    title ={"Bicleaner{AI}: Bicleaner Goes Neural"},
    author ={"Zaragoza-Bernabeu, Jaume  and
      Ram{\'\i}rez-S{\'a}nchez, Gema  and
      Ba{\~n}{\'o}n, Marta  and
      Ortiz Rojas, Sergio"},
    booktitle ={"Proceedings of the Thirteenth Language Resources and Evaluation Conference"},
    month = jun,
    year ={"2022"},
    address ={"Marseille, France"},
    publisher ={"European Language Resources Association"},
    url ={"https://aclanthology.org/2022.lrec-1.87"},
    pages ={"824--831"},
    abstract ={"This paper describes the experiments carried out during the development of the latest version of Bicleaner, named Bicleaner AI, a tool that aims at detecting noisy sentences in parallel corpora. The tool, which now implements a new neural classifier, uses state-of-the-art techniques based on pre-trained transformer-based language models fine-tuned on a binary classification task. After that, parallel corpus filtering is performed, discarding the sentences that have lower probability of being mutual translations. Our experiments, based on the training of neural machine translation (NMT) with corpora filtered using Bicleaner AI for two different scenarios, show significant improvements in translation quality compared to the previous version of the tool which implemented a classifier based on Extremely Randomized Trees."},}What is New?Bicleaner AI is aBicleanerfork that uses neural networks.
It comes with two types of models, lite models for fast scoring and full models for high performance.
Lite models useA Decomposable Attention Model for Natural Language Inference (Parikh et al.).
Full models use fine-tuned XLMRoberta (Unsupervised Cross-lingual Representation Learning at Scale).The use of XLMRoberta and 1:10 positive to negative ratio were inspired in the winner of WMT20 Parallel Corpus Filtering Task paper (Filtering noisy parallel corpus using transformers with proxy task learning).Installation & RequirementsPython >= 3.7TensorFlow >= 2.6.5CUDA 11.2 (for training and inference with full models)Bicleaner AI is written in Python and can be installed usingpip.
It also requires theKenLMPython bindings with support for 7-gram language models.
You can easily install it by running the following commands:pipinstallbicleaner-ai
pipinstall--config-settings="--build-option=--max_order=7"https://github.com/kpu/kenlm/archive/master.zipHardrules usesFastSpellthat requirespython-devandlibhunspell-dev:sudoaptinstallpython-devlibhunspell-devHunspell dictionaries used by default are automatically installed.
If you need to change default configuration for language identification, seehttps://github.com/mbanon/fastspell#configuration.After installation, three binary files (bicleaner-ai-train,bicleaner-ai-classify,bicleaner-ai-download) will be located in yourpython/installation/prefix/bindirectory. This is usually$HOME/.local/binor/usr/local/bin/.TensorFlowTensorFlow 2 will be installed as a dependency andGPU supportis required for training.pipwill install latest TensorFlow but older versions>=2.6.5are supported and can be installed if your machine does not meet TensorFlow CUDA requirements.
Seethistable for the CUDA and TensorFlow versions compatibility.
But current allowed versions only supoportCUDA 11.2.
In case you want a different TensorFlow version, you can downgrade using:pipinstalltensorflow==2.6.5TensorFlow logging messages are suppressed by default, in case you want to see them you have to explicitly setTF_CPP_MIN_LOG_LEVELenvironment variable.
For example:TF_CPP_MIN_LOG_LEVEL=0bicleaner-ai-classifyWARNING: If you are experiencing slow downs because Bicleaner AI is not running in the GPU, you should check those logs to see if TensorFlow is loading all the libraries correctly.Optional requirementsFor Serbo-Croatian languages, models work better with transliteration. To be able score transliterated text, install optional dependency:pip install bicleaner-ai[transliterate]Note that this won't transliterate the output text, it will be used only for scoring.CleaningGetting Startedbicleaner-ai-classifyaims at detecting noisy sentence pairs in a parallel corpus. It
indicates the likelihood of a pair of sentences being mutual translations (with a value near to 1) or not (with a value near to 0). Sentence pairs considered very noisy are scored with 0.By default, the input file (the parallel corpus to be classified) expects at least four columns, being:col1: URL 1col2: URL 2col3: Source sentencecol4: Target sentencebut the source and target sentences column index can be customized by using the--scoland--tcolflags. Urls are not mandatory.The generated output file will contain the same lines and columns that the original input file had, adding an extra column containing the Bicleaner AI classifier score.Download a modelBicleaner AI has two types of models, full and lite models.
Full models are recommended, as they provide much higher quality.
If speed is a hard constraint to you, lite models could be an option (take a look at the speedcomparison).See available full modelshereand available lite modelshere.You can download the model with:bicleaner-ai-download en fr fullThis will downloadbitextor/bicleaner-ai-full-en-frmodel from HuggingFace and store it at the cache directory.Or you can download a lite model with:bicleaner-ai-download en fr lite ./bicleaner-modelsThis will download and store the en-fr lite model at./bicleaner-models/en-fr.Since 2.3.0 version, full models also accept a local path to download, instead of the HF cache directory.
In that case, to use the model, provide the local path instead of the HF identifier.To read more information about how HF cache works, please read theofficial documentation.ClassifyingTo classify a tab separated file containing English sentences in the first column and French sentences in the second column, usebicleaner-ai-classify\--scol1--tcol2corpus.en-fr.tsv\corpus.en-fr.classifed.tsv\bitextor/bicleaner-ai-full-en-frwhere--scoland--tcolindicate the location of source and target sentence,corpus.en-fr.tsvthe input file,corpus.en-fr.classified.tsvoutput file andbitextor/bicleaner-ai-en-fris the HuggingFace model name.
Each line of the new file will contain the same content as the input file, adding a column with the score given by the Bicleaner classifier.Note that, to use a lite model, you need to provide model path in your local file system, instead of HuggingFace model name.ParametersThe complete list of parameters is:positional arguments:input: Tab-separated files to be classified (default line format:URL1 URL2 SOURCE_SENTENCE TARGET_SENTENCE [EXTRA_COLUMNS], tab-separated). When input is -, reads standard input.output: Output of the classification (default: standard output). When output is -, writes standard output.metadata: Training metadata (YAML file), generated bybicleaner-ai-trainor downloaded as a part of a language pack.
There's a script that can download and unpack it for you, use:$bicleaner-ai-downloadencslite./modelsto download English-Czech lite language pack to the ./models directory and unpack it.optional arguments:-h, --help: show this help message and exitOptional:-S SOURCE_TOKENIZER_COMMAND: Source language tokenizer full command (including flags if needed). If not given, Sacremoses tokenizer is used (withescape=Falseoption).-T TARGET_TOKENIZER_COMMAND: Target language tokenizer full command (including flags if needed). If not given, Sacremoses tokenizer is used (withescape=Falseoption).--scol SCOL: Source sentence column (starting in 1). If--headeris set, the expected value will be the name of the field (default: 3 if--headeris not set else src_text)--tcol TCOL: Target sentence column (starting in 1). If--headeris set, the expected value will be the name of the field (default: 4 if--headeris not set else trg_text)--tmp_dir TMP_DIR: Temporary directory where creating the temporary files of this program (default: default system temp dir, defined by the environment variable TMPDIR in Unix)-b BLOCK_SIZE, --block_size BLOCK_SIZE: Sentence pairs per block (default: 10000)--lm_threshold LM_THRESHOLD: Threshold for language model fluency scoring. All sentence pairs whose LM fluency score falls below the threshold are removed (classifier score set to 0), unless the option --keep_lm_result is set. (default: 0.5)--score_only: Only output one column which is the bicleaner score (default: False)--calibrated: Output calibrated scores (default: False)--raw_output: Return raw output without computing positive class probability. (default: False)--disable_hardrules: Disables the bicleaner_hardrules filtering (only bicleaner_classify is applied) (default: False)--disable_lm_filter: Disables LM filtering.--disable_porn_removal: Disables porn removal.--disable_minimal_length: Don't apply minimal length rule (default: False).Logging:-q, --quiet: Silent logging mode (default: False)--debug: Debug logging mode (default: False)--logfile LOGFILE: Store log to a file (default: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>)-v, --version: show version of this script and exitTraining classifiersIn case you need to train a new classifier (i.e. because it is not available in the language packs provided atbicleaner-ai-data), you can usebicleaner-ai-train.bicleaner-ai-trainis a Python tool that allows you to train a classifier which predicts
whether a pair of sentences are mutual translations or not and discards too noisy sentence pairs. Visit ourWikifor a detailed example on Bicleaner AI training.RequirementsIn order to train a new classifier, you must provide:A clean parallel corpus (500k pairs of sentences is the recommended size).Monolingual corpus for the source and the target language (not necessary forxlmrclassifier).Gzipped lists of monolingual word frequencies. You can check their format by downloading any of the available language packs.The SL list of word frequencies with one entry per line. Each entry must contain the following 2 fields, split by space, in this order: word frequency (number of times a word appears in text), SL word.The TL list of word frequencies with one entry per line. Each entry must contain the following 2 fields, split by space, in this order: word frequency (number of times a word appears in text), TL word.These lists can easily be obtained from a monolingual corpus and a command line in bash:$catmonolingual.SL\|sacremoses-lSLtokenize-x\|awk'{print tolower($0)}'\|tr' ''\n'\|LC_ALL=Csort|uniq-c\|LC_ALL=Csort-nr\ \|grep-v'[[:space:]]*1'\|gzip>wordfreq-SL.gz
$catmonolingual.TL\|sacremoses-lTLtokenize-x\|awk'{print tolower($0)}'\|tr' ''\n'\|LC_ALL=Csort|uniq-c\|LC_ALL=Csort-nr\ \|grep-v'[[:space:]]*1'\|gzip>wordfreq-TL.gzOptionally, if you want the classifier to include a porn filter, you must also provide:File with training dataset for porn removal classifier. Each sentence must contain at the beginning the__label__negativeor__label__positiveaccording to FastText convention. It should be lowercased and tokenized.ParametersIt can be used as follows.bicleaner-ai-train[-h]-mMODEL_DIR-sSOURCE_LANG-tTARGET_LANG[--mono_trainMONO_TRAIN]--parallel_trainPARALLEL_TRAIN--parallel_devPARALLEL_DEV[-SSOURCE_TOKENIZER_COMMAND][-TTARGET_TOKENIZER_COMMAND][-FTARGET_WORD_FREQS][--block_sizeBLOCK_SIZE][-pPROCESSES][-gGPU][--mixed_precision][--save_train_dataSAVE_TRAIN_DATA][--distilled][--seedSEED][--classifier_type{dec_attention,transformer,xlmr}][--batch_sizeBATCH_SIZE][--steps_per_epochSTEPS_PER_EPOCH][--epochsEPOCHS][--patiencePATIENCE][--pos_ratioPOS_RATIO][--rand_ratioRAND_RATIO][--womit_ratioWOMIT_RATIO][--freq_ratioFREQ_RATIO][--fuzzy_ratioFUZZY_RATIO][--neighbour_mixNEIGHBOUR_MIX][--porn_removal_trainPORN_REMOVAL_TRAIN][--porn_removal_testPORN_REMOVAL_TEST][--porn_removal_filePORN_REMOVAL_FILE][--porn_removal_side{sl,tl}][--noisy_examples_file_slNOISY_EXAMPLES_FILE_SL][--noisy_examples_file_tlNOISY_EXAMPLES_FILE_TL][--lm_dev_sizeLM_DEV_SIZE][--lm_file_slLM_FILE_SL][--lm_file_tlLM_FILE_TL][--lm_training_file_slLM_TRAINING_FILE_SL][--lm_training_file_tlLM_TRAINING_FILE_TL][--lm_clean_examples_file_slLM_CLEAN_EXAMPLES_FILE_SL][--lm_clean_examples_file_tlLM_CLEAN_EXAMPLES_FILE_TL][-q][--debug][--logfileLOGFILE]positional arguments:input: Tab-separated bilingual input file (default: Standard input)(line format: SOURCE_SENTENCE TARGET_SENTENCE, tab-separated)optional arguments:-h, --help: show this help message and exitMandatory:-m MODEL_DIR, --model_dir MODEL_DIR: Model directory, metadata, classifier and SentencePiece models will be saved in the same directory (default: None)-s SOURCE_LANG, --source_lang SOURCE_LANG: Source language (default: None)-t TARGET_LANG, --target_lang TARGET_LANG: Target language (default: None)--mono_train MONO_TRAIN: File containing monolingual sentences of both languages shuffled together, used to train SentencePiece embeddings. Not required for XLMR. (default: None)--parallel_train PARALLEL_TRAIN: TSV file containing parallel sentences to train the classifier (default: None)--parallel_dev PARALLEL_DEV: TSV file containing parallel sentences for development (default: None)Options:-S SOURCE_TOKENIZER_COMMAND, --source_tokenizer_command SOURCE_TOKENIZER_COMMAND: Source language tokenizer full command (default: None)-T TARGET_TOKENIZER_COMMAND, --target_tokenizer_command TARGET_TOKENIZER_COMMAND: Target language tokenizer full command (default: None)-F TARGET_WORD_FREQS, --target_word_freqs TARGET_WORD_FREQS: R language gzipped list of word frequencies (needed for frequence based noise) (default: None)--block_size BLOCK_SIZE: Sentence pairs per block when apliying multiprocessing in the noise function (default: 10000)-p PROCESSES, --processes PROCESSES: Number of process to use (default: 71)-g GPU, --gpu GPU: Which GPU use, starting from 0. Will set the CUDA_VISIBLE_DEVICES. (default: None)--mixed_precision: Use mixed precision float16 for training (default: False)--save_train_data SAVE_TRAIN_DATA: Save the generated dataset into a file. If the file already exists the training dataset will be loaded from there. (default: None)--distilled: Enable Knowledge Distillation training. It needs pre-built training set with raw scores from a teacher model. (default: False)--seed: SEED           Seed for random number generation. By default, no seeed is used. (default: None)--classifier_type {dec_attention,transformer,xlmr}: Neural network architecture of the classifier (default: dec_attention)--batch_size BATCH_SIZE: Batch size during classifier training. If None, default architecture value will be used. (default: None)--steps_per_epoch STEPS_PER_EPOCH: Number of batch updates per epoch during training. If None, default architecture value will be used or the full dataset size. (default: None)--epochs EPOCHS: Number of epochs for training. If None, default architecture value will be used. (default: None)--patience PATIENCE: Stop training when validation has stopped improving after PATIENCE number of epochs (default: None)--pos_ratio POS_RATIO: Ratio of positive samples used to oversample on validation and test sets (default: 1)--rand_ratio RAND_RATIO: Ratio of negative samples misaligned randomly (default: 3)--womit_ratio WOMIT_RATIO: Ratio of negative samples misaligned by randomly omitting words (default: 3)--freq_ratio FREQ_RATIO: Ratio of negative samples misaligned by replacing words by frequence (needs --target_word_freq) (default: 3)--fuzzy_ratio FUZZY_RATIO: Ratio of negative samples misaligned by fuzzy matching (default: 0)--neighbour_mix NEIGHBOUR_MIX: If use negative samples misaligned by neighbourhood (default: False)--porn_removal_train PORN_REMOVAL_TRAIN: File with training dataset for FastText classifier. Each sentence must contain at the beginning the '__label__negative' or '__label__positive' according to FastText convention. It should be lowercased and tokenized. (default: None)--porn_removal_test PORN_REMOVAL_TEST: Test set to compute precision and accuracy of the porn removal classifier (default: None)--porn_removal_file PORN_REMOVAL_FILE: Porn removal classifier output file (default: porn_removal.bin)--porn_removal_side {sl,tl}: Whether the porn removal should be applied at the source or at the target language. (default: sl)--noisy_examples_file_sl NOISY_EXAMPLES_FILE_SL: File with noisy text in the SL. These are used to estimate the perplexity of noisy text. (default: None)--noisy_examples_file_tl NOISY_EXAMPLES_FILE_TL: File with noisy text in the TL. These are used to estimate the perplexity of noisy text. (default: None)--lm_dev_size LM_DEV_SIZE: Number of sentences to be removed from clean text before training LMs. These are used to estimate the perplexity of clean text. (default: 2000)--lm_file_sl LM_FILE_SL: SL language model output file. (default: None)--lm_file_tl LM_FILE_TL: TL language model output file. (default: None)--lm_training_file_sl LM_TRAINING_FILE_SL: SL text from which the SL LM is trained. If this parameter is not specified, SL LM is trained from the SL side of the input file, after removing --lm_dev_size sentences. (default: None)--lm_training_file_tl LM_TRAINING_FILE_TL: TL text from which the TL LM is trained. If this parameter is not specified, TL LM is trained from the TL side of the input file, after removing --lm_dev_size sentences. (default: None)--lm_clean_examples_file_sl LM_CLEAN_EXAMPLES_FILE_SL: File with clean text in the SL. Used to estimate the perplexity of clean text. This option must be used together with --lm_training_file_sl and both files must not have common sentences. This option replaces --lm_dev_size. (default: None)--lm_clean_examples_file_tl LM_CLEAN_EXAMPLES_FILE_TL: File with clean text in the TL. Used to estimate the perplexity of clean text. This option must be used together with --lm_training_file_tl and both files must not have common sentences. This option replaces --lm_dev_size. (default: None)Logging:-q, --quiet: Silent logging mode (default: False)--debug: Debug logging mode (default: False)--logfile LOGFILE: Store log to a file (default: <_io.TextIOWrapper name='' mode='w' encoding='UTF-8'>)Examplebicleaner-ai-train\--parallel_traincorpus.en-cs.train\--parallel_devcorpus.en-cs.dev\--mono_trainmono.en-cs\-mmodels/en-cs\-sen\-tcs\-Fwordfreqs-cs.gz\--lm_file_slmodels/en-cs/lm.en--lm_file_tlmodels/en-cs/lm.cs\--porn_removal_trainporn-removal.txt.en--porn_removal_filemodels/en-cs/porn-model.en\This will train a lite classifier for English-Czech using the corpuscorpus.en-cs.train, thecorpus.en-cs.devas development set and the monolingual corpusmono.en-csto train the vocabulary embeddings.
All the model files created during training, the language model files, the porn removal file, and themetadata.yamlwill be stored in the model directorymodels/en-cs.To train full models you would need to use--classifier_type xlmrand--mono_trainis not needed.Synthetic noiseBy default the training will userand_ratio,womit_ratioandfreq_ratiooptions with a value of 3.
Bothwomit_ratioandfreq_ratiowill use Sacremoses tokenizer by default.
So, for languages that are not supported by this tokenizer or are poorly supported,source_tokenizer_commandand/ortarget_tokenizer_commandshould be provided.
Also note that, if a tokenizer command is used, the word frequencies need to be tokenized in the same way to allow noise based on frequency work correctly.If no tokenization is available for your languages, you can disable these noise option that use tokenization and use fuzzy mathing noise:--womit_ratio 0 --freq_ratio 0 --fuzzy_ratio 6.Setting the number of threadsTo set the maximum number of threads/processes to be used during training or classifying,--processesoption is no longer available.
You will need to setBICLEANER_AI_THREADSenvironment variable to the desired value.
For example:BICLEANER_AI_THREADS=12 bicleaner-ai-classify ...If the variable is not set, the program will use all the available CPU cores.SpeedA comparison of the speed in number of sentences per second between different types of models and hardware:modelspeed CPUx1speed GPUx1full1.78 rows/sec200 rows/seclite600 rows/sec10,000 rows/secCPU: Intel Core i9-9960X single core (lite model batch 16, full model batch 1)GPU: Nvidia V100 (lite model batch 2048, full model batch 16)All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.