Package: opustrainer
OpusTrainerThe purpose of the trainer is to provide the user with a flexible way of scheduling various sources of input data, as well as augment the training data with tittle casing, all caps, etc. This is particularly useful when you have multiple data sources and you want to pretrain the model first on backtranslated data, gradually add other sources of data, and finally fine tune, all in one go.Alternatively, this tool is particularly suited to training multilingual models, as it provides an easy way to define the desired mixture of datasets from different language sources.InstallationYou've got two options: Install directly from PyPI:pipinstallopustraineror clone this repository, and install it in editable mode so you can change the source, but still use all the commands:gitclonegit@github.com:hplt-project/opustrainer.gitcdopustrainer
pipinstall-e.Usage%./trainer.py--help
usage:trainer.py[-h]--configCONFIG[--temporary-directoryTEMPORARY_DIR][--stateSTATE_FILE][--do-not-resume][--sync][trainer-command[arguments]]Feedsmariantsvdatafortraining.

options:-h,--helpshowthishelpmessageandexit--configCONFIG,-cCONFIGYMLconfigurationinput.--temporary-directoryTEMPORARY_DIR,-tTEMPORARY_DIRTemporarydir,usedforshufflingandtrackingstate--stateSTATE_FILEPathtotrainerstatefilewhichstoreshowmuchofeachdatasethasbeenread.Defaultsto${CONFIG}.state--syncDonotshuffleinthebackground--do-not-resume,-dDonotresumefromtheprevioustrainingstate--no-shuffle,-nDonotshuffle,fordebuggingOnce you fix the paths in the configuration file,train_config.ymlyou can run a test case by doing:./trainer.py-ctrain_config.yml/path/to/marian-cmarian_config--any--other--flagsYou can check resulting mixed file in/tmp/test. If your neural network trainer doesn't support training fromstdin, you can use this tool to generate a training dataset and then disable data reordering or shuffling at your trainer implementation, as your training input should be balanced.At the start of the training all datasets are shuffled. Each time a dataset's end is reached, it is re-shuffled. Shufflingin the system temp directorybut can be repositioned using--temporary-directoryor theTMPDIRenvironment variable. By default, the training state is kept in the same place as the configuration file. If training is interrupted, re-running the trainer should resume from where it was (depending on how much your neural network trainer has buffered, that part will be skipped).Configuration fileDefine your training process via a configuration file. You define the datasets on top, the stages and then for each stage a mixing criteria and a stage termination criteria. An example configuration file is provided below. The path to thetraineris a path to any neural network trainer that supports having stdin as training input format.# Datasets are already TSV files. We support reading gzip'd files, as well as multiple dataset file per name
datasets:
  clean: test/data/clean
  medium: test/data/medium
  dirty: test/data/dirty

stages:
  - start
  - mid
  - end

start:
  - clean 0.8
  - medium 0.2
  - dirty 0
  - until clean 2 # Until two epochs of clean

mid:
  - clean 0.6
  - medium 0.3
  - dirty 0.1
  - until medium 1

end:
  - clean 0.4
  - medium 0.3
  - dirty 0.3
  - until dirty 5 # use `inf` to mean until forever

modifiers:
- UpperCase: 0.05 # Apply uppercase randomly to 5% of sentences. See below
- TitleCase: 0.05

seed: 1111
trainer: /path/to/trainer/run.pyExtended stage configurationIf you want to change which modifiers are used for a specific stage, you can the extended stage configuration format. If amodifiersis mentioned here, it will override the curriculum-wide definedmodifiersfor just this stage.In the extended format, the list of datasets is defined in themixkey. You can optionally add amodifierskey. For example:start:mix:-clean 0.8-medium 0.2-dirty 0-until clean 2# Until two epochs of cleanmodifiers:-UpperCase:0.05-TitleCase:0.05Note that you can use YAML references if you wish to extensively combine global and local modifiers.ModifiersModifiers are randomly applied to the sentences that go into the trainer. Each modifier has a probability associated with it that is the chance that a sentence is modified by the modifier. E.g. a modifier with a probability of 0.05 will affect about 1 in every 20 sentences.Modifiers are applied one after another, in the order you defined them, all with their own probability regardless of the modifiers that got applied before it. E.g. if you have the following configuration:modifiers:-UpperCase:0.05-TitleCase:0.05This means that 1 in 20 sentences will be uppercased, and 1 in 20 will be titlecased. And effectively0.05 * 0.05so 1 in 400 will first be uppercased and then titlecased.UpperCaseTurns the entire source and target sentence to upper case, e.g. 'heLLo' becomes 'HELLO'.modifiers:-UpperCase:0.05TitleCaseMakes the first letter of every word uppercase, and the rest lowercase. Words are split by spaces. E.g. 'heLLo' becomes 'Hello'.modifiers:-TitleCase:0.05TyposIntroduce typos in the source side of the sentence pair.The probability of the modifier itself is the chance a sentence is affected. The probabilities of each of the types of typos describes the chance a word is affected. Each type of typo occurs at most once in a sentence.You can specify a probability for each modifier individually. If any of the typo classes is omitted, it has a probability of 0. Alternatively, you can omit all typo classes. Then all of them will have a default 10% probability.modifiers:-Typos:0.05char_swap:0.1# Swaps two random consecutive word characters in the string.missing_char:0.1# Skips a random word character in the string.extra_char:0.1# Adds an extra, keyboard-neighbor, letter next to a random word character.nearby_char:0.1# Replaces a random word character with keyboard-neighbor letter.similar_char:0.1# Replaces a random word character with another visually similar character.skipped_space:0.1# Skips a random space from the string.random_space:0.1# Adds a random space in the string.repeated_char:0.1# Repeats a random word character.unichar:0.1# Replaces a random consecutive repeated letter with a single letter.TagsAdds a placeholder tag to the source sentence that can be used by the model to hint how it should translate that word. The word to hint is chosen at random from the target sentence. Only words with a 1-to-1 mapping between source and target are considered.This modifier needs a third column in the training data with per-word alignment information.-Tags:0.05custom_detok_src:nullcustom_detok_trg:zhtemplate:"__source__{src}__target__{trg}__done__"All options are optional.You can specify custom detokenizer languages usingcustom_detok_srcandcustom_detok_trgif the dataset you're reading from has been tokenized by the Moses tokenizer. This can be helpful to do for languages that do not use spaces to delimit words. The default tokenisation strategy is splitting/joining by spaces.The format for telling the translation model the intention to translate a word in a certain way can be controlled bytemplate. Here{src}and{trg}are replaced by the selected words from the source and target side of the sentence pair.PrefixPrepends a random subsection of the target sentence before the source sentence.This is useful for teaching the model to force decode a specific string if the user is absolutely certain it has to appear in the output. For exampleI like pie. Me gustan los pasteles.becomes__start__ los pasteles __end__ I like pie. Me gustan los pasteles.Note: The Prefix modifier must always be used as the last modifier, but ideally never used together with "Tags".modifiers:-Prefix:0.5min_words:2max_words:5template:"__start__{trg}__end__"TagsIn order to train models supporting vocabulary "hints" we provide a tags system where we laverage word alignment information to provide "hints" for the translation model to what it should produce on the Target side. The work is very similar toDinu et al. 2019. An example. Given an alignment augmented tsv training line:I like pies! \t Ich mag Kuchen! \t 0-0 1-1 2-2The machine translation traing system would see something like:I like <tag0> mag <tag0> pies! \t Ich mag Kuchen!Where the numeric ID of the tag is drawn from a random distribution from the total number of tags. The probability asigned to theTagsmodifier determines how likely is it for a tag augmentation to appear on any given word.Finally, if your corpus is augmented with alignment info, but you don't want to use any tags, just set the probability of this modifier to 0.Generating vocabulary and tags before trainingIn the future, this will be handled by a training Pipeline, but until then here's the basic scripts usedFor producing alignment augmented corpus use this script:#!/bin/bash -v# Usage: ./align_corpus.sh source_corpus target_corpus src trg# install fast alignmkdir-pbin# download and compile fast_alignif[!-ebin/fast_align];thengitclonehttps://github.com/clab/fast_alignmkdir-pfast_align/buildcdfast_align/buildcmake..make-j4cpfast_alignatools../../bincd../../fi# Prepare the corpus for fast aligntest-s$2/corpus.tmp.${3}-${4}.falign||cat$1|sed's/\t/ ||| /'>$2/corpus.tmp.${3}-${4}.falign# Align ittest-s$2/align.${3}-${4}.s2t||bin/fast_align-vod-i$2/corpus.tmp.${3}-${4}.falign>$2/align.${3}-${4}.s2ttest-s$2/align.${3}-${4}.t2s||bin/fast_align-vodr-i$2/corpus.tmp.${3}-${4}.falign>$2/align.${3}-${4}.t2stest-s$2/corpus.${3}-${4}.aln||bin/atools-i$2/align.${3}-${4}.s2t-j$2/align.${3}-${4}.t2s-cgrow-diag-final-and>$2/corpus.${3}-${4}.alnFor creating vocabulary with tags support, use this script:#!/usr/bin/env bash#Usage ./vocab.sh en de path-to-corpora char-cov vocab_sizechar_cov=${4:-'0.9995'}# Default char coveragevocab_size=${5:-'32000'}# Default vocab size# Set up some constants# Language pairssrc=$1trg=$2prefix="--model_prefix=model.${src}-${trg}"# Placeholders arrayplaceholders="--user_defined_symbols=__source__,__target__,__done__,__start__,__end__"# Character coverage. CJK is recommended to have 0.9995, vocab languages proabbly you want 1.char_cov="--character_coverage=${char_cov}"# First clone and compile SPMspm_exec="sentencepiece/build/src/spm_train"if[!-e${spm_exec}];thengitclonehttps://github.com/google/sentencepiece.gitcdsentencepiecemkdirbuildcdbuildcmake..make-j4cd..cd..if[!-e${spm_exec}];thenecho"Failed to compile sentencepiece"exit1fifi$spm_exec--bos_id=-1--eos_id=0--unk_id=1${placeholders}${char_cov}${prefix}--vocab_size=${vocab_size}--input=${3}--input_sentence_size=20000000--byte_fallback#--input_format=tsv seems brokenFuture workTerminology support (using a dictionary), where augmentation happens not by using alignment scores but by taking values from a dictionary.A one click run trainingAcknowledgementsThis project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]