Package: nmtpytorch
Clickhereto join to our Slack channel for questions.nmtpytorchallows training of various end-to-end neural architectures including
but not limited to neural machine translation, image captioning and automatic
speech recognition systems. The initial codebase was inTheanoand was
inspired from the famousdl4mt-tutorialcodebase.nmtpytorchis mainly developed by theLanguage and Speech TeamofLe Mans Universitybut
receives valuable contributions from theGrounded Sequence-to-sequence Transduction TeamofFrederick Jelinek Memorial Summer Workshop 2018:Loic Barrault, Ozan Caglayan, Amanda Duarte, Desmond Elliott, Spandana Gella, Nils Holzenberger,
Chirag Lala, Jasmine (Sun Jae) Lee, Jindřich Libovický, Pranava Madhyastha,
Florian Metze, Karl Mulligan, Alissa Ostapenko, Shruti Palaskar, Ramon Sanabria, Lucia Specia and Josiah Wang.If you usenmtpytorch, you may want to cite the followingpaper:@article{nmtpy2017,
  author    = {Ozan Caglayan and
               Mercedes Garc\'{i}a-Mart\'{i}nez and
               Adrien Bardet and
               Walid Aransa and
               Fethi Bougares and
               Lo\"{i}c Barrault},
  title     = {NMTPY: A Flexible Toolkit for Advanced Neural Machine Translation Systems},
  journal   = {Prague Bull. Math. Linguistics},
  volume    = {109},
  pages     = {15--28},
  year      = {2017},
  url       = {https://ufal.mff.cuni.cz/pbml/109/art-caglayan-et-al.pdf},
  doi       = {10.1515/pralin-2017-0035},
  timestamp = {Tue, 12 Sep 2017 10:01:08 +0100}
}Installationnmtpytorchcurrently requirespython>=3.6andtorch>=0.4.1.
We are not planning to support Python 2.x.IMPORTANT:After installingnmtpytorch, youneedto runnmtpy-install-extrato download METEOR related files into your${HOME}/.nmtpyfolder.
This step is only required once.pipYou can installnmtpytorchfromPyPIusingpip(orpip3depending on your
operating system and environment):$ pip install nmtpytorchThis will automatically fetch and install the dependencies as well. For thetorchdependency it will specifically install thetorch 1.0.0package fromPyPIthat
shipsCUDA 9.0within.condaWe provide anenvironment.ymlfile in the repository that you can use to create
a ready-to-use anaconda environment fornmtpytorch:$ conda update --all
$ git clone https://github.com/lium-lst/nmtpytorch.git
$ conda env create -f nmtpytorch/environment.ymlDevelopment ModeFor continuous development and testing, it is sufficient to runpython setup.py developin the root folder of your GIT checkout. From now on, all modifications to the source
tree are directly taken into account without requiring reinstallation.DocumentationWe currently only provide some preliminary documentation in ourwiki.Release NotesThis release supports Pytorch >= 0.4.1 including the recent 1.0 release. The relevantsetup.pyandenvironment.ymlfiles will default to1.0.0installation.v4.0.0 (18/12/2018)Critical:NumpyDatasetnow returns tensors of shapeHxW, N, Cfor 3D/4D convolutional features,1, N, Cfor 2D feature files. Models should be adjusted to adapt to this new shaping.Anorder_fileper split (ord: path/to/txt file with integer per line) can be given from the configurations to change the feature order of numpy tensors to flexibly revert, shuffle, tile, etc. them.Better dimension checking to ensure that everything is OK.AddedLabelDatasetfor single label input/outputs with associatedVocabularyfor integer mapping.Addedhandle_oom=(True|False)argument for[train]section to recover fromGPU out-of-memory (OOM)errors during training. This is disabled by default, you need to enable it from the experiment configuration file. Note that it is still possible to get an OOM during validation perplexity computation. If you hit that, reduce theeval_batch_sizeparameter.Addedde-hyphenpost-processing filter to stitch back the aggressive hyphen splitting of Moses during early-stopping evaluations.Added optional projection layer and layer normalization toTextEncoder.Addedenc_lnorm, sched_samplingoptions toNMTto enable layer normalization for encoder and usescheduled samplingat a given probability.ConditionalDecodercan now be initialized with max-pooled encoder states or the last state as well.You can now experiment with different decoders forNMTby changing thedec_variantoption.Collect all attention weights inself.historydictionary of the decoders.Addedn-bestoutput tonmtpy translatewith the argument-N.Changed the way-Sworks fornmtpy translate. Now you need to give the split name with-sall the time but-Sis used to override the input data sources defined for that split in the configuration file.Removed decoder-initialized multimodal NMTMNMTDecInit. Same functionality exists within theNMTmodel by using the model optiondec_init=feats.New model MultimodalNMT:that supports encoder initialization, decoder initialization, both, concatenation of embeddings with visual features, prepending and appending. This model covers almost all the models fromLIUM-CVC's WMT17 multimodal systemsexcept the multiplicative interaction variants such astrgmul.New model MultimodalASR:encoder-decoder initialized ASR model. See thepaperNew Model AttentiveCaptioning:Similar but not an exact reproduction of show-attend-and-tell, it uses feature files instead of raw images.New model AttentiveMNMTFeaturesFA:LIUM-CVC's WMT18 multimodal systemi.e. filtered attentionNew (experimental) model NLI:A simple LSTM-based NLI baseline forSNLIdataset:directionshould be defined asdirection: pre:Text, hyp:Text -> lb:Labelpre, hypandlbkeys point to plain text files with one sentence per line. A vocabulary should be constructed even for the labels to fit the nmtpy architecture.accshould be added toeval_metricsto compute accuracy.v3.0.0 (05/10/2018)Major release that brings support forPytorch 0.4and drops support for0.3.Training and testing onCPUsare now supported thanks to easier device
semantics of Pytorch 0.4: just give-d cputonmtpyto switch to CPU mode.
NOTE: Training on CPUs is only logical for debugging, otherwise it's very slow.NOTE:device_idis no longer a configuration option. It should be removed
from your old configurations.Multi-GPU is not supported. Always restrict to single GPU usingCUDA_VISIBLE_DEVICESenvironment variable.You can now override the config options used to train a model during
inference: Example:nmtpy translate (...) -x model.att_temp:0.9nmtpy trainnow detects invalid/old[train]options and refuses to
train the model.New sampler:ApproximateBucketBatchSamplerSimilar to the defaultBucketBatchSamplerbut more efficient for sparsely
distributed sequence-lengths as in speech recognition. It bins similar-length
items to buckets. It no longer guarantees that the batches are completely
made of same-length sequences socare has to be taken in the encodersto support packing/padding/masking.TextEncoderalready does this automatically
while speech encoderBiLSTMpdoes not care.EXPERIMENTAL: You can decode an ASR system using the approximate sampler
although the model does not take care of the padded positions (a warning
will be printed at each batch).
The loss is 0.2% WER for a specific dataset that we tried. So although the computations
in the encoder becomes noisy and not totally correct, the model can handle
this noise quite robustly:$ nmtpy translate -s val -o hyp -x model.sampler_type:approximate best_asr.ckptThis type of batching cuts ASR decoding time almost by a factor of 2-3.Other changesVocabularies generated bynmtpy-build-vocabnow contains frequency
information as well. The code is backward-compatible with old vocab files.Batchobjects should now be explicitly moved to the allocated device
using.device()method. Seemainloop.pyandtest_performance()from
theNMTmodel.Training no longer shows the cached GPU allocation fromnvidia-smioutput
as it was in the end a hacky thing to callnvidia-smiperiodically. We
plan to usetorch.cuda.*to get an estimate on memory consumption.NOTE: Multi-process data loading is temporarily disabled as it was
crashing from time to time sonum_workers > 0does not have an effect
in this release.Attentionis separated intoDotAttentionandMLPAttentionand a
convenience functionget_attention()is provided to select between them
during model construction.get_activation_fn()should be used to select between non-linearities
dynamically instead of doinggetattr(nn.functional, activ). The latter
will not work fortanhandsigmoidin the next Pytorch releases.Simplification:ASRmodel is now derived fromNMT.v2.0.0 (26/09/2018)Ability to install throughpip.Advanced layers are now organized into subfolders.New basic layers: Convolution over sequence, MaxMargin.New attention layers: Co-attention, multi-head attention, hierarchical attention.New encoders: Arbitrary sequence-of-vectors encoder, BiLSTMp speech feature encoder.New decoders: Multi-source decoder, switching decoder, vector decoder.New datasets: Kaldi dataset (.ark/.scp reader), Shelve dataset, Numpy sequence dataset.Added learning rate annealing: Seelr_decay*options inconfig.py.Removed subword-nmt and METEOR files from repository. We now depend on
the PIP package for subword-nmt. For METEOR,nmtpy-install-extrashould
be launched after installation.More multi-task and multi-input/outputtranslateandtrainingregimes.New early-stopping metrics: Character and word error rate (cer,wer) and ROUGE (rouge).Curriculum learning option for theBucketBatchSampler, i.e. length-ordered batches.New models:ASR: Listen-attend-and-spell like automatic speech recognitionMultitask*: Experimental multi-tasking & scheduling between many inputs/outputs.v1.4.0 (09/05/2018)Addenvironment.ymlfor easy installation usingconda. You can now
create a ready-to-usecondaenvironment by just callingconda env create -f environment.yml.MakeNumpyDatasetmemory efficient by keepingfloat16arrays as they are
until batch creation time.RenameMulti30kRawDatasettoMulti30kDatasetwhich now supports both
raw image files and pre-extracted visual features file stored as.npy.Add CNN feature extraction script underscripts/.Add doubly stochastic attention toShowAttendAndTelland multimodal NMT.New modelMNMTDecinitto initialize decoder with auxiliary features.New modelAMNMTFeatureswhich is the attentive MMT but with features file
instead of end-to-end feature extraction which was memory hungry.v1.3.2 (02/05/2018)Updates toShowAttendAndTellmodel.v1.3.1 (01/05/2018)Removed oldMulti30kDataset.Sort batches by source sequence length instead of target.FixShowAttendAndTellmodel. It should now work.v1.3 (30/04/2018)AddedMulti30kRawDatasetfor training end-to-end systems from raw images as input.AddedNumpyDatasetto read.npy/.npztensor files as input features.You can now pass-Stonmtpy trainto produce shorter experiment files with not all the hyperparameters in file name.New post-processing filter optionde-spmfor Google SentencePiece (SPM) processed files.sacrebleuis now a dependency as it is now accepted as an early-stopping metric.
It only makes sense to use it with SPM processed files since they are detokenized
once post-processed.Addedsklearnas a dependency for some metrics.Addedmomentumandnesterovparameters to[train]section for SGD.ImageEncoderlayer is improved in many ways. Please see the code for further details.Added unmerged upstreamPRforModuleDict()support.METEORwill now fallback to English if language can not be detected from file suffixes.-fnow produces a separate numpy file for token frequencies when building vocabulary files withnmtpy-build-vocab.Added new commandnmtpy testfor non beam-search inference modes.Removednmtpy resumecommand and addedpretrained_fileoption for[train]to initialize model weights from a checkpoint.Addedfreeze_layersoption for[train]to give comma-separated list of layer name prefixes to freeze.Improved seeding: seed is now printed in order to reproduce the results.Added IPython notebook for attention visualization.LayersNew shallowSimpleGRUDecoderlayer.TextEncoder: Ability to setmaxnormandgradscaleof embeddings and work with or without sorted-length batches.ConditionalDecoder: Make it work with GRU/LSTM, allow settingmaxnorm/gradscalefor embeddings.ConditionalMMDecoder: Same as above.nmtpy translate--avoid-doubleand--avoid-unkremoved for now.Added Google's length penalty normalization switch--lp-alpha.Added ensembling which is enabled automatically if you give more than 1 model checkpoints.New machine learning metric wrappers inutils/ml_metrics.py:Label-ranking average precisionlrapCoverage errorMean reciprocal rankv1.2 (20/02/2018)You can now use$HOMEand$USERin your configuration files.Fixed an overflow error that would cause NMT with more than 255 tokens to fail.METEOR worker process is now correctly killed after validations.Many runs of an experiment are now suffixed with a unique random string instead of incremental integers to avoid race conditions in cluster setups.Replacedutils.nn.get_network_topology()with a newTopologyclassthat will parse thedirectionstring of the model in a more smart way.IfCUDA_VISIBLE_DEVICESis set, theGPUManagerwill always honor it.Dropped creation of temporary/advisory lock files under/tmpfor GPU reservation.Time measurements during training are now structered into batch overhead, training and evaluation timings.DatasetsAddedTextDatasetfor standalone text file reading.AddedOneHotDataset, a variant ofTextDatasetwhere the sequences are not prefixed/suffixed with<bos>and<eos>respectively.Added experimentalMultiParallelDatasetthat merges an arbitrary number of parallel datasets together.nmtpy translate.nodbland.nounksuffixes are now added to output files for--avoid-doubleand--avoid-unkarguments respectively.A model-agnostic enoughbeam_search()is now separated out into its own filenmtpytorch/search.py.max_lendefault is increased to 200.v1.1 (25/01/2018)New experimentalMulti30kDatasetandImageFolderDatasetclassestorchvisiondependency added for CNN supportnmtpy-coco-metricsnow computes one METEOR withoutnorm=TrueMainloop mechanism is completely refactored withbackward-incompatibleconfiguration option changes for[train]section:patience_deltaoption is removedAddedeval_batch_sizeto define batch size for GPU beam-search during trainingeval_freqdefault is now3000which means per3000minibatcheseval_metricsnow defaults toloss. As before, you can provide a list
of metrics likebleu,meteor,lossto compute all of them and early-stop
based on the firstAddedeval_zero (default: False)which tells to evaluate the model
once on dev set right before the training starts. Useful for sanity
checking if you fine-tune a model initialized with pre-trained weightsRemovedsave_best_n: we no longer save the bestNmodels on dev set
w.r.t. early-stopping metricAddedsave_best_metrics (default: True)which will save best models
on dev set w.r.t each metric provided ineval_metrics. This kind of
remedies the removal ofsave_best_ncheckpoint_freqnow to defaults to5000which means per5000minibatches.Addedn_checkpoints (default: 5)to define the number of last
checkpoints that will be kept ifcheckpoint_freq > 0i.e. checkpointing enabledAddedExtendedInterpolationsupport to configuration files:You can now define intermediate variables in.conffiles to avoid
typing same paths again and again. A variable can be referenced
from within itssectionusingtensorboard_dir: ${save_path}/tbnotation
Cross-section references are also possible:${data:root}will be replaced
by the value of therootvariable defined in the[data]section.Added-p/--pretrainedtonmtpy trainto initialize the weights of
the model using another checkpoint.ckpt.Improved input/output handling fornmtpy translate:-saccepts a comma-separated test setsdefinedin the configuration
file of the experiment to translate them at once. Example:-s val,newstest2016,newstest2017The mutually exclusive counterpart of-sis-Swhich receives a
single input file of source sentences.For both cases, an output prefixshould now beprovided with-o.
In the case of multiple test sets, the output prefix will be appended
the name of the test set and the beam size. If you just provide a single file with-Sthe final output name will only reflect the beam size information.Two new arguments fornmtpy-build-vocab:-f: Stores frequency counts as well inside the finaljsonvocabulary-x: Does not add special markers<eos>,<bos>,<unk>,<pad>into the vocabularyLayers/ArchitecturesAddedFusion()layer toconcat,sum,mulan arbitrary number of inputsAddedexperimentalImageEncoder()layer to seamlessly plug a VGG or ResNet
CNN usingtorchvisionpretrained modelsAttentionlayer arguments improved. You can now select the bottleneck
dimensionality for MLP attention withatt_bottleneck. Thedotattention isstill not testedand probably broken.New layers/architectures:AddedAttentiveMNMTwhich implements modality-specific multimodal attention
from the paperMultimodal Attention for Neural Machine TranslationAddedShowAttendAndTellmodelChanges inNMT:dec_initdefaults tomean_ctx, i.e. the decoder will be initialized
with the mean context computed from the source encoderenc_lnormwhich was just a placeholder is now removed since we do not
provided layer-normalization for nowBeam Search is completely moved to GPU