Package: yoyodyne
Yoyodyne 🪀Yoyodyne provides neural models for small-vocabulary sequence-to-sequence
generation with and without feature conditioning.These models are implemented usingPyTorchandLightning.While we provide classic LSTM and transformer models, some of the provided
models are particularly well-suited for problems where the source-target
alignments are roughly monotonic (e.g.,transducer) and/or where source and
target vocabularies have substantial overlap (e.g.,pointer_generator_lstm).PhilosophyYoyodyne is inspired byFairSeq(Ott et al. 2019) but differs on several key points of design:It is for small-vocabulary sequence-to-sequence generation, and therefore
includes no affordances for machine translation or language modeling.
Because of this:It has no plugin interface and the architectures provided are intended
to be reasonably exhaustive.There is little need for data preprocessing; it works with TSV files.It has support for using features to condition decoding, with
architecture-specific code to handle feature information.It uses validation accuracy (not loss) for model selection and early
stopping.Releases are made regularly.🚧 UNDER CONSTRUCTION 🚧: It has exhaustive test suites.🚧 UNDER CONSTRUCTION 🚧: It has performance benchmarks.InstallFirst install dependencies:pip install -r requirements.txtThen install:pip install .It can then be imported like a regular Python module:importyoyodyneUsageTrainingTraining is performed by theyoyodyne-trainscript. One
must specify the following required arguments:--model_dir: path for model metadata and checkpoints--experiment: name of experiment (pick something unique)--train: path to TSV file containing training data--val: path to TSV file containing validation dataThe user can also specify various optional training and architectural arguments.
See below or runyoyodyne-train --helpfor more
information.PredictionPrediction is performed by theyoyodyne-predictscript.
One must specify the following required arguments:--model_dir: path for model metadata--experiment: name of experiment--checkpoint: path to checkpoint--predict: path to file containing data to be predicted--output: path for predictionsThe--predictfile can either be a TSV file or an ordinary TXT file with one
source string per line; in the latter case, specify--target_col 0. Runyoyodyne-predict --helpfor more information.Data formatThe default data format is a two-column TSV file in which the first column is
the source string and the second the target string.source   targetTo enable the use of a feature column, one specifies a (non-zero) argument to--features_col. For instance in theSIGMORPHON 2017 shared
task, the first column is the
source (a lemma), the second is the target (the inflection), and the third
contains semi-colon delimited feature strings:source   target    feat1;feat2;...this format is specified by--features_col 3.Alternatively, for theSIGMORPHON 2016 shared
taskdata:source   feat1,feat2,...    targetthis format is specified by--features_col 2 --features_sep , --target_col 3.In order to ensure that targets are ignored during prediction, one can specify--target_col 0.Reserved symbolsYoyodyne reserves symbols of the form<...>for internal use.
Feature-conditioned models also use[...]to avoid clashes between feature
symbols and source and target symbols. Therefore, users should not provide any
symbols of the form<...>or[...].Model checkpointingCheckpointing is handled byLightning.
The path for model information, including checkpoints, is specified by a
combination of--model_dirand--experiment, such that we build the pathmodel_dir/experiment/version_n, where each run of an experiment with the samemodel_dirandexperimentis namespaced with a new version number. A version
stores everything needed to reload the model, including the hyperparameters
(model_dir/experiment_name/version_n/hparams.yaml) and the checkpoints
directory (model_dir/experiment_name/version_n/checkpoints).By default, each run initializes a new model from scratch, unless the--train_fromargument is specified. To continue training from a specific
checkpoint, thefull path to the checkpointshould be specified with for the--train_fromargument. This creates a new version, but starts training from
the provided model checkpoint.During training, we save the best--save_top_kcheckpoints (by default, 1)
ranked according to accuracy on the--valset. For example,--save_top_k 5will save the top 5 most accurate models.ModelsThe user specifies the overall architecture for the model using the--archflag. The value of this flag specifies the decoder's architecture and whether or
not an attention mechanism is present. This flag also specifies a default
architecture for the encoder(s), but it is possible to override this with
additional flags. Supported values for--archare:attentive_lstm: This is an LSTM decoder with LSTM encoders (by default)
and an attention mechanism. The initial hidden state is treated as a learned
parameter.lstm: This is an LSTM decoder with LSTM encoders (by default); in lieu of
an attention mechanism, the last non-padding hidden state of the encoder is
concatenated with the decoder hidden state.pointer_generator_lstm: This is an LSTM decoder with LSTM encoders (by
default) and a pointer-generator mechanism. Since this model contains a copy
mechanism, it may be superior to an ordinary attentive LSTM when the source
and target vocabularies overlap significantly. Note that this model requires
that the number of--encoder_layersand--decoder_layersmatch.pointer_generator_transformer: This is a transformer decoder with
transformer encoders (by default) and a pointer-generator mechanism. Likepointer_generator_lstm, it may be superior to an ordinary transformer when
the source and target vocabularies overlap significantly. When using
features, the user may wish to specify the n umber of features attention
heads (with--features_attention_heads; default:1).transducer: This is an LSTM decoder with LSTM encoders (by default) and a
neural transducer mechanism. On model creation, expectation maximization is
used to learn a sequence of edit operations, and imitation learning is used
to train the model to implement the oracle policy, with roll-in controlled
by the--oracle_factorflag (default:1). Since this model assumes
monotonic alignment, it may be superior to attentive models when the
alignment between input and output is roughly monotonic and when input and
output vocabularies overlap significantly.transformer: This is a transformer decoder with transformer encoders (by
default). Sinusodial positional encodings and layer normalization are used.
The user may wish to specify the number of attention heads (with--source_attention_heads; default:4).The user can override the default encoder architectures. One can override the
source encoder using the--source_encoderflag:feature_invariant_transformer: This is a variant of the transformer
encoder used with features; it concatenates source and features and uses a
learned embedding to distinguish between source and features symbols.linear: This is a linear encoder.lstm: This is a LSTM encoder.transformer: This is a transformer encoder.When using features, the user can also specify a non-default features encoder
using the--features_encoderflag (linear,lstm,transformer).For all models, the user may also wish to specify:--decoder_layers(default:1): number of decoder layers--embedding(default:128): embedding size--encoder_layers(default:1): number of encoder layers--hidden_size(default:512): hidden layer sizeBy default, LSTM encoders are bidirectional. One can disable this with the--no_bidirectionalflag.Training optionsA non-exhaustive list includes:Batch size:--batch_size(default:32)--accumulate_grad_batches(default: not enabled)Regularization:--dropout(default:0.2)--label_smoothing(default:0.0)--gradient_clip_val(default: not enabled)Optimizer:--learning_rate(default:0.001)--optimizer(default:"adam")--beta1(default:0.9): $\beta_1$ hyperparameter for the Adam
optimizer (--optimizer adam)--beta2(default:0.99): $\beta_2$ hyperparameter for the Adam
optimizer (--optimizer adam)--scheduler(default: not enabled)Duration:--max_epochs--min_epochs--max_steps--min_steps--max_time--patienceSeeding:--seedWeights & Biases:--log_wandb(default:False): enables Weights & Biases trackingSchedulersBy default, Yoyodyne uses a constant learning rate during training, but best
practice is to gradually decreasing learning rate as the model approaches
convergence using ascheduler. Three (non-null)
schedulers are supported and are selected with--scheduler:lineardecay: linearly decreases the learning rate (multiplying it by--start_factor) for--total_decay_stepssteps, then decreases the
learning rate by--end_factor.reduceonplateau: reduces the learning rate (multiplying it by--reduceonplateau_factor) after--reduceonplateau_patienceepochs with
no improvement (when the loss stops decreasing if--reduceonplateau loss,
or when the validation accuracy stops increasing if--reduceonplateaumode accuracy) until the learning rate is less than or
equal to--min_learning_rate.warmupinvsqrt: linearly increases the learning rate from 0 to--learning_ratefor--warmup_stepssteps, then decreases learning rate
according to an inverse root square schedule.Simulating large batchesAt times one may wish to train with a larger batch size than will fit in "in
core". For example, suppose one wishes to fit with a batch size of 4,096, but
this gives an out of memory exception. Then, with minimal overhead, one could
simulate an effective batch size of 4,096 by using batches (--batch_size) of
1,024, accumulating gradients from 4 batches
(--accumulate_grad_batches)
per update:yoyodyne-train --batch_size 1024 --accumulate_grad_batches 4 ...Automatic tuningyododyne-train --auto_lr_finduses a heuristic (Smith
2017) to propose an
initial learning rate. Batch auto-scaling is not supported.Hyperparameter tuningNo neural model should be deployed without proper hyperparameter tuning.However, the default options give a reasonable initial settings for an attentive
biLSTM. For transformer-based architectures, experiment with multiple encoder
and decoder layers, much larger batches, and the warmup-plus-inverse square root
decay scheduler.Weights & Biases tuningwandb_sweepsshows how to useWeights &
Biasesto run hyperparameter sweeps.AcceleratorsHardware
acceleratorscan be used during training or prediction. In addition to CPU (the default) and
GPU (--accelerator gpu),other
acceleratorsmay also be supported but not all have been tested yet.PrecisionBy default, training uses 32-bit precision. However, the--precisionflag
allows the user to perform training with half precision (16) or with thebfloat16half precision
formatif
supported by the accelerator. This may reduce the size of the model and batches
in memory, allowing one to use larger batches.ExamplesTheexamplesdirectory contains interesting examples, including:wandb_sweepsshows how to useWeights &
Biasesto run hyperparameter sweeps.For developersDevelopers, developers, developers!- Steve BallmerThis section contains instructions for the Yoyodyne maintainers.ReleasingCreate a new branch. E.g., if you want to call this branch "release":git checkout -b releaseSync your fork's branch to the upstream master branch. E.g., if the upstream
remote is called "upstream":git pull upstream masterIncrement the version field inpyproject.toml.Stage your changes:git add pyproject.toml.Commit your changes:git commit -m "your commit message here"Push your changes. E.g., if your branch is called "release":git push origin releaseSubmit a PR for your release and wait for it to be merged intomaster.Tag themasterbranch's last commit. The tag should begin withv; e.g.,
if the new version is 3.1.4, the tag should bev3.1.4. This can be done:on GitHub itself: click the "Releases" or "Create a new release" link on
the right-hand side of the Yoyodyne GitHub page) and follow the
dialogues.from the command-line usinggit tag.Build the new release:python -m buildUpload the result to PyPI:twine upload dist/*ReferencesOtt, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and
Auli, M. 2019.fairseq: a fast, extensible toolkit for sequence
modeling. InProceedings of the 2019
Conference of the North American Chapter of the Association for Computational
Linguistics (Demonstrations), pages 48-53.Smith, L. N. 2017.Cyclical learning rates for training neural
networks. In2017 IEEE
Winter Conference on Applications of Computer Vision, pages 464-472.