Transformer for Language Modeling | Towards Data Science

Recent trends in Natural Language Processing have been building upon one of the biggest breakthroughs in the history of the field: the Transformer. The Transformer is a model architecture researched mainly by Google Brain and Google Research. It was initially shown to achieve state-of-the-art in the translation task but was later shown to be effective in just about any NLP task when it became massively adopted. The transformer architecture consists of a stack of encoders and decoders with self-attention layers that help the model pay attention to respective inputs. You can learn more about transformers in the original paper here.
In this post, we will be showing you how to implement the transformer for the language modeling task. Language modeling is the task of assigning probability to sentences in a language. The goal for language modeling is for the model to assign high probability to real sentences in our dataset so that it will be able to generate fluent sentences that are close to human-level through a decoder scheme. We will be using the Fairseq library for implementing the transformer.
towardsdatascience.com
In this article, we will be again using the CMU Book Summary Dataset to train the Transformer model. You can refer to Step 1 of the blog post to acquire and prepare the dataset. After preparing the dataset, you should have the train.txt, valid.txt, and test.txt files ready that correspond to the three partitions of the dataset.
If you haven’t heard of Fairseq, it is a popular NLP library developed by Facebook AI for implementing custom models for translation, summarization, language modeling, and other generation tasks. You can check out my comments on Fairseq here.
Now, in order to download and install Fairseq, run the following commands:
git clone https://github.com/pytorch/fairseqcd fairseqpip install --editable ./
You can also choose to install NVIDIA’s apex library to enable faster training if your GPU allows:
git clone https://github.com/NVIDIA/apexcd apexpip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \  --global-option="--fast_multihead_attn" ./
Now, you have successfully installed Fairseq and finally we are all good to go!
To preprocess the dataset, we can use the fairseq command-line tool, which makes it easy for developers and researchers to directly run operations from the terminal. To preprocess our data, we can use fairseq-preprocess to build our vocabulary and also binarize the training data.
cd fairseq/DATASET=/path/to/datasetfairseq-preprocess \--only-source \--trainpref $DATASET/train.txt \--validpref $DATASET/valid.txt \--testpref $DATASET/test.txt \--destdir data-bin/summary \--workers 20
After executing the above commands, the preprocessed data will be saved in the directory specified by the --destdir .
Finally, we can start training the transformer! To train a model, we can use the fairseq-train command:
CUDA_VISIBLE_DEVICES=0 fairseq-train --task language_modeling \data-bin/summary \--save-dir checkpoints/transformer_summary \--arch transformer_lm --share-decoder-input-output-embed \--dropout 0.1 \--optimizer adam --adam-betas '(0.9, 0.98)' --weight-decay 0.01 --clip-norm 0.0 \--lr 0.0005 --lr-scheduler inverse_sqrt --warmup-updates 4000 --warmup-init-lr 1e-07 \--tokens-per-sample 512 --sample-break-mode none \--max-tokens 2048 --update-freq 16 \--fp16 \--max-update 50000 \--max-epoch 12
In our case, we specify the GPU to use as the 0th (CUDA_VISIBLE_DEVICES), task as language modeling (--task), the data in data-bin/summary , the architecture as a transformer language model (--arch ), the number of epochs to train as 12 (--max-epoch ) , and other hyperparameters. After training, the best checkpoint of the model will be saved in the directory specified by --save-dir .
12 epochs will take a while, so sit back while your model trains! Of course, you can also reduce the number of epochs to train according to your needs. The following output is shown when the training is complete:
Note that in each epoch, the relevant numbers are shown, such as loss and perplexity. These could be helpful for evaluating the model during the training process.
After your model finishes training, you can evaluate the resulting language model using fairseq-eval-lm :
fairseq-eval-lm data-bin/summary \--path checkpoints/transformer_summary/checkpoint_best.pt \--max-sentences 2 \--tokens-per-sample 512 \--context-window 400
Here the test data will be evaluated to score the language model (the train and validation data are used in the training phase to find the optimized hyperparameters for the model). The following shows the command output after evaluation:
As you can see, the loss of our model is 9.8415 and perplexity is 917.48 (in base 2).
After training the model, we can try to generate some samples using our language model. To generate, we can use the fairseq-interactive command to create an interactive session for generation:
fairseq-interactive data-bin/summary \--task language_modeling \--path checkpoints/transformer_summary/checkpoint_best.pt \--beam 5
During the interactive session, the program will prompt you an input text to enter. After the input text is entered, the model will generate tokens after the input. A generation sample given The book takes place as input is this:
The book takes place in the story of the story of the story of the story of the story of the story of the story of the story of the story of the story of the characters...
The generation is repetitive which means the model needs to be trained with better parameters. The above command uses beam search with beam size of 5. We can also use sampling techniques like top-k sampling:
fairseq-interactive data-bin/summary \--task language_modeling \--path checkpoints/transformer_summary/checkpoint_best.pt \--sampling --beam 1 --sampling-topk 10
and top-p sampling:
fairseq-interactive data-bin/summary \--task language_modeling \--path checkpoints/transformer_summary/checkpoint_best.pt \--sampling --beam 1 --sampling-topp 0.8
Note that when using top-k or top-sampling, we have to add the beam=1 to suppress the error that arises when --beam does not equal to--nbest . This seems to be a bug.
In this blog post, we have trained a classic transformer model on book summaries using the popular Fairseq library! Although the generation sample is repetitive, this article serves as a guide to walk you through running a transformer on language modeling. Take a look at my other posts if interested :D
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
[1] A. Vaswani, N. Shazeer, N. Parmar, etc., Attention Is All You Need (2017), 31st Conference on Neural Information Processing Systems
[2] L. Shao, S. Gouws, D. Britz, etc., Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models (2017), Empirical Methods in Natural Language Processing
[3] A. Fan, M. Lewis, Y. Dauphin, Hierarchical Neural Story Generation (2018), Association of Computational Linguistics
[4] A. Holtzman, J. Buys, L. Du, etc., The Curious Case of Neural Text Degeneration (2019), International Conference on Learning Representations
[5] Fairseq Github, Facebook AI Research
[6] Fairseq Documentation, Facebook AI Research