Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

1 INTRODUCTION . Representation learning with deep recurrent neural networks has revolutionized natural language processing and replaced many of the expert-designed , linguistic features used previously . Recently , researchers have begun to investigate the properties of learned representations by training auxiliary classifiers that use the hidden states of frozen , pretrained models to perform other tasks . These investigations have shown that when deep LSTMs ( Hochreiter & Schmidhuber , 1997 ) are trained on tasks like translation , they learn substantial syntactic and semantic information about their input sentences , including part-of-speech ( Shi et al. , 2016 ; Belinkov et al. , 2017a ; b ; Blevins et al. , 2018 ) . These intriguing findings lead us to ask the following questions : 1 . How does the training task affect how well models learn syntactic properties ? Which tasks are better at inducing these properties ? 2 . How does the amount of data the model is trained on affect these results ? When does training on more data help ? We investigate these questions by holding the data source and model architecture constant , while varying both the training task and the amount of training data . Specifically , we examine models trained on English-German ( En-De ) translation , language modeling , skip-thought ( Kiros et al. , 2015 ) , and autoencoding , and also compare to an untrained LSTM model as a baseline . We control for the data domain by exclusively training on datasets from the 2016 Conference on Machine Translation ( WMT ; Bojar et al. , 2016 ) . We train models on all tasks using the parallel En-De corpus , which allows us to make fair comparisons across tasks . We also train models on a subset of the this corpus to examine the effect of training data volume on learned representations . Additionally , we augment the parallel dataset with a large monolingual corpus from WMT to examine how the performance of the unsupervised tasks ( all but translation ) scale with more data . Throughout our work , we focus on the syntactic evaluation tasks of part-of-speech ( POS ) tagging and Combinatorial Categorical Grammar ( CCG ) supertagging . Supertagging is considered a build- ing block for parsing as these tags constrain the ways in which words can compose , largely determining the parse of the sentence . CCG supertagging thus allows us to measure the degree to which models learn syntactic structure above the word . We focus our analysis on representations learned by language models and by the encoders of sequence-to-sequence models , as translation encoders have been found to learn richer representations of POS and morphological information than translation decoders ( Belinkov et al. , 2017a ) . We find that for POS and CCG tagging , bidirectional language models ( BiLMs ) —created by separately training forward and backward language models , and concatenating their hidden states— outperform models trained on all other tasks . Even BiLMs trained on relatively small amounts of data ( 1 million sentences ) outperform translation and skip-thought models trained on larger datasets ( 5 million and 63 million sentences respectively ) . Our inclusion of an untrained LSTM baseline allows us to study the effect of training on hidden state representations of LSTMs . We find , surprisingly , that when we use all of the available labeled tag data to train our auxiliary task classifiers , our best trained models ( BiLMs ) only outperform the randomly initialized , untrained LSTMs by a few percentage points . When we reduce the amount of classifier training data though , the performance of the randomly initialized LSTM model drops far below those of trained models . We hypothesize that this occurs because training the classifiers on large amounts of auxiliary task data allows them to memorize configurations of words seen in the training set and their associated tags . We test this hypothesis by training classifiers to predict the identity of neighboring words from a given hidden state , and find that randomly initialized models outperform all trained models on this task . Our findings demonstrate that our best trained models do well on the tagging tasks because they are truly learning representations that conform to our notions of POS and CCG tagging , and not simply because the classifiers we train are able to recover neighboring word identity information . 2 RELATED WORK . Evaluating Learned Representations Adi et al . ( 2016 ) introduce the idea of examining sentence vector representations by training auxiliary classifiers to take sentence encodings and predict attributes like word order . Belinkov et al . ( 2017a ) build on this work by examining the hidden states of LSTMs trained on translation and find that they learn substantial POS and morphological information without direct supervision for these linguistic properties . Beyond translation , Blevins et al . ( 2018 ) find that deep LSTMs learn hierarchical syntax when trained on a variety of tasks—including semantic role labeling , language modeling , and dependency parsing . However , the models examined by Blevins et al . ( 2018 ) were also trained on different datasets , so it ’ s unclear if the differences in syntactic task performance are due to the training objectives or simply differences in the training data . By controlling for model size and the quantity and genre of the training data , we we are able to make direct comparisons between tasks on their ability to induce syntactic information . Transfer Learning of Representations Much of the work on sentence-level pretraining has focused on sentence-to-vector models and evaluating learned representations on how well they can be used to perform sentence-level classification tasks . A prominent early success in this area with unlabeled data is skip-thought ( Kiros et al. , 2015 ) , the technique of training a sequence-to-sequence model to predict the sentence preceding and following each sentence in a running text . InferSent ( Conneau et al. , 2017 ) —the technique of pretraining encoders on natural language inference data— yields strikingly better performance when such labeled data is available . Work in transfer learning of representations has recently moved beyond strict sentence-to-vector mappings . Newer models that incorporate LSTMs or Transformer networks pretrained on datarich tasks , like translation and language modeling , have achieved state-of-the-art results on many tasks—including semantic role labeling , natural language inference , and coreference resolution ( Peters et al. , 2018 ; McCann et al. , 2017 ; Howard & Ruder , 2018 ; Radford et al. , 2018 ) . Although comparisons have previously been made between translation and language modeling as pretraining tasks ( Peters et al. , 2018 ; Wang et al. , 2018 ) , we investigate this issue more thoroughly by controlling for the quantity and content of the training data . Training Dataset Size The performance of neural models depends immensely on the amount of training data used . Koehn & Knowles ( 2017 ) find that when training machine translation models on corpora with fewer than 15 million words ( English side ) , statistical machine translation approaches outperform neural ones . Similarly , Hestness et al . ( 2017 ) study the affect of training data volume on performance for several tasks—including translation and image classification . They find that for small amounts of data , neural models perform about as well as chance . After a certain threshold , model performance improves logarithmically with the amount of training data , but this eventually plateaus . With this in mind , we also vary the amount of training data to investigate the relationship between performance and data volume for each task . Randomly Initialized Models Conneau et al . ( 2018 ) use randomly initialized LSTMs as a baseline when studying sentence-to-vector embedding models . They find that untrained models outperform many trained models on several auxiliary tasks , including predicting word content . Similarly in vision , untrained convolutional networks have been shown to capture many low-level image statistics and can be used for image denoising ( Ulyanov et al. , 2017 ) . Our method of training auxiliary classifiers on randomly initialized RNNs builds on the tradition of reservoir computing , in which randomly initialized networks or “ reservoirs ” are fixed and only “ read-out ” classifier networks are trained ( Lukoševičius & Jaeger , 2009 ) . Echo state networks—reservoir computing with recurrent models—have been used for tasks like speech recognition , language modeling , and time series prediction ( Verstraeten et al. , 2006 ; Tong et al. , 2007 ; Sun et al. , 2017 ) . 3 METHODS . 3.1 MAIN TRAINING DATA . We use the parallel English-German ( En-De ) dataset from the 2016 ACL Conference on Machine Translation ( WMT ) shared task on news translation ( Bojar et al. , 2016 ) . This dataset contains 5 million ordered sentence translation pairs . We also use the 2015 English monolingual news discussion dataset from the same WMT shared task , which contains approximately 58 million ordered sentences . To examine how the volume of training data affects learned representations , we use four corpus sizes : 1 , 5 , 15 , and 63 million sentences ( translation is only trained on the smaller two sizes ) . We create the 1 million sentence corpora from the 5 million sentence dataset by sampling ( i ) sentence pairs for translation , ( ii ) English sentences for autoencoders , and ( iii ) ordered English sentence pairs for skip-thought and language models1 . Similarly , we create the 15 million sentence corpora for the unsupervised tasks by sampling sentences from the entire corpus of 63 million sentences . We use word-level representations throughout and use the Moses package ( Koehn et al. , 2007 ) to tokenize and truecase our data . Finally , we limit both the English and German vocabularies to the 50k most frequent tokens in the training set . 1Note , in training we initialize the language model LSTM hidden states with the final state after reading the previous sentence . 3.2 MODEL ARCHITECTURE AND TRAINING . We train all our models using OpenNMT-py ( Klein et al. , 2017 ) and use the default options for model sizes , hyperparameters , and training procedure—except we increase the size of the LSTMs , make the encoders bidirectional , and use validation-based learning rate decay instead of a fixed schedule . Specifically , all our models ( except language models ) are 1000D , two-layer encoder-decoder LSTMs with bidirectional encoders ( 500D per direction ) and 500D embeddings . We train models both with and without attention ( Bahdanau et al. , 2015 ) . For language models , we train a 1000D forward language model and a bidirectional language model—two 500D language models ( forward and backward ) trained separately , whose hidden states are concatenated . All models , including our untrained baseline , are initialized from a uniform distribution ( −0.1 , 0.1 ) , the default in OpenNMT . We use the same training procedure for all our models . We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets , and evaluate approximately every 5 million sentences when training on the larger datasets . We use SGD with an initial learning rate of 1 . Whenever a model ’ s validation loss increases relative to the previous evaluation , we halve the learning rate and stop training when the learning rate reaches 0.515 . For each training task and dataset size , we select the model with the lowest validation perplexity to perform auxiliary task evaluations on . We report model performance in terms of perplexity and BLEU ( Papineni et al. , 2002 ) in Table 1 . For translation we use beam search ( B = 5 ) when decoding .

Summary: I have mixed feelings about this paper. On one hand, it’s a thorough and well-written experimental paper, something which is really important but is also clearly underappreciated in the machine learning community. On the other, it was not really obvious to me why some of objectives tested here are interesting: LM objectives like ELMo have seen a lot of uptake in the NLP community (and this is definitely an NLP paper), but most of the others—like skip-thought, MT, and autoencoders—have not. So the basic research question doesn’t seem like an especially burning one. The trends in Fig. 2 show that these alternatives underperform an LM objective, which suggests that the NLP community can keep using that objective without worry—and everything else in the figure seems as we would expect.