Language Model Pre-training for Hierarchical Document Representations

Hierarchical neural architectures can efficiently capture long-distance dependencies and have been used for many document-level tasks such as summarization , document segmentation , and sentiment analysis . However , effective usage of such a large context can be difficult to learn , especially in the case where there is limited labeled data available . Building on the recent success of language model pretraining methods for learning flat representations of text , we propose algorithms for pre-training hierarchical document representations from unlabeled data . Unlike prior work , which has focused on pre-training contextual token representations or context-independent sentence/paragraph representations , our hierarchical document representations include fixed-length sentence/paragraph representations which integrate contextual information from the entire documents . Experiments on document segmentation , document-level question answering , and extractive document summarization demonstrate the effectiveness of the proposed pre-training algorithms . 1 INTRODUCTION . While many natural language processing ( NLP ) tasks have been posed as isolated prediction problems with limited context ( often a single sentence or paragraph ) , this does not reflect how humans understand natural language . When reading text , humans are sensitive to much more context , such as the rest of the document or even other relevant documents . In this paper , we focus on tasks that require document-level understanding . We build upon two existing separate lines of work that are useful for these tasks : ( 1 ) document-level models with hierarchical architectures , which include sentence representations contextualized with respect to entire documents ( Ruder et al. , 2016 ; Cheng & Lapata , 2016 ; Koshorek et al. , 2018 ; Yang et al. , 2016 ) , and ( 2 ) contextual representation learning with language-model pretraining ( Peters et al. , 2018 ; Radford et al. , 2018 ) . In this work , we show for the first time that these two ideas can be successfully combined . We find that pretraining hierarchical document representations raises several unique challenges compared to existing representation learning . First , it is not clear how to pretrain the higher level representations . Using existing techniques , it is possible to initialize the lower level of the hierarchical representation , to represent words in the context of individual sentences e.g . using ELMo ( Peters et al. , 2018 ) , or , in addition , initialize sentence vectors that are not dependent on the context from the full document using Skip-Thought vectors ( Kiros et al. , 2015 ) , but it is not possible to initialize the full hierarchical neural structure . Second , bidirectional representations are standard in state-of-the-art NLP tasks ( Wu et al. , 2016 ; Weissenborn et al. , 2017 ) , and may be particularly important for document-level representations due to the long-distance dependencies . However , current pretraining algorithms often use left-to-right and right-to-left language models to pretrain the representations . This restricts the expressiveness of the hierarchical representations , as left and right contextual information do not fuse together when forming higher-level hierarchical representations . In this paper , we address these challenges by proposing two novel approaches to pre-train documentlevel hierarchical representations . Both methods pre-train representations from unlabeled documents containing thousands of tokens . Our first approach generalizes ( Peters et al. , 2018 ) to pre-train hierarchical left-to-right and right-to-left document representations . To allow the hierarchical representations to learn to fuse left and right contextual information from abundant unlabeled text , our second approach is a new pretraining algorithm called masked language model , which allows efficient training of bidirectional hierarchical document representations . We evaluate the impact of the novel aspects of our pre-trained representations on three tasks : document segmentation , answer passage retrieval for document-level question answering , and extractive text summarization . We first pretrain document-level representations on unlabeled documents and then fine-tune with task-specific labeled data and a light-weight task network . Experiments show that pre-training hierarchical document representations brings significant benefits on all tasks , and that pretraining the higher level of the document representation results in larger improvements than pre-training only the locally contextualized lower word level in most tasks . On the CNN/Daily Mail summarization task , we obtain strong results without using reinforcement learning methods employed by the previous state-of-the-art models . On the TriviaQA answer passage retrieval task , our model improves over a strong baseline ( Clark & Gardner , 2018 ) by 6 % absolute . 2 BACKGROUND . In this section , we review pretraining of contextual token representations using language models and discuss how the language model probability decomposition imposes constraints on the architectures . Language Model Pretraining Given a sequence of tokens ( x1 , . . . , xn ) , a representation module Vθ encodes each word into a vector : Vθ ( x1 , . . . , xn ) = ( v1 , . . . , vn ) , where each word-level contextual representation vi could potentially depend on the whole sequence . Common choices for the representation modules are LSTMs , CNNs , and self-attention architectures ( Hochreiter & Schmidhuber , 1997 ; Vaswani et al. , 2017 ; LeCun & Bengio , 1998 ) . A representation module is parameterized by an underlying neural architecture with parameters θ . The typical left-to-right language model aims to maximize the probability of observed sequences : n∏ t=1 P ( xt|x1 , . . . xt−1 ; θ ) , ( 1 ) The pretrained parameters θ can then be used to initialize the model for a downstream task . Uni-directionality constraint The formulation of the language model implicitly imposes a directionality constraint on the Vθ , which we term uni-directionality constraint . In Equation 1 , the word xt can only depend on the ( representations of ) the previous words . Therefore , the contextual representations v1 , . . . , vt−1 can not depend on the representations of xt , . . . , xn , resulting in the uni-directionality constraint on the representations . One common type of representation modules that satisfy the uni-directionality constraint are unidirectional recurrent neural networks ( RNNs ) . Given a sequence of words , a left-to-right LSTM model generates a sequence of representations Vθ ( x1 , . . . , xn ) = ( −→ h 1 , . . . , −→ h n ) that satisfies the uni-directionality constraints , where −→ h t only depends on −→ h 1 , −→ h 2 , . . . −→ h t−1 . Modules that satisfy the uni-directionality constraint can also be derived from self-attention , convolution , and feedforward architectures ( e.g . ( Radford et al. , 2018 ) ) . As mentioned in the introduction , for many downstream tasks , bidirectional representations bring large improvements over uni-directional ones . Therefore , Peters et al . ( 2018 ) , pre-trained two encoders using two language models : a left-to-right and a right-to-left model , using a left-to-right and a right-to-left LSTM encoder , respectively . The token representations are formed by the concatenation of outputs from the two uni-directional encoders . We term this style of language model pre-training L+R-LM . More specifically , the representation of the t-th word becomes vt = [ −→ h t ←− h t ] , where ←− h t is generated from a right-to-left LSTM . Tasks using these input representations must learn to merge the left and right contextual information from scratch using only the labeled data . The general strategy we adopt in this paper is to pretrain the hierarchical representations on unlabeled data and then to fine-tune the representations with the task-specific labeled data . Therefore , the uni-directionality constraint will also impacts the task-specific networks . 3 HIERARCHICAL DOCUMENT-LEVEL REPRESENTATIONS . As our goal is to learn fixed-length contextualized representations of text segments to be used in document-level tasks , we choose hierarchical neural architectures that build representations of tokens and text segments in context . Here we describe the specific architectures studied in this paper . The novel contribution of our work – pre-training such representations from unlabeled text , will be detailed in Section 4 . We choose LSTM-based architectures with a standard hierarchical structure that has been useful for capturing long-term context in document-level tasks ( Serban et al. , 2016 ; Koshorek et al. , 2018 ) . We experiment with two related document-level representations . The first one , called BI-HLSTM , is a standard hierarchical bidirectional LSTM encoder . BI-HLSTM does not satisfy the uni-directionality constraint and can not be pre-trained using a left-to-right language model likelihood objective function . Our second document-level representation is L+R-HLSTM , which consists of two concatenated uni-directional versions of BI-HLSTM , and can be seen as a hierarchical document-level extension of ELMo ( Peters et al. , 2018 ) . Both document-level representations view documents as sequences of text blocks , each consisting of a sequence of tokens . The text blocks can be sentences , paragraphs , or sections , depending on the task . Here we experiment with sentences and paragraphs as text blocks . The BI-HLSTM architecture fuses left and right contextual information more effectively and directly corresponds to state-of-the-art architectures used for multiple NLP tasks . BI-HLSTM BI-HLSTM has a two-level hierarchy . Every block is first independently processed by a bidirectional LSTM ( Bi-LSTM ) to encode the local context . The contextual representations of the words in each block are then pooled and projected to a single vector to represent the block . The vector representations of the sequence of blocks in the document are then processed by a global BiLSTM , resulting in block representations contextualized by the entire document . The overall design is shown in Figure 1a , and the encoding procedure is formalized next . A document D consists of K blocks D = { U1 , U2 , . . . UK } , and each block consists of a sequence of tokens where Ui = { xi1 , xi2 , . . . xiT } .1 For a block Ui , we start with character representations of the words , where x̄it = CNNchar ( x i t ) . Then character-based word representations are processed by the local Bi-LSTM , where { hi1 , . . . , hiT } = BI-LSTMLocal ( { x̄i1 , . . . , x̄iT } ) . 1In our experiments , the value of T depends on i . For simplicity , we slightly abuse notation and use the same T for all blocks . Before generating the globally-contextualized representations , the encoder generates representations of all blocks by pooling the locally-contextualized word representations generated from the local encoder . The block representations are obtained using both max and average-pooling on the word representations , followed by a feed-forward transformation : ci = FFNN ( POOLmax ( { hi1 , . . . , hiT } ) ; POOLavg ( { hi1 , . . . , hiT } ) ) . The resulting local block representations ci are then processed by the global Bi-LSTM to generate document-contextualized block representations pi : { p1 , . . . , pK } = BI-LSTMGlobal ( { c1 , . . . , cK } ) . In downstream tasks , this document representation is used by inputting the text block contextual vector representations { pi } to light-weight task-specific networks . L+R-HLSTM As previously mentioned , when using left-to-right language models to pre-train representations , the representation of the i-th word can only depend on the representations of the previous words ( i−1 ) words . Similarly , the representation of the j-th text block can only depend on the representations of the previous text blocks . To design a document-level representation similar to the one defined by the BI-HLSTM but that can still be trained using a uni-directional language model decomposition , we replace all of the bi-directional LSTMs in BI-HLSTM with either left-to-right or right-to-left LSTMs . The left-to-right version of the resulting hierarchical encoder is presented in Figure 1b . Given a document D = { U1 , U2 , . . . UK } , let the encoding results of the left-to-right HLSTM be denoted as { −→p 1 , . . . , −→p K } = L-TO-R-HLSTM ( D ; θl ) . We use another right-to-left hierarchical encoder to produce representations : { ←−p 1 , . . . , ←−p K } = R-TO-L-HLSTM ( D ; θr ) . We then concatenate the final representations of the two encoders : p̂i = [ −→p i ←−p i ] . We term the resulting document-level encoder L+R-HLSTM . Note that the L+R-HLSTM combines the left and right information only at the top level , and there is no interaction between the two unidirectional contextualizers . While at the bottom-most layer , the BI-HLSTM similarly concatenates left and right information , the second local BI-LSTM layer can fuse the two directions . Similarly , the left and right contexts at the text segment level are fused at the second global BI-LSTM layer . In principle , both BI-HLSTM and L+R-HLSTM can capture long distance dependencies via their hierarchical structure , but it is unclear whether such dependencies can be effectively learned from limited amounts of supervised data labeled for down-stream tasks of interest . In the next section , we discuss how such document encoders can be trained to provide rich text segment and token representations using large unlabeled document collections .

Summary: In this work, the authors explore different ways to pre-train contextualized word and sentence representations for use in other tasks. They propose two main methods: a straight-forward extension of the ElMO model for hierarchical uni-directional language models, and a de-noising auto-encoder type method which allows to train bi-directional representations. The learned contextual representations are evaluated on three downstream tasks, demonstrating the superiority of the bi-directional training setting, and beating strong baselines on extractive summarization.