A Mutual Information Maximization Perspective of Language Representation Learning
Authors: Lingpeng Kong
Cyprien de Masson d'Autume
Lei Yu
Wang Ling
Zihang Dai
Dani Yogatama
Venue:  (2020)

Abstract: We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence). Our formulation provides an alternative perspective that unifies classical word embedding models (e.g., Skip-gram) and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide an example by drawing inspirations from related methods based on mutual information maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between a global sentence representation and n-grams in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and translate progress across multiple domains (e.g., natural language processing, computer vision, audio processing).

Published as a conference paper at ICLR 2020
A MUTUAL INFORMATION MAXIMIZATION PERSPEC- TIVE OF LANGUAGE REPRESENTATION LEARNING
Lingpeng Kong*, Cyprien de Masson d' d'Autume* Wang Ling Lei Yu Zihang Dai
Dani Yogatama*
DeepMind Carnegie Mellon University Google Brain
London, United Kingdom
(lingpenk, cyprien, lingwang, leiyu, zihangd, dyogat ama @google com
ABSTRACT
We show state-of-the-art word representation learning methods maximize an ob- jective function that a lower bound on the mutual information between different parts of word sequence (i.e., sentence). Our formulation provides alternative perspective that unifies classical word embedding models (e.g., Skip-gram and modern contextual embeddings (e.g., BERT, XLNet). In addition to enhancing our theoretical understanding of these methods, our derivation leads to a principled framework that can be used to construct new self-supervised tasks. We provide example by drawing inspirations from related methods based on mutual informa- tion maximization that have been successful in computer vision, and introduce a simple self-supervised objective that maximizes the mutual information between global sentence representation and n-gramm in the sentence. Our analysis offers a holistic view of representation learning methods to transfer knowledge and trans- late progress across multiple domains (e.g., natural language processing, computer vision, audio processing).
INTRODUCTION
Advances representation learning have driven progress natural language processing. Performance on many downstream tasks have improved considerably, achieving parity with human baselines in benchmark leaderboards such as SQuAD (Rajpurkar et al. 2016 2018) and GLUE (Wang et al.) 2019). The main ingredient the "pretrain and fine-tune" approach, where a large text encoder trained on an unlabeled corpus with superviee training objectives and used to initialize task-specific model. Such an approach has also been shown to reduce the number of training examples that is needed to achieve good performance on the task of interest (Yogatama et al. 2019).
In contrast to first-generation models that learn word type embeddings (Mikolov et al. 2013 Pen nington et al. 2014), recent methods have focused contextual token representations--ic.. learning an encoder to represent words in context. Many of these encoders are trained with language modeling objective, where the representation of a context trained to be predictive of a target token by maximizing the log likelihood of predicting this token (Dai Le 2015| Howard & Ruder 2018 Radford al. 2018 2019p. In vanilla language modeling objective, the target token is always the next token that follows the context. Peters et al. (2018) propose an improvement by adding reverse objective that also predicts the word token that precedes the context. Following this trend, current state-of-the-ar encoders such as BERT (Devlin et 2018) and XLNet (Yang et al. (019) are also trained with variants of the language modeling objective: masked language modeling and permutation language modeling.
In this paper, we provide an alternative view and show that these methods also maximize lower bound on the mutual information between different parts of word sequence. Such framework is inspired by the InfoMax principle (Linsker) 1988) and has been the main driver of progress in self-supervised representation learning other domains such as computer vision, audio processing.
and reinforcement learning (Belghazi et al. 2018 van den Oord et al. 2019 Hjelm et al. 2019

---

Published as a conference paper at ICLR 2020
Bachman al. 2019 O Connor & Veeling 2019). Many of these methods are trained to maximize particular lower bound called InfoNCE (van den Oord et al. 2019)-ale known as contrastive learning (Arora et al., 2019). 'he main idea behind contrastive learning is to divide an input data into multiple (possibly overlapping) views and maximize the mutual information between encoded representations of these views, using views derived from other inputs as negative samples. In $2 we provide an overview of representation learning with mutual information maximization. We then show how the skip-gree grapgrram objective ($3.J) Mikolov al. 2013), masked language modeling ($3.2) Devlin al. 2018). and permutation language modeling $3.3 Yang et al. 2019), fit in this framework.
In addition to providing a principled theoretical understanding that bridges progress in multiple areas, our proposed framework also gives rise to a general class of word representation learning models which serves basis for designing and combining self self-supervised training objectives create better language representations. As an example, we show how to use this framework to construct simple superiseer objective that maximizes the mutual information between a sentence and n-grams in the sentence ($4). We combine it with a variant of the masked language modeling objective and show that the resulting representation performs better, particularly on tasks such question answering and linguistics acceptability ($5).
2 MUTUAL INFORMATION MAXIMIZATION
Mutual information measures dependencies between random variables. Given two random variables and B, can be understood as how much knowing A reduces the uncertainty in or vice versa. Formally, the mutual information between A and B is:
(AA -) H(A) B) ( - H(B A).
Consider A and B to be different views of an input data (e.g., a word and its context, two different partitions of sentence). Consider function f that takes A a and training is to learn parameters of the function that maximizes [(A, B).
its input. The goal of
Maximizing mutual information directly is generally intractable when the function f consists of modern encoders such as neural networks (Paninski] 2003), so we need to resort lower bound on [(A, B). One particular lower bound that has been shown work well in practice is InfoNCE [Logeswaran & Lee 2018 yan den Oord et al. 2019 which is based on Noise Contrastive Estimation (NCE; Gutmann Hyvarinen 2012)
InfoNCE is defined as:
T I(A, B) 2 BAA.B) bo
log Σ exp fo(a,
log B
(B)
bEB
where a and are different views of an input sequence, fe E R is a function parameterized by (e.g., dot product between encoded representations of word and its context, a dot product between encoded representations of two partitions of sentence), and B is a set of samples drawn from proposal distribution (B). The set contains the positive sample and |B] negative samples.
Learning representations based this objective is also known contrastive learning- Arora et al. (2019) show representations learned by such method have provable performance guarantees and reduce sample complexity on downstream tasks.
We note that InfoNCE is related to cross-entropy. When B always includes all possible values of the random variable B (i.e., B = B) and they are uniformly distributed, maximizing InfoNCE is analogous maximizing the standard cross-entropy loss:
E (AA
bo(a,
log Σ exp So(a,
6EB
Alternattve bounds include Donsker- Donsker-Vardhan representation Donsker & Varadhan 1983 and Jensen- Shannon estimator (Nowozin et al. (216)), but focus on InfoNCE here.
2 See van den Oord et al. 2019); ; Poole et al. (2019) for detailed derivations of InfoNCE as bound on mutual information.
2

---

Published as a conference paper at ICLR 2020
Eq. 2 above shows that InfoNCE is related to maximizing po(b a), and it approximates the summation over elements in B (i.c., the partition function) by negative sampling. As function of the negative samples, the InfoNCE bound is tighter when ' contains more samples (as can be seen in Eq. I above by inspecting the log [B] term). Approximating a softmax over a large vocabulary with negative samples is a popular technique that has been widely used in natural language processing in the past. We discuss it here make the connection under this framework clear.
3 MODELS
We describe how Skip-gram, BERT, and XLNet fit into the mutual information maximization framework as instances of InfoNCE. In the following, we assume that fe(a, b) = ge (a), where 0 = {w, $}. Denote the vocabulary set by and the size of the vocabulary by V. For word representation learning,
seek to learn encoder parameterized by to represent each word in sequence æ = ( FT } in d dimensions. For each of the models we consider in this paper, and are formed by taking different parts of x: (e.g., a = To and = IT).
3.1 SKIP-GRAM
We first start with simple word representation learning model Skip- gramp (Mikolov et al. 2013). Skip-gram is a method for learning word representations that relies on the assumption that a good representation of word should be predictive of its context. The objective function that maximized in Skip- gram is: ap()..)) [p(x) =2)], where is word token and z² is context word of 2,.
Let be the context word to be predicted s² and a be the input word Ty. Recall that fo b) is In (b)
The skip-gram objective function can be written an instance of InfoNCE (Eq. I where (b) and (a) are embedding lookup functions that map each word type to Rd. (i.c.. R.d).
(a). (b)TTW(()
Io (b), 9w (a)
p(a) az)) can either be computed using a standard softmax the entire vocabulary with negative sampling (when the vocabulary is very large). These two approaches correspond to different choices of B. In the softmax approach, ' is the full vocabulary set V and each word in V is uniformly distributed. In negative sampling, B is set of negative samples drawn from e.g., unigram distribution.
While Skip-gram has been widely accepted as an instance contrastive learning (Mikolov et al. 2013 Mnih & Kavukcuoglu 2013), we include it here to illustrate its connection with modern approaches such as BERT and XLNet described subsequently. We can see that the two views of an input sentence that are considered in Skip-gram are two words that appear in the same sentence, and they are encoded using simple lookup functions.
3.2 BERT
Devlin et al. (2018) introduce two self-supervised tasks for learning contextual word representations: masked language modeling and next sentence prediction. Previous work suggests that the next sentence prediction objective is not necessary to train a high quality BERT encoder and the masked
language modeling appears to be the key to learn good representations (Liu et al.) 2019: Joshi et al., 2019 Lample & Conneau 2019), so we focus on masked language modeling here, However, we also show how next sentence prediction fits into our framework in Appendix
In masked language modeling, given a sequence of word tokens of length T, x = (Z1,... BERT replaces 15% of the tokens in the sequence with (i) a mask symbol 80% of the time, (ii random word 10% of the time, (iii) its original word. For each replaced token, it introduces a term in the masked language modeling training objective to predict the original word given the perturbed sequence à; {aI,.... ,B,,... .Tr,...)))). (i.e., the sequence æ masked at w.). This training objective can be written as: (.)(,..) [p(#2 6.))
}.
3

---

Published as a conference paper at ICLR 2020
Following our notation in $2 have fo(a,b) b) (b) gw (a). Let b be masked word 2; and be the masked sequence ã. Consider a Transformer encoder parameterized by and denote Iw (.t,) E Rd as function that returns the final hidden state corresponding to the i-th token (i.e., the masked token) after running é; through the Transformer. Let Pv V Rd be lookup function that maps each word type into vector and B be the full vocabulary set V. Under this formulation, the masked language modeling objective maximizes Eq. [ different choices of masking probabilities can be understood as manipulating the joint distributions p(a. b). In BERT, the two views of sentence correspond to masked word in the sentence and its masked context.
Contextual non-contextual. It is generally understood that the main difference between Skip- gram and BERT is that Skip-gram learns representations of word types (i.e., the representation for word is always the same regardless of the context it appears in) and BERT learns representations of word tokens. We note that under our formulation for either -kip-gram or BERT, the encoder that we want to learn appears in Iw. and 96 is not used after training. We show that Skip-gram and BERT maximizes a similar objective, and the main difference is in the choice of the encoder that forms -w- context dependent Transformer encoder that takes sequence as its input for BERT and a simple word embedding lookup for Skip-gram.
3.3 XLNET
Yang et al, 2019) propose permutation language modeling objective to learn contextual word repre- sentations. This objective considers all possible factorization permutations of a joint distribution of a sentence. Given a sentence a (.i,..., FTT) there are ways to factorize its joint distribution! Given sentence æ, denote a permutation by 2. XLNet optimizes the objective function:
Ex(x)
Ep(x)
Σ
log D
xx)
As running example, consider a permutation order 3, 1, 5, 2, for sentence I1, 42, T3, T4 Ug. Given the order, XLNet is only trained to predict the last S tokens in practice. For S = 1, the context sequence used for training is 22, 03, F5. with 24 being the target word.
In addition to replacing the Transformer encoder with Transformer XI. (Dai al. 2019), a key architectural innovation of XLNet is the two-stream self-attention. In two-stream self attention, shared encoder is used to compute two sets of hidden representations from one original sequence. They are called the query stream and the content stream. In the query stream, the input sequence is masked at the target position, whereas the content stream sees the word the target position. Words at future positions for the permutation order under consideration are also masked in both streams. These masks are implemented as two attention mask matrices. During training, the final hidden representation for a target position from the query stream is used to predict the target word.
Since there is only set of encoder parameters for both streams, show that the permutation language modeling objective from the masked language modeling objective with an architectural change in the encoder. Denote hidden representation by hţ, where indexes the position and k indexes the layer, and consider the training sequence 21, 22, Wa, 26 and the permutation order 3,
arrive at
2, 4. In BERT, we compute attention scores to obtain hy from h for every t (i.e., t= 1...., 7), where hς is the embedding for the mask symbol. In XLNet, the attention scores for future words the permutation order are masked to O. For example, when we compute h only the attention score from - is considered (since the permutation order is 3, 1, 5, 2, 4). For Ln, we use h - and h R-1 XLNet does not require a mask symbol embedding since the attention score from a masked token is always zeroed with attention mask (implemented as a matrix). As result, we can consider XLNet training as masked language modeling with stochastic attention masks the encoder.
is now straightforward to see that the permutation language modeling objective is an instance of Eq where b is target token x; and is masked sequence é; {I1,..., B...... (T Similar to
For example, we can factorize p(æ)= = p(æ: }p(#2 mi)... p(ær Ii.... 1T-1) p(er)p(Fr- Tr) ..., p( 3r,-..; 22). and many others.
4

---

Published as a conference paper at ICLR 2020
Table 1: Summary of methods as instances of contrastive learning. See text for details.
Objective
b
p(a, b)
Sw
Fe
Skip-gram
word
word
word and its context
lookup
lookup
MLM
context
masked word
masked tokens probability
Transformer
lookup
NSP
sentence sentence
(non-)conseenl joonsccvveeee sentences factorization permutation sentence and its n-grams
Transformer
lookup
XLNet
context
masked word
TXL+
lookup
DIM
context
masked ri-grams
Transformer
not used
BERT, we have Transformer encoder parameterized by and denote Iw (á;; E IR as function that returns the final hidden state corresponding to the i-th token (i.e., the masked token) after running è; through the Transformer. Let Pv V - R be a lookup function that maps each word type into a vector and ' = 'B be the full vocabulary set V. The main difference between BERT and XLNet that the encoder that forms Iw used in XLNet implements attention masking based on a sampled permutation order when building its representations. addition, XLNet and BERT also differ in the choice of p( (a, b) since each of them has its own masking procedure. However, we can see that both XLNNt and BERT maximize the same objective.
4
INFOWORD
Our analysis on Skip-Gram, BERT, and XLNet shows that their objective functions are different instances of InfoNCE in Eq although they are typically trained using the entire vocabulary set for instead of negative sampling. These methods differ in how they choose which views of a sentence they use as a and b, the data distribution p(a, 6), and the architecture of the encoder for computing Jw which we summarize in Table Seen under this unifying framework, we can observe that progress in the field has largely been driven by using a more powerful encoder to represent 9w- While we only provide derivations for Skip-gram, BERT, and XLNet, language-modelle based pretraining-objectives such as those used in ELMo (Peters et al., 2018) and GPT-2 Radford et al.) 2019) can be formulated under this framework.
straightforward to show that other
Our framework also allows us to draw connections to other mutual information maximization representation learning methods that have been successful in other domains (e.g.. computer vision, audio processing, reinforcement learning). In this section, we discuss example derive insights to design a simple self-supervised objective for learning better language representations.
Deep InfoMax (DIM: Hjelm et al. 2019) a mutual information maximization based representation learning method for images. DIM shows that maximizing the mutual information between an image representation and local regions of the image improves the quality of the representation. The complete objective function that DIM maximizes consists of multiple terms. Here, we focus on a term in the objective that maximizes the mutual information between local features and global features. We describe the main idea of this objective for learning representations from one-dimensional sequence, although is originally proposed to learn from a two-dimensional object.
Given a sequence = {w11 22...., Tr), we consider the "global" representation of the sequence to be the hidden state of the first token (assumed to be a special start of sentence symbol) after contextually encoding the sequence (=x) and the local representations to be the encoded repre- sentations of each word in the sequence gup (x.). We can use the contrastive learning framework to design task that maximizes the mutual information between this global representation vector and its corresponding "local" representations using local representations from other sequences (f:) as negative samples. This is analogous to training the global representation vector a sentence to choose which words appear in the sentence and which words are from other sentences [ However, we feed the original sequence to the encoder and take the hidden state of the first token as the global
Alternattvely, the global representation can be the averaged representations of words in the sequence although we do not explore this in our experiments.
5 We can see that this supe-supe task is related to the next sentence prediction objective in BERT. However, instead of leaming global representation (assumed he the representation of the first taken in BERT)
5

---

Published as a conference paper at ICLR 2020
representation, the task becomes trivial since the global representation is built using all the words in the sequence. We instead use masked sequence a = -t = .......,,..................... and b = It.
State-of-the-art methods based on language modeling objectives consider all negative samples since the second view of the imput data (i.e., the part denoted by b in Eq. that are used is simple and it consists of only target word-hence the size of the negative set is still manageable. A major benefit of the contrastive learning framework is that we only need to be able to take negative samples for 6 Denote an
training. Instead of individual words, we can use n-grams as the local representations
n-gram by Daj and a masked sequence masked at position i to j by Big We define JDM as:
DIM
Eppee. m.n)) Iw (ftas) gw (arrg) (e
log Σ E8
(
where Bgy is a sentence masked at position to j. Déy is an n-gram spanning from i to j. and Tys is an n-gram from set $ that consists of the positive sample Fayj and negative n-grams from other sentences in the corpus. We use one Transformer to encode both views, so we do not need Gup here.
Since the main goal of representation learning is train an encoder parameterized by w. it is possible to combine multiple self-supervised tasks into objective function in the contrastive learning framework. Our model, which we denote INFOWORD, combines the above objective-whin designed improve sentence and span representations- with a masked language modeling objective JMLM for learning word representations. The only difference between our masked language modeling objective and the standard masked language modeling objective is that we use negative sampling to construct V by sampling from the unigram distribution. We have:
is
JMLM =))) (w,) g (2x4) - -log Σ exp((w((t,) (wx))))
B.EV
where è; sentence masked at position and a; is the i-th token in the sentence.
Our overall objective function is weighted combination of the two terms above:
JInFoWoRD = AMLM?MLM + ADIMIDIM
where AMLM and ADIM are hyperparameters that balance the contribution of each term.
EXPERIMENTS
In this section, evaluate the effects of training masked language modeling with negative sampling and adding JIMM to the quality of learned representations.
5.1 SETUP
We largely follow the same experimental setup as the original BERT model (Devlin et al. 2018 We have two Transformer architectures similar to BERT Base and BERTLARGE BERT BASE has 12 hidden layers, 768 hidden dimensions, and 12 attention heads (110 million parameters): whereas BERT LARGE has 24 hidden layers, 1024 hidden dimensions, and 16 attention heads (340 million parameters).
For each of the Transformer variant above, we compare three models in our experiments:
BERT: The original BERT model publicly available in https://github.. com/
BERT-NCE: Our reimplementation of BERT. It differs from the original implementation in several ways: (1) we only use the masked language modeling objective and remove next
sentence prediction, (2) we negative sampling instead of softmax, and (3) only use one sentence for each training example in a batch.
be predictive of whether sentences consecutive sentences, learns its global representation be predictive of words in the original sentence. "Local image patches used in DIM analogous n-grams in sentence.

---

Published as a conference paper at ICLR 2020
INFOWORD: Our model described in $4 The main difference between INFOWORD and BERT-NCE is the addition of DIIM to the objective function. We discuss how we mask the data for DIM in $5.2
5.2 PRETRAINING
We use the same training corpora and apply the same preprocessing and tokenization BERT. We create masked sequences for training with JIM follows. We iteratively sample regrams from sequence until the masking budget (15% of the sequence length) has been spent. At each sampling iteration, we first sample the length of the n--ram (i.e., n in n-grams) from a Gaussian distribution N(5, 1) clipped I (minimum length) and 10 (maximum length). Since BERT tokenizes words into subwords, we measure the r-gram length at the word level and compute the masking budget at the subword level. This procedure inspired by the masking approach in Joshi et al. (2019).
For negative sampling, we use words and n-grams from other sequences in the same batch as negative samples (for MLM and DIM respectively). There are approximately 70,000 subwords and 10,000 n- n-grams (words and phrases) in a batch. We discuss hyperparameter details in Appendix B
5.3 FINE-TUNING
We evaluate on two benchmarks: GLUE (Wang et al. 2019) and SQuAD(Raipurkar et al. 2016). We train a task-specific decoder and fine-tune pretrained models for each dataset that we consider. We describe hyperparameter details in Appendix B
GLUE is a set of natural language understanding tasks that includes sentiment analysis, linguistic acceptability, paraphrasing, and natural language inference. Each task is formulated as classification task. The tasks in GLUE are either a single-sentence classification task or sentence pair classification task. We follow the same setup the original BERT model and add start of sentence symbol (i.e., the CLS symbol) to every example and use separator symbol (i.e., the SEP symbol) to separate two concatenated sentences (for sentence pair classification tasks). We add a linear transformation and softmax layer to prediet the correct label (class) from the representation of the first token of the sequence.
SQuAD is a reading comprehension dataset constructed from Wikipedia articles. We report results on SQuAD 1.1. Here, also follow the same setup as the original BERT model and predict an answer span the start and end indices of the correct answer in the context. We use standard span predictor as the decoder, which we describe in details in Appendix C
5.4 RESULTS
We show our main results in Table and Table Our BERT reimplementation with negative sampling underperforms the original BERT model GLUE but significantly better on SQuAD. However, we think that the main reasons for this performance discrepancy are the different masking procedures (we use span-based masking instead of whole-word masking) and the different ways training examples are presented the model (we use one consecutive sequence instead of two sequences separated by the separator symbol). Comparing BERT-NCE and INFOWORD WOD, we observe the benefit of the new self-supervised objective UDIM (better overall GLUE and SQuAD results), particularly on tasks such as question answering and linguistics acceptability that seem to require understanding of longer phrases. In order to better understand our model, we investigate its performance with varying numbers of training examples and different values of ADIM the SQuAD development set and show the results in Figure [ (for models with the BASE configuration). We can see that INFOWORD consistently outperforms BERT-NCE and the performance gap is biggest when the dataset is smallest, suggesting the benefit of having better pretrained representations when there are fewer training examples.
7

---

Published as a conference paper at ICLR 2020
Table 2: Summary of results on GLUE
MNLI
GLUE
Model
CoLA
SST-2
MRPC
QQP
QNLI
RTE
(M/MM)
AVG
BASE
BERT
52.1
93.5
88.9
71.2
84.6/83.4
90.5
66.4
78.8
BERT-NCE
50.8
93.0
88.6
70.5
83.2/83.0
90.9
65.9
78.2
INFOWORD
53.3
92.5
88.7
71.0
83.7/82.4
91.4
68.3
78.9
LARGE
BERT
60.5
94.9
89.3
72.1
86.7/85.9
92.7
70.1
81.5
BERT-NCE
54.7
93.1
89.5
71.2
85.8/85.0
92.7
72.5
80.6
INFOWORD
57.5
94.2
90.2
71.3
85.8/84.8
92.6
72.0
81.1
Table 3: Summary of results on SQuAD 1.1.
DEV
TEST
Model
Fi
EM
Fi
EM
BERT
88.5
80.8
BASE
BERT-NCE
90.2
83.3
90.9
84.4
INFOWORD
90.7
84.0
91.4
84.7
LARGE
BERT
90.9
84.1
91.3
84.3
BERT-NCE
92.0
85.9
92.7
86.6
INFOWORD
92.6
86.6
93.1
87.3
5.5 DISCUSSION
Span-based models. We show how to design a simple self-supervised task in the InfoNCE frame- work that improves downstream performance several datasets. Learning language representations to predict contiguous masked tokens has been explored in other context, and the objective introduced in DIM is related to these span-based models such as SpanBERT (Joshi et al., 2019) and MASS (Song al. 2019). While our experimental goal to demonstrate the benefit of contrastive learning for constructing self-supervised tasks, we note that INFOW ORD is simpler to train and exhibits similar trends to SpanBERT that outperforms baseline models. We leave exhaustive comparisons to these methods to future work.
Mutual information maximization. A recent study has questioned whether the success of In- foNCE as an objective function due to its property as lower bound on mutual information and provides an alternative hypothesis based on metric learning (Tschannen al.) 2019). Regardless of the prevailing perspective, InfoNCE is widely accepted as good representation learning objective, and formulating state-of-the-r language representation learning methods under this framework offers valuable insights that unifies many popular representation learning methods.
Regularization. Image representation learning methods often incorporate a regularization term in its objective function to encourage learned representations to look like prior distribution (Hjelm) al. 2019 Bachman al. 2019). This is useful for incorporating prior knowledge into repre- sentation learning model. For example, the DeepInfoMax model has term in its objective that encourages the learned representation from the encoder to match uniform prior. Regularization is not commonly used when learning language representations. Our analysis and the connection we draw to representation learning methods used in other domains provide an insight into possible ways to incorporate prior knowledge into language representation learning models.
Future directions. The InfoNCE framework provides a holistic way to view progress in language representation learning. The framework is very flexible and suggests several directions that can be explored to improve existing methods. We show that progress in the field has been largely driven by innovations in the encoder which forms 9w InfoNCE is based on maximizing the mutual information between different views of an input data, and it facilitates training on structured views long as can perform negative sampling (van den Oord et al. 2019 Bachman et al.) 2019). Our analysis demonstrates that existing methods based on language modeling objectives only consider a single target word as one of the views. We think that incorporating more complex views (e.g.. higher-order skip n-grams, syntactic and semantie parses, etc.) and designing appropriate self-supervised tasks
8

---

Published as a conference paper at ICLR 2020
500
8-
6. 8-
F1
90.4
BERT-NCE
10
9003
8.
training exumples
Figure 1: The left plot shows F1 scores of BERT-NCE and INFOWORD we increase the percentage of training examples on SQuAD (dev). The right plot shows Fi scores of INFOWORD on SQuAD (dev) as a function of ADIM-
a promising future direction. related area that is also underexplored is designing methods to obtain better negative samples.
CONCLUSION
We analyzed state-of-the-art language representation learning methods from the perspective of mutual information maximization. We provided a unifying view of classical and modern word embedding models and showed how they relate to popular representation learning methods used in other domains. We used this framework to construct a new self-supervised task based on maximizing the mutual information between the global representation and local representations of a sentence. We demonstrated the benefit of this task via experiments GLUE and SQuAD.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning. In Proc. of ICMI., 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint 1906. 000600090, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual information neural estimation. In Proc. of ICML, 2018.
Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning, In Proc. NIPS, 2015.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Xra:sformer-XL: Attentive language models beyond fixed-length context. In Proc. of ACL, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Pruc. of NAACL. 2018.
M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Commuicatiion on Pure and Applied Mathematics, 36(2):183 122- 1983.
Michael U. Gutmann and Aapo Hyvarinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13: 207-361,220 2012.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Proc. of ICLR, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proc. of ACI., 2018

---

Published as a conference paper at ICLR 2020
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-tranning by representing and predicting spans. arXiv preprint 1907.10529, 2019.
Diederik P. Kingma and Jimmy Lei Ba. Adam: a method for stochastic optimization. In Proc. of ICLR, 2015.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint 1901.07291, 2019.
Ralph Linsker. orff-ngaanitttion in perceptual network. Computer, 21(3):105-111 1988.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint 1907.11692, 2019.
Lajanugen Logeswaran and Honglak Lee, An efficient framework for learning sentence representa- tions. In Proc. of ICLR, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proc. NIPS, 2013.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Proc. of NIPS, 2013.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Proc. of NIPS, 2016.
Sindy Lowe Peter O'Connor and Bastiaan S. Veeling. Greedy infomax for biologically plausible self-supervised representation learning. In Proc. of NeurlPS, 2019.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191-.. 1253, 2003.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proc. of EMNLP, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. of NAACL, 2018.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational lower bounds of mutual information. In Proc. of ICML, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. Technical report, OpenAl, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAl, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proc. of ACI, 2018.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. Proc. of ICML, 2019.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, and Sylvain Gelly. On mutual information maximization for representation learning. arXiv preprint 1907.13623 2019.
Aaron van Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807.03748, 2019.
10

---

Published as a conference paper at ICLR 2020
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understainding. In Proc. of ICIR, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V.
Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint 1906.08237, 2019.
Dani Yogatama, Cyprien de Masson d' Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. Learning and evaluating general linguistic intelligence. arXiv preprint 1901,11373 2019.
A NEXT SENTENCE PREDICTION
We show that the next sentence prediction objective used BERT is an instance of contrastive learning in this section. In next sentence prediction, given two sentences xt and æ², the task is to predict whether these are two consecutive sentences or not. Training data for this task is created by sampling a random second sentence 2 from the corpus to be used as a negative example 50% the time.
Consider discriminator (i.e., classifier with parameters Φ) that takes encoded representations of concatenated x' and x² and returns score. We denote this discriminator by d$ (²²) The next sentence prediction objective function is:
Bp(=²_a²) _=²) [log (oggdaaaaa ((x', xx²))) + loog11 (9w ( -
This objective cectionn-wicee is used for training BERT is known in the literature as "local" Noise Contrastive Estimation (Gutmann & Hyvarinen, 2012). Since summing over all possible negative sentences is intractable, BERT approximates this by using binary classifier to distinguish real samples and noisy samples.
alternative approximation using a binary classifier is use "global NCE", which is what InfoNCE is based on. Here, we have:
Ep(n' .x?) 20 Iw (wx x)))
log Σ exp(1 ((=) ,,,)))))))
where we sample negative sentences from the corpus and combine with the positive sentence to construct 22. To make the connection of this objective function with InfoNCE in Eq. explicit, let and be two consecutive sentences 3 and Ø2. Let fa b) be &b 9w ([a, b)), where &p Rd is trainable parameter, (a, b] denotes a concatenation of and b. Consider Transformer encoder parameterized by and let 9w (Ia, b]) E Id be function that returns the final hidden state of the first token after running the concatenated sequence to the Transformer. Note that the encoder that we want to learn only depends on Iw. so both of these approximations can be used for training next sentence prediction.
T
B HYPERPARAMETERS
Pretraining. We use Adam (Kingma & Ba 2015) with 61 = 0.9, B3 = 0.98 and c 1c - 6. The batch size for training 1024 with a maximum sequence length 512. We train for 400,000 steps (including 18,000 warmup steps) with a weight decay rate of 0.01. We set the learning rate for all variants of the BASE models and le for the LARGE models. We set AMLM to 1.0 and tune ADIM E {0.4, 0.6, 0.8, 1.0].
GLUE. We the maximum sequence length 128. For each GLUE task, we use the respective development set to choose the learning rate from (5e 1e- 2e-5 5, 35 5e-5). and the batch size from {16, 32). The number of training epochs is set to 4 for CoLA and 10 for other tasks, following Joshi et al. (2019). We run each hyperparameter configuration times and evaluate the best model on the test set (once).
11

---

Published as a conference paper at ICLR 2020
SQuAD. We set the maximum sequence length to 512 and train for 4 epochs. We use the develop- ment set to choose the learning rate from {5e leee 2e- 3,3e 5e 5) and the batch size from (16,32}.
C QUESTION ANSWERING DECODER
We use standard span predictor as follows. Denote the length of the context paragraph by M, and zcnntext {sgamieet , } Denote the encoded representation of the m-th token in the context by xoomlest The question answering decoder introduces two sets of parameters: Wstart and Wend The probability of each context token being the start of the answer is computed as: exp( X cinnt)
p(start It.m
The probability of the end index of the answer is computed analogously using Wend- The predicted answer is the span with the highest probability after multiplying the start and end probabilities.
(x)) Inzo exp(w, W M
12


Reviews:
[{"review_id": "Skx9YZG-qB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper first gives a concise yet precise summary of maximizing one of variational lower bounds of mutual information, InfoNCE, then it provides an alternative view to explain case by case why word embedding Skip-gram, BERT, XLNet work in practice can be viewed by InfoNCE framework, thus we have a good understand for these methods. Moreover it introduces a self-learning method  that maximizes the mutual information between a global sentence representation and n-grams in the sentence based on deep InfoMax framework instead. Experiments show that it is better then BERT and BERT-NCE. It's known that InfoNCE increases bias but reduce variance, the same is true for deep InfoMax. Do you observe this in your experiments? If so, please provide.\n\nThe paper is well-written and easy to follow. The originality is relative low though, since it is mainly an application of  deep InfoMax to language modeling, not inventing a new algorithm and applying to language modeling.\n\nIn equations 1 and 2, should a, b be written in capital? Since they represent random variables.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hyemr5VVtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper gives a big picture view on training objectives used to obtain static and contextualized word embeddings. This is very handy since classical static word embeddings, such as SGNS and GloVe, have been studied theoretically in a number of works (e.g., Levy and Goldberg, 2014; Arora et al., 2016; Hashimoto et al., 2016; Gittens et al., 2017; Allen and Hospedales, 2019; Assylbekov and Takhanov, 2019), but not much has been done for the modern contextualized embedding models such ELMo and BERT - I personally know only the work of Wang and Cho (2019), and please correct me if I am wrong.\n\n\"There is nothing as practical as a good theory\", and the authors confirm this statement: their theory suggests them to modify the training objective of the masked language modeling in a certain way and this modification proves to benefit the embeddings in general when evaluated on standard tasks.\n\nI don't have any major issues to raise. A minor comment is that the mutual information I(., .) being a function of two variables suddenly became a function of a single variable in Eq. (1) and in the text which precedes it.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1eWGbyRYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to make a clear connection between the InfoNCE learning objective (which is a lower bound of the mutual information) and multiple language models like BERT and XLN. Then based on the observation that classical LM can be seen as instances of InfoNCE, they propose a new (InfoWord) model relying on the same principles, but taking inspiration from other models also based on InfoNCE. Mainly, the proposed model  differs both in the nature of the a and b variables used in InfoNCE, and also on the fact that it uses negative sampling instead of softmax. Experiments are made on two tasks and compared to a classical BERT model, and on the BERT-NCE model that is a BERT variant proposed by the authors which is somehow in-between BERT and InfoWord. They show that their approach works quite well. \n\nI have a very mitigated opinion on the paper. I) First, I really like the idea of trying to unify different models under the same learning principles, and then show that these models can be seen as specific instances of generic principles. But the way it is presented and explained lacks of clarity: for instance in Section 2, some notations are not well defined (e.g what is f?) . Moreover, the way classical models are casted under the InfoNCE principle is badly written: it assumes that readers have a very good knowledge of the models, and the paper does not show well the mapping between the loss function of each model and the InfoNCE criterion. It gives technical details that could (in my opinion) get ignored, and I would clearly prefer to catch the main differences between the different models that being flooded by technical details. So, my suggestion would be to improve the writing of this section to make the message stronger and relevant for a larger audience. II) The Infoword model can be seen as a simple instance of word masking based models, and as an extension of deep infomax for sequences (it would be certainly nice to describe a little bit what Deep InfoMax is to facilitate the reading).  Here again, the article moves from technical details (e.g \"hidden state of the first token (assumed to be a special start of sentence symbol \") without providing formal definitions. Having a first loss function after paragraph 4 could help to understand the principle of this model (before restricting the model to n-grams).  Moreover, the equation J_DIM seems to be wrong since it contains g_\\omega twice while I think (but maybe I am wrong) that it has also to be defined by g_\\psi. J_MLM is also not clear since x_i is never defined (I assume it is x_{i:i}). At last,  after unifying multiple models under one common learning objective, the authors propose to mix two different losses which is strange (the effect of the second term is slightly studied in the experimental section) without allowing us to understand why it is important to have this second loss function and why the first one is not sufficient enough. At last, I am pretty sure to not be able to reproduce the model described in the paper (adding a section on that in the supplementary material would help), and many concrete aspects are described too fast (like the way to sample negative pairs). \n\nConcerning the experimental section, experiments are convincing and show that the model is able to achieve a performance which is close to classical models. In my opinion, tis section has to be interpreted as  a proof that the proposed unified vision is a good way to easily define new and efficient models. \n\nTo summarize, the unification under the InfoNCE principle is interesting,  but the way the paper is written makes it very difficult to follow, and the description of the proposed model is unclear (making the experiments difficult to reproduce) and lacks of a better discussion about the interest of mixing multiple loss. \n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores several embedding models (Skip-gram, BERT, XLNet) and describes a framework for comparing, and in the end, unifying them.  The framework is such that it actually suggests new ways of creating embeddings, and draws connections to methodology from computer vision.

One of the reviewers had several questions about the derivations in your paper and was worried about the paper's clarity.  But all of the reviewers appreciated the contributions of the paper, which joins multiple seemingly disparite models under into one theoretical framework.

The reviewers were positive about the paper, and in particular were happy to see the active response of authors to their questions and willingness to update the paper with their suggested improvements.
Decision: Accept (Spotlight)