Venue: NAACL 2019

# pair2vec: Compositional Word-Pair Embeddings

for Cross-Sentence Inference

Mandar Joshi\({}^{\dagger}\) **Eunsol Choi\({}^{\dagger}\)****Omer Levy\({}^{\ddagger}\)****Daniel S. Weld\({}^{\dagger}\)****Luke Zettlemoyer\({}^{\dagger\ddagger}\)\({}^{\dagger}\) Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA {mandar90,eunsol,weld,lsz}@cs.washington.edu**

Facebook AI Research, Seattle {omerlevy,lsz}@fb.com

###### Abstract

Reasoning about implied relationships (e.g. paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems. This paper proposes new methods for learning and using embeddings of word _pairs_ that implicitly represent background knowledge about such relationships. Our pairwise embeddings are computed as a compositional function on word representations, which is learned by maximizing the pointwise mutual information (PMI) with the contexts in which the two words co-occur. We add these representations to the cross-sentence attention layer of existing inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or replacing existing word embeddings. Experiments show a gain of 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI. Our representations also aid in better generalization with gains of around 6-7% on adversarial SQuAD datasets, and 8.8% on the adversarial entailment test set by Glockner et al. (2018).

## 1 Introduction

Reasoning about relationships between pairs of words is crucial for cross sentence inference problems such as question answering (QA) and natural language inference (NLI). In NLI, for example, given the premise "_golf is prohibitively expensive_", inferring that the hypothesis "_golf is a cheap pastime_" is a contradiction requires one to know that _expensive_ and _cheap_ are antonyms. Recent work Glockner et al. (2018) has shown that current models, which rely heavily on unsupervised single-word embeddings, struggle to learn such relationships. In this paper, we show that they can be learned with word _pair_ vectors (pair2vec1), which are trained unsupervised, and which significantly improve performance when added to existing cross-sentence attention mechanisms.

Footnote 1: [https://github.com/mandarjoshi90/pair2vec](https://github.com/mandarjoshi90/pair2vec)

Unlike single-word representations, which typically model the co-occurrence of a target word \(x\) with its context \(c\), our word-pair representations are learned by modeling the three-way co-occurrence between words \((x,y)\) and the context \(c\) that ties them together, as seen in Table 1. While similar training signals have been used to learn models for ontology construction Hearst (1992); Snow et al. (2005); Turney (2005); Shwartz et al. (2016) and knowledge base completion Riedel et al. (2013), this paper shows, for the first time, that large scale learning of pairwise embeddings can be used to directly improve the performance of neural cross-sentence inference models.

More specifically, we train a feedforward network \(R(x,y)\) that learns representations for the individual words \(x\) and \(y\), as well as how to compose them into a single vector. Training is done by maximizing a generalized notion of the pointwise mutual information (PMI) among \(x\), \(y\), and their context \(c\) using a variant of negative sampling Mikolov et al. (2013a). Making \(R(x,y)\) a

\begin{table}
\begin{tabular}{p{42.7pt} c c} \hline \hline
**X** & **Y** & **Contexts** \\ \hline \multirow{2}{*}{_hot_} & \multirow{2}{*}{_cold_} & with **X** and **Y** baths too **X** or too **Y** neither **X** nor **Y** \\  & & & **in X**, Y** metropolitan area in **Y** \\  & & **X** International Airport in **Y** \\ \hline \multirow{2}{*}{_crop_} & \multirow{2}{*}{_wheat_} & food **X** are maize, **Y**, etc dry **X**, such as **Y**, more **X** circles appeared in **Y** fields \\ \cline{3-3}  & & **X** OS comes with **Y** play the **X** team at **Y** \\ \cline{3-3}  & & **X** is developed by **Y** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example word pairs and their contexts.



compositional function on individual words alleviates the sparsity that necessarily comes with embedding pairs of words, even at a very large scale.

We show that our embeddings can be added to existing cross-sentence inference models, such as BiDAF++ Seo et al. (2017); Clark and Gardner (2018) for QA and ESIM Chen et al. (2017) for NLI. Instead of changing the word embeddings that are fed into the encoder, we add the pretrained _pair_ representations to _higher layers_ in the network where cross sentence attention mechanisms are used. This allows the model to use the background knowledge that the pair embeddings implicitly encode to reason about the likely relationships between the pairs of words it aligns.

Experiments show that simply adding our word-pair embeddings to existing high-performing models, which already use ELMo Peters et al. (2018), results in sizable gains. We show 2.72 F1 points over the BiDAF++ model Clark and Gardner (2018) on SQuAD 2.0 Rajpurkar et al. (2018), as well as a 1.3 point gain over ESIM Chen et al. (2017) on MultiNLI Williams et al. (2018). Additionally, our approach generalizes well to adversarial examples, with a 6-7% F1 increase on adversarial SQuAD Jia and Liang (2017) and a 8.8% gain on the Glockner et al. (2018) NLI benchmark. An analysis of pair2vec on word analogies suggests that it complements the information in single-word representations, especially for encyclopedic and lexicographic relations.

## 2 Unsupervised Pretraining

Extending the distributional hypothesis to word pairs, we posit that similar word _pairs_ tend to occur in similar contexts, and that the contexts provide strong clues about the likely relationships that hold between the words (see Table 1). We assume a dataset of \((x,y,c)\) triplets, where each instance depicts a word pair \((x,y)\) and the context \(c\) in which they appeared. We learn two compositional representation functions, \(R(x,y)\) and \(C(c)\), to encode the pair and the context, respectively, as \(d\)-dimensional vectors (Section 2.1). The functions are trained using a variant of negative sampling, which tries to embed word pairs \((x,y)\) close to the contexts \(c\) with which they appeared (Section 2.2).

### Representation

Our word-pair and context representations are both fixed-length vectors, composed from individual words. The word-pair representation function \(R(x,y)\) first embeds and normalizes the individual words with a shared lookup matrix \(E_{a}\):

These vectors, along with their element-wise product, are fed into a four-layer perceptron:

\[R(x,y)=MLP^{4}(\mathbf{x},\mathbf{y},\mathbf{x}\circ\mathbf{y})\]

The context \(c=c_{1}...c_{n}\) is encoded as a \(d\)-dimensional vector using the function \(C(c)\). \(C(c)\) embeds each token \(c_{i}\) with a lookup matrix \(E_{c}\), contextualizes it with a single-layer Bi-LSTM, and then aggregates the entire context with attentive pooling:

\[\begin{split}\mathbf{c}_{i}&=E_{c}(c_{i})\\ \mathbf{h_{1}}...\mathbf{h_{n}}&=\text{BiLSTM}( \mathbf{c}_{1}...\mathbf{c}_{n})\\ w&=\text{softmax}_{i}(\mathbf{kh_{i}})\\ C(c)&=\sum_{i}w_{i}\mathbf{W}\mathbf{h}_{i}\end{split}\]

where \(\mathbf{W}\in\mathbb{R}^{d\times d}\) and \(\mathbf{k}\in\mathbb{R}^{d}\). All parameters, including the lookup tables \(E_{a}\) and \(E_{c}\), are trained.

Our representation is similar to two recently-proposed frameworks by Washio and Kato (2018, 2019), but differs in that: (1) they use dependency paths as context, while we use surface form; (2) they encode the context as either a lookup table or the last state of a unidirectional LSTM. We also use a different objective, which we discuss next.

### Objective

To optimize our representation functions, we consider two variants of negative sampling Mikolov et al. (2013): bivariate and multivariate. The original bivariate objective models the two-way distribution of context and (monolithic) word pair co-occurrences, while our multivariate extension models the three-way distribution of word-word-context co-occurrences. We further augment the multivariate objective with typed sampling to upsample harder negative examples. We discuss the impact of the bivariate and multivariate objectives (and other components) in Section 4.3.

Bivariate Negative SamplingOur objective asspires to make \(R(x,y)\) and \(C(c)\) similar (have high inner products) for \((x,y,c)\) that were observed together in the data. At the same time, we wish 

to keep our pair vectors _dis_-similar from random context vectors. In a straightforward application of the original (bivariate) negative sampling objective, we could generate a negative example from each observed \((x,y,c)\) instance by replacing the original context \(c\) with a randomly-sampled context \(c^{N}\) (Table 2, \(J_{2NS}\)).

Assuming that the negative contexts are sampled from the empirical distribution \(P(\cdot,\cdot,c)\) (with \(P(x,y,c)\) being the portion of \((x,y,c)\) instances in the dataset), we can follow Levy and Goldberg (2014) to show that this objective converges into the pointwise mutual information (PMI) between the word pair and the context.

\[R(x,y)\cdot C(c)=\log\frac{P(x,y,c)}{k_{c}P(x,y,\cdot)P(\cdot,\cdot,c)}\]

This objective mainly captures co-occurrences of monolithic pairs and contexts, and is limited by the fact that the training data, by construction, only contains pairs occurring within a sentence. For better generalization to cross-sentence tasks, where the pair distribution differs from that of the training data, we need a multivariate objective that captures the full three-way (\(x,y,c\)) interaction.

Multivariate Negative SamplingWe introduce negative sampling of target words, \(x\) and \(y\), in addition to negative sampling of contexts \(c\) (Table 2, \(J_{3NS}\)). Our new objective also converges to a novel multivariate generalization of PMI, different from previous PMI extensions that were inspired by information theory Van de Cruys (2011) and heuristics Jameel et al. (2018).2 Following Levy and Goldberg (2014), we can show that when replacing target words in addition to contexts, our objective will converge3 to the optimal value in Equation 1:

Footnote 2: See supplementary material for their exact formulations.

Footnote 3: A full proof is provided in the supplementary material.

\[R(x,y)\cdot C(c)=\log\frac{P(x,y,c)}{Z_{x,y,c}} \tag{1}\]

where \(Z_{x,y,c}\), the denominator, is:

\[Z_{x,y,c} =k_{c}P(\cdot,\cdot,c)P(x,y,\cdot)\] \[+k_{x}P(x,\cdot,\cdot)P(\cdot,y,c)\] \[+k_{y}P(\cdot,y,\cdot)P(x,\cdot,c) \tag{2}\]

This optimal value deviates from previous generalizations of PMI by having a linear mixture of marginal probability products in its denominator. By introducing terms such as \(P(x,\cdot,c)\) and \(P(\cdot,y,c)\), the objective penalizes spurious correlations between words and contexts that disregard the other target word. For example, it would assign the pattern "_X is a Y_" a high score with (_banana_, _fruit_), but a lower score with (_cat_, _fruit_).

Typed SamplingIn multivariate negative sampling, we typically replace \(x\) and \(y\) by sampling from their unigram distributions. In addition to this, we also sample uniformly from the top 100 words according to cosine similarity using distributional word vectors. This is done to encourage the model to learn relations between specific instances as opposed to more general types. For example, using _California_ as a negative sample for _Oregon_ helps the model to learn that the pattern "_X is located in Y_" fits the pair (_Portland_, _Oregon_), but not the pair (_Portland_, _California_). Similar adversarial constraints were used in knowledge base completion Toutanova et al. (2015) and word embeddings Li et al. (2017).4

Footnote 4: Applying typed sampling also changes the value to which our objective will converge, and will replace the unigram probabilities in Equation (2) to reflect the type-based distribution.

## 3 Integrating pair2vec into Models

We first present a general outline for incorporating pair2vec into attention-based architectures, and then discuss changes made to BiDAF++ and ESIM. The key idea is to inject our pairwise representations into the attention layer by reusing the cross-sentence attention weights. In addition to attentive pooling over single word representations, we also pool over cross-sentence word pair em

\begin{table}
\begin{tabular}{l l} \hline \hline
**Bivariate** & \(J_{2NS}\left(x,y,c\right)=\log\sigma\left(R(x,y)\cdot C(c)\right)+\sum_{i=1}^{ k_{c}}\log\sigma\left(-R(x,y)\cdot C(c_{i}^{N})\right)\) \\
**Multivariate** & \(J_{3NS}\left(x,y,c\right)=J_{2NS}\left(x,y,c\right)+\sum_{i=1}^{k_{r}}\log \sigma\left(-R(x_{i}^{N},y)\cdot C(c)\right)+\sum_{i=1}^{k_{y}}\log\sigma \left(-R(x,y_{i}^{N})\cdot C(c)\right)\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The bivariate and multivariate negative sampling objectives. The superscript \({}^{N}\) marks randomly sampled components, with \(k_{*}\) being the negative sample size per instance. The equations present per-instance objectives.

 



### General Approach

Pair RepresentationWe assume that we are given two sequences \(a=a_{1}...a_{n}\) and \(b=b_{1}...b_{m}\). We represent the word-pair embeddings between \(a\) and \(b\) using the pretrained pair2vec model as:

\[\mathbf{r}_{i,j}=\left[\frac{R(a_{i},b_{j})}{\|R(a_{i},b_{j})\|};\frac{R(b_{j}, a_{i})}{\|R(b_{j},a_{i})\|}\right] \tag{3}\]

We include embeddings in both directions, \(R(a_{i},b_{j})\) and \(R(b_{j},a_{i})\), because the many relations can be expressed in both directions; e.g., hypernymy can be expressed via "_X is a type of Y_" as well as "_Y such as X_". We take the \(L_{2}\) normalization of each direction's pair embedding because the heavy-tailed distribution of word pairs results in significant variance of their norms.

Base ModelLet \(\mathbf{a}_{1}...\mathbf{a}_{n}\) and \(\mathbf{b}_{1}...\mathbf{b}_{m}\) be the vector representations of sequences \(a\) and \(b\), as produced by the input encoder (e.g. ELMo embeddings contextualized with model-specific BiLSTMs). Furthermore, we assume that the base model computes soft word alignments between \(a\) and \(b\) via co-attention (4, 5), which are then used to compute \(b\)-aware representations of \(a\):

\[s_{i,j} =f_{att}(\mathbf{a}_{i},\mathbf{b}_{j}) \tag{4}\] \[\alpha =\text{softmax}_{j}(s_{i,j})\] (5) \[\overline{\mathbf{b}}_{i} =\sum_{j=0}^{m}\alpha_{ij}\mathbf{b}_{j}\] (6) \[\mathbf{a}_{i}^{inf} =\big{[}\mathbf{a}_{i};\overline{\mathbf{b}}_{i}\big{]} \tag{7}\]

The symmetric term \(\mathbf{b}_{j}^{inf}\) is defined analogously. We refer to \(\mathbf{a}^{inf}\) and \(\mathbf{b}^{inf}\) as the inputs to the _inference_ layer, since this layer computes some function over aligned word pairs, typically via a feed-forward network and LSTMs. The inference layer is followed by aggregation and output layers.

Injecting pair2vecWe conjecture that the inference layer effectively learns word-pair relationships from training data, and it should, therefore, help to augment its input with pair2vec. We augment \(\mathbf{a}_{i}^{inf}\) (7) with the pair vectors \(\mathbf{r}_{i,j}\) (3) by concatenating a weighted average of the pair vectors \(r_{i,j}\) involving \(a_{i}\), where the weights are the same \(\alpha_{i,j}\) computed via attention in (5):

\[\mathbf{r}_{i} =\sum_{j}\alpha_{i,j}\mathbf{r}_{i,j} \tag{8}\] \[\mathbf{a}_{i}^{inf} =\big{[}\mathbf{a}_{i};\overline{\mathbf{b}}_{i};\mathbf{r}_{i} \big{]} \tag{9}\]

The symmetric term \(\mathbf{b}_{j}^{inf}\) is defined analogously.

### Question Answering

We augment the inference layer in the BiDAF++ model with pair2vec. BiDAF++ is an improved version of the BiDAFNoAnswer Seo et al. (2017); Levy et al. (2017) which includes self-attention and ELMo embeddings from Peters et al. (2018). We found this variant to be stronger than the baselines presented in Rajpurkar et al. (2018) by over 2.5 F1. We use BiDAF++ as a baseline since its architecture is typical for QA systems, and, until recently, was state-of-the-art on SQuAD 2.0 and other benchmarks.

BiDAF++Let \(\mathbf{a}\) and \(\mathbf{b}\) be the outputs of the passage and question encoders respectively (in place of the standard \(\mathbf{p}\) and \(\mathbf{q}\) notations). The inference layer's inputs \(\mathbf{a}_{i}^{inf}\) are defined similarly to

Figure 1: A typical architecture of a cross-sentence inference model (left), and how pair2vec is added to it (right). Given two sequences, \(a\) and \(b\), existing models create \(b\)-aware representations of words in \(a\). For any word \(a_{i}\), this typically involves the BiLSTM representation of word \(a_{i}\) (\(\mathbf{a}_{i}\)), and an attention-weighted sum over \(b\)’s BiLSTM states with \(a_{i}\) as the query (\(\overline{\mathbf{b}}_{i}\)). To these, we add the word-pair representation of \(a_{i}\) and each word in \(b\), weighted by attention (\(\mathbf{r}_{i}\)). Thicker attention arrows indicate stronger word pair alignments (e.g. _cheap_, _expensive_).



the generic model's in (7), but also contain an aggregation of the elements in \({\bf a}\), with better-aligned elements receiving larger weights:

\[\mu=\text{softmax}_{i}(\max_{j}s_{i,j}) \tag{10}\] \[\mathbf{\hat{a}}_{i}=\sum_{i}\mu_{i}\mathbf{a}_{i}\] (11) \[\mathbf{a}^{inf}_{i}=\left[\mathbf{a}_{i};\overline{\mathbf{b}}_{ i};\mathbf{a}_{i}\circ\overline{\mathbf{b}}_{i};\mathbf{\hat{a}}\right] \tag{12}\]

In the later layers, \(\mathbf{a}^{inf}\) is recontextualized using a BiGRU and self attention. Finally a prediction layer predicts the start and end tokens.

BiDAF++ with pair2vecTo add our pair vectors, we simply concatenate \(\mathbf{r}_{i}\) (3) to \(\mathbf{a}^{inf}_{i}\) (12):

\[\mathbf{a}^{inf}_{i}=\left[\mathbf{a}_{i};\overline{\mathbf{b}}_{i};\mathbf{ a}_{i}\circ\overline{\mathbf{b}}_{i};\mathbf{\hat{a}};\mathbf{r}_{i}\right] \tag{13}\]

### Natural Language Inference

For NLI, we augment the ESIM model Chen et al. (2017), which was previously state-of-the-art on both SNLI Bowman et al. (2015) and MultiNLI Williams et al. (2018) benchmarks.

EsimLet \({\bf a}\) and \({\bf b}\) be the outputs of the premise and hypothesis encoders respectively (in place of the standard \({\bf p}\) and \({\bf h}\) notations). The inference layer's inputs \(\mathbf{a}^{inf}_{i}\) (and \(\mathbf{b}^{inf}_{j}\)) are defined similarly to the generic model's in (7):

\[\mathbf{a}^{inf}_{i}=\left[\mathbf{a}_{i};\overline{\mathbf{b}}_{i};\mathbf{a }_{i}\circ\overline{\mathbf{b}}_{i};\mathbf{a}_{i}-\overline{\mathbf{b}}_{i}\right] \tag{14}\]

In the later layers, \(\mathbf{a}^{inf}\) and \(\mathbf{b}^{inf}\) are projected, recontextualized, and converted to a fixed-length vector for each sentence using multiple pooling schemes. These vectors are then passed on to an output layer, which predicts the class.

ESIM with pair2vecTo add our pair vectors, we simply concatenate \(\mathbf{r}_{i}\) (3) to \(\mathbf{a}^{inf}_{i}\) (14):

\[\mathbf{a}^{inf}_{i}=\left[\mathbf{a}_{i};\overline{\mathbf{b}}_{i};\mathbf{ a}_{i}\circ\overline{\mathbf{b}}_{i};\mathbf{a}_{i}-\overline{\mathbf{b}}_{i}; \mathbf{r}_{i}\right] \tag{15}\]

A similar augmentation of ESIM was recently proposed in KIM Chen et al. (2018). However, their pair vectors are composed of WordNet features, while our pair embeddings are learned directly from text (see further discussion in Section 6).

## 4 Experiments

For experiments on QA (Section 4.1) and NLI (Section 4.2), we use our full model which includes multivariate and typed negative sampling. We discuss ablations in Section 4.3

DataWe use the January 2018 dump of English Wikipedia, containing 96M sentences to train pair2vec. We restrict the vocabulary to the 100K most frequent words. Preprocessing removes all out-of-vocabulary words in the corpus. We consider each word pair within a window of 5 in the preprocessed corpus, and subsample5 instances based on pair probability with a threshold of \(5\cdot 10^{-7}\). We define the context as one word each to the left and right, and all the words in between each pair, replacing both target words with placeholders \(X\) and \(Y\) (see Table 1). More details can be found in the supplementary material.

Footnote 5: Like in word2vec, subsampling reduces the size of the dataset and speeds up training. For this, we define the word pair probability as the product of unigram probabilities.

### Question Answering

We experiment on the SQuAD 2.0 QA benchmark Rajpurkar et al. (2018), as well as the adversarial datasets of SQuAD 1.1 Rajpurkar et al. (2016); Jia and Liang (2017). Table 3 shows the performance of BiDAF++, with ELMo , before and after adding pair2vec. Experiments on SQuAD 2.0 show that our pair representations improve performance by 2.72 F1. Moreover, adding pair2vec also results in better generalization on the adversarial SQuAD datasets with gains of 7.14 and 6.11 F1.

### Natural Language Inference

We report the performance of our model on MultiNLI and the adversarial test set from Glocnner et al. (2018) in Table 5. We outperform the

\begin{table}
\begin{tabular}{l c r r} \hline \hline Benchmark & ESIM & + pair2vec & \(\Delta\) \\ \hline Matched & 79.68 & 81.03 & +1.35 \\ Mismatched & 78.80 & 80.12 & +1.32 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on MultiNLI, with and without pair2vec. All models have ELMo.

 

ESIM + ELMo baseline by 1.3% on the matched and mismatched portions of the dataset.

We also record a gain of 8.8% absolute over ESIM on the Glockner et al. (2018) dataset, setting a new state of the art. Following standard practice Glockner et al. (2018), we train all models on a combination of SNLI Bowman et al. (2015) and MultiNLI Glockner et al. (2018) show that with the exception of KIM Chen et al. (2018), which uses WordNet features, several NLI models fail to generalize to this setting which involves lexical inference. For a fair comparison with KIM on the Glocker test set, we replace ELMo with GLoVE embeddings, and still outperform KIM by almost halving the error rate.

### Ablations

Ablating parts of pair2vec shows that all components of the model (Section 2) are useful. We ablate each component and report the EM and F1 on the development set of SQuAD 2.0 (Table 6). The full model, which uses a 4-layer MLP for \(R(x,y)\) and trains with multivariate negative sampling, achieves the highest F1 of 72.68.

We experiment with two alternative composition functions, a 2-layer MLP (_Composition: 2 Layers_) and element-wise multiplication (_Composition: Multiply_), which yield significantly smaller gains over the baseline BiDAF++ model. This demonstrates the need for a deep composition function. Eliminating sampling of target words \((x,y)\) from the objective (_Objective: Bivariate NS_) results in a drop of 0.7 F1, accounting for about a quarter of the overall gain. This suggests that while the bulk of the signal is mined from the pair-context interactions, there is also valuable information in other interactions as well.

We also test whether specific pre-training of word _pair_ representations is useful by replacing pair2vec embeddings with the vector offsets of pre-trained word embeddings (_Unsupervised: Pair Dist_). We follow the PairDistance method for word analogies Mikolov et al. (2013b), and represent the pair \((x,y)\) as the L2 normalized difference of single-word vectors: \(({\bf x}-{\bf y})/\|{\bf x}-{\bf y}\|\). We use the same fastText Bojanowski et al. (2017) word vectors with which we initialized pair2vec before training. We observe a gain of only 0.34 F1 over the baseline.

## 5 Analysis

In Section 4, we showed that pair2vec adds information complementary to single-word representations like ELMo. Here, we ask what this extra information is, and try to characterize which word relations are better captured by pair2vec. To that end, we evaluate performance on a word analogy dataset with over 40 different relation types (Section 5.1), and observe how pair2vec fills hand-crafted relation patterns (Section 5.2).

\begin{table}
\begin{tabular}{l l c} \hline Model & EM (\(\Delta\)) & F1 (\(\Delta\)) \\ \hline pair2vec (Full Model) & 69.20 & 72.68 \\ \hline Composition: 2 Layers & 68.35 (-0.85) & 71.65 (-1.03) \\ Composition: Multiply & 67.10 (-2.20) & 70.20 (-2.48) \\ Objective: Bivariate NS & 68.63 (-0.57) & 71.98 (-0.70) \\ Unsupervised: Pair Dist & 67.07 (-2.13) & 70.24 (-2.44) \\ \hline No pair2vec (BiDAF) & 66.66 (-2.54) & 69.90 (-2.78) \\ \hline \end{tabular}
\end{table}
Table 6: Ablations on the Squad 2.0 development set show that argument sampling as well as using a deeper composition function are useful.

Figure 2: Accuracy as a function of the interpolation parameter \(\alpha\) (see Eq. (16)). The \(\alpha\)=0 configuration relies only on fastText Bojanowski et al. (2017), while \(\alpha\)=1 reflects pair2vec.

 



### Quantitative Analysis: Word Analogies

Word Analogy DatasetGiven a word pair \((a,b)\) and word \(x\), the word analogy task involves predicting a word \(y\) such that \(a:b::x:y\). We use the Bigger Analogy Test Set BATS, Gladkova et al. (2016) which contains four groups of relations: encyclopedic semantics (e.g., person-profession as in _Einstein-physicist_), lexicographic semantics (e.g., antonymy as in _cheap-expensive_), derivational morphology (e.g., noun forms as in _oblige-obligation_), and inflectional morphology (e.g., noun-plural as in _bird-birds_). Each group contains 10 sub-relations.

MethodWe interpolate pair2vec and 3CosAdd Mikolov et al. (2013); Levy et al. (2014) scores on fastText embeddings, as follows:

score \[(y)=\alpha\cdot\text{cos}(\mathbf{r}_{a,b},\mathbf{r}_{x,y})\\ +(1-\alpha)\cdot\text{cos}(\mathbf{b}-\mathbf{a}+\mathbf{x}, \mathbf{y})\] (16)

where **a**, **b**, **x**, and **y** represent fastText embeddings6 and \(\mathbf{r}_{a,b}\), \(\mathbf{r}_{x,y}\) represent the pair2vec embedding for the word pairs \((a,b)\) and \((x,y)\), respectively; \(\alpha\) is the linear interpolation parameter. Following prior work Mikolov et al. (2013), we return the highest-scoring \(y\) in the entire vocabulary, excluding the given words \(a\), \(b\), and \(x\).

Footnote 6: The fastText embeddings in the analysis were retrained using the same Wikipedia corpus used to train pair2vec to control for the corpus when comparing the two methods.

ResultsFigure 2 shows how the accuracy on each category of relations varies with \(\alpha\). For all four groups, adding pair2vec to 3CosAdd results in significant gains. In particular, the biggest relative improvements are observed for encyclopedic (356%) and lexicographic (51%) relations.

Table 7 shows the specific relations in which pair2vec made the largest absolute impact. The gains are particularly significant for relations where fastText embeddings provide limited signal. For example, the accuracy for _substance meronyms_ goes from 3.8% to 14.5%. In some cases, there is also a synergistic effect; for instance, in _noun+less_, pair2vec alone scored 0% accuracy, but mixing it with 3CosAdd, which got 4.8% on its own, yielded 16% accuracy.

These results, alongside our experiments in Section 4, strongly suggest that pair2vec encodes information complementary to that in single-word embedding methods such as fastText and ELMo.

### Qualitative Analysis: Slot Filling

To further explore how pair2vec encodes such complementary information, we consider a setting similar to that of knowledge base completion: given a Hearst-like context pattern \(c\) and a single word \(x\), predict the other word \(y\) from the entire vocabulary. We rank candidate words \(y\) based on the scoring function in our training objective: \(R(x,y)\cdot C(c)\). We use a fixed set of example relations and manually define their predictive context patterns and a small set of candidate words \(x\).

Table 8 shows the top three \(y\) words. The model embeds \((x,y)\) pairs close to contexts that reflect their relationship. For example, substituting _Portland_ in the city-state pattern ("_in X, Y._"), the top two words are _Oregon_ and _Maine_, both US states with cities named Portland. When used with the city-city pattern ("_from X to Y._"), the top two words are _Salem_ and _Astoria_, both cities in Oregon. The word-context interaction often captures multiple relations; for example, _Monet_ is used to refer to the painter (_professional_) as well as his paintings.

As intended, pair2vec captures the three-way word-word-context interaction, and not just the two-way word-context interaction (as in single-word embeddings). This profound difference allows pair2vec to complement single-word embeddings with additional information.

## 6 Related Work

Pretrained Word EmbeddingsMany state-of-the-art models initialize their word representations using pretrained embeddings such as word2vec Mikolov et al. (2013) or ELMo Peters et al. (2018). These representations are typically trained using an interpretation of the Distributional Hy

\begin{table}
\begin{tabular}{l c r r} \hline Relation & 3CosAdd & +pair2vec & \(\alpha^{*}\) \\ \hline Country:Capital Name:Occupation & 1.2 & 86.1 & 0.9 \\ Name:Occupation & 1.8 & 44.6 & 0.8 \\ Name:Nationality & 0.1 & 42.0 & 0.9 \\ UK City:Country & 0.7 & 31.7 & 1.0 \\ Country:Language & 4.0 & 28.4 & 0.8 \\ Verh 3pSp:Ved & 49.1 & 61.7 & 0.6 \\ Verb Ying:Ved & 61.1 & 73.3 & 0.5 \\ Verb Inf:Yed & 58.5 & 70.1 & 0.5 \\ Noun+less & 4.8 & 16.0 & 0.2 \\ Substance Meronym & 3.8 & 14.5 & 0.6 \\ \hline \end{tabular}
\end{table}
Table 7: The top 10 analogy relations for which interpolating with pair2vec improves performance. \(\alpha^{*}\) is the optimal interpolation parameter for each relation.



pothesis (Harris, 1954) in which the bivariate distribution of target words and contexts is modeled. Our work deviates from the word embedding literature in two major aspects. First, our goal is to represent word _pairs_, not individual words. Second, our new PMI formulation models the _trivariate_ word-word-context distribution. Experiments show that our pair embeddings can complement single-word embeddings.

Mining Textual PatternsThere is extensive literature on mining textual patterns to predict relations between words (Hearst, 1992; Snow et al., 2005; Turney, 2005; Riedel et al., 2013; Van de Cruys, 2014; Toutanova et al., 2015; Shwartz and Dagan, 2016). These approaches focus mostly on relations between pairs of nouns (perhaps with the exception of VerbOcean Chklovski and Pantel, 2004)). More recently, they have been expanded to predict relations between unrestricted pairs of words (Jameel et al., 2018; Espinosa Anke and Schockaert, 2018), assuming that each word-pair was observed together during pretraining. Washio and Kato (2018, 2018) relax this assumption with a compositional model that can represent any pair, as long as each word appeared (individually) in the corpus.

These methods are evaluated on either intrinsic relation prediction tasks, such as BLESS Baroni and Lenci, 2011) and CogALex Santus et al., 2016), or knowledge-base population benchmarks, e.g. FB15 (Bordes et al., 2013). To the best of our knowledge, our work is the first to integrate pattern-based methods into modern high-performing semantic models and evaluate their impact on complex end-tasks like QA and NLI.

Integrating Knowledge in Complex Models Ahn et al. (2016) integrate Freebase facts into a language model using a copying mechanism over fact attributes. Yang and Mitchell (2017) modify the LSTM cell to incorporate WordNet and NELL knowledge for event and entity extraction. For cross-sentence inference tasks, Weissenborn et al. (2017), Bauer et al. (2018), and Mihaylov and Frank (2018) dynamically refine word representations by reading assertions from ConceptNet and Wikipedia abstracts. Our approach, on the other hand, relies on a relatively simple extension of existing cross-sentence inference models. Furthermore, we do not need to dynamically retrieve and process knowledge base facts or Wikipedia texts, and just pretrain our pair vectors in advance.

KIM Chen et al. (2017) integrates word-pair vectors into the ESIM model for NLI in a very similar way to ours. However, KIM's word-pair vectors contain only hand-engineered word-relation indicators from WordNet, whereas our word-pair vectors are automatically learned from unlabeled text. Our vectors can therefore reflect relation types that do not exist in WordNet (such as _profession_) as well as word pairs that do not have a direct link in WordNet (e.g. _bronze_ and _statue_); see Table 8 for additional examples.

\begin{table}
\begin{tabular}{c c l l} \hline \hline
**Relation** & **Context** & **X** & **Y (Top 3)** \\ \hline Antonymy/Exclusion & either X or Y & \begin{tabular}{l} accept \\ hard \\ \end{tabular} & 
\begin{tabular}{l} _reject_, _refuse_, recognise \\ _soft_, _brittle_, _polished_ \\ \end{tabular} \\ \hline Hypermyr & including X and other Y & copper & ones, _metals_, _mines_ \\  & & google & apps, _brousers_, _searches_ \\ \hline Hyponymy & X like Y & cities & solaris, _speyer_, _medina_ \\  & & browsers & _chrome_, _fracfrox_, _netscape_ \\ \hline Co-hyponymy & , X , Y , & copper & _malachite_, _flint_, ivory \\  & & google & _microsoft_, _bing_, _yahoo_ \\ \hline City-State & in X , Y . & portland & _oregon_, _maine_, _dorset_ \\  & & dallas & _tx_, _texas_, va \\ \hline City-City & from X to Y . & portland & _salem_, _astoria_, oggensburg \\  & & dallas & _denton_, _allatoona_, _addison_ \\ \hline Profession & X , a famous Y , & ronaldo & _footballer_, Portuguese, _player_ \\  & & monet & _painter_, painting, butterfly \\ \hline \hline \end{tabular}
\end{table}
Table 8: Given a context \(c\) and a word \(x\), we select the top 3 words \(y\) from the entire vocabulary using our scoring function \(R(x,y)\cdot C(c)\). The analysis suggests that the model tends to rank correct matches (italics) over others.

 

Conclusion and Future Work

We presented new methods for training and using word _pair_ embeddings that implicitly represent background knowledge. Our pair embeddings are computed as a compositional function of the individual word representations, which is learned by maximizing a variant of the PMI with the contexts in which the the two words co-occur. Experiments on cross-sentence inference benchmarks demonstrated that adding these representations to existing models results in sizable improvements for both in-domain and adversarial settings.

Published concurrently with this paper, BERT Devlin et al. (2018), which uses a masked language model objective, has reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition. Potential avenues for future work include multitasking BERT with pair2vec in order to more directly incorporate reasoning about word pair relations into the BERT objective.

## Acknowledgments

We would like to thank Anna Rogers Gladkova), Qian Chen, Koki Washio, Pranav Rajpurkar, and Robin Jia for their help with the evaluation. We are also grateful to members of the UW and FAIR NLP groups, and anonymous reviewers for their thoughtful comments and suggestions.

## References

* Ahn et al. (2016) Sungjin Ahn, Heeyoul Choi, Tanel Paramana, and Yoshua Bengio. 2016. A neural knowledge language model. _arXiv preprint arXiv:1608.00318_.
* Baroni and Lenci (2011) Marco Baroni and Alessandro Lenci. 2011. How we blessed distributional semantic evaluation. In _Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics_, GEMS '11, pages 1-10, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Bauer et al. (2018) Lisa Bauer, Yicheng Wang, and Mohit Bansal. 2018. Commonsense for generative multi-hop question answering tasks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4220-4230, Brussels, Belgium. Association for Computational Linguistics.
* Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. _Transactions of the Association for Computational Linguistics_, 5:135-146.
* Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multirelational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 26_, pages 2787-2795. Curran Associates, Inc.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642. Association for Computational Linguistics.
* Chen et al. (2018) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018. Neural natural language inference models enhanced with external knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2406-2417. Association for Computational Linguistics.
* Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1657-1668. Association for Computational Linguistics.
* Chklovski and Pantel (2004) Timothy Chklovski and Patrick Pantel. 2004. Verbocean: Mining the web for fine-grained semantic verb relations. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_.
* Clark and Gardner (2018) Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 845-855. Association for Computational Linguistics.
* Van de Cruys (2011) Tim Van de Cruys. 2011. Two multivariate generalizations of pointwise mutual information. In _Proceedings of the Workshop on Distributional Semantics and Compositionality_, pages 16-20. Association for Computational Linguistics.
* Van de Cruys (2014) Tim Van de Cruys. 2014. A neural network approach to selectional preference acquisition. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 26-35. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Anke and Schockaert (2018) Luis Espinosa Anke and Steven Schockaert. 2018. Seven: Augmenting word embeddings with unsupervised relation vectors. In _Proceedings of the 

27th International Conference on Computational Linguistics_, pages 2653-2665. Association for Computational Linguistics.
* Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. _arXiv preprint arXiv:1803.07640_.
* Gladkova et al. (2016) Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: What works and what doesn't. In _Proceedings of the NAACL-HLT SRW_, pages 47-54, San Diego, California, June 12-17, 2016. ACL.
* Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 650-655. Association for Computational Linguistics.
* Harris (1954) Zellig Harris. 1954. Distributional structure. _Word_, 10(23):146-162.
* Hearst (1992) Marti Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In _Proceedings of the 14th International Conference on Computational Linguistics_, Nantes, France.
* Jameel et al. (2018) Shoaib Janeel, Zied Bouraoui, and Steven Schockaert. 2018. Unsupervised learning of distributional relation vectors. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 23-33. Association for Computational Linguistics.
* Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2021-2031. Association for Computational Linguistics.
* Levy et al. (2014) Omer Levy, Ido Dagan, and Jacob Goldberger. 2014. Focused entailment graphs for open ie propositions. In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning_, pages 87-97. Association for Computational Linguistics.
* Volume 2_, NIPS'14, pages 2177-2185, Cambridge, MA, USA. MIT Press.
* Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017_, pages 333-342.
* Li et al. (2017) Bofang Li, Tao Liu, Zhe Zhao, buzhou Tang, Aleksandr Drozd, Anna Rogers, and Xiaoyong Du. 2017. Investigating different syntactic context types and context representations for learning word embeddings. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2421-2431. Association for Computational Linguistics.
* Mihaylov and Frank (2018) Todor Mihaylov and Anette Frank. 2018. Knowledge-able reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 821-832. Association for Computational Linguistics.
* Mikolov et al. (2013a) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States._, pages 3111-3119.
* Mikolov et al. (2013b) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 746-751. Association for Computational Linguistics.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237. Association for Computational Linguistics.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789. Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Benjamin M. Martin, and Andrew McCallum. 2013. Relation extraction with matrix factorization and universal schemas. In _Joint Human Language Technology Conference/Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLTNAACL)_ 

Enrico Santus, Anna Gladkova, Stefan Evert, and Alessandro Lenci. 2016. The CogALex-V shared task on the corpus-based identification of semantic relations. In _Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon (CogALex - V)_, pages 69-79. The COLING 2016 Organizing Committee.
* Seo et al. (2017) Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishrizi. 2017. Bidirectional attention flow for machine comprehension. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Shwartz and Dagan (2016) Vered Shwartz and Ido Dagan. 2016. Path-based vs. distributional information in recognizing lexical semantic relations. In _Proceedings of the 5th Workshop on Cognitive Aspects of the Lexicon, CoagALex@COLING 2016, Osaka, Japan, December 12, 2016_, pages 24-29.
* Shwartz et al. (2016) Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016. Improving hypernymy detection with an integrated path-based and distributional method. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2389-2398. Association for Computational Linguistics.
* Snow et al. (2005) Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In L. K. Saul, Y. Weiss, and L. Bottou, editors, _Advances in Neural Information Processing Systems 17_, pages 1297-1304. MIT Press.
* Toutanova et al. (2015) Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1499-1509.
* Turney (2005) Peter D. Turney. 2005. Measuring semantic similarity by latent relational analysis. In _Proceedings of the 19th International Joint Conference on Artificial Intelligence_, IJCAI'05, pages 1136-1141, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Washio and Kato (2018a) Koski Washio and Tsuneaki Kato. 2018a. Filling missing paths: Modeling co-occurrences of word pairs and dependency paths for recognizing lexical semantic relations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1123-1133. Association for Computational Linguistics.
* Washio and Kato (2018b) Koki Washio and Tsunaeaki Kato. 2018b. Neural latent relational analysis to capture lexical semantic relations in a vector space. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018_. Association for Computational Linguistics.
* Weissenborn et al. (2017) Dirk Weissenborn, Tomas Kocisky, and Chris Dyer. 2017. Dynamic integration of background knowledge in neural NLU systems. _arXiv preprint arXiv:1706.02596_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Yang and Mitchell (2017) Bishan Yang and Tom Mitchell. 2017. Leveraging knowledge bases in LSTMs for improving machine reading. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1436-1446. Association for Computational Linguistics.

## Appendix A Implementation Details

HyperparametersFor both word pairs and contexts, we use 300-dimensional word embeddings initialized with FastText Bojanowski et al. (2017). The context representation uses a single-layer BiLSTM with a hidden layer size of 100. We use 2 negative context samples and 3 negative argument samples for each pair-context tuple.

For pre-training, we used stochastic gradient descent with an initial learning rate of 0.01. We reduce the learning rate by a factor of 0.9 if the loss does not decrease for 300K steps. We use a batch size of 600, and train for 12 epochs.7

Footnote 7: On Titan X GPUs, the training takes about a week.

For both end-task models, we use AllenNLP's implementations Gardner et al. (2018) with default hyperparameters; we did not change any setting before or after injecting pair2vec. We use 0.15 dropout on our pretrained pair embeddings.

## Appendix B Multivariate Negative Sampling

### Relation to Multivariate PMI

In this appendix, we elaborate on mathematical details of multivariate negative sampling to support our claims in Section 2.2, and also discuss its relation to other PMI multivariate formulations (Table 9).

### Global Objective

Equation (Table 2, \(J_{3NS}\)) in Section 2.2 characterizes the local objective for each data instance. To understand the mathematical properties of this objective, we must first describe the global objective in terms of the entire dataset. However, this 

cannot be done by simply summing the local objective for each \((x,y,c)\), since each such example may appear multiple times in our dataset. Moreover, due to the nature of negative sampling, the number of times an \((x,y,c)\) triplet appears as a positive example will almost always be different from the number of times it appears as a negative one. Therefore, we must determine the frequency in which each triplet appears in each role.

We first denote the number of times the example \((x,y,c)\) appears in the dataset as \(\#(x,y,c)\); this is also the number of times \((x,y,c)\) is used as a positive example. We observe that the expected number of times \((x,y,c)\) is used as a corrupt \(x\) example is \(k_{x}P(x,\cdot,\cdot)\#(\cdot,y,c)\), since \((x,y,c)\) can only be created as a corrupt \(x\) example by randomly sampling \(x\) from an example that already contained \(y\) and \(c\). The number of times \((x,y,c)\) is used as a corrupt \(y\) or \(c\) example can be derived analogously. Therefore, the global objective of our ternary negative sampling approach is:

\[J^{\text{Global}}_{3NS}= \sum_{(x,y,c)}J^{x,y,c}_{3NS} \tag{17}\] \[J^{x,y,c}_{3NS}= \#(x,y,c)\cdot\log\sigma\left(S_{x,y,c}\right)\] \[+Z^{\prime}_{x,y,c}\cdot\log\sigma\left(-S_{x,y,c}\right)\] (18) \[Z^{\prime}_{x,y,c}= k_{x}P(x,\cdot,\cdot)\#(\cdot,y,c)J^{x,y,c}_{-}\] \[+k_{y}P(\cdot,y,\cdot)\#(x,\cdot,c)J^{x,y,c}_{-}\] \[+k_{c}P(\cdot,\cdot,c)\#(x,y,\cdot)J^{x,y,c}_{-}\] (19) \[S_{x,y,c}= R(x,y)\cdot C(c) \tag{20}\]

With the global objective, we can now ask what is the optimal value of \(S_{x,y,c}\) (20) by comparing the partial derivative of (17) to zero. This derivative is in fact equal to the partial derivative of (18), since it is the only component of the global objective in which \(R(x,y)\cdot C(c)\) appears:

\[0=\frac{\partial J^{\text{Global}}_{3NS}}{\partial S_{x,y,c}}=\frac{\partial J ^{x,y,c}_{3NS}}{\partial S_{x,y,c}}\]

The partial derivative of (18) can be expressed as:

\[0=\#(x,y,c)\cdot\sigma\left(-S_{x,y,c}\right)-Z^{\prime}_{x,y,c}\cdot\sigma \left(S_{x,y,c}\right)\]

which can be reformulated as:

\[S_{x,y,c}=\log\frac{\#(x,y,c)}{Z^{\prime}_{x,y,c}}\]

By expanding the fraction by \(1/\#(\cdot,\cdot,\cdot)\) (i.e. dividing by the size of the dataset), we essentially convert all the frequency counts (e.g. \(\#(x,y,z)\)) to empirical probabilities (e.g. \(P(x,y,z)\)), and arrive at Equation (1) in Section 2.2.

### Other Multivariate PMI Formulations

Previous work has proposed different multivariate formulations of PMI, shown in Table 9. Van de Cruys (2011) presented specific interaction information (\(SI_{1}\)) and specific correlation (\(SI_{2}\)). In addition to those metrics, Jameel et al. (2018) experimented with \(SI_{3}\), which is the bivariate PMI between \((x,y)\) and \(c\), and with \(SI_{4}\). Our formulation deviates from previous work, and, to the best of our knowledge, cannot be trivially expressed by one of the existing metrics.

\begin{table}
\begin{tabular}{l l c} \hline \hline  & \(SI_{1}(x,y,c)\) & \(\log\frac{P(x,y,c)P(y,c)P(\cdot,y,c)}{p(x,\cdot,\cdot)P(\cdot,y,c)P(\cdot,c)P( x,y,c)}\) \\ (Van de Cruys, 2011) & \(SI_{2}(x,y,c)\) & \(\log\frac{P(x,y,c)}{p(x,\cdot,\cdot)P(\cdot,y,c)P(\cdot,y,c)}\) \\  & \(SI_{3}(x,y,c)\) & \(\log\frac{P(x,y,c)}{p(x,\cdot,\cdot)P(\cdot,y,c)P(\cdot,c)}\) \\ \hline   (Jameel et al., 2018) & \(SI_{3}(x,y,c)\) & \(\log\frac{P(x,y,c)}{p(x,\cdot,\cdot)P(\cdot,c)P(\cdot,c)}\) \\  & \(SI_{4}(x,y,c)\) & \(\log\frac{P(x,y,c)P(\cdot,c)}{p(x,\cdot,c)P(\cdot,y,c)}\) \\ \hline (This Work) & \(MVPMI(x,y,c)\) & \(\log\frac{P(x,y,c)}{k_{x}P(\cdot,\cdot,c)P(x,y,\cdot)+k_{x}P(x,\cdot,\cdot)P( \cdot,y,c)+k_{y}P(\cdot,\cdot,\cdot)P(x,\cdot,c)}\) \\ \hline \hline \end{tabular}
\end{table}
Table 9: Multivariate generalizations of PMI.