Venue: NAACL 2016

# Embedding Lexical Features via Low-Rank Tensors

 Mo Yu\({}^{*}\)

Harbin Institute of Technology

IBM Watson

yum@us.ibm.com &Mark Dredze

HLTCOE

Johns Hopkins University

mdredze@cs.jhu.edu &Raman Arora

Johns Hopkins University

arora@cs.jhu.edu &Matthew R. Gormley

Carnegie Mellon University

mgormley@cs.cmu.edu

###### Abstract

Modern NLP models rely heavily on engineered features, which often combine word and contextual information into complex lexical features. Such combination results in large numbers of features, which can lead to overfitting. We present a new model that represents complex lexical features--comprised of parts for words, contextual information and labels--in a tensor that captures conjunction information among these parts. We apply low-rank tensor approximations to the corresponding parameter tensors to reduce the parameter space and improve prediction speed. Furthermore, we investigate two methods for handling features that include \(n\)-grams of mixed lengths. Our model achieves state-of-the-art results on tasks in relation extraction, PP-attachment, and preposition disambiguation.

+
Footnote †: Paper submitted during Mo Yu’s PhD study at HIT.

## 1 Introduction

Statistical NLP models usually rely on hand-designed features, customized for each task. These features typically combine lexical and contextual information with the label to be scored. In relation extraction, for example, there is a parameter for the presence of a specific relation occurring with a feature conjoining a word type (lexical) with dependency path information (contextual). In measuring phrase semantic similarity, a word type is conjoined with its position in the phrase to signal its role. Figure 0(b) shows an example in dependency parsing, where multiple types (words) are conjoined with POS tags or distance information.

To avoid model over-fitting that often results from features with lexical components, several smoothed lexical representations have been proposed and shown to improve performance on various NLP tasks; for instance, word embeddings Bengio et al. (2006) help improve NER, dependency parsing and semantic role labeling Miller et al. (2004); Koo et al. (2008); Turian et al. (2010); Sun et al. (2011); Roth and Woodsend (2014); Hermann et al. (2014).

However, using only word embeddings is not sufficient to represent complex lexical features (e.g. \(\phi\) in Figure 0(c)). In these features, the same word embedding conjoined with different non-lexical properties may result in features indicating different labels; the corresponding lexical feature representations should take the above interactions into consideration. Such important interactions also increase the risk of over-fitting as feature space grows exponentially, yet how to capture these interactions in representation learning remains an open question.

To address the above problems,1 we propose a general and unified approach to reduce the feature space by constructing low-dimensional feature representations, which provides a new way of combining word embeddings, traditional non-lexical properties, and label information. Our model exploits the inner structure of features by breaking the feature into multiple parts: lexical, non-lexical and (optional) label. We demonstrate that the full feature is an outer product among these parts. Thus, a parameter tensor scores each feature to produce a prediction. Our model then reduces the number of param

eters by approximating the parameter tensor with a low-rank tensor: the Tucker approximation of Yu et al. (2015) but applied to each embedding type (view), or the Canonical/Parallel-Factors Decomposition (CP). Our models use fewer parameters than previous work that learns a separate representation for each feature Ando and Zhang (2005); Yang and Eisenstein (2015). CP approximation also allows for much faster prediction, going from a method that is cubic in rank and exponential in the number of lexical parts, to a method linear in both. Furthermore, we consider two methods for handling features that rely on \(n\)-grams of mixed lengths.

Our model makes the following contributions when contrasted with prior work:

Lei et al. (2014) applied CP to combine different views of features. Compared to their work, our usage of CP-decomposition is different in the application to feature learning: (1) We focus on dimensionality reduction of existing, well-verified features, while Lei et al. (2014) generates new features (usually different from ours) by combining some "atom" features. Thus their work may ignore some useful features; it relies on binary features as supplementary but our model needs not. (2) Lei et al. (2014)'s factorization relies on views with explicit meanings, e.g. head/modifier/arc in dependency parsing, making it less general. Therefore its applications to tasks like relation extraction are less obvious.

Compared to our previous work Gormley et al. (2015); Yu et al. (2015), this work allows for higher-order interactions, mixed-length n-gram features, lower-rank representations. We also demonstrate the strength of our new model via applications to new tasks.

The resulting method learns smoothed feature representations combining lexical, non-lexical and label information, achieving state-of-the-art performance on several tasks: relation extraction, preposition semantics and PP-attachment.

## 2 Notation and Definitions

We begin with some background on notation and definitions. Let \({\mathcal{T}}\in\mathbb{R}^{d_{1}\times\cdots\times d_{K}}\) be a \(K\)-way tensor (i.e., a tensor with \(K\) views). In this paper, we consider the tensor \(k\)-mode product, i.e. multiplying a tensor \({\mathcal{T}}\in\mathbb{R}^{d_{1}\times\cdots\times d_{K}}\) by a matrix \({\mathbf{x}}\in\mathbb{R}^{d_{k}\times J}\) (or a vector if \(J=1\)) in mode (view) \(k\). The product is denoted by \({\mathcal{T}}\times_{k}{\mathbf{x}}\) and is of size \(d_{1}\times\cdots\times d_{k-1}\times J\times d_{k+1}\times\cdots\times d_{K}\). Elementwise, we have

\[({\mathcal{T}}\times_{k}{\mathbf{x}})_{i_{1}...i_{k-1}~{}j~{}i_{k+1}...i_{K}} =\sum_{i_{k}=1}^{d_{k}}{\mathcal{T}}_{i_{1}...i_{k}...i_{K}}{\mathbf{x}}_{i_{ k}j},\]

for \(j=1,\ldots,J\). A mode-\(k\) fiber \({\mathcal{T}}_{i_{1}...i_{k-1}\bullet i_{k+1}...i_{K}}\) of \({\mathcal{T}}\) is the \(d_{k}\) dimensional vector obtained by fixing all but the \(k\)th index. The mode-\(k\) unfolding \({\mathcal{T}}_{(k)}\) of \({\mathcal{T}}\) is the \(d_{k}\times\prod_{i\neq k}d_{i}\) matrix obtained by concatenating all the \(\prod_{i\neq k}d_{i}\) mode-\(k\) fibers along columns.

Given two matrices \(\text{W}_{\!1}\in\mathbb{R}^{d_{1}\times r_{1}},\text{W}_{\!2}\in\mathbb{R}^{d _{2}\times r_{2}}\), we write \(\text{W}_{\!1}\otimes\text{W}_{\!2}\) to denote the Kronecker product between W\({}_{\!1}\) and W\({}_{\!2}\) (outer product for vectors). We define the Frobenius product (matrix dot product) \({\mathbf{A}}\odot{\mathbf{B}}=\sum_{i,j}A_{ij}B_{ij}\) between two matrices with

Figure 1: An example of lexical features used in dependency parsing. To predict the “PMOD” arc (the dashed one) between “see” and “with” in (a), we may rely on lexical features in (b). Here \(p\), \(c\), \(g\) are indices of the word “with”, its child (“telescope”) and a candidate head. Figure (c) shows what the fifth feature (\(\phi\)) is like, when the candidate is “see”. As is common in multi-class classification tasks, each template generates a different feature for each label y. Thus a feature \(\phi=\text{w}_{g}\wedge\text{w}_{c}\land\text{u}\land\text{y}\) is the conjunction of the four parts. Figure (d) is the one-hot representation of \(\phi\), which is equivalent to the outer product (i.e. a 4-way tensor) among the four one-hot vectors. \({\mathbf{v}}(x)=1\) means the vector \({\mathbf{v}}\) has a single non-zero element in the \(x\) position.

 

the same sizes; and define element-wise (Hadamard) multiplication \({\bf a}\circ{\bf b}\) between vectors with the same sizes.

Tucker Decomposition:Tucker Decomposition represents a \(d_{1}\times d_{2}\times\ldots\times d_{K}\) tensor \({\cal T}\) as:

\[{\cal T}=g\times_{1}\mbox{W}_{1}\times_{2}\mbox{W}_{2}\ldots\times_{K}\mbox{ W}_{K} \tag{1}\]

where each \(\times_{i}\) is the tensor \(i\)-mode product and each W\({}_{i}\) is a \(r_{i}\times d_{i}\) matrix. Tensor \(g\) with size \(r_{1}\times r_{2}\times\ldots\times r_{K}\) is called the _core tensor_. We say that \({\cal T}\) has a Tucker rank \((r_{(1)},r_{(2)},\ldots,r_{(K)})\), where \(r_{(i)}=\text{rank}({\cal T}_{(i)})\) is the rank of mode-\(i\) unfolding. To simplify learning, we define the Tucker rank as \(r_{(i)}\)=rank(**g\({}_{(i)}\)**), which can be bounded simply by the dimensions of \(g\), i.e. \(r_{(i)}\leq r_{i}\); this allows us to enforce a rank constraint on \({\cal T}\) simply by restricting the dimensions \(r_{i}\) of \(g\), as described in SS6.

CP Decomposition:CP decomposition represents a \(d_{1}\!\times\!d_{2}\!\times\!\ldots\!\times\!d_{K}\) tensor \({\cal T}\) as a sum of rank-one tensors (i.e. a sum of outer products of \(K\) vectors):

\[{\cal T}=\sum_{j=1}^{r}\mbox{W}_{1}[j,:]\otimes\mbox{W}_{2}[j,:]\otimes\ldots \otimes\mbox{W}_{K}[j,:] \tag{2}\]

where each W\({}_{i}\) is an \(r\times d_{i}\) matrix and W\({}_{i}[j,:]\) is the vector of its \(j\)-th row. For CP decomposition, the rank \(r\) of a tensor \({\cal T}\) is defined to be the number of rank-one tensors in the decomposition. CP decomposition can be viewed as a special case of Tucker decomposition in which \(r_{1}=r_{2}=\ldots=r_{K}=r\) and \(g\) is a superdiagonal tensor.

## 3 Factorization of Lexical Features

Suppose we have feature \(\phi\) that includes information from a label y, multiple lexical items w\({}_{1},\ldots,\mbox{w}_{n}\) and non-lexical property u. This feature can be factorized as a conjunction of each part: \(\phi=\mbox{y}\wedge\mbox{u}\land\mbox{w}_{1}\land\ldots\wedge\mbox{w}_{n}\). The feature fires when all \((n+2)\) parts fire in the instance (reflected by the \(\land\) symbol in \(\phi\)). The one-hot representation of \(\phi\) can then be viewed as a tensor \(e_{\phi}=\mbox{y}\otimes\mbox{u}\otimes\mbox{w}_{1}\otimes\cdots\otimes\mbox{ w}_{n}\), where each feature part is also represented as a one-hot vector.2 Figure 0(b) illustrates this case with two lexical parts.

Footnote 2: \(\mbox{u},\mbox{y},\mbox{w}_{i}\) denote one-hot vectors instead of symbols.

Given an input instance \({\bf x}\) and its associated label y, we can extract a set of features \(S({\bf x},\mbox{y})\). In a traditional log-linear model, we view the instance \({\bf x}\) as a bag-of-features, i.e. a feature vector \(F({\bf x},\mbox{y})\). Each dimension corresponds to a feature \(\phi\), and has value 1 if \(\phi\in S({\bf x},\mbox{y})\). Then the log-linear model scores the instance as \(s({\bf x},\mbox{y};{\bf w})={\bf w}^{T}F({\bf x},\mbox{y})=\sum_{\phi\in S({ \bf x},\mbox{y})}s(\phi;{\bf w})\), where \({\bf w}\) is the parameter vector. We can re-write \(s({\bf x},\mbox{y};{\bf w})\) based on the factorization of the features using tensor multiplication; in which \({\bf w}\) becomes a parameter tensor \({\cal T}\):

\[s({\bf x},\mbox{y};{\bf w})=s({\bf x},\mbox{y};{\cal T})=\sum_{\phi\in S({\bf x },\mbox{y})}s(\phi;{\cal T}) \tag{3}\]

Here each \(\phi\) has the form \((\mbox{y},\mbox{u},\mbox{w}_{1},\ldots,\mbox{w}_{n})\), and

\[s(\phi;{\cal T})={\cal T}\times_{l}\mbox{y}\times_{f}\mbox{u}\times_{\mbox{w}_ {1}}\mbox{w}_{1}...\times_{\mbox{w}_{n}}\mbox{w}_{n}. \tag{4}\]

Note that one-hot vectors w\({}_{i}\) of words themselves are large (|w\({}_{i}|>500\)k), thus the above formulation with parameter tensor \({\cal T}\) can be very large, making parameter estimation difficult. Instead of estimating only the values of the dimensions which appear in training data as in traditional methods, we will reduce the size of tensor \({\cal T}\) via a low-rank approximation. With different approximation methods, (4) will have different equivalent forms, e.g. (6), (7) in SS4.1.

Optimization objective:The loss function \(\ell\) for training the log-linear model uses (3) for scores, e.g., the log-loss \(\ell({\bf x},\mbox{y};{\cal T})=-\log\frac{\exp\{s({\bf x},\mbox{y};{\cal T}) \}}{\sum_{y^{\prime}\in L}\exp\{s({\bf x},\mbox{y};{\cal T})\}}\). Learning can be formulated as the following optimization problem:

minimize: \[\sum_{({\bf x},\mbox{y})\in{\cal D}}\ell({\bf x},y;{\cal T})\] subject to: \[\left\{\begin{array}{c}\text{rank}({\cal T})\leq(r_{1},r_{2},...,r_{ n+2})\\ \qquad\qquad\qquad\qquad\quad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad  

to better model generalization. Note that there are examples where features have different numbers of multiple lexical parts, such as both unigram and bigram features in PP-attachment. We will use two different methods to handle these features (SS5).

**Remarks (advantages of our factorization)** Compared to prior work, e.g. Lei et al. (2014); Lei et al. (2015), the proposed factorization has the following advantages:

1. **Parameter explosion** when mapping a view with lexical properties to its representation vector (as will be discussed in 4.3): Our factorization allows the model to treat word embeddings as inputs to the views of lexical parts, dramatically reducing the parameters. Prior work cannot do this since its views are mixtures of lexical and non-lexical properties. Note that Lei et al. (2014) uses embeddings by _concatenating_ them to specific views, which increases dimensionality, but the improvement is limited.
2. **No weight-sharing** among conjunctions with same lexical property, like the child-word "word(\(c\))" and its conjunction with head-postag "word(\(c\)) \(\wedge\) word(\(g\))" in Figure 1(b). The factorization in prior work treats them as _independent_ features, greatly increasing the dimensionality. Our factorization builds representations of both features based on the embedding of "word(\(c\))", thus utilizing their connections and reducing the dimensionality.

The above advantages are also key to overcome the problems of prior work mentioned at the end of SS1.

## 4 Feature Representations via Low-rank Tensor Approximations

Using one-hot encodings for each of the parts of feature \(\phi\) results in a very large tensor. This section shows how to compute the score in (4) without constructing the full feature tensor using two tensor approximation methods (SS4.1 and SS4.2).

We begin with some intuition. To score the original (full rank) tensor representation of \(\phi\), we need a parameter tensor \({\cal T}\) of size \(d_{1}\times d_{2}\times\ldots\times d_{n+2}\), where \(d_{3}=\cdots=d_{n+2}=|V|\) is the vocabulary size, \(n\) is the number of lexical parts in the feature and \(d_{1}=|L|\) and \(d_{2}=|F|\) are the number of different labels and non-lexical properties, respectively. (SS5 will handle \(n\) varying across features.) Our methods reduce the tensor size by embedding each part of \(\phi\) into a lower dimensional space, where we represent each label, non-lexical property and words with an \(r_{1},r_{2}\), \(r_{3},\ldots,r_{n+2}\) dimensional vector respectively (\(r_{i}\ll d_{i}\), \(\forall i\)). These embedded features can then be scored by much smaller tensors. We denote the above transformations as matrices W\({}_{l}\in\mathbb{R}^{r_{1}\times d_{1}}\), W\({}_{f}\in\mathbb{R}^{r_{2}\times d_{2}}\), W\({}_{i}\in\mathbb{R}^{r_{i}+2\times d_{i+2}}\) for \(i=1,\ldots,n\), and write corresponding low-dimensional hidden representations as \({\bf h}_{y}^{(l)}={\rm W}_{l}{\bf y}\), \({\bf h}_{u}^{(f)}={\rm W}_{f}{\bf u}\) and \({\bf h}_{w}^{(i)}={\rm W}_{i}{\bf w}\).

In our methods, the above transformations of embeddings are _parts of low-rank tensors_ as in (5), so the embeddings of non-lexical properties and labels can be trained simultaneously with the low-rank tensors. Note that for one-hot input encodings the transformation matrices are essentially lookup tables, making the computation of these transformations sufficiently fast.

### Tucker Form

For our first approximation, we assume that tensor \({\cal T}\) has a low-rank Tucker decomposition: \({\cal T}=g\times_{l}\) W\({}_{l}\times_{f}\) W\({}_{f}\times_{{\rm w}_{1}}\) W\({}_{1}\times_{{\rm w}_{2}}\cdots\times_{{\rm w}_{n}}\) W\({}_{n}\). We can then express the scoring function (4) for a feature \(\phi=({\rm y},{\rm u},{\rm w}_{1},\ldots{\rm w}_{n})\) with \(n\)-lexical parts, as:

\[s({\rm y},{\rm u},{\rm w}_{1},\cdots,{\rm w}_{n};g,{\rm W}_{l}, {\rm W}_{f},\{{\rm W}_{i}\}_{i=1}^{n})\\ =g\times_{l}{\bf h}_{y}^{(l)}\times_{f}{\bf h}_{u}^{(f)}\times_{{ \rm w}_{1}}{\bf h}_{w_{1}}^{(1)}\cdots\times_{{\rm w}_{n}}{\bf h}_{w_{n}}^{( n)}, \tag{6}\]

which amounts to first projecting \({\rm u},{\rm y}\), and \({\rm w}_{i}\) (for all \(i\)) to lower dimensional vectors \({\bf h}_{u}^{(f)},{\bf h}_{y}^{(l)},{\bf h}_{w_{i}}^{(i)}\), and then weighting these hidden representations using the flattened core tensor \(g\). The low-dimensional representations and the corresponding weights are learned jointly using a discriminative (supervised) criterion. We call the model based on this representation the _Low-Rank Feature Representation with Tucker form_, or lrfr\({}_{n}\)-tucker.

### CP Form

For the Tucker approximation the number of parameters in (6) scale exponentially with the number of lexical parts. For instance, suppose each \({\bf h}_{w_{i}}^{(i)}\) has di 

mensionality \(r\), then \(|g|\propto r^{n}\). To address scalability and further control the complexity of our tensor based model, we approximate the parameter tensor using CP decomposition as in (2), resulting in the following scoring function:

\[s(\text{y},\text{u},\text{w}_{1},\cdots,\text{w}_{n};\text{W}_{l} ,\text{W}_{f},\{\text{W}_{i}\}_{i=1}^{n})=\\ \sum_{j=1}^{r}\left(\mathbf{h}_{\text{y}}^{(l)}\circ\mathbf{h}_{ \text{u}}^{(\mathsf{F})}\circ\mathbf{h}_{\text{w}_{1}}^{(1)}\circ\dots\circ \mathbf{h}_{\text{w}_{n}}^{(n)}\right)_{j}. \tag{7}\]

We call this model _Low-Rank Feature Representation with CP form_ (lrfr\({}_{n}\)-cp).

### Pre-trained Word Embeddings

One of the computational and statistical bottlenecks in learning these lrfr\({}_{n}\) models is the vocabulary size; the number of parameters to learn in each matrix W\({}_{i}\) scales linearly with \(|V|\) and would require very large sets of labeled training data. To alleviate this problem, we use pre-trained continuous word embeddings Mikolov et al. (2013) as input embeddings rather than the one-hot word encodings. We denote the \(m\)-dimensional word embeddings by \(e_{\text{w}}\); so the transformation matrices W\({}_{i}\) for the lexical parts are of size \(r_{i}\times m\) where \(m\ll|V|\).

We note that when sufficiently large labeled data is available, our model allows for fine-tuning the pre-trained word embeddings to improve the expressive strength of the model, as is common with deep network models.

RemarksOur lrfrs introduce embeddings for non-lexical properties and labels, making them better suit the common setting in NLP: rich linguistic properties; and large label sets such as open-domain tasks Hoffmann et al. (2010). The lrfr-cp better suits \(n\)-gram features, since when \(n\) increases 1, the only new parameters are the corresponding W\({}_{i}\). It is also very efficient during prediction (\(O(nr)\)), since the cost of transformations can be ignored with the help of look-up tables and pre-computing.

## 5 Learning Representations for \(n\)-gram Lexical Features of Mixed Lengths

For features with \(n\) lexical parts, we can train an lrfr\({}_{n}\) model to obtain their representations. However, we often have features of varying \(n\) (e.g. both unigrams (\(n\)=1) and bigrans (\(n\)=2) as in Figure 1). We require representations for features with arbitrary different \(n\) simultaneously.

We propose two solutions. The first is a straightforward solution based on our framework, which handles each \(n\) with a \((n+2)\)-way tensor. This strategy is commonly used in NLP, e.g. Taub-Tabib et al. (2015) have different kernel functions for different order of dependency features. The second is an approximation method which aims to use a single tensor to handle all \(n\)s.

Multiple Low-Rank TensorsSuppose that we can divide the feature set \(S(\textbf{x},\text{y})\) into subsets \(S_{1}(\textbf{x},\text{y}),S_{2}(\textbf{x},\text{y}),\ldots,S_{n}(\textbf{ x},\text{y})\) which correspond to features with one lexical part (unigram features), two lexical parts (bigram features), ... and \(n\) lexical parts (\(n\)-gram features), respectively. To handle these types of features, we modify the training objective as follows:

\[\underset{T_{1},T_{2},\cdots,T_{n}}{\text{minimize}}\sum_{(\textbf{x},\text{ y})\in D}\ell(\textbf{x},\text{y};\mathcal{T}_{1},\mathcal{T}_{2},\ldots,.. .\mathcal{T}_{n}), \tag{8}\]

where the score of a training instance \((\textbf{x},\text{y})\) is defined as \(s(\textbf{x},\text{y};\mathcal{T})=\sum_{i=1}^{n}\sum_{\phi\in S_{i}(\textbf{ x},\text{y})}s(\phi;\mathcal{T}_{i})\). We use the Tucker form low-rank tensor for \(\mathcal{T}_{1}\), and the CP form for \(\mathcal{T}_{i}~{}(\forall i>1)\). We refer to this method as lrfr\({}_{1}\)-tucker & lrfr\({}_{2}\)-cp.

Word ClustersAlternatively, to handle different numbers of lexical parts, we replace some lexical parts with discrete word clusters. Let \(c(\text{w})\) denote the word cluster (e.g. from Brown clustering) for word w. For bigran features we have:

\[s(\text{y},\text{u},\text{w}_{1},\text{w}_{2};\mathcal{T})\] \[=s(\text{y},\text{u}\wedge c(\text{w}_{1}),\text{w}_{2};\mathcal{ T})+s(\text{y},\text{u}\wedge c(\text{w}_{2}),\text{w}_{1};\mathcal{T})\] \[=\mathcal{T}\times_{l}\text{y}\times_{f}(\text{u}\wedge c(\text{ w}_{1}))\times_{\text{w}}e_{\text{w}_{2}}\] \[\qquad\quad+\mathcal{T}\times_{l}\text{y}\times_{f}(\text{u} \wedge c(\text{w}_{2}))\times_{\text{w}}e_{\text{w}_{1}} \tag{9}\]

where for each word we have introduced an additional set of non-lexical properties that are conjunctions of word clusters and the original non-lexical properties. This allows us to reduce an \(n\)-gram feature representation to a unigram representation. The advantage of this method is that it uses a single low-rank tensor to score features with different numbers of lexical parts. This is particularly helpful when we have very limited labeled data. We denote this method as lrfr\({}_{1}\)-Brown, since we use Brown clusters in practice. In the experiments we use the 

Tucker form for lrfr\({}_{1}\)-Brown.

## 6 Parameter Estimation

The goal of learning is to find a tensor \(\mathcal{T}\) that solves problem (5). Note that this is a non-convex objective, so compared to the convex objective in a traditional log-linear model, we are trading better feature representations with the cost of a harder optimization problem. While stochastic gradient descent (SGD) is a natural choice for learning representations in large data settings, problem (5) involves rank constraints, which require an expensive proximal operation to enforce the constraints at each iteration of SGD. We seek a more efficient learning algorithm. Note that we fixed the size of each transformation matrix W\({}_{i}\in\mathbb{R}^{r_{i}\times d_{i}}\) so that the smaller dimension (\(r_{i}<d_{i}\)) matches the upper bound on the rank. Therefore, the rank constants are always satisfied through a run of SGD and we in essence have an unconstrained optimization problem. Note that in this way we do not guarantee orthogonality and full-rank of the learned transformation matrices. These properties are assumed in general, but are not necessary according to Kolda and Bader (2009).

The gradients are computed via the chain-rule. We use AdaGrad Duchi et al. (2011) and apply L2 regularization on all W\({}_{i}\)s and \(g\), except for the case of \(r_{i}\)=\(d_{i}\), where we will start with W\({}_{i}=\mathbf{I}\) and regularize with \(\|\text{W}_{i}\) - \(\mathbf{I}\|_{2}\). We use early-stopping on a development set.

## 7 Experimental Settings

We evaluate lrfr on three tasks: relation extraction, PP attachment and preposition disambiguation (see Table 1 for a task summary). We include detailed feature templates in Table 2.

PP-attachment and relation extraction are two fundamental NLP tasks, and we test our models on the largest English data sets. The preposition disambiguation task was designed for compositional semantics, which is an important application of deep learning and distributed representations. On all these tasks, we compare to the state-of-the-art.

We use the same word embeddings in Belinkov et al. (2014) on PP-attachment for a fair comparison. For the other experiments, we use the same 200-\(d\) word embeddings in Yu et al. (2015).

Relation ExtractionWe use the English portion of the ACE 2005 relation extraction dataset Walker et al. (2006). Following Yu et al. (2015), we use both gold entity spans and types, train the model on the news domain and test on the broadcast conversation domain. To highlight the impact of training data size we evaluate with all 43,518 relations (entity mention pairs) and a reduced training set of the first 10,000 relations. We report precision, recall, and F1.

We compare to two baseline methods: 1) a log-linear model with a rich binary feature set from Sun et al. (2011) and Zhou et al. (2005) as described in Yu et al. (2015)Baseline; 2) the embedding model fcm of Gormley et al. (2015), which uses rich linguistic features for relation extraction. We use the same feature templates and evaluate on fine-grained relations (sub-types, 32 labels) Yu et al. (2015). This will evaluate how lmfr can utilize non-lexical linguistic features.

PP-attachmentWe consider the prepositional phrase (PP) attachment task of Belinkov et al. (2014),3 where for each PP the correct head (verbs or nouns) must be selected from content words before the PP (within a 10-word window). We formulate the task as a ranking problem, where we optimize the score of the correct head from a list of candidates with varying sizes.

Footnote 3: [http://groups.csail.mit.edu/rbg/code/pp](http://groups.csail.mit.edu/rbg/code/pp)

PP-attachment suffers from data sparsity because of bi-lexical features, which we will model with methods in SS5. Belikov et al. show that rich features - POS, WordNet and VerbNet - help this task. The combination of these features give a large number of non-lexical properties, for which embeddings of non-lexical properties in lmfr should be useful.

We extract a dev set from section 22 of the PTB following the description in Belinkov et al. (2014).

Preposition DisambiguationWe consider the preposition disambiguation task proposed by Ritter et al. (2014). The task is to determine the spatial relationship a preposition indicates based on the two objects connected by the preposition. For example, "the apple on the refrigerator" indicates the "support by Horizontal Surface" relation, while "the apple on the branch" indicates the "Support from Above" relation. Since the meaning of a preposition depends 

on the combination of both its head and child word, we expect conjunctions between these word embeddings to help, i.e. features with two lexical parts.

We include three baselines: point-wise addition (SUM) Mitchell and Lapata (2010), concatenation Ritter et al. (2014), and an SVM based on handcrafted features in Table 2. Ritter et al. show that the first two methods beat other compositional models.

Hyperparametersare all tuned on the dev set. The chosen values are learning rate \(\eta=0.05\) and the weight of L2 regularizer \(\lambda=0.005\) for lrfr, except for the third lrfr in Table 3 which has \(\lambda=0.05\). We select the rank of lrfr-tucker with a grid search from the following values: \(r_{1}=\{10,20,d_{1}\}\), \(r_{2}=\{20,50,d_{2}\}\) and \(r_{3}=\{50,100,200\}\). For lrfr-cp, we select \(r=\{50,100,200\}\). For the _PP-attachement_ task there is no \(r_{1}\) since it uses a ranking model. For the _Preposition Disambiguation_ we do not choose \(r_{1}\) since the number of labels is small.

## 8 Results

Relation ExtractionAll lrfr-tucker models improve over Baseline and fcm (Table 3), making these the best reported numbers for this task. However, lrfr-cp does not work as well on the features with only one lexical part. The Tucker-form does a better job of capturing interactions between different views. In the limited training setting, we find that lrfr-cp does best.

Additionally, the primary advantage of the CP approximation is its reduction in the number of model parameters and running time. We report each model's running time for a single pass on the development set. The lrfr-cp is by far the fastest. The first three lrfr-tucker models are slightly slower than fcm, because they work on dense non-lexical property embeddings while fcm benefits from sparse vectors.

PP-attachmentTable 4 shows that lrfr (89.6 and 90.3) improves over the previous best standalone system HPCD (88.7) by a large margin, with exactly the same resources. Belinkov et al. (2014) also reported results of parsers and parser re-rankers, which can access to additional resources (complete parses for training and complete sentences as inputs) so it is unfair to compare them with the standalone systems like HPCD and our lrfr. Nonetheless

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
* & 1,213 / 607 \\ Preposition Disambiguation & Ritter et al. (2014) & Ritter et al. (2014) & 6 & 9/3 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of each task. PP-attachment and preposition disambiguation have both unigram and bigram features. Therefore we list the numbers of non-lexical properties for both types.
 

less lrfr\({}_{1}\)-tucker & lrfr\({}_{2}\)-cp (90.3) still outperforms the state-of-the-art parser RBG (88.4), reranker Charniak-RS (88.6), and the combination of the state-of-the-art parser and compositional model RBG + HPCD (90.1). Thus, even with fewer resources, lrfr becomes the new best system.

Not shown in the table: we also tried lrfr\({}_{1}\)-tucker &lrfr\({}_{2}\)-cp with _postag features only_ (89.7), and with grand-head-modifier conjunctions removed (89.3) . Note that compared to lrfr, RBG benefits from binary features, which also exploit grand-head-modifier structures. Yet the above reduced models still work better than RBG (88.4) without using additional resources.4 Moreover, the results of lrfr can still be potentially improved by combining with binary features. The above results show the advantage of our factorization method, which allows for utilizing pre-trained word embeddings, and thus can benefit from semi-supervised learning.

Footnote 4: Still this is not a fair comparison since we have different training objectives. Using RBG’s factorization and training with our objective will give a fair comparison and we leave it to future work.

**Preposition Disambiguation**lrfr improves (Table 5) over the best methods (SUM and Concatenation) in Ritter et al. (2014) as well as the SVM based on the original lexical features (85.1). In this task lrfr\({}_{1}\)-Brown better represents the unigram and bigram lexical features, compared to the usage of two low-rank tensors (lrfr\({}_{1}\)-tucker & lrfr\({}_{2}\)-cp). This may be because lrfr\({}_{1}\)-Brown has fewer parameters, which is better for smaller training sets.

We also include a control setting (lrfr\({}_{1}\)-Brown - Control), which has a full rank parameter tensor with the same inputs on each view as lrfr\({}_{1}\)-Brown, but represented as one hot vectors without transforming to the hidden representations hs. This is equivalent to an SVM with the compound cluster features as in Koo et al. (2008). It performs much worse than lrfr\({}_{1}\)-Brown, showing the advantage of using word embeddings and low-rank tensors.

SummaryFor unigram lexical features, lrfr\({}_{n}\)-tucker achieves better results than lrfr\({}_{n}\)-cp. However, in settings with fewer training examples,

\begin{table}
\begin{tabular}{|p{85.4pt}|c|c|c|c|c|c|c|c|} \hline  & \multicolumn{4}{c|}{**Parameters**} & \multicolumn{3}{c|}{**Full Set (\(|D|\)=43,518)**} & \multicolumn{3}{c|}{**Reduced Set (\(|D|\)=10,000)**} & **Prediction** \\ \cline{2-10}
* & \(\mathbf{r_{1}}\) & \(\mathbf{r_{2}}\) & \(\mathbf{r_{3}}\) & **P*
* & **R*
* & **F1*
* & **P*
* & **R*
* & **F1*
* & **Time (ms)*
* \\ \hline Baseline &
* & - & - & 60.2 & **51.2*
* & 55.3 & - & - & - & - \\ FCM & 32/N & 264/N & 200/N & 62.9 & 49.6 & 55.4 & **61.6*
* & 37.1 & 46.3 & 2,242 \\ \hline lrfr\({}_{1}\)-tucker & 32/N & 20/Y & 200/Y & 62.1 & **52.7*
* & **57.0*
* & 51.5 & 40.8 & 45.5 & 3,076 \\ lrfr\({}_{1}\)-tucker & 32/N & 20/Y & 200/N & **63.5*
* & 51.1 & 56.6 & 52.8 & 40.1 & 45.6 & 2,972 \\ lrfr\({}_{1}\)-tucker & 20/Y & 20/Y & 200/Y & 62.4 & 51.0 & 56.1 & 52.1 & 41.2 & 46.0 & 2,538 \\ lrfr\({}_{1}\)-tucker & 32/Y & 20/Y & 50/Y & 57.4 & 52.4 & 54.8 & 49.7 & **46.1*
* & 47.8 & 1,198 \\ lrfr\({}_{1}\)-cp & \multirow{2}{*}{200/Y} & \multirow{2}{*}{61.3} & \multirow{2}{*}{50.7} & \multirow{2}{*}{55.5} & \multirow{2}{*}{58.3} & \multirow{2}{*}{41.6} & \multirow{2}{*}{**48.6**} & \multirow{2}{*}{**502**} \\ \end{tabular}
\end{table}
Table 3: Results on test for relation extraction. Y(es)/N(o) indicates whether embeddings are updated during training.

\begin{table}
\begin{tabular}{|p{113.8pt}|c|} \hline
**System** & \multicolumn{3}{c|}{**Resources Used**} & **Acc** \\ \hline SVM Belinkov et al. (2014) & distance, word, embedding, clusters, POS, WordNet, VerbNet HPCD Belinkov et al. (2014) & distance, embedding, POS, WordNet, VerbNet & 86.7 \\ \hline lrfr\({}_{1}\)-tucker & lrfr\({}_{2}\)-CP & distance, embedding, POS, WordNet, VerbNet & **90.3** \\ lrfr\({}_{1}\)-Brown & distance, embedding, clusters, POS, WordNet, VerbNet & 89.6 \\ \hline RBG Lei et al. (2014) & dependency parser & 88.4 \\ Charniak-RS McClosky et al. (2006) & dependency parser + re-ranker & 88.6 \\ RBG + HPCD (combined model) & dependency parser + distance, embedding, POS, WordNet, VerbNet & 90.1 \\ \hline \end{tabular}
\end{table}
Table 4: PP-attachment test accuracy. The baseline results are from Belinkov et al. (2014).

 

features with more lexical parts (\(n\)-grams), or when faster predictions are advantageous, lrfr\({}_{n}\)-cp does best as it has fewer parameters to estimate. For \(n\)-grams of variable length, lrfr\({}_{1}\)-tucker &lrfr\({}_{2}\)-cp does best. In settings with fewer training examples, lrfr\({}_{1}\)-Brown does best as it has only one parameter tensor to estimate.

## 9 Related Work

**Dimensionality Reduction for Complex Features** is a standard technique to address high-dimensional features, including PCA, alternating structural optimization Ando and Zhang (2005), denoising autoencoders Vincent et al. (2008), and feature embeddings Yang and Eisenstein (2015). These methods treat features as atomic elements and ignore the inner structure of features, so they learn separate embedding for each feature without shared parameters. As a result, they still suffer from large parameter spaces when the feature space is very huge.5

Footnote 5: For example, a state-of-the-art dependency parser Zhang and McDonald (2014) extracts about 10 million features; in this case, learning 100-dimensional feature embeddings involves estimating approximately a billion parameters.

Another line of research studies the inner structures of lexical features: e.g. Koo et al. (2008), Turian et al. (2010), Sun et al. (2011), Nguyen and Grishman (2014), Roth and Woodsend (2014), and Hermann et al. (2014) used pre-trained word embeddings to replace the lexical parts of features ; Srikumar and Manning (2014), Gormley et al. (2015) and Yu et al. (2015) propose splitting lexical features into different parts and employing tensors to perform classification. The above can therefore be seen as special cases of our model that only embed a certain part (view) of the complex features. This restriction also makes their model parameters form a full rank tensor, resulting in data sparsity and high computational costs when the tensors are large.

**Composition Models (Deep Learning)** build representations for structures based on their component word embeddings Collobert et al. (2011); Bordes et al. (2012); Socher et al. (2012); Socher et al. (2013b). When using only word embeddings, these models achieved successes on several NLP tasks, but sometimes fail to learn useful syntactic or semantic patterns beyond the strength of combinations of word embeddings, such as the dependency relation in Figure 1(a). To tackle this problem, some work designed their model structures according to a specific kind of linguistic patterns, e.g. dependency paths Ma et al. (2015); Liu et al. (2015), while a recent trend enhances compositional models with linguistic features. For example, Belinkov et al. (2014) concatenate embeddings with linguistic features before feeding them to a neural network; Socher et al. (2013a) and Hermann and Blunsom (2013) enhanced Recursive Neural Networks by refining the transformation matrices with linguistic features (e.g. phrase types). These models are similar to ours in the sense of learning representations based on linguistic features and embeddings.

**Low-rank Tensor Models for NLP** aim to handle the conjunction among different views of features Cao and Khudanpur (2014); Lei et al. (2014); Chen and Manning (2014). Yu and Dredze (2015) proposed a model to compose phrase embeddings from words, which has an equivalent form of our CP-based method under certain restrictions. Our work applies a similar idea to exploiting the inner structure of complex features, and can handle \(n\)-gram features with different \(n\)s. Our factorization (SS3) is general and easy to adapt to new tasks. More importantly, it makes the model benefit from pre-trained word embeddings as shown by the PP-attachment results.

## 10 Conclusion

We have presented lrfr, a feature representation model that exploits the inner structure of complex lexical features and applies a low-rank tensor to efficiently score features with this representation. lmfr attains the state-of-the-art on several tasks, including relation extraction, PP-attachment, and preposition disambiguation. We make our implementation available for general use.6

Footnote 6: [https://github.com/Gorov/LowRankFCM](https://github.com/Gorov/LowRankFCM)

### Acknowledgements

A major portion of this work was done when MY was visiting MD and RA at JHU. This research was supported in part by NSF grant IIS-1546482.

 



## References

* [Ando and Zhang2005] Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. _The Journal of Machine Learning Research_, 6.
* Belinkov et al. [2014] Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014. Exploring compositional architectures and word vector representations for prepositional phrase attachment. _Transactions of the Association for Computational Linguistics_, 2.
* Bengio et al. [2006] Yoshua Bengio, Holger Schwenk, Jean-Sebastien Senecal, Frederic Morin, and Jean-Luc Gauvain. 2006. Neural probabilistic language models. In _Innovations in Machine Learning_. Springer.
* Bordes et al. [2012] Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2012. A semantic matching energy function for learning with multi-relational data. _Machine Learning_.
* Cao and Khudanpur [2014] Yuan Cao and Sanjeev Khudanpur. 2014. Online learning in tensor space. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Chen and Manning [2014] Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of EMNLP_.
* Collobert et al. [2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _JMLR_, 12.
* Duchi et al. [2011] John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. _The Journal of Machine Learning Research_, 12.
* Gormley et al. [2015] Matthew R. Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
* Hermann and Blunsom [2013] Karl Moritz Hermann and Phil Blunsom. 2013. The role of syntax in vector space models of compositional semantics. In _Association for Computational Linguistics_.
* Hermann et al. [2014] Karl Moritz Hermann, Dipanjan Das, Jason Weston, and Kuzman Ganchev. 2014. Semantic frame identification with distributed word representations. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Hoffmann et al. [2010] Raphael Hoffmann, Congle Zhang, and Daniel S. Weld. 2010. Learning 5000 relational extractors. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_.
* Kolda and Bader [2009] Tamara G Kolda and Brett W Bader. 2009. Tensor decompositions and applications. _SIAM review_, 51(3).
* Koo et al. [2008] Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In _Proceedings of ACL_.
* Lei et al. [2014] Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Lei et al. [2015] Tao Lei, Yuan Zhang, Lluis Marquez, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Liu et al. [2015] Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng WANG. 2015. A dependency-based neural network for relation classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_.
* Ma et al. [2015] Mingbo Ma, Liang Huang, Bowen Zhou, and Bing Xiang. 2015. Dependency-based convolutional neural networks for sentence embedding. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_.
* McClosky et al. [2006] David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In _Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics_.
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems_, pages 3111-3119.
* Miller et al. [2004] Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In _Proceedings of HLT-NAACL_.
* Mitchell and Lapata [2010] Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. _Cognitive science_, 34(8).
* Nguyen and Grishman [2014] Thien Huu Nguyen and Ralph Grishman. 2014. Employing word representations and regularization for domain adaptation of relation extraction. In _Association for Computational Linguistics (ACL)_.
* Ritter et al. [2014] Samuel Ritter, Cotie Long, Denis Paperno, Marco Baroni, Matthew Botvinick, and Adele Goldberg. 2014.



Leveraging preposition ambiguity to assess representation of semantic interaction in cdsm. In _NIPS Workshop on Learning Semantics_.
* Roth and Woodsend (2014) Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In _Proceedings of EMNLP_.
* Socher et al. (2012) Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In _Proceedings of EMNLP-CoNLL 2012_.
* Socher et al. (2013a) Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. 2013a. Parsing with compositional vector grammars. In _Proceedings of ACL_.
* Socher et al. (2013b) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of EMNLP_.
* Srikumar and Manning (2014) Vivek Srikumar and Christopher D Manning. 2014. Learning distributed representations for structured output prediction. In _Advances in Neural Information Processing Systems_.
* Sun et al. (2011) Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011. Semi-supervised relation extraction with large-scale word clustering. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_.
* Taub-Tabib et al. (2015) Hillel Taub-Tabib, Yoav Goldberg, and Amir Globerson. 2015. Template kernels for dependency parsing. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Turian et al. (2010) Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In _Association for Computational Linguistics_.
* Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In _Proceedings of the 25th international conference on Machine learning_.
* Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. ACE 2005 multilingual training corpus. _Linguistic Data Consortium, Philadelphia_.
* Yang and Eisenstein (2015) Yi Yang and Jacob Eisenstein. 2015. Unsupervised multi-domain adaptation with feature embeddings. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 672-682, Denver, Colorado, May-June. Association for Computational Linguistics.
* Yu and Dredze (2015) Mo Yu and Mark Dredze. 2015. Learning composition models for phrase embeddings. _Transactions of the Association for Computational Linguistics_, 3.
* Yu et al. (2015) Mo Yu, Matthew R. Gormley, and Mark Dredze. 2015. Combining word embeddings and feature embeddings for fine-grained relation extraction. In _North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Zhang and McDonald (2014) Hao Zhang and Ryan McDonald. 2014. Enforcing structural diversity in cube-pruned dependency parsing. In _Proceedings of ACL_.
* Zhou et al. (2005) GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang. 2005. Exploring various knowledge in relation extraction. In _Proceedings of ACL_.