Title: TOWARDS DECOMPOSED LINGUISTIC REPRESENTATION WITH HOLOGRAPHIC REDUCED REPRESENTATION

Abstract: The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures. In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced Representation (HRR). This allows us to inject structures directly into our wordlevel and chunk-level representations. Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics. 1
1 INTRODUCTION Recent advances in representation learning have been unequivocally led by the long strides of progress in deep learning and its distributed representations. In many tasks of Natural Language Processing (NLP), researchers have convincingly shown that distributed representations are capable of encoding the complex structure of textual inputs (for example Mikolov et al. (2010; 2013); Józefowicz et al. (2016); Sutskever et al. ( 2014)). The dominant approach for many NLP tasks is the encoder-decoder paradigm that uses neural networks to learn the transformations from many smaller comprising units to one complex embedding, and vice versa 2 . The underlying structure, in a rather crude fashion, is assumed to be represented by this complex embedding. In many cases, such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability and transferability. Transparency and interpretability require that the operations of encoding and decoding have clear conceptual meaning, and transferability generally necessitates a separation of transferable features (e.g., domain-invariants) from the rest. On account of shortcomings, much previous work has been devoted to inducing disentangled representations (

:

minimal supervision, and has been proven hugely beneficial for many other NLP tasks. The versatility of language modeling demonstrates that some linguistic regularities much be present, and the training signal is sufficient for them to arise. We carefully design a language model with HRR that explicitly encodes the underlying structure as role-filler pairs on both word-level and chunk-level, and show that HRR provides an inductive bias towards the learning of decomposed representations. We demonstrate that on both Penn Treebank (PTB) and a subset of One-Billion-Word LM data set (1B), our model can effectively separate certain aspects of word or chunk representation, which roughly corresponds to a division between syntax and semantics. We perform various analyses on the learned embeddings, and validate that they indeed capture distinct linguistic regularities.

Our paper is structured as follows. Section 2 gives a background overview of VSA and HRR; Section 3 details our proposed models; Experimental results are shown in Section 4, followed by related work in Section 5 and a conclusion in Section 6.

2. BACKGROUND:

Vector Symbolic Architecture (VSA) is a family of models that enable connectionist models to perform symbolic processing, while encoding complex structures in distributed representations. A set of algebraic operations defined by these approaches allow them to compose, decompose and manipulate symbolic structures.

Our paper focuses on such an approach, namely Holographic Reduced Representation (HRR) proposed by (Plate, 1995) . HRR uses three operations: circular convolution, circular correlation and element-wise addition, to perform encoding, decoding and composition, respectively. The cicircular convolution (denoted by the operator * ) of two vectors x and y of dimension d, is defined as z = x * y, in which

z i = d k=1 x i mod d y (i-k) mod d , i = 1 . . . d

x * y is called the binding of x and y, or the encoding of the pair (x, y). Using this operation, the composition of a set of role-filler pairs (r 1 , f 1 ), (r 2 , f 2 ), . . . , (r m , f m ) is represented as H = r 1 * f 1 + r 2 * f 2 + . . . r m * f m , where + is element-wise addition used as a composition operator. The previous example John loves his mom can be represented as r agent * f John + r predicate * f loves + r patient * f his mom .

By definition, HRR guarantees that the composed representation remains a vector of dimension d, regardless of how many items are bound together by the * operation. This avoids the parameter explosion problem of other VSA approaches such as as Tensor Product Represenation (TPR) (Smolensky, 1990) , and makes HRR a more practical choice for representing compositional structures.

To decode from an HRR and retrieve a role or filler representation, an approximate inverse operation of the circular convolution, named circular correlation, is defined as t = x # y in which

EQUATION

Now given a memory trace z = x * y, the correlation operation allows us to retrieve y from the cue x via y ≈ x # z. We do not detail the exact conditions when this retrieval holds, but refer interested readers to the original paper (Plate (1995) )

3. HOLOGRAPHIC REDUCED REPRESENTATION FOR LANGUAGE MODELING:

We incorporate HRR into language models on two levels: word level and chunk level. Our HRRenabled language models (HRRLM) posit an explicit decomposition of word or chunk representations, which enables our model to capture different aspects of linguistic regularities. Before delving into the details of our model, we first introduce notations and provide a brief account of the commonly used RNN-based LM.

3.1. RNNLM:

RNN-based LMs estimate the probability of any given sentence s = w 1 w 2 . . . w n using an RNN. At each step t, RNN encodes the history w 1 w 2 . . . w t into a vector h, and tries to predict the next token w t+1 . Prediction is generally modeled by a linear layer followed by softmax operation. Specifically,

EQUATION

where E(•) is the embedding operation to embed a symbol into a continuous space R d , and V is the entire or a sampled subset of the vocabulary. The scoring function is defined by dot product, and therefore maximizing this probability encourages related words to form clusters, and away from other words in the embedded space.

3.2. WORD-LEVEL HRRLM:

Encoding We first use HRR to directly encode the underlying structures of words. We assume there is a decomposition of representations along N directions. Specifically, we embed a word w as

EQUATION

where r word i 's are basis role embeddings, shared by all words. Each basis role embedding is bound to its distinct set of filler embeddings, modeled by Ẽi . The motivation is that when properly trained, different bindings should capture disparate aspects of word representation. For instance, the first binding might be relevant for syntactic categories, and the second one for semantic relatedness. With this particular decomposition, the word getting should be close to other gerunds such as giving and forgetting in the first embedding space, and get, got or received in the second. In this case, the composite (i.e., the sum) of these bindings essentially encodes getting = {semantics: GET, syntax: GERUND}.

We additionally assume that each set of filler embeddings Ẽi resides in a separate linear subspace. This is achieved by modeling each Ẽi (w) with a linear combination of its associated basis filler embeddings f i,j . Specifically,

Ẽi (w) = d j=1 s w i,j f i,j = [f i,1 ; f i,2 ; . . . ; f i,d ]      s w i,1 s w i,2 . . . s w i,d      = F i s w i

where s w i ∈ R d is a word-specific d -dimensional vector. In other words, F i projects s w i ∈ R d to Ẽi (w) ∈ R d . This assumption has two advantages. First, the total number of parameters for the embedding layer is now V N d . We can set a smaller value d to prevent overparameterization, while maintaining a d-dimensional vector as the input to RNN. Second, by having separate bases for different role-filler bindings, we introduce an inductive bias for the model to learn a decomposition of word representation. This requirement makes sense intuitively -a decomposition of representation usually necessitates a separation of feature space. Our preliminary experiments show that this separation is essential for obtaining decomposed representations. The entire encoding operation is illustrated in the bottom half of Figure 1(b) .

Decoding The composite embedding E(w) is fed as input to RNN. From its output h given by the top hidden layer, we decode all the filler vectors using circular correlation, and factorize the scoring function into N parts. Specifically, we use the same word level loss as in Equation ( 2), but replace the scoring function in Equation (3) as a sum of dot products in every filler space: where α i 's are scalar hyperparameters that we use to break the symmetry of the scoring function. Specifically when N = 2, α 1 is set to a constant 1.0, and α 2 is linearly annealed from 0.0 at the start of training to 1.0 at a specified time step T , then remains constant afterwards. Note that dot products are only computed between co-indexed filler embeddings, namely between f i and Ẽi (w). This ensures the model only learns relatedness in the i-th subspace, without interference from other subspaces. The entire word-level HRRLM is illustrated is Figure 1(b) .

f i = r word i # h, score(h, w) = N i=1 α i f i • Ẽi (w) (5)

Regularization on basis embeddings Basis embeddings are chosen so that they are not correlated with each other. This decorrelation is promoted by adding an isometric regularization term to the basis embeddings. For instance, we define an isometric regularization penalty for the basis fillers F i 's as:

Φ(F 1 , F 2 , . . . , F N ) = β( F i F i -I 2 + F i F j 2 ), ∀1 ≤ i, j ≤ N, i = j,

where I is an identity matrix, and β is a hyperparameter. Similarly, we add a regularization term for r word i 's. As an alternative, we also consider using fixed random vectors for basis embeddings since high-dimensional random vectors are approximately orthogonal to each other.

3.3. CHUNK-LEVEL HRRLM:

A direct extension to the chunk-level model using the proposed techniques above is not desirable, due to two major difficulties. First, Equation (4) stipulates that each unique word type is assigned a vectorial parameter, which is computationally infeasible due to the vast number of unique chunks. Second, the same chunk can carry different semantic roles. For instance, the two sentences His mom loves John and John loves his mom have the same noun phrases, but with their roles of agent and patient switched. In light of these difficulties, we take on a compositional approach to address the first issue, and use a context-sensitive role to address the second.

Encoding Unlike the word-level HRR representation in Equation ( 4) where roles are fixed, the chunk roles are represented by linear combinations of M basis role embeddings. This formulation allows us to model context-dependency. Specifically, we construct HRR representations for a chunk c = w 1 w 2 . . . w m in three steps: STEP 1 (ROLE PREDICTION) We first predict a context-sensitive role tuple for each word w k in the chunk:

r w k = (r w k 1 , r w k 2 , . . . , r w k M ) = (a w k 1 r chunk 1 , a w k 2 r chunk 2 , . . . , a w k M r chunk M ).

a w k i 's represent the distribution of chunk roles, which are predicted by the same RNN used to predict the next tokens. Specifically, we feed the output vector from RNN through a linear layer and then a softmax layer. r chunk i 's are chunk-level basis role embeddings, shared by all chunks3 . Step 1 corresponds to the role prediction module in Figure 1(c) .

STEP 2 (PROJECTION AND BINDING) For each of the basis roles, we predict their associated filler embeddings. This is done by projecting the HRR word representation Ẽ(w k ) (Equation ( 4)) into M vectors. These fillers are then bound with their corresponding roles and summed together. Specifically,

g w k 1 , g w k 2 , . . . , g w k M = [W 1 ; W 2 ; . . . ; W M ] Ẽ(w k ), Êc (w k ) = M i=1 r w k i * g w k i = M i=1 (a w k i r chunk i ) * g w k i .

Intuitively, the first two steps aims to evaluate the representational contribution of w k to the entire chunk c. Following the practice in constructing a word-level model (Equation ( 4)), we also require the separation of different role-filler bindings. We also note that the binding Êc (w k ) embeds the word w k into a space that is specific to the chunk c. The first two steps are illustrated in Figure 1 (c).

STEP 3 (ENCODING) After obtaining bindings for all words within a chunk, the chunk embedding is defined as

Ê(c) = m k=1

Êc (w k ) (Figure 1(d) ). It can be easily verified that

EQUATION

Note that Ê(c) has the same form as Equation ( 4), and Êi (c) can be interpreted as the chunk filler embedding for the i-th chunk role. However, chunk embeddings are different in two key aspects. First, the filler embeddings for chunks are projected from the word embeddings, instead of being a set of independently trainable parameters. This compositional approach addresses the first issue regarding the vast number of unique chunks. Second, chunk embeddings rely on weights a w i 's from role prediction, which provide a natural vehicle for carrying contextual information, therefore addressing the second issue of context-sensitivity.

Chunk prediction Comparable to language modeling that uses next word prediction as supervision, our chunk-level HRR model uses next chunk prediction (CP in Figure 1(a) ). Specifically in our experiments, we simply concatenate the chunk embeddings from the last two steps, and feed it through a linear layer followed by tanh activation:

Ê(c τ ) = tanh [U 1 ; U 2 ] [ Ê(c τ -1 ); Ê(c τ -2 )] .

Decoding Similar to word-level HRR (Equation ( 5)), we decode the filler embeddings from the predicted chunk embedding using r chunk i as cue, and then use the decoded embeddings to factorize the scoring function. The same form of loss function is used. To provide negative examples in the denominator of the softmax Equation (2), we use all the chunks in the mini-batch. These chunks form a pseudo "chunk vocabulary" that is constructed on the fly. Role annealing, and regularization on basis embeddings are also applied.

Chunk boundaries Chunk boundaries have to be supplied in order to construct a chunk embedding. We reply on a third-party chunker to provide such annotations. This is analogous to our word-level model, only that word boundaries are trivially provided by whitespace in English.

4. EXPERIMENTS:



4.1. SETUP:

Data Sets We train all models on both Penn Treebank (PTB) (Marcus et al., 1994) and One-Billion-Word Benchmark dataset (1B) (Chelba et al., 2014) . For 1B, we experimented with three training sizes: the full dataset, one-tenth of the dataset, and one-hundredth. We evaluate all models on the following three aspects:

• Perplexity: We report perplexity to evaluate language models.

• Intrinsic evaluation: We use a test suite consisting of 18 word embedding benchmark datasets to intrinsically evaluate the quality of word embeddings (Jastrzebski et al., 2017) .4 • Extrinsic evaluation: We evaluate all word embeddings on six downstream tasks following Nayak et al. ( 2016): Part-of-speech tagging (POS), chunking, named entity recognition (NER), sentiment analysis (SA), question classification (QC), and natural language Inference (NLI). 5Training details For our models, We mainly experimented with two word-level and two chunklevel roles. The basis embeddings are either trained with isometric constraint, or fixed as constant after random initialization.

We also note that for PTB, we experimented with either contiguous input or noncontiguous input.

For contiguous input, we follow the common practice in the literature to feed the input sentences in the order that they appear in the document, and initialize the hidden state of LSTM with the last state from the last batch. We refer interested readers to the appendix for more training details.

4.2. PERPLEXITY RESULTS:

Although not a prime motivation of our approach, Table 1 shows that our HRR models can outperform the baseline in terms of perplexity, especially when we do not have enough training data. On PTB, the best HRR model obtains a gain of 2.2 with noncontiguous input (Fixed-big 88.0 vs Baseline 90.2), and 1.9 with contiguous input (Fixed-big-cont 74.7 vs Baseline-cont 76.6). We also note that increasing the number of basis fillers has a positive impact on HRR models (Fixed-big 88.0 vs Fixed-medium 90.2 vs Fixed-small 97.0). On 1B, we observe that HRR models benefit from having trainable bases with isometric regularization (107.2 vs 112.3 on 1/100 1B, and 55.9 vs 57.9 on full 1B).

4.3. WORD LEVEL ANALYSIS:

We then demonstrate that our word-level HRRLM can effectively separate certain attributes of word representation. Specifically, we look at our word-level models with two word-level roles, and find that the first set of filler embeddings captures mostly syntax-related categories, especially tense and agreement for verbs, whereas the second set focuses more on the semantic content of words. This decomposition is illustrated in Figure 2 , where we visualize both sets of filler embeddings for the most frequent 2500 words in PTB via t-SNE (Maaten & Hinton, 2008 ). In the first set (in blue), verbs with different inflectional markers form distinct clusters. For instance, bare forms, gerunds, preterites, and third-person-singular verbs each form a visually distinguishable group. In contrast, for the second set of filler embeddings (in orange), semantically related words tend to be close regardless of their morphosyntactic markers. For instance, give, gave, and giving are close in the second space.

We proceed to evaluate the quality of the learned word embeddings both intrinsically and extrinsically. As Table 2 summarizes, our HRR models consistently outperform the baseline on the intrinsic evaluation task (numbers on the left). The best HRR model Isometric-big obtains a gain of 9.0%, 11.6%, and 16.2% on 1/100 1B, 1/10 1B and full 1B, respectively. In addition, the HRR model with trainable bases has a noticeable advantage over the model with fixed bases (2.5% on average). On the other hand, the HRR models have only a marginal advantage over the baseline (numbers on the right), which echos findings in (Schnabel et al., 2015) which claims that intrinsic improvement for word embeddings does not necessarily correlate with improvement in downstream tasks. Detailed results for each dataset and downstream task are available in A breakdown of the intrinsic evaluation reveals that we obtain huge gains on the word analogy task, especially in the verb-related categories such as plural verbs and past tense. On the Semantic-Syntactic analogy dataset Mikolov et al. (2013) , the accuracy (top 1) for plural verbs is 0.897 if we consider the most frequent 200K words. If we limit the vocabulary to the more common words (10K), the accuracy for present participles reaches 0.964. This is consistent with our previous observation that the first set of filler embeddings effectively captures the representation of different verb forms.

To further quantify how decomposed are our word-level filler embeddings, we use them to classify whether two words are related in terms of either semantics or syntax. Specifically, we use a threshold θ for cosine similarity to determine whether any pair of words (w, w ) are related. We then use different values of θ to plot ROC curve and compute its AUC. For the ground truths, we use the syntactic categories from the Semantic-Syntactic dataset, and extract them from each a : b = c : d example.

Two sets of categorizations are obtained, driven by semantics and syntax respectively. For instance, for the example make : making = give : giving, we obtain ({making, giving}, {make, give}) for syntactic considerations, and ({making, make}, {giving, give}) for semantics. For the verb-related categories, Table 4 shows that the first set of filler embeddings does noticeably better than the second in the first experiment, while the second set is much better at the second experiment. Figure 3 shows two such categories which confirm that the decomposition does make sense on a crude syntax-semantics level. More details of the intrinsic evaluation results can be found in Table 8 

4.4. CHUNK LEVEL ANALYSIS:

We train a chunk-level HRR model by initializing it with a pretrained word-level model on PTB. 6 We evaluate the quality of chunk embeddings in two ways. First, we perform a human analysis focused on the phrase the company, which is the most frequent noun phrase in PTB. We randomly select 100 occurrences, and cluster their chunk embeddings using K-means into 5 categories based on the chunk-level filler embeddings. 7 For each sentence cluster we manually identify the dominating role which the company played in that cluster of sentences. Table 5 shows that there is a clear role for the phrases in each of the clusters. We also performed a t-SNE analysis using the second filler embedding, in which we observed the company is close to many semantically related nouns such as stock, market and etc. From this analysis we see that the first role in chunk-level HRR is more sensitive to the different syntactic roles the company plays dependent on the context, while the second role is more associated with its related semantic concepts.

Secondly, to further investigate if our model consistently captures chunk-level roles, we run the model on the Wall Street Journal part of the OntoNotes dataset (Weischedel et al., 2013) in which semantic role labels for each sentence are provided, and compare chunk embedding clusters against the role labels. We are mainly interested in 4 coarser-grained labels corresponding to AGENT, PATIENT, PREDICATE and MODIFIER. We provide two metrics to measure the quality of the obtained clusters. First, for each chunk c, we find its k nearest neighbors (k = 1, 5, 10, 25) according to their filler embeddings, and report the percentage of the cases where c shares the same role labels as its neighbor. This hit-rate metric gives us a sense of how accurately chunk clusters capture meaningful semantic roles. In addition, we compute purity score for the learned clusters against the ground truth clusters. Both metric scores are reported in Table 6 , from which we can see that chunk embedding indeed correctly captures semantic roles more than half of the time. Moreover, the first set of fillers are more accurate at clustering these roles than the second, and also deteriorates better when we include more neighbors to compute the hit rates.

5. RELATED WORK:

Perhaps most related to our work are recent attempts to integrate tensor product structure with neural networks (Palangi et al., 2017; Huang et al., 2018) . While these works and ours share a common goal of incorporating neural models with symbolic structures, there are several difference. First of all, we make use of HRR instead of tensor product as basis for our representation to enable long sequence encoding without parameter explosion. Moreover, we aim to induce linguistic structures from a task that requires as little supervision as possible like language modeling, whereas their work is focused on question answering which provides stronger guidance signal from labeled data. On the other hand, recent work (Shen et al., 2018 ) also proposes a novel network architecture that is capable of learning syntactic roles and semantics jointly, but it is not based on structured representation like HRR.

There have been many attempts of using symbolic architectures like HRR and TPR for linguistic analysis, see for example (Jones & Mewhort, 2007; De Vine & Bruza, 2010; Recchia et al., 

APPENDIX C CHUNK LEVEL ANALYSIS:

To obtain clustering results, we used sklearn with the default parameters and settings provided by the toolbox. For the OntoNotes experiment, we removed sentences that have unknown tokens of more than 10 percent, and removed chunks that are longer than six words. In addition, we removed some rare role labels (out of 11), and collapsed some object-related roles (e.g., direct object, and indirect object) into PATIENT. This resulted in a total of around 4K chunks, and four remaining coarse labels AGENT, PATIENT, PREDICATE, and MODIFIER, which correspond to the original tags ARG0, ARG1, V and ARGM. 

Code will be released to the public.

For attention mechanisms, there is actually a distribution over multiple embeddings, rather than one single complex embedding.

Note that chunk-level bases r chunk i 's are different parameters from word-level bases r word i 's

https://github.com/kudkudak/word-embeddings-benchmarks.

https://github.com/NehaNayak/veceval

The perplexity for the chunk-level model is 89.5 vs 88.0 for the word-level model. The degradation of performance for next word prediction is noticeable. However, we do note that such degradation is relatively common for some multitask settings.

More details about the experimental setup for chunk level analysis are provided in Appendix C.

Prince & Smolensky, 1997;Clark et al., 2008;Clark & Pulman, 2010;Grefenstette et al., 2011). HRR itself as a variable binding and association mechanism, has also been integrated with neural networks with different motivations like associative memory modeling(Danihelka et al., 2016), relationship reasoning(Weiss et al., 2016) etc. While most of the work mainly focuses on symbolic and formal analysis via algebraic operations and logic derivations, our work aims to enable neural language models to learn linguistic roles by taking advantage of the HRR properties.6 CONCLUSIONIn this paper, we employ HRR to provide a principled decomposition of representation. We design our HRR language models to work on both word-level and chunk-level. Our analysis revealed that by introducing an inductive bias, our models can learn disentangled representations, which roughly corresponds to syntax and semantics.

https://www.clips.uantwerpen.be/pages/MBSP

Figures and Tables:

Figure 1: Architecture of HRRLMs. (a) Chunk-level HRRLM. (b)Word-level HRRLM. (c) Step 1 and 2 of chunk-level model encoding. (d) Step 3 of chunk-level model encoding. See Section 3 for details.

Figure 2: t-SNE visualizations for two sets of filler embeddings (blue and orange).

Figure 3: Examples of ROC curves for two categories. The left figure is for a syntax-driven categorization of gerunds, while the right is a semantics-driven categorization of plural verbs. Isometricbig-f1 means the first filler embedding, and Isometric-big-f2 the second.

Figure 4: ROC for every syntactic category in the first set of categorization experiments (syntaxdriven).14

Figure 5: ROC for every syntactic category in the second set of categorization experiments (semantics-driven).15

Perplexity results on PTB and 1B data sets. All HRR models have two word-level roles, and each role has 320, 200, and 50 basis filler embeddings for Fixed-big, Fixed-medium and Fixedsmall, respectively.

Table content: <table><tr><td>Model</td><td colspan="3">PTB 1/100 1B 1/10 1B</td><td>1B</td></tr><tr><td>Baseline</td><td>90.2</td><td>120.4</td><td>64.5</td><td>56.0</td></tr><tr><td>Baseline-cont</td><td>76.6</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Fixed-small</td><td>97.0</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="2">Fixed-medium 90.2</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Fixed-big</td><td>88.0</td><td>112.3</td><td>64.2</td><td>57.9</td></tr><tr><td colspan="2">Fixed-big-cont 74.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Isometric-big</td><td>88.2</td><td>107.2</td><td>63.9</td><td>55.9</td></tr></table>

Table 7 in Appendix B. Averaged scores for intrinsic evaluation (18 datasets, numbers on the left) and extrinsic evaluation (6 tasks, numbers on the right).

Table content: <table><tr><td>Model</td><td>1/100 1B</td><td>1/10 1B</td><td>1B</td></tr><tr><td>Baseline</td><td>0.386|0.818</td><td>0.454|0.816</td><td>0.427|0.817</td></tr><tr><td>Fixed-big</td><td>0.466|0.823</td><td>0.552|0.817</td><td>0.543|0.821</td></tr><tr><td colspan="4">Isometric-big 0.476|0.820 0.570|0.819 0.589|0.822</td></tr></table>

Accuracy (top 1) on the Semantic-Syntactic analogy benchmark. The most frequent 200K words are used. We use COSMUL for all analogy testsLevy & Goldberg (2014)

Table content: <table/>

Average AUC on verb-related categories for the baseline, and also the two sets of filler embeddings from Isometric-big. All models are trained on the full 1B dataset. The first row corresponds to the first syntactic-driven categorization experiment, and the second row to the semantic-driven experiment.

Table content: <table><tr><td>in</td></tr></table>

Cluster analysis for a random selection of 100 sentences containing the company. The most dominant role in each cluster is identified by a human judge.

Table content: <table><tr><td>Metric</td><td colspan="2">Filler 1 Filler 2</td></tr><tr><td>Hit rate@1</td><td>0.604</td><td>0.572</td></tr><tr><td>Hit rate@5</td><td>0.604</td><td>0.553</td></tr><tr><td colspan="2">Hit rate@10 0.600</td><td>0.544</td></tr><tr><td colspan="2">Hit rate@25 0.586</td><td>0.531</td></tr><tr><td>Purity</td><td>0.677</td><td>0.610</td></tr></table>

Hit rate and purity score analysis for chunk embeddings. See Sec.4.4 for more details.

Table content: <table/>

Word analogy test results on different datasets.

Table content: <table><tr><td>2015;</td></tr></table>

Detailed word analogy test results in each category.

Table content: <table/>

Peer review (iclr2019):
 Summary: ======== Theis paper proposes a method for learning decomposable representations in the context of a language modeling task. Using holographic reduced representations (HRR), a word embedding is composed of a role and a filler. The embedding is then fed to an LSTM language model. There is also an extension to chunk-level representations. Experimentally, the model achieves perplexity comparable to a (weak) baseline LSTM model. The analysis of the learned representations shows a separation into syntactic and semantic roles. The paper targets an important problem, that of learning decomposable representations. As far as I know, it introduces a novel perspective using HRR and does so in the context of language modeling, which is a core NLP task. The analysis of the learned representations is quite interesting. I do have some concerns with regards to the quality of the language model, the clarity of some of the model description, and the validity of using HRR in this scenario. Please see detailed comments below. Comments: ========= 1. Section 2 refers to Plate (1995) for the conditions when the approximate decoding via correlation holds. I think it*s important to mention these conditions and discuss whether they apply to the language modeling case. In particular, Plate mentions that the elements of each vector need to be iid with mean zero and variance 1/n (where n is the length of the vector). Is this true for the present case? Typically, word embeddings and LSTM states are do not exhibit this distribution. Are there other conditions that are (not) met? 2. Learning separate bases for different role-filler bindings is said to encourage the model to learn a decomposition of word representation. On the other hand, if I understand correctly, this means that word embeddings are not shared between roles, because s^w_i is also a role-specific vector (not just a word-specific vector). Is that a cause of concern? 3. It*s not clear to me where in the overall model the next word is predicted. Figure 1b has an LSTM that predicts filler embeddings. Does this replace predicting the next word in a vanilla LSTM? Equation 5 still computes a word score. Is this used to compute the probability of the next word as in equation 2? 4. Comparison to other methods for composing words. Since much of the paper is concerned with composing words, it seem natural to compare the methods (and maybe some of the results) to methods for composing words. Some examples include [2] and the line of work on recursive neural networks by Socher et al., but there are many others. 5. Perplexity results: - The baseline results (100.5 ppl on PTB) are very weak for an LSTM. There are multiple papers showing that a simple LSTM can do much better. The heavily tuned LSTM of [1] gets 59.6 but even less tuned LSTMs go under 80 or 80 ppl. See some results in [1]. This raises a concern that the improvements from the HRR model may not be significant. Would they hold in a more competitive model? - Can you speculate or analyze in more detail why the chunk-level model doesn*t perform well, and why adding more fillers doesn*t help in this case? 6. Motivation: - The introduction claims that the dominant encoder-decoder paradigm learns "transformations from many smaller comprising units to one complex emedding, and vice versa". This claim should be qualified by the use of attention, where there is not a single complex embedding, rather a distribution over multiple embeddings. - Introduction, first paragraph, claims that "such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability or transferability" - what do you mean by these concepts and how exactly is the current approach limited with respect to them? Giving a bit more details about this point here or elsewhere in the paper would help motivate the work. 7. Section 3.3 was not so clear to me: - In step 1, what are these r_i^{chunk}? Should we assume that all chunks have the same role embeddings, despite them potentially being syntactically different? How do you determine where to split output vectors from the RNN to two parts? What is the motivation for doing this? - In prediction, how do you predict the next chunk embedding? Is there a different loss function for this? - Please provide more details on decoding, such as the mentioned annealing and regularization. - Finally, the reliance on a chunker is quite limiting. These may not be always available or of high quality. 8. The analysis in section 4.3 is very interesting and compelling. Figure 2 makes a good point. I would have liked to see more analysis along these lines. For example, more discussion of the word analogy results, including categories where HRR does not do better than the baseline. Also consider other analogy datasets that capture different aspects. 9. While I agree that automatic evaluation at chunk-level is challenging, I think more can be done. For instance, annotations in PTB can be used to automatically assign roles such as those in table 4, or others (there are plenty of annotations on PTB), and then to evaluate clustering along different annotations at a larger scale. 10. The introduction mentions a subset of the one billion word LM dataset (why a subset?), but then the rest of the papers evaluates only on PTB. Is this additional dataset used or not? 11. Introduction, first paragraph, last sentence: "much previous work" - please cite such relevant work on inducing disentangled representations. 12. Please improve the visibility of Figure 1. Some symbols are hard to see when printed. 13. More details on the regularization on basis embeddings (page 4) would be useful. 14. Section 3.3 says that each unique word token is assigned a vectorial parameter. Should this be word type? 15. Why not initialize the hidden state with the last state from the last batch? I understand that this is done to assure that the chunk-level models only consider intra-sentential information, but why is this desired? 16. Have you considered using more than two roles? I wonder how figure 2 would look in this case. Writing, grammar, etc.: ====================== - End of section 1: Our papers -> Our paper - Section 2: such approach -> such an approach; HRR use -> HRR uses; three operations -> three operations*:* - Section 3.1: "the next token w_t" - should this be w_{t+1)? - Section 3.2, decoding: remain -> remains - Section 3.3: work token -> word token - Section 4.1: word analogy task -> a word analogy task; number basis -> numbers of basis - Section 4.2: that the increasing -> that increasing - Section 4.3: no space before comma (first paragraph); on word analogy task -> on a word analogy task; belong -> belongs - Section 4.4: performed similar -> performed a similar; luster -> cluster - Section 5: these work -> these works/papers/studies; share common goal -> share a common goal; we makes -> we make; has been -> have been References ========== [1] Melis et al., On the State of the Art of Evaluation in Neural Language Models [2] Mitchell and Lapata, Vector-based Models of Semantic Composition