Title: TOWARDS DECOMPOSED LINGUISTIC REPRESENTATION WITH HOLOGRAPHIC REDUCED REPRESENTATION

Abstract: The vast majority of neural models in Natural Language Processing adopt a form of structureless distributed representations. While these models are powerful at making predictions, the representational form is rather crude and does not provide insights into linguistic structures. In this paper we introduce novel language models with representations informed by the framework of Holographic Reduced Representation (HRR). This allows us to inject structures directly into our wordlevel and chunk-level representations. Our analyses show that by using HRR as a structured compositional representation, our models are able to discover crude linguistic roles, which roughly resembles a classic division between syntax and semantics. 1
1 INTRODUCTION Recent advances in representation learning have been unequivocally led by the long strides of progress in deep learning and its distributed representations. In many tasks of Natural Language Processing (NLP), researchers have convincingly shown that distributed representations are capable of encoding the complex structure of textual inputs (for example Mikolov et al. (2010; 2013); Józefowicz et al. (2016); Sutskever et al. ( 2014)). The dominant approach for many NLP tasks is the encoder-decoder paradigm that uses neural networks to learn the transformations from many smaller comprising units to one complex embedding, and vice versa 2 . The underlying structure, in a rather crude fashion, is assumed to be represented by this complex embedding. In many cases, such crude way of representing the structure is unsatisfactory, due to a lack of transparency, interpretability and transferability. Transparency and interpretability require that the operations of encoding and decoding have clear conceptual meaning, and transferability generally necessitates a separation of transferable features (e.g., domain-invariants) from the rest. On account of shortcomings, much previous work has been devoted to inducing disentangled representations (

:

minimal supervision, and has been proven hugely beneficial for many other NLP tasks. The versatility of language modeling demonstrates that some linguistic regularities much be present, and the training signal is sufficient for them to arise. We carefully design a language model with HRR that explicitly encodes the underlying structure as role-filler pairs on both word-level and chunk-level, and show that HRR provides an inductive bias towards the learning of decomposed representations. We demonstrate that on both Penn Treebank (PTB) and a subset of One-Billion-Word LM data set (1B), our model can effectively separate certain aspects of word or chunk representation, which roughly corresponds to a division between syntax and semantics. We perform various analyses on the learned embeddings, and validate that they indeed capture distinct linguistic regularities.

Our paper is structured as follows. Section 2 gives a background overview of VSA and HRR; Section 3 details our proposed models; Experimental results are shown in Section 4, followed by related work in Section 5 and a conclusion in Section 6.

2. BACKGROUND:

Vector Symbolic Architecture (VSA) is a family of models that enable connectionist models to perform symbolic processing, while encoding complex structures in distributed representations. A set of algebraic operations defined by these approaches allow them to compose, decompose and manipulate symbolic structures.

Our paper focuses on such an approach, namely Holographic Reduced Representation (HRR) proposed by (Plate, 1995) . HRR uses three operations: circular convolution, circular correlation and element-wise addition, to perform encoding, decoding and composition, respectively. The cicircular convolution (denoted by the operator * ) of two vectors x and y of dimension d, is defined as z = x * y, in which

z i = d k=1 x i mod d y (i-k) mod d , i = 1 . . . d

x * y is called the binding of x and y, or the encoding of the pair (x, y). Using this operation, the composition of a set of role-filler pairs (r 1 , f 1 ), (r 2 , f 2 ), . . . , (r m , f m ) is represented as H = r 1 * f 1 + r 2 * f 2 + . . . r m * f m , where + is element-wise addition used as a composition operator. The previous example John loves his mom can be represented as r agent * f John + r predicate * f loves + r patient * f his mom .

By definition, HRR guarantees that the composed representation remains a vector of dimension d, regardless of how many items are bound together by the * operation. This avoids the parameter explosion problem of other VSA approaches such as as Tensor Product Represenation (TPR) (Smolensky, 1990) , and makes HRR a more practical choice for representing compositional structures.

To decode from an HRR and retrieve a role or filler representation, an approximate inverse operation of the circular convolution, named circular correlation, is defined as t = x # y in which

EQUATION

Now given a memory trace z = x * y, the correlation operation allows us to retrieve y from the cue x via y ≈ x # z. We do not detail the exact conditions when this retrieval holds, but refer interested readers to the original paper (Plate (1995) )

3. HOLOGRAPHIC REDUCED REPRESENTATION FOR LANGUAGE MODELING:

We incorporate HRR into language models on two levels: word level and chunk level. Our HRRenabled language models (HRRLM) posit an explicit decomposition of word or chunk representations, which enables our model to capture different aspects of linguistic regularities. Before delving into the details of our model, we first introduce notations and provide a brief account of the commonly used RNN-based LM.

3.1. RNNLM:

RNN-based LMs estimate the probability of any given sentence s = w 1 w 2 . . . w n using an RNN. At each step t, RNN encodes the history w 1 w 2 . . . w t into a vector h, and tries to predict the next token w t+1 . Prediction is generally modeled by a linear layer followed by softmax operation. Specifically,

EQUATION

where E(•) is the embedding operation to embed a symbol into a continuous space R d , and V is the entire or a sampled subset of the vocabulary. The scoring function is defined by dot product, and therefore maximizing this probability encourages related words to form clusters, and away from other words in the embedded space.

3.2. WORD-LEVEL HRRLM:

Encoding We first use HRR to directly encode the underlying structures of words. We assume there is a decomposition of representations along N directions. Specifically, we embed a word w as

EQUATION

where r word i 's are basis role embeddings, shared by all words. Each basis role embedding is bound to its distinct set of filler embeddings, modeled by Ẽi . The motivation is that when properly trained, different bindings should capture disparate aspects of word representation. For instance, the first binding might be relevant for syntactic categories, and the second one for semantic relatedness. With this particular decomposition, the word getting should be close to other gerunds such as giving and forgetting in the first embedding space, and get, got or received in the second. In this case, the composite (i.e., the sum) of these bindings essentially encodes getting = {semantics: GET, syntax: GERUND}.

We additionally assume that each set of filler embeddings Ẽi resides in a separate linear subspace. This is achieved by modeling each Ẽi (w) with a linear combination of its associated basis filler embeddings f i,j . Specifically,

Ẽi (w) = d j=1 s w i,j f i,j = [f i,1 ; f i,2 ; . . . ; f i,d ]      s w i,1 s w i,2 . . . s w i,d      = F i s w i

where s w i ∈ R d is a word-specific d -dimensional vector. In other words, F i projects s w i ∈ R d to Ẽi (w) ∈ R d . This assumption has two advantages. First, the total number of parameters for the embedding layer is now V N d . We can set a smaller value d to prevent overparameterization, while maintaining a d-dimensional vector as the input to RNN. Second, by having separate bases for different role-filler bindings, we introduce an inductive bias for the model to learn a decomposition of word representation. This requirement makes sense intuitively -a decomposition of representation usually necessitates a separation of feature space. Our preliminary experiments show that this separation is essential for obtaining decomposed representations. The entire encoding operation is illustrated in the bottom half of Figure 1(b) .

Decoding The composite embedding E(w) is fed as input to RNN. From its output h given by the top hidden layer, we decode all the filler vectors using circular correlation, and factorize the scoring function into N parts. Specifically, we use the same word level loss as in Equation ( 2), but replace the scoring function in Equation (3) as a sum of dot products in every filler space: where α i 's are scalar hyperparameters that we use to break the symmetry of the scoring function. Specifically when N = 2, α 1 is set to a constant 1.0, and α 2 is linearly annealed from 0.0 at the start of training to 1.0 at a specified time step T , then remains constant afterwards. Note that dot products are only computed between co-indexed filler embeddings, namely between f i and Ẽi (w). This ensures the model only learns relatedness in the i-th subspace, without interference from other subspaces. The entire word-level HRRLM is illustrated is Figure 1(b) .

f i = r word i # h, score(h, w) = N i=1 α i f i • Ẽi (w) (5)

Regularization on basis embeddings Basis embeddings are chosen so that they are not correlated with each other. This decorrelation is promoted by adding an isometric regularization term to the basis embeddings. For instance, we define an isometric regularization penalty for the basis fillers F i 's as:

Φ(F 1 , F 2 , . . . , F N ) = β( F i F i -I 2 + F i F j 2 ), ∀1 ≤ i, j ≤ N, i = j,

where I is an identity matrix, and β is a hyperparameter. Similarly, we add a regularization term for r word i 's. As an alternative, we also consider using fixed random vectors for basis embeddings since high-dimensional random vectors are approximately orthogonal to each other.

3.3. CHUNK-LEVEL HRRLM:

A direct extension to the chunk-level model using the proposed techniques above is not desirable, due to two major difficulties. First, Equation (4) stipulates that each unique word type is assigned a vectorial parameter, which is computationally infeasible due to the vast number of unique chunks. Second, the same chunk can carry different semantic roles. For instance, the two sentences His mom loves John and John loves his mom have the same noun phrases, but with their roles of agent and patient switched. In light of these difficulties, we take on a compositional approach to address the first issue, and use a context-sensitive role to address the second.

Encoding Unlike the word-level HRR representation in Equation ( 4) where roles are fixed, the chunk roles are represented by linear combinations of M basis role embeddings. This formulation allows us to model context-dependency. Specifically, we construct HRR representations for a chunk c = w 1 w 2 . . . w m in three steps: STEP 1 (ROLE PREDICTION) We first predict a context-sensitive role tuple for each word w k in the chunk:

r w k = (r w k 1 , r w k 2 , . . . , r w k M ) = (a w k 1 r chunk 1 , a w k 2 r chunk 2 , . . . , a w k M r chunk M ).

a w k i 's represent the distribution of chunk roles, which are predicted by the same RNN used to predict the next tokens. Specifically, we feed the output vector from RNN through a linear layer and then a softmax layer. r chunk i 's are chunk-level basis role embeddings, shared by all chunks3 . Step 1 corresponds to the role prediction module in Figure 1(c) .

STEP 2 (PROJECTION AND BINDING) For each of the basis roles, we predict their associated filler embeddings. This is done by projecting the HRR word representation Ẽ(w k ) (Equation ( 4)) into M vectors. These fillers are then bound with their corresponding roles and summed together. Specifically,

g w k 1 , g w k 2 , . . . , g w k M = [W 1 ; W 2 ; . . . ; W M ] Ẽ(w k ), Êc (w k ) = M i=1 r w k i * g w k i = M i=1 (a w k i r chunk i ) * g w k i .

Intuitively, the first two steps aims to evaluate the representational contribution of w k to the entire chunk c. Following the practice in constructing a word-level model (Equation ( 4)), we also require the separation of different role-filler bindings. We also note that the binding Êc (w k ) embeds the word w k into a space that is specific to the chunk c. The first two steps are illustrated in Figure 1 (c).

STEP 3 (ENCODING) After obtaining bindings for all words within a chunk, the chunk embedding is defined as

Ê(c) = m k=1

Êc (w k ) (Figure 1(d) ). It can be easily verified that

EQUATION

Note that Ê(c) has the same form as Equation ( 4), and Êi (c) can be interpreted as the chunk filler embedding for the i-th chunk role. However, chunk embeddings are different in two key aspects. First, the filler embeddings for chunks are projected from the word embeddings, instead of being a set of independently trainable parameters. This compositional approach addresses the first issue regarding the vast number of unique chunks. Second, chunk embeddings rely on weights a w i 's from role prediction, which provide a natural vehicle for carrying contextual information, therefore addressing the second issue of context-sensitivity.

Chunk prediction Comparable to language modeling that uses next word prediction as supervision, our chunk-level HRR model uses next chunk prediction (CP in Figure 1(a) ). Specifically in our experiments, we simply concatenate the chunk embeddings from the last two steps, and feed it through a linear layer followed by tanh activation:

Ê(c τ ) = tanh [U 1 ; U 2 ] [ Ê(c τ -1 ); Ê(c τ -2 )] .

Decoding Similar to word-level HRR (Equation ( 5)), we decode the filler embeddings from the predicted chunk embedding using r chunk i as cue, and then use the decoded embeddings to factorize the scoring function. The same form of loss function is used. To provide negative examples in the denominator of the softmax Equation (2), we use all the chunks in the mini-batch. These chunks form a pseudo "chunk vocabulary" that is constructed on the fly. Role annealing, and regularization on basis embeddings are also applied.

Chunk boundaries Chunk boundaries have to be supplied in order to construct a chunk embedding. We reply on a third-party chunker to provide such annotations. This is analogous to our word-level model, only that word boundaries are trivially provided by whitespace in English.

4. EXPERIMENTS:



4.1. SETUP:

Data Sets We train all models on both Penn Treebank (PTB) (Marcus et al., 1994) and One-Billion-Word Benchmark dataset (1B) (Chelba et al., 2014) . For 1B, we experimented with three training sizes: the full dataset, one-tenth of the dataset, and one-hundredth. We evaluate all models on the following three aspects:

• Perplexity: We report perplexity to evaluate language models.

• Intrinsic evaluation: We use a test suite consisting of 18 word embedding benchmark datasets to intrinsically evaluate the quality of word embeddings (Jastrzebski et al., 2017) .4 • Extrinsic evaluation: We evaluate all word embeddings on six downstream tasks following Nayak et al. ( 2016): Part-of-speech tagging (POS), chunking, named entity recognition (NER), sentiment analysis (SA), question classification (QC), and natural language Inference (NLI). 5Training details For our models, We mainly experimented with two word-level and two chunklevel roles. The basis embeddings are either trained with isometric constraint, or fixed as constant after random initialization.

We also note that for PTB, we experimented with either contiguous input or noncontiguous input.

For contiguous input, we follow the common practice in the literature to feed the input sentences in the order that they appear in the document, and initialize the hidden state of LSTM with the last state from the last batch. We refer interested readers to the appendix for more training details.

4.2. PERPLEXITY RESULTS:

Although not a prime motivation of our approach, Table 1 shows that our HRR models can outperform the baseline in terms of perplexity, especially when we do not have enough training data. On PTB, the best HRR model obtains a gain of 2.2 with noncontiguous input (Fixed-big 88.0 vs Baseline 90.2), and 1.9 with contiguous input (Fixed-big-cont 74.7 vs Baseline-cont 76.6). We also note that increasing the number of basis fillers has a positive impact on HRR models (Fixed-big 88.0 vs Fixed-medium 90.2 vs Fixed-small 97.0). On 1B, we observe that HRR models benefit from having trainable bases with isometric regularization (107.2 vs 112.3 on 1/100 1B, and 55.9 vs 57.9 on full 1B).

4.3. WORD LEVEL ANALYSIS:

We then demonstrate that our word-level HRRLM can effectively separate certain attributes of word representation. Specifically, we look at our word-level models with two word-level roles, and find that the first set of filler embeddings captures mostly syntax-related categories, especially tense and agreement for verbs, whereas the second set focuses more on the semantic content of words. This decomposition is illustrated in Figure 2 , where we visualize both sets of filler embeddings for the most frequent 2500 words in PTB via t-SNE (Maaten & Hinton, 2008 ). In the first set (in blue), verbs with different inflectional markers form distinct clusters. For instance, bare forms, gerunds, preterites, and third-person-singular verbs each form a visually distinguishable group. In contrast, for the second set of filler embeddings (in orange), semantically related words tend to be close regardless of their morphosyntactic markers. For instance, give, gave, and giving are close in the second space.

We proceed to evaluate the quality of the learned word embeddings both intrinsically and extrinsically. As Table 2 summarizes, our HRR models consistently outperform the baseline on the intrinsic evaluation task (numbers on the left). The best HRR model Isometric-big obtains a gain of 9.0%, 11.6%, and 16.2% on 1/100 1B, 1/10 1B and full 1B, respectively. In addition, the HRR model with trainable bases has a noticeable advantage over the model with fixed bases (2.5% on average). On the other hand, the HRR models have only a marginal advantage over the baseline (numbers on the right), which echos findings in (Schnabel et al., 2015) which claims that intrinsic improvement for word embeddings does not necessarily correlate with improvement in downstream tasks. Detailed results for each dataset and downstream task are available in A breakdown of the intrinsic evaluation reveals that we obtain huge gains on the word analogy task, especially in the verb-related categories such as plural verbs and past tense. On the Semantic-Syntactic analogy dataset Mikolov et al. (2013) , the accuracy (top 1) for plural verbs is 0.897 if we consider the most frequent 200K words. If we limit the vocabulary to the more common words (10K), the accuracy for present participles reaches 0.964. This is consistent with our previous observation that the first set of filler embeddings effectively captures the representation of different verb forms.

To further quantify how decomposed are our word-level filler embeddings, we use them to classify whether two words are related in terms of either semantics or syntax. Specifically, we use a threshold θ for cosine similarity to determine whether any pair of words (w, w ) are related. We then use different values of θ to plot ROC curve and compute its AUC. For the ground truths, we use the syntactic categories from the Semantic-Syntactic dataset, and extract them from each a : b = c : d example.

Two sets of categorizations are obtained, driven by semantics and syntax respectively. For instance, for the example make : making = give : giving, we obtain ({making, giving}, {make, give}) for syntactic considerations, and ({making, make}, {giving, give}) for semantics. For the verb-related categories, Table 4 shows that the first set of filler embeddings does noticeably better than the second in the first experiment, while the second set is much better at the second experiment. Figure 3 shows two such categories which confirm that the decomposition does make sense on a crude syntax-semantics level. More details of the intrinsic evaluation results can be found in Table 8 

4.4. CHUNK LEVEL ANALYSIS:

We train a chunk-level HRR model by initializing it with a pretrained word-level model on PTB. 6 We evaluate the quality of chunk embeddings in two ways. First, we perform a human analysis focused on the phrase the company, which is the most frequent noun phrase in PTB. We randomly select 100 occurrences, and cluster their chunk embeddings using K-means into 5 categories based on the chunk-level filler embeddings. 7 For each sentence cluster we manually identify the dominating role which the company played in that cluster of sentences. Table 5 shows that there is a clear role for the phrases in each of the clusters. We also performed a t-SNE analysis using the second filler embedding, in which we observed the company is close to many semantically related nouns such as stock, market and etc. From this analysis we see that the first role in chunk-level HRR is more sensitive to the different syntactic roles the company plays dependent on the context, while the second role is more associated with its related semantic concepts.

Secondly, to further investigate if our model consistently captures chunk-level roles, we run the model on the Wall Street Journal part of the OntoNotes dataset (Weischedel et al., 2013) in which semantic role labels for each sentence are provided, and compare chunk embedding clusters against the role labels. We are mainly interested in 4 coarser-grained labels corresponding to AGENT, PATIENT, PREDICATE and MODIFIER. We provide two metrics to measure the quality of the obtained clusters. First, for each chunk c, we find its k nearest neighbors (k = 1, 5, 10, 25) according to their filler embeddings, and report the percentage of the cases where c shares the same role labels as its neighbor. This hit-rate metric gives us a sense of how accurately chunk clusters capture meaningful semantic roles. In addition, we compute purity score for the learned clusters against the ground truth clusters. Both metric scores are reported in Table 6 , from which we can see that chunk embedding indeed correctly captures semantic roles more than half of the time. Moreover, the first set of fillers are more accurate at clustering these roles than the second, and also deteriorates better when we include more neighbors to compute the hit rates.

5. RELATED WORK:

Perhaps most related to our work are recent attempts to integrate tensor product structure with neural networks (Palangi et al., 2017; Huang et al., 2018) . While these works and ours share a common goal of incorporating neural models with symbolic structures, there are several difference. First of all, we make use of HRR instead of tensor product as basis for our representation to enable long sequence encoding without parameter explosion. Moreover, we aim to induce linguistic structures from a task that requires as little supervision as possible like language modeling, whereas their work is focused on question answering which provides stronger guidance signal from labeled data. On the other hand, recent work (Shen et al., 2018 ) also proposes a novel network architecture that is capable of learning syntactic roles and semantics jointly, but it is not based on structured representation like HRR.

There have been many attempts of using symbolic architectures like HRR and TPR for linguistic analysis, see for example (Jones & Mewhort, 2007; De Vine & Bruza, 2010; Recchia et al., 

APPENDIX C CHUNK LEVEL ANALYSIS:

To obtain clustering results, we used sklearn with the default parameters and settings provided by the toolbox. For the OntoNotes experiment, we removed sentences that have unknown tokens of more than 10 percent, and removed chunks that are longer than six words. In addition, we removed some rare role labels (out of 11), and collapsed some object-related roles (e.g., direct object, and indirect object) into PATIENT. This resulted in a total of around 4K chunks, and four remaining coarse labels AGENT, PATIENT, PREDICATE, and MODIFIER, which correspond to the original tags ARG0, ARG1, V and ARGM. 

Code will be released to the public.

For attention mechanisms, there is actually a distribution over multiple embeddings, rather than one single complex embedding.

Note that chunk-level bases r chunk i 's are different parameters from word-level bases r word i 's

https://github.com/kudkudak/word-embeddings-benchmarks.

https://github.com/NehaNayak/veceval

The perplexity for the chunk-level model is 89.5 vs 88.0 for the word-level model. The degradation of performance for next word prediction is noticeable. However, we do note that such degradation is relatively common for some multitask settings.

More details about the experimental setup for chunk level analysis are provided in Appendix C.

Prince & Smolensky, 1997;Clark et al., 2008;Clark & Pulman, 2010;Grefenstette et al., 2011). HRR itself as a variable binding and association mechanism, has also been integrated with neural networks with different motivations like associative memory modeling(Danihelka et al., 2016), relationship reasoning(Weiss et al., 2016) etc. While most of the work mainly focuses on symbolic and formal analysis via algebraic operations and logic derivations, our work aims to enable neural language models to learn linguistic roles by taking advantage of the HRR properties.6 CONCLUSIONIn this paper, we employ HRR to provide a principled decomposition of representation. We design our HRR language models to work on both word-level and chunk-level. Our analysis revealed that by introducing an inductive bias, our models can learn disentangled representations, which roughly corresponds to syntax and semantics.

https://www.clips.uantwerpen.be/pages/MBSP

Figures and Tables:

Figure 1: Architecture of HRRLMs. (a) Chunk-level HRRLM. (b)Word-level HRRLM. (c) Step 1 and 2 of chunk-level model encoding. (d) Step 3 of chunk-level model encoding. See Section 3 for details.

Figure 2: t-SNE visualizations for two sets of filler embeddings (blue and orange).

Figure 3: Examples of ROC curves for two categories. The left figure is for a syntax-driven categorization of gerunds, while the right is a semantics-driven categorization of plural verbs. Isometricbig-f1 means the first filler embedding, and Isometric-big-f2 the second.

Figure 4: ROC for every syntactic category in the first set of categorization experiments (syntaxdriven).14

Figure 5: ROC for every syntactic category in the second set of categorization experiments (semantics-driven).15

Perplexity results on PTB and 1B data sets. All HRR models have two word-level roles, and each role has 320, 200, and 50 basis filler embeddings for Fixed-big, Fixed-medium and Fixedsmall, respectively.

Table content: <table><tr><td>Model</td><td colspan="3">PTB 1/100 1B 1/10 1B</td><td>1B</td></tr><tr><td>Baseline</td><td>90.2</td><td>120.4</td><td>64.5</td><td>56.0</td></tr><tr><td>Baseline-cont</td><td>76.6</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Fixed-small</td><td>97.0</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="2">Fixed-medium 90.2</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Fixed-big</td><td>88.0</td><td>112.3</td><td>64.2</td><td>57.9</td></tr><tr><td colspan="2">Fixed-big-cont 74.7</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Isometric-big</td><td>88.2</td><td>107.2</td><td>63.9</td><td>55.9</td></tr></table>

Table 7 in Appendix B. Averaged scores for intrinsic evaluation (18 datasets, numbers on the left) and extrinsic evaluation (6 tasks, numbers on the right).

Table content: <table><tr><td>Model</td><td>1/100 1B</td><td>1/10 1B</td><td>1B</td></tr><tr><td>Baseline</td><td>0.386|0.818</td><td>0.454|0.816</td><td>0.427|0.817</td></tr><tr><td>Fixed-big</td><td>0.466|0.823</td><td>0.552|0.817</td><td>0.543|0.821</td></tr><tr><td colspan="4">Isometric-big 0.476|0.820 0.570|0.819 0.589|0.822</td></tr></table>

Accuracy (top 1) on the Semantic-Syntactic analogy benchmark. The most frequent 200K words are used. We use COSMUL for all analogy testsLevy & Goldberg (2014)

Table content: <table/>

Average AUC on verb-related categories for the baseline, and also the two sets of filler embeddings from Isometric-big. All models are trained on the full 1B dataset. The first row corresponds to the first syntactic-driven categorization experiment, and the second row to the semantic-driven experiment.

Table content: <table><tr><td>in</td></tr></table>

Cluster analysis for a random selection of 100 sentences containing the company. The most dominant role in each cluster is identified by a human judge.

Table content: <table><tr><td>Metric</td><td colspan="2">Filler 1 Filler 2</td></tr><tr><td>Hit rate@1</td><td>0.604</td><td>0.572</td></tr><tr><td>Hit rate@5</td><td>0.604</td><td>0.553</td></tr><tr><td colspan="2">Hit rate@10 0.600</td><td>0.544</td></tr><tr><td colspan="2">Hit rate@25 0.586</td><td>0.531</td></tr><tr><td>Purity</td><td>0.677</td><td>0.610</td></tr></table>

Hit rate and purity score analysis for chunk embeddings. See Sec.4.4 for more details.

Table content: <table/>

Word analogy test results on different datasets.

Table content: <table><tr><td>2015;</td></tr></table>

Detailed word analogy test results in each category.

Table content: <table/>

Peer review (iclr2019):
 The paper proposes a new approach for neural language models based on holographic reduced representations (HRRs). The goal of the approach is to learn disentangled representations that separate different aspects of a term, such as its semantic and its syntax. For this purpose the paper proposes models both on the word and chunk level. These models aim disentangle the latent space by structuring the latent space into different aspects via role-filler bindings. Learning disentangled representations is a promising research direction that fits well into ICLR. The paper proposes interesting ideas to achieve this goal in neural language models via HRRs. Compositional models like HRRs make a lot of sense for disentangling structure in the embedding space. Some of the experimental results seem to indicate that the proposed approach is indeed capable to discover rough linguistic roles. However, I am currently concerned about different aspects of the paper: - From a modeling perspective, the paper seems to conflate two points: a) language modeling vie role-filler/variable-binding models and b) holographic models as specific instance of variable bindings. The benefits of HRRs (compared e.g., to tensor-product based models) are likely in terms of parameter efficiency. However, the benefits from a variable-binding approach for disentanglement should remain across the different binding operators. It would be good to separate these aspects and also evaluate other binding operators like tensors products in the experiments. - It is also not clear to me in what way we can interpret the different filler embeddings. The paper seems to argue that the two spaces correspond to semantics and syntax. However, this seems in no way guaranteed or enforced in the current model. For instance, on a different dataset, it could entirely be possible that the embedding spaces capture different aspects of polysemy. However, this is a central point of the paper and would require a more thorough analysis, either by a theoretical motivation or a more comprehensive evaluation across multiple datasets. - In its current form, I found the experimental evaluation not convincing. The qualitative analysis of filler embeddings is indeed interesting and promising. However, the comparisons to baseline models is currently lacking. For instance, perplexity results are far from state of the art and more importantly below serious baselines. For instance, the RNN+LDA baseline from Mikolov (2012) achieves already a perplexity of 92.0 on PTB (best model in the paper is 92.4). State-of-the-art models acheive perplexities around 50 on PTB. Without an evaluation against proper baselines I find it difficult to accurately assess the benefits of these models. While language modeling in terms of perplexity is not necessarily a focus of this paper, my concern translates also to the remaining experiments as they use the same weak baseline. - Related to my point above, the experimental section would benefit significantly if the paper also included evaluations on downstream tasks and/or evaluated against existing methods to incorporate structure in language models. Overall, I found that the paper pursues interesting and promising ideas, but is currently not fully satisfying in terms of evaluation and discussion.