HUBERT Untangles BERT to Improve Transfer across NLP Tasks
Authors: Mehrad Moradshahi
Hamid Palangi
Monica S. Lam
Paul Smolensky
Jianfeng Gao
Venue:  (2020)
Keywords: Tensor Product Representation
BERT
Transfer Learning
Neuro-Symbolic Learning

Abstract: We introduce HUBERT which combines the structured-representational power of Tensor-Product Representations (TPRs) and BERT, a pre-trained bidirectional transformer language model. We validate the effectiveness of our model on the GLUE benchmark and HANS dataset. We also show that there is shared structure between different NLP datasets which HUBERT, but not BERT, is able to learn and leverage. Extensive transfer-learning experiments are conducted to confirm this proposition.

Under review as a conference paper at ICLR 2020
HUBERT UNTANGLES BERT TO IMPROVE TRANSFER ACROSS NLP TASKS
A Anonymous authors Paper under double-blind review
ABSTRACT
We introduce HUBERT" which combines the structured reprsstttoee power of Tensor-Product Representations (TPRs) and BERT, pre-trained bidirectional Transformer language model. We show that there is shared structure between different NIP datasets that HUBERT, but not BERT, is able learn and leverage. We validate the effectiveness of our model on the GIUE benchmark and HANS dataset. Our experiment results show that untangling data-specific semantics from general language structure is key for better transfer among NLP tasks 2
INTRODUCTION
Built on the Transformer architecture (Vaswani al. 2017). the BERT model (Devlin et al. 2018 has demonstrated great power for providing general-purpose vector embeddings of natural language: its representations have served as the basis of many successful deep Natural Language Processing (NLP) models on a variety of tasks (e.g.. Liu et al.] 2019a bj Zhang et al.] 2019). Recent studies
(Coenen al., 2019: Hewitt & Manning 2019] Lin al.] | Tenney 20199 have shown that BERT representations carry considerable information about grammatical structure, which, by design, is deep and general encapsulation of linguistic information. Symbolic computation over structured symbolic representations such as parse trees has long been used to formalize linguistic knowledge. To strengthen the generality of BERT's representations, we propose to import into its architecture this type of computation.
Symbolic linguistic representations support the important distinction between content and form in- formation. The form consists of structure devoid of content, such as unlabeled tree, collection of nodes defined by their structural positions or roles (Newell, 1980), such as root, left-child-of-roo fohh--cildd-o--eff-chhhll
moot etc. In a particular linguistic expression such as "Kim referred to herself during the speech". these purely-structural roles are filled with particular content-bearing symbols, including terminal words like Kim and non-terminal categories like NounPhrase, These role fillers have their own identities, which are preserved as they move from role to role across expressions: Kim retains its referent and its semantic properties whether it fills the subject or the object role in a sentence. Structural roles too maintain their distinguishing properties as their fillers change: the root role dominates the refl-child-of-rate role regardless of how these roles are filled.
Thus it is natural to ask whether BERT's representations can be usefully factored into content x form, i.e., filler x role, dimensions. To answer this question, we recast it as: can BERT's representa- tions be usefully unpacked into Tensor-Produs Representations (TPRs)? A TPR is a collection of constituents, each of which is the binding of a filler to a structural role. Specifically, we let BERT's final-layer vector-encoding of each token of input string be factored explicitly into filler bound to role: both the filler and the role are embedded in a continuous vector space, and they are bound together according to the principles defining TPRs: with the tensor product. This factorization ef- fectively untangles the fillers from their roles, these two dimensions having been fully entangled in the BERT encoding itself. We then see whether disentangling BERT representations into TPRs facilitates their general use in a range of NLP tasks.
Concretely, as illustrated in Figure I we create HUBERT by adding a TPR layer on top BERT; this layer takes the final-layer BERT embedding of each input token and transforms it into the tensor
HUBBETTE is recursive acronym for HUBERT Untangles BERT. code and will made available publication.

---

Under review as a conference paper at ICLR 2020
product of a filler umbedding-vector and a role embedding-vector The model learns separate fillers from roles in an unsupervised fashion, trained end-to-end to perform an NLP task.
If the BERT representations truly veneral-purposs. for NLP, the TPR re-coding should reflect this generality. In particular, the formal, grammatical knowledge we expect to be carried by the roles should be generally useful across wide range of downstream tasks. We thus examine transfer learning, asking whether the roles learned in the service of one NLP task can facilitate learning when carried over to another task.
In brief, overall we find in our experiments on the NLP benchmarks of GLUE (Wagg et al. 2018) and HANS (McCoy et al. 2019) that HUBERT's recasting of BERT encodings as TPRs does indeed lead to effective knowledge transfer across NLP tasks, while the bare BERT encodings do not. Specifically, after pre-training on the MNLI dataset in GLUE, we observe positive gains ranging from 0.60% to 12.28% when subsequently fine-tuning on QNLI, QQP, RTE, SST, and SNLI tasks. This is due to transferring TPR knowledge- in particular the learned roles-relaane relative to transferring just BERT parameters which have gains ranging from minus 0.33% to positive 2.53%.
Additionally, on average, we gain 5.7% improvement on the demanding non-entailment class of the HANS challenge dataset. Thus TPR's disentangling of fillers from roles, motivated by the nature of symbolic representations, does yield more general deep linguistic representations as measured by cross-task transfer.
The paper is structured as follows. First we discuss the prior work on TPRs deep learning and its previous applications in Section 2 We then introduce the model design in Section 3 and present our
experimental results in Section
We conclude in Section
RELATED WORK
Building on the successes of symbolic Al and linguistics since the mid-1950s, there has been long line of work exploiting symbolic and discrete structures in neural networks since the 1990s.
Along with Holographic Reduced Representations (Plate 1995) and Vector-Symbolic Architectures (Levy & Gayler 2008). Tensor Product Representations (TPRs) provide the capacity to represent the discrete linguistic structure in continuous, distributed manner, where grammatical form and semantic content can be disentangled (Smolensky) 1990, Smolensky & Legendre, 22066). In Lee
eet al. (2016), TPR-like representations were used solve the bAbl tasks (Weston al. 2016
achieving close to 100% accuracy in all but one of these tasks. [Schlag & Schmidhuber (2018)
also achieved success on the bAbl tasks, using third-order TPRs encode and process knowledge- graph triples. In Palangi et al. (2018), a new structured recurrent unit (TPRN) was proposed to learn grammatically -aaammttcaall--mteepetabbbbb representations using weak supervision from (context, question, answer) triplets in the SQuAD dataset (Rajpurkar et al. 2016). In Huang et al. (2018). unbinding operations of TPRs were used to perform image captioning. None of this previous work, however, examined the generality of learned linguistic knowledge through transfer learning.
Transfer learning for transformer-base models has been studied recently: Keskar et al. (0199) and Wang et al. (2019) report improvements in accuracy over BERT after training on an intermediate task from GLUE; an approach which has come to be known as Supplementary Training on Intermediate Labeled data Tasks (STILTs). However, as shown in more recent work (Phang et al., 2018). the results do not follow consistent pattern when using different corpora for fine-tuning BERT, and degraded downstream transfer is often observed. Even for data-rich tasks like QNLI, regardless of the intermediate task and multi-tasking strategy, the baseline results do not improve. This calls for new model architectures with better knowledge transfer capability.
3 MODEL DESCRIPTION
Applying the TPR scheme to encode the individual words (or sub-word tokens) fed to BERT, word is represented as the tensor product of a vector embedding its content-its filler (or symbol aspect-and a vector embedding the structural role it plays in the sentence. Given the results of
We use filler symhol to indicate content symholic representation throughout the paper.
2

---

Under review as a conference paper at ICLR 2020
Palangi et al. (0018), expect the symbol to capture the semantic contribution of the word while the structural role captures its grammatical role:
x (t) X (1)
(I)
Assuming we have ng symbols with dimension ds and ng roles with dimension dR, (0) E Ids xdn the tensor representation for token i, s(²) Rdds the (presumably semantic) symbol repre- sentation and r(t) E Rd is the (presumably grammatical) role representation for token s(() may be either the embedding of one symbol or a linear combination of different symbols using softmax symbol selector, and similarly for r(() In other words, Eq. | can also be represented as (+)) = (t)) Rd where S E XRdsxns and R E Rda xnR are matrices the columms of which contain the global symbol and role embeddings, common for all tokens, and either learned from scratch or initialized by transferring from other tasks, as explained in Section 4
( E IR x is the bind- ing matrix which selects specific roles and symbols (embeddings) from R and S and binds them together. We assume that for a single-word representation, the binding matrix B is rank 1, so we can decompose it into two separate vectors, one soft-selecting a symbol and the other a role, and rewrite equation as (t) S(ag aR be interpreted as attention weights over different roles (columns of R) and symbols (columns of S).
(+) (f) R where a R and a "2 can respectively
For each input token x²(), get its contextual representations of grammatical role (a and mantic symbol (a. by fusing the contextual information from the role and symbol representations of its surrounding tokens.
We explore two options for mapping the input token from the current time-step, and the tensor (t) representation from the previous time-step. dg and ag Long Short-Term Memory (LSTM) architecture (Hochreiter & Schmidhuber 1997) and a one-layer Transformer. All the models share the general architecture depicted Figure [ except for BERT and BERT-LSTM, where the TPR layer is absent. Our conclusion based on initial experiments was that the Transformer layer results in better integration and homogeneous combination with the other Transformer layers in BERT, as will be described shortly.
Pr(c
Linear classifier
Aggrogation layer
TPR layer
b(t)
BERT model
09)
(0)
09)
Figure 1: General architecture for all models: HUBERT models have TPR layer; BERT and BERT-
LSTM don't. BERT and TPR layers can be shared between tasks but the classifier is task-specific.
The TPR layer with LSTM architecture works as follows (see also Figure 2
discussed in Sec. 4.2
We calculate the hidden states (h h (t) and cell states (c() .c to the following equations:
(() and for each time-step according
hg (0) (t) = LSTMss (v)) cg ((-")));
h CR LSTMR(M, (veo(x ch ((-1))) (2)
where U, is the final-layer BERT embedding of the t-th word, and vec(.) flattens the input tensor into a vector. Each LSTM's input cell state is the previous LSTM's cell output state. Each LSTM's input hidden state, however, is calculated by binding the previous cell's role and symbol vectors.
3

---

Under review as a conference paper at ICLR 2020
In the TPR layer with Transformer architecture, we calculate the output representations (hgg) k12)) using a Transformer Encoder layer:
i Trransformrrg(:))::
(e) R
Transformerz((5))
Each Transformer layer consists of multi-head attention layer, followed by a residual block (with dropout), a layer normalization block, feed-forward layer, another residual block (with dropout), and a final layer normalization block. (See Figure 3] discussed in Sec. 4.2 and the original Trans- former paper. Vaswani et al. (0017), for more details.)
Given that each word is usually assigned to few grammatical roles and semantic concepts (ideally one), an inductive bias is enforced using a softmax temperature (T) to make a and as sparse. Note that in the limit of very low temperatures, we will end up with one-hot vectors which pick only one filler and one role
(c) ag softmax((ssll)lllo (Wsh (6) (T);
(t) aR
softmax(WRRh) /T)
(4)
Here Ws and WR are linear-layer weights. For the final output of the transformer model, we explored different aggregation strategies to construct the final sentence embedding:
P(cff) = softmax Wj Agg((x x(1)
x (N2))
(5)
where P(c]f) is probability distribution over class labels, is the final sentence representation, W, is the classifier weight matrix, and N is the maximum sequence length. Agg(-) defines the merging strategy. We experimented with different aggregation strategies: max-pooling, mean- pooling, masking all but the input-initial [CLS] token, and concatenating all tokens and projecting down using linear layer. Devlin et al. (2018), the final representation for the [CLS] token used as the sentence representation. However, during our experiments, we observed better results when concatenating the final embeddings for all tokens and then projecting down smaller dimension, as this exposes the classifier to the full span of token information.
The formal symmetry between symbols and roles evident in Eq. T is broken in two ways.
First, we choose hyper-parameters so that the number of symbols is greater than the number of roles. Thus each role on average used more often than each symbol, encouraging more general information (such as structural position) to be encoded in the roles, and more specific information (such as word semantics) to be encoded the symbols. (This effect was evident in the analogous TPR learning model of Palangi et al. (2018).)
Second, to enable the symbol that fills any given role to exactly recoverable from a TPR which it appears along with other symbols, the role vectors should be linearly independent: this expresses the intuition that distinct structural roles carry independent information. Fillers, however, are not expected be independent in this sense, since many fillers may have similar meanings and be quasi-intercaangeable. So for the role matrix R, but not the filler matrix S, we add a regularization term the training loss which encourages the R matrix be orthogonal:
C C= - 1 co]log P(clf) + AQRRR - -da 12 + R R - R-1n Im)
(6)
Here L indicaatss the loss function, I3 is the identity matrix with k rows and k columns, and 1[.] is the indicator function: it is when the predicted class c matches the correct class e" label, and 0 other- wise. Following the practice in Bansal et al. 2018) use double soft orthogonality regularization to handle both over-complete and under-complete matrices R.
Note parameters omitted simplicity presentation.
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
We performed extensive experiments to answer the following questions:
1. Does adding TPR layer top of BERT (as in the previous section) impact its perfor- mance positively or negatively? We are specifically interested in MNLI for this experiment because it is large-scale compared to other GLUE tasks and is more robust model noise (i.e., different randomly-initialized models tend to converge to the same final score on this task). This task is also used as the source task during transfer learning. This experiment is mainly sanity check to verify that the specific TPR decomposition added does not hurt task performance.
2. Does transferring the BERT model's parameters, fine-tuned on one of the GLUE tasks, help the other tasks in the Natural Language Understanding (NLU) benchmarks (Bowman al. 2015 Wang et al. 2018)? Based on hypothesis of the advantage of disentangling content from form, the learned symbols and roles should be transferable across natural language tasks. Does transferring role (R) and/or symbol (S) embeddings (described in the previous section) improve transfer learning on BERT across the GLUE tasks?
3. Is the ability to transfer the TPR layer limited to GLUE tasks? Can be generalized? To answer this question we evaluated our models on challenging diagnostic dataset outside of GLUE called HANS (McCoy al. 2019).
4.1 DATASET
We conduct three major experiments answer the above questions: comparison of architectures on the MNLI dataset, a study of transfer learning between GLUE tasks (Wang et al. 2018). and finally model diagnosis using HANS (McCoy al. 2019); these are discussed in Sections 4. and 4.4 respectively.
Section A. provides more detailed analysis of the dataset. Table 4 (see Appendix) shows the dataset statistics and details for GLUE, SNLI, and HANS.
4.2 ARCHITECTURE COMPARISON ON MNLI
Our experiments are done with four different model architectures. The general architecture of the models is depicted in Figure ! The TPR layer is absent in BERT and BERT-LSTM. In the figure, BERT base model which has 12 Transformer encoder layers. The aggregation layer computes the final sentence representation (see Eq. [5) The linear classifier is task-specific and is not shared between tasks during transfer learning.
the BERT model indicates the pre-trained
BERT: This is our baseline model which consists of BERT, an aggregation layer on top, and a final linear classifier.
We augment the BERT model by adding unidirectional LSTM Recurrent layer (Hochreiter Schmidhuber, 1997 Sutskever al. 2014) on top. The inputs to the LSTM are token representations encoded by BERT. We then take the final hidden state of the LSTM and feed it into a classifier to get the final predictions. Since this model has an additional layer augmented on top of BERT, it can serve baseline for TPR models introduced below.
BERT-LSTM:
HUBERT (LSTM): We use two separate LSTM networks to compute symbol and role representa- tion for each token. Figure2 how final token embedding (x+()) constructed at each time step: this plays the role of the LSTM hidden state (t). (In the figures, 'O" denotes matrix -vector multiplication.) The results (Table show that this decomposition improves the accuracy on MNLI compared to both the BERT and BERT-LSTM models. Training recurrent models usually diffi- cult, due to exploding or vanishing gradients, and has been studied for many years (Le et al. 2015] Vorontsov et al., 2017). With the introduction of the gating mechanism LSTM and GRU cells, this problem was alleviated. In our model, we have tensor-product operation which binds role and symbol vectors. We observed that during training the values comprising these vectors can reach very small numbers (<c 10 10-+), and after binding, the final embedding vectors have values roughly in the order 10-8, This makes difficult for the classifier to distinguish between similar but different sentences. Additionally, backpropagation not effective since the gradients are too small.
5

---

Under review as a conference paper at ICLR 2020
We avoided this problem by linearly scaling all values by large number (~ 1K) and making that scaling value trainable so that the model can adjust it for better performance.
LSTM cell
LSTM cell
x
AS
W
00
LSTM cell
LSTM cell
UR)
Figure 2: TPR layer architecture for HUBERT (LSTM). R and S are global Role and Symbol embedding matrices which are learned and re-used at each time-step.
HUBERT (Transformer): In this model, instead of using recurrent layer, we deploy the power of
Transformers /Vaswani al. 2017 to encode roles and symbols (see Figure 3 This lets us attend (k) to all the tokens when calculating R and ag
(k) and thus better capture long-distanes dependencies. also speeds up training as all embeddings are computed in parallel for each sentence. Furthermore, it naturally solves the vanishing and exploding gradients problem, by taking advantage of residual blocks (He al., 2015) to facilitate hackpropagation and Layer Normalization (Lei Ba et al. 2016) to prohibit value shifts. It also integrates well with the rest of the BERT model and presents more homogencous architecture
We first do an architecture comparison study the four models, each built on BERT (base model). We fine-tune BERT on the MNLI task, which we will then use as our primary source training task for testing transfer learning. We report the final accuracy on the MNLI development set.
Table D summarizes the results. Both HUBERT models are able to maintain the same performance as our bascline (BERT). This confirms that adding TPR heads will not degrade the model's accuracy and can even improve it (in our case when evaluated on MNLI matched development set). Although HUBERT (Transformer) and HUBERT (LSTM) have roughly the same accuracy, choose HU- BERT (Transformer) to perform our transfer learning experiments, since eliminates the limitations of HUBERT (LSTM) (as discussed above) and reduces the training and inference time significantly (> 4X).
Model
BERT
BERT-LSTM
HUBERT (LSTM)
HUBERT (Transformer)
Accuracy (%)
84.15
84.17
84.26
84.30
Table MNLI (matched) dev set accuracy for different models.
4.3 TRANSFER LEARNING
We compare the transfer-learning performance of HUBERT (Transformer) against BERT. We follow the same training procedure for each model and compare the final development set accuracy on the target corpus. The training procedure is as follows: For Baseline, we train three instances of each model on the target corpus and then select the one with the highest accuracy on target dev set (We vary the random seed and the order in which the training data is sampled for each instance.) These results are reported for each model in the Baseline Acc. column in Table 2 For Fine-tuned, in separate experiment, we first mune one instance of each model on the source corpus and use these
Thee results reported here corresonn to an implementation using additional Transformer encuder layer on top of the TPR layer, we the input values this layer only. versions of the model will omit layer.

---

Under review as a conference paper at ICLR 2020
S
eg
(N)
softroor
softrea
WR
W
W
W/y
W/
W
Nx
Ag
Residual black
Residual block
Feed Forward layer
Feed Forward layer
Residual Layer norm
Residual Layer
Multi-Head Attention
Multi-Heal Attention
Figure 3: TPR layer architecture for HUBERT (Transformer). and are global Role and Symbol embeddings which are learned and shared for all token positions.
updated parameters to initialize a second instance the same model. The initialized model will then be trained and tested the target corpus. In this setting. have three subsets of parameters to choose from when transferring values from the source model to the target model: BERT parameters, the Role embeddings R, and the Filler embeddings F. Each of these subsets can independently be transferred or not, leading to a total of combinations excluding the option in which none of them are transferred. We chose the model which has the highest absolute accuracy on the target dev dataset. These results are reported for each model under Fine-tuned Acc. Note that the transferred parameters are not frozen, but updated during training on the target corpus.
Table 2 summarizes the results for these transfer learning experiments when the source task is MNLI. Gain shows the difference between Fine-tuned model's 's accuracy and Base- line's accuracy. For HUBERT (Transformer), we observe substantial gain across all 5 target corpora after transfer. However, for BERT we have a drop for QNLI, QQP, and SST.
MNLI source:
These observations confirm our hypothesis that recasting the BERT encodings as TPRs leads to better generalization across down-stream NLP tasks.
Almost all tasks benefit from transferring roles except for QNLI. This may be due to the structure of this dataset, as it is modified version of a question-anee mnestion-answering dataset (Rajpurkar al. (016) and has been re-designed to be an NLI task. Transferring the filler embeddings helps with only QNLI and RTE. Transferring BERT parameters in conjunction with fillers or roles surprisingly boosts accuracy for QNLI and SST, where we had negative gains for the BERT model, suggesting that TPR decomposition can also improve BERT's parameter transfer.
The patterns here quite different as the source task is now paraphrase task (instead of inference) and TPR needs to encode a new structure. Again transferring roles gives positive results except for RTE. Filler vectors learned from QQP are transferable compared to MNLI and gives a boost to all tasks except for SNLI Surprisingly, transferring BERT parameters is hurting the results now even when TPR present. However, for cases in which we also transferred BERT parameters (not shown), the Gains were still higher than for BERT, confirming the results obtained when MNLI was the source task
QQP as source:
"useeline slightly Table due to using different scaling value this source

---

Under review as a conference paper at ICLR 2020
Model
Target Corpus
Transfer BERT
Transfer Filler
Transfer Role
Baseline Acc. (%)
Fine-
Gain (%)
tuned
Acc. (%)
BERT
QNLI
True
91.60
91.27
0.33
BERT
QQP
True
91.45
91.12
-.33
BERT
RTE
True
71.12
73.65
+ 2.53
BERT
SNLI
Truc
90.45
90.69
+ 0.24
BERT
SST
Truc
93.23
92.78
0.45
HUBERT (Transformer)
QNLI
Truc
True
Falsc
90.56
91.16
+ 0.60
HUBERT (Transformer)
QQP
Falsc
False
True
90.81
91.42
0.61
HUBERT (Transformer)
RTE
True
True
True
61.73
74.01
12.28
HUBERT (Transformer)
SNLI
True
False
True
90.66
91.36
+ 0.70
HUBERT (Transformer)
SST
True
False
True
91.28
92.43
+ 1.15
Table 2: Transfer learning results for GLUE tasks. The source corpus is MNLI. Baseline accuracy is when Transfer BERT, Filler, and Role are all False, equivalent to no transfer. Fine-tuned accuracy the best accuracy among all possible transfer options.
We also verified that our TPR layer is not hurting the performance by comparing the test set results for HUBERT (Transformer) and BERT. The results are obtained by submitting models to the GLUE evaluation server. The results are presented in Table 6
4.4 MODEL DIAGNOSIS
We also evaluated HUBERT (Transformer) on probing dataset outside of GLUE called HANS (McCoy et al. 2019) Results are presented in Table 3 HANS is a diagnosis dataset that probes various syntactie heuristics which many of the state-of-the-art models turn they perform poorly on cases that don't follow those heuristics. There are three heuristics measured in HANS which are as follows: Lexical overlap where premise entails any hypothesis built from subset of words in the premise, Subsequence where premise entails any contiguous subsequences of it, and Constituent where premise entails all complete subtrees in its parse tree. Our results indicate that TPR models are less prone to adopt these heuristics, resulting in versatile models with better domain adaptation. Following McCoy al. (2019), we combined the predictions of neutral and contradictory into a non-entailment class, since HANS uses two classes instead of three. Note that no subset of the HANS data is used for training
exploit, and thus
We observed that HUBERT (Transformer) model trained MNLI did not diminish BERT's near-perfect performance on correctly-entaledd cases (which follow the heuristics). In fact, it in- creased the accuracy of Lexical and Subsequence heuristics. On the problematic Non-En Eottlmmtilm cases, however, BERT outperforms HUBERT (Transformer). Since HUBERT has more parameters than BERT it can better fit the training data. Thus, suspect that HUBERT attends more to the heuristics that MNLI has in its design, and gets a lower score on sentences that don't 't follow those heuristics. But to examine the tnowledge-transfer power of TPR, we additionally fine-tuned each model on SNLI and tested again on HANS. (For HUBERT (Transformer), we only transfer roles and fillers). On Non-entailment cases, for the HUBERT model, the Lexical accuracy improved dras- tically: by 61.62% (6, 162 examples). Performance on cases violating the Subsequence heuristic improved by 1.44% (144 examples) and performance on those violating the Constituent heuristic improved by 5.4% (540 examples). These improvements on Non-entailment case came at the cost of small drops Entailment accuracy. This pattern of transfer in stark contrast with the BERT results. Although the results on Entailment cases are improved, the accuracies for Subsequence and Constituent Non-Entailment cases drop significantly, showing that BERT is failing to integrate new knowledge gained from SNLI with previously learned information from MNLI. This shows that here, HUBERT (Transformer) can leverage information from new source of data efficiently. The huge improvement the Lexical Non-entailment cases speak to the power of TPRs to generate role-specific word embeddings: the Lexical heuristic amounts essentially to performing inference
We observed high variance on the MNLI dev set can have quite different accuracies on HANSS. To account for this, we ran our experiments with leust 3 different seeds and reported scores each model.
HANS both BERT HUBERT, instance, two models that achieve similar scores
8

---

Under review as a conference paper at ICLR 2020
on bag-of-words representation, where mere lexical overlap between premise and a hypothesis yields a prediction entailment.
Correct: Entailment
Correct: Non-Entaimmet
Model
Acc. (%)
Lcx. (%)
Sub. (%)
Const. (%)
Lex. (%)
Sub.
Const. (%)
BERT
63.59
95.32
99.32
99.44
53.40
8.86
25.20
BERT
61.03 4
98.70
99,961 1
100.00
$5.22 1
2.92 +
9,40 J
HUBERT (Transformer)
52.31
98.30
99.92
99.40
8,40
2.32
5.52
HUBERT (Transformerr) +
63.22 f
95.52
99.76]
99.32
70.02 f
3.76
10.92 f
Table HANS results for BERT and HUBERT (Transformer) models. Ace. indicates the average of the results each sub-task in HANS. Each model is fine-tuned on MNLI. '+' indicates that the model additionally fine-tuned on the SNLI corpus. 1 indicates an increase and & indicates a decrease in accuracy after the model is fine-tuned SNLI.
5 CONCLUSION
In this work we showed that BERT cannot effectively transfer its knowledge across NLP tasks, even the two tasks are fairly closely related. To resolve this problem, we proposed HUBERT: this adds a decomposition layer top of BERT which disentangles symbols from their roles in BERT's representations. The HUBERT architecture exploits Pensor-Product Representations, in which each word's representation is constructed by binding together two separated properties, the word's (se- mantic) content and its structural (grammatical) role. In extensive empirical studies, HUBERT showed consistent improvement in knowledge-transfer across various linguistic tasks. HUBERT+ outperformed BERT+ on the challenging HANS diagnosis dataset, which attests to the power of its learned, disentangled structure. The results from this work, along with recent observations reported in Kovaleva et al. 12019); McCoy et al. (2019); Clark et al. (2019) 9 Michel et al. (2019), call for bet- ter model designs enabling synergy between linguistic knowledge obtained from different language tasks.
ACKNOWLEDGMENTS
We would like thank R. Thomas McCoy from Johns Hopkins University and Alessandro Sordoni from Microsoft Research for sharing and discussing their recent results on HANS, and Xiaodong Liu from Microsoft Research for thoughtful discussions.
REFERENCES
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang, Can we gain more from orthogonality regular- izations in training deep cnns? In Proceedings of the 32nd International Conference on Neural Information Processing Systems. pp. 4266-4276. Curran Associates Inc., 2018.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv: 05326, 2015.
K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What Does BERT Look At? An Analysis of BERT's Attention. arXiv e-prints, June 2019.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wat- tenberg. Visualizing and measuring the geometry of BERT. arXiv preprint arXiv:1906.022 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arxiv:1810.04800, 2018.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. arXiv e-prints, December 2015.

---

Under review as a conference paper at ICLR 2020
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa- pers), pp. 4129-4138, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735- 1780, 1997.
Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Oliver Wu. Tensor product generation networks for deep NLP modeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Pupers), pp. 1263-1273, 2018.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09280 2019.
D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, December 2014.
O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the Dark Secrets of BERT. arXiv e-prints, August 2019.
Q. V. Le, N. Jaitly, and G. E. Hinton. A Simple Way to Initialize Recurrent Networks of Rectified Linear Units. arXiv e-prints, April 2015.
Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. Reasoning in vector space: An exploratory study of question answering. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
J. Lei Ba, J. R. Kiros, and G. E. Hinton. Layer Normalization. arXiv e-prints, July 2016.
Simon D Levy and Ross Gayler. Vector symbolic architectures: A new building material for artificial general intelligence. In Proceedings of the 2008 Conference Artificial General Intelligence 2008: Proceedings of the First AGI Conference, pp. 414-418. IOS Press, 2008.
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside BERT's linguistic knowledge. arXiv preprint arXiv:1906.1 100.0.99,..0698, 2019.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 4487-4496, Florence, Italy, July 2019a. Association for Computational Linguistics. BtRLhttss:// www..alwwwa.... org/ http://www..g..atooogggg//////// http///hhoooooyy/1/// 1441
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019b.
approach. arXiv preprint 17Xiv:1907.1166 2019b.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint 190.00190..01007, 2019.
Michel, O. Levy, and G. Neubig, Are Sixteen Heads Really Better than One? arXiv e-prints, May 2019.
Allen Newell. Physical symbol systems. Cognitive science, 4(2):135-183, 1980.
Hamid Palangi, Paul Smolensky, Xiaodong He, and Li Deng. Question-answering with granmatically-ineerrttabe representations. In AAAI, 2018.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 1.01088, 2018.
Tony A Plate. Holographic reduced representations. JEEE Transactions on Neural networks, 6(3): 623-641, 1995.
10

---

Under review as a conference paper at ICLR 2020
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ Questions for Machine Com- prehension of Text. arXiv e-prints, June 2016.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016.
Imanol Schlag and Jürgen Schmidhuber. Learning reason with third order tensor products. In Advances in Neural Information Processing Systems 31, pp. 9981-9993. 2018.
Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46(1):159 216, 1990.
Paul Smolensky and Géraldine Legendre. The Harmonic Mind: From Neural Computation to Sieimalliyy-hhorettte GranmarrVllume I: Cognitive Architecture (Bradford Books). The MIT Press, 2006. ISBN 0262195267.
I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to Sequence Learning with Neural Networks. arXiv e-prints, September 2014.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. arXiv preprint wrXiv:1905.059900 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
E. Vorontsov, C. Trabelsi, S. Kadoury, and C. Pal. On orthogonality and learning recurrent networks with long term dependencies. arXiv e-prints, Jamuary 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv 2018.
preprint
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, al. Can you tell me how get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4465-4476, 2019.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. Semantics-aware bert for language understanding. arXiv preprint arXiv: I 19...92299.0.... 2019.
A APPENDIX
A.1 DATASET DETAILS
In this section, we briefly describe the datasets we use to train and evaluate our model. GLUE is collection of different NLP tasks that currently serve as a good benchmark for different proposed language models. The tasks can be broadly categorized into single sentence tasks (e.g. CoLAA and SST) and paired sentence tasks (e.g. MNLI and QQP). In the former setting, the model makes a binary decision whether single input satisfies certain property or not. For CoLA, the property is grammatical acceptability; for SST, the property is positive sentiment.
The 7 other tasks in GLUE are paired sentence tasks in which the model strives find relationship (binary or ternary) between two sentences. QNLI, WNLI, and RTE are inference tasks, in which given premise and hypothesis, the model predicts whether the hypothesis is congruent with the premise (i.e. entailment) or not (i.c. conffict). Although QNLI and WNLI are not originally designed inference tasks, they have been re-designed to have similar configuration other
11

---

separate corpus and report our Under review as a conference paper at ICLR 2020
NLI tasks. This way, single classifier can be used to judge whether the right answer is in the hypothesis (e.g. for QNLI) or whether a pronoun is replaced with the correct antecedent (e.g. for WNLI). MNLI is an additional NLI task in which three classes being used instead of two to represent the relation between two sentences. The third class shows neutrality when the model is not confident that the relation is either entailment contradiction. The last. three tasks measure sentence similarity. In MRPC the model decides if two sentences are paraphrases of each other, In QQP, given two questions, the model decides whether they are equivalent and are asking for the same information. All the tasks discussed so far fall under the classification category. where the model produces a probabilistic distribution over the possible class outcomes and the highest value is selected. STS-B, however, is regression task where the model produces real number between and 5, indicating the two sentences" semantic similarity. Since our model is designed only for classification tasks, we skip this dataset.
Corpus
Task
single' pair single
# Train
# Dev
# Test
Labels
CoLAA
Acceptahility
8.5K
IK
IK
SST
Sentiment
single
67K
872
1.8K
MRPC
Paraphrase
pair
3.7K
408
1.7K
QQP
Paraphrase
pair
364K
40K
391K
MNLI
Inference
puir
393K
20K
20K
QNLI
Inference
pair
108K
5.7K
5.7K
RTE
Inference
pair
2.5K
276
3K
WNLI
Inference
pair
634
71
146
SNLI
Inference
poir
549K
9.8K
9.8K
HANS
Inference
pair
30K
Table 4: Details of the GLUE (excluding STS-B), SNLI and HANS corpora
We observed lot of variance in the accuracy (15%) for models trained on WNLI, MRPC, and CoLA. As mentioned in the GLUE webpage there are some issues with the dataset, which makes many SOTA models perform worse than majority-votinn. We found that MRPC results are highly dependent on the initial random seed and order of sentences in the shuffled training data which mainly caused by the small number of training samples (Table 4.. CoLA is the only task in GLUE which examines grammatical correctness rather than sentiment, and thus it makes it harder to benefit from the knowledge learned from other tasks. The train and test set are also constructed in an adversarial way which makes it very challenging. For example, the sentence "Bill pushed Harry off the sofa for hours." " is labeled as incorrect in the train split but very similar sentence "Bill pushed Harry off the sofa." is labeled as correct in the test split. Hence, we only conduct our experiments on the remaining 5 datasets from GLUE.
We also take advantage of an additional NLI dataset called SNLI. It is distributed in the same format as MNLI and recommended by Waan et al. (2018) to be used in conjunction with MNLI during training. However, in our experiments, treat this dataset results on it individually.
To further test the capabilities of model, evaluate our model probing dataset McCoy ct al. 2019). It introduces three different syntactic heuristics and claims that most of SOTA neural NLI models exploit these statistical clues to form their judgments extensive experiments that these models obtain very low accuracies for sentences cleverly crafted to defeat the models which exploit these heuristics. Lexical overlap, Subsequence, and Constituent the three categories examined, each containing 10 sub-categories.
each example. It shows through
A.2 TEST RESULTS
Table 5 shows the transfer learning results when the source corpus is QQP. Table 6 shows the test results for BERT and HUBERT (Transformer). The top 5 rows are for MNLI as source corpus and bottom 5 rows are for QQP as source corpus.
12

---

Under review as a conference paper at ICLR 2020
Model
Target
Transfer BERT
Transfer Filler
Transfer Role
Baseline Acc. (%)
Fine-
Gain (%)
Corpus
tuned
Acc. (%)
BERT
QNLI
True
91.60
90.96
0.64
BERT
MNLI
True
84.15
84.41
0.26
BERT
RTE
True
71.12
62.45
8.67
BERT
SNLI
True
90.45
90.88
0.43
BERT
SST
True
93.23
92.09
1.14
HUBERT (Transformer)
QNLI
False
True
True
88.32
90.55
2.23
HUBERT (Transformer)
MNLI
False
True
True
84.30
85.24
0.94
HUBERT (Transformer)
RTF
True
False
61.73
65.70
+.97
HUBERT (Transformer)
SNLT
False
False
True
90.63
91.20
HUBERT (Transformer)
SST
True
True
True
86.12
91.06
+ 4.94
Table 5: Transfer learning results for GLUE tasks. The source corpus is QQP. Baseline accuracy is for when Transfer BERT, Filler, and Role are all False, which is equivalent to no transfer. Fine-tuned accuracy is the best accuracy among all possible transfer options.
HUBERT
Source Corpus
Target Corpus
Transfer
Transfer
Transfer
BERT Acc.
HUBERT
BERT
Filler
Role
(%)
Acc. (%)
MNLI
QNLI
Truc
True
False
90.50
90.50
MNLI
QQP
False
False
True
89.20
89.30
MNLI
RTE
True
True
True
66.40
69.30
MNLI
SNLI
True
False
True
89.20
90.35
MNLI
SST
True
False
True
93.50
QQP
QNLI
False
True
True
90.50
90.70
QQP
MNLI
False
True
84.60
QQP
RTE
False
True
False
66.40
63.20
QQP
SNLI
False
False
True
89.20
90.36
QQP
SST
True
True
True
93.50
91.00
Table 6: Test set results for HUBERT (Transformer) and BERT. BERT accuracy indicates test results on target corpus (without transfer) for bert-base-uncased which directly taken from the GLUE leaderboard. Fine-tuned accuracy are the test results for best performing HUBERT (Transformer) model on target dev after transfer (see Tables and 5).
13

---

Under review as a conference paper at ICLR 2020
A.3 IMPLEMENTATION DETAILS
Our implementations are in PyTorch and based on the HuggingFacd repository which is a library of tatte-of-thearre NLP models, and BERT's original codebase T0 In all of our experiments, we used bert-base -uncased model which has 12 Transformer Encoder layers with 12 attention heads each and the hidden layer dimension of 768. BERT's ' word-piece tokenizer was used to preprocess the sentences. We used Adamax (Kingma & Ba, 2014) as our optimizer with a learning rate of 5 x 10-S and used linear warm-up schedule for 0.1 proportion of training. In all our experiments used the same value for dimension and number of roles and symbols (ds: 32, dæ: 32, n.s: 50, nu: 35). These parameters were chosen from the best performing BERT models over MNLI. We used the gradient accumulation method to speed up training (in which accumulate the gradients for two consecutive batches and then update the parameters in one step). Our models were trained with batch size of 256 distributed over 4 V100 GPUs. Each model was trained for 10 epochs, both on the source task and the target task (for transfer learning experiments).
We performed hyper-pane parameter tuning for both BERT and HUBERT models on the MNLI dev set. As for the dimension of roles and symbols we did grid search over these values: [10, 30, 60] We fixed the number of roles to 35 and searched among these values for number of fillers: [50, 100, 150]. We additionally performed some light tuning on learning rate, temperature value, and scaling value. To control for the randomness in our results, we ran our experiments by fine-tuning BERT with 3 different seeds and choosing the best results among them. However, for HUBERT we used the same seed for all experiments and only changed the initial weights of layers the model. We observed small variance in baseline and fine-tuned accuracy across different runs for BERT model. Specifically when MNLI is the source corpus, the standard deviation for QNLI, QPP, RTE, SNI, and SST as target corpora was 0.2%, 0.4%, 0.5%, 0.4%, 0.4% respectively.
A.4 INTERPRETATION OF LEARNED ROLES
To gain better understanding of what R (the global role matrix) is learning we analyze the attention scores (a over this matrix generated by the model for each token. The final role vector (r(1) for each token is being calculated by performing a matrix product between the attention vector and Role matrix as discussed in Section
(+) Ra
( R
a is a vector of nf dimension indicating the importance of each Role in constructing the final role vector p()), First, for each sentence in the dataset we collect the Part Of Speech (POS) tags for each token (or sub-token) coming out of BERT model. We obtain the POS tags from the last sub-tokens of each word. This requires us to take extra care when processing the data and keep a dictionary that maps the index of each token in the original sentence to the indices of all its sub-tokens. Then, we gather the roles each sub-token is being attracted to. To account for the distributed nature of the attention scores, instead of choosing the role with highest value in the attention distribution, we select the top K values, concatenate them, and treat this new tuple as a single role. We then calculate the number of roles assigned to each POS tag. Figure shows this distribution where each color shows a specific role tuple.
The number of all possible POS tags in MNLI is 36 which nicely aligns with the 35 number of roles we chose in our experiments. See this link for description of each tag. To make the visualizations simpler we merged tags indicating similar grammar roles into one coming up with a total of 21 tags (e.g. combining NN, NNS, NNP, and NNPS into one category: NN).
We can observe some interesting patterns by looking at the frequency of the roles assigned to each tag. For example, we observe that the "yellow" role is attracted to ' tag which represents punctua- tion marks such as a full stop, question mark, exclamation mark, etc. On the other hand, the "blue" role is attracted almost exclusively to NN. Additionally the "green" role is mainly present in NN, VB, and JJ which are linguistically the three most important POS categories. However, the "purple"
9 butps::/giduu...om/// hwtp:///ww...ogiooooooooppttaaaa
10 bttps://tthub.co
Penn Treehank POS
14

---

Under review as a conference paper at ICLR 2020
role is more or less used by all different POS categories suggesting low correlation between that role and POS tags.
50000
40000
Number of Roles
30000
20000
10000
NN
VB
CC
D7
PRP
0
MD
FW
RP
POS
UH
EX
PDT
POS Tags
Figure 4: POS tags vs Role frequencies when selecting the two roles with highest value in the attention distribution. Subsequently the tags referring to similar grammar roles are merged into one category to generate better visualization.
15


Reviews:
[{"review_id": "rJgzyIYGqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an alternative way of reusing pretrained BERT for downstream tasks rather than the traditional method of fine-tuning the embeddings equivalent to the CLS token. \n\nFor each bert embedded token, the proposed method aims at disentangling semantic information of the word from its structural role. Authors provide two ways to provide this disentagling using LSTM or transformer blocks. with several design choices such as: *  a regularization term to encourages the roles matrix to be orthogonal and hence each role carry independent information *  design the roles and symbols matrices so that the number of symbols is greater than the number of roles\n\nIn evaluation authors design several experiments to show that: \n* Does transferring disentangled role & symbol embeddings improve transfer learning\n* the effectiveness of the TPR layer on performance?\n* Transfer beyond Glue tasks? \n\nWhile those experiments provide empirical gains of the design choices, authors don't show enough study to attribute those  empirical gains to the presented design choices: \n\nOne large claim in the paper is that empirical gains in the ability of transfer between similar tasks MNLI and GLUE is because of disentangling the semantics from the role representations. We don't know if the TPR layer really manages to do that, this could have been easily verified using for example clustering word senses of the same word. \n\nThe empirical gains in transfer learning can be simply attributed to: \n- More params it seems adding an LSTM over bert embeddings already does some improvement, I would have loved to see this more exploited but it wasn't. This aligns with some recent findings that BERT is undertrained (Liu et al. 2019) https://arxiv.org/abs/1907.11692  \n- Variance in the results (authors report only results of one single run not mean and std of several runs).\n- More budget given to hyper-parameter search for the models proposed in the paper.  Hyper param budget isn't also reported in the paper. \n- other factors, not the ones associated with the claims in the paper: for example what authors claim is an ablation study was comparing several different models together. It would have been more interesting to see for example the effect of making the # symbols = # roles or removing the orthogonality loss from the roles matrix.\n\nConclusion: The paper introduces large claims and empirical results that correlate with, however the provided experiments are not done with enough control to attribute gains to the design choices provided in the paper. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l6scnPYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a fine-tune technique to help BERT models to learn & capture form and content information on textual data (without any form of structural parsing needed). They key addition to the classic BERT model is the introduction of the R and S embeddings. R &S are supposed to learn the information in text that is traditionally represented as the structural positions and the content-bearing  symbols in those positions. \n\nIn order to effectively learn R and S embeddings, the authors propose two possible ways to do so: LSTM (Fig 2) and 1-layer Transformer (Fig 3). The main experiments are based on 1-layer transformer HUBERT b/c from a single test in Table 1, the transformer variant appears to be working better than the LSTM variant.\n\nMy main concern regarding this paper is two-fold: limited novelty and insignificant performance gain.\nThe authors did a great job motivating the need for separating role and filler in the intro. However, in neither implementation of HUBERT, I do not see how the structural information (e.g., a parse tree) is directly incorporated into the learning of HUBERT.\n\nRegarding the performance, it seems HUBERT is gaining very little over the BERT baseline. please refer to my specific question below.\n\n\nQuestions:\nWhat are the numeric values for d_S, d_R, n_S, n_R (defined under Section 3 on page 2) in experiment ? I think d_S, d_R are determined at author's discretion (just like the dimensionality of, say, the LSTM hidden layer). But how are n_S and n_R determined?\n\nPage 7, first paragraph: what is Filler embeddings F? F is not defined in either version the proposed HUBERT( Figure 2 or Figure 3). Did the authors mean S?\n\nTable 2. Why do the first 5 rows and the bottom 5 rows have different baseline Acc. ? Shouldn't we always use the best accuracy as baseline for comparison? If we look at the HUBERT Fine-tuned Acc., in many cases, they are actually worse than the best baseline acc. available. (i.e., QNLI , QQP, and SST).\n\nOther comments:\nTypo on page one: “[] To strengthen the generality of ….”\nFigure 1 is never referred in main text.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gqjnahKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a layer on top of BERT which is motivated by a desire to disentangle content (meaning of the tokens) and form (structural roles of the tokens).  Figure 1 shows this clearly. The paper considers two variants of the disentangling layer (TPR), one with LSTMs (figure 2) and the other with attention (figure 3). The aim in both is to obtain a decomposition of the form x(t) = S a_s(v_t) a_r(v_t) R where S and R are shared matrices of parameters and v is the output of BERT. \n\nThe model is well motivated and includes clear reasonable design ideas, including choosing hyper-parameters so that the number of symbols (s) is greater than the number of roles (r), and forcing only the roles to be independent (eqn 6). \n\nMinor: I would have preferred that figure 1 appeared earlier in page 3. This would help as the authors forgot to define v in eqn 2. One has to wait for the figure. Having said this, the paper is extremely clear in the notation and does an excellent job at defining dimensions for all the quantities of interest.\n\nI read the paper eagerly and with excitement until I got to the results. First, it wasn't clear to me how well motivated is the idea of fine-tuning on intermediate tasks. I understand the authors are just trying to make a point that BERT does worse than their model in this case and that this is not good for transfer, but still I find this to be artificially constructed.\n \n The variations in the numbers seem small and possibly attributable to other factors. For this reason, I feel the authors should have continued showing results for the other baselines from the first experiment. I would also have loved to see some visualizations for a, r, A and R in the appendix. Some visualization and anecdotal results might have helped me see that the motivation is backed up by the results. I hope the authors have the time to do this and consider the extra experiments.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces additional layers on top BERT type models for disentangling of semantic and positional information.  The paper demonstrates (small) performance gains in transfer learning compared to pure BERT baseline.

Both reviewers and authors have engaged in a constructive discussion of the merits of the proposed method. Although the reviewers appreciate the ideas and parts of the paper the consensus among the reviewers is that the evaluation of the method is not clearcut enough to warrant publication.

Rejection is therefore recommended. Given the good ideas presented in the paper and the promising results the authors are encouraged to take the feedback into account and submit to the next ML conference.   
Decision: Reject