Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an alternative way of reusing pretrained BERT for downstream tasks rather than the traditional method of fine-tuning the embeddings equivalent to the CLS token. 

For each bert embedded token, the proposed method aims at disentangling semantic information of the word from its structural role. Authors provide two ways to provide this disentagling using LSTM or transformer blocks. with several design choices such as: *  a regularization term to encourages the roles matrix to be orthogonal and hence each role carry independent information *  design the roles and symbols matrices so that the number of symbols is greater than the number of roles

In evaluation authors design several experiments to show that: 
* Does transferring disentangled role & symbol embeddings improve transfer learning
* the effectiveness of the TPR layer on performance?
* Transfer beyond Glue tasks? 

While those experiments provide empirical gains of the design choices, authors don't show enough study to attribute those  empirical gains to the presented design choices: 

One large claim in the paper is that empirical gains in the ability of transfer between similar tasks MNLI and GLUE is because of disentangling the semantics from the role representations. We don't know if the TPR layer really manages to do that, this could have been easily verified using for example clustering word senses of the same word. 

The empirical gains in transfer learning can be simply attributed to: 
- More params it seems adding an LSTM over bert embeddings already does some improvement, I would have loved to see this more exploited but it wasn't. This aligns with some recent findings that BERT is undertrained (Liu et al. 2019) https://arxiv.org/abs/1907.11692  
- Variance in the results (authors report only results of one single run not mean and std of several runs).
- More budget given to hyper-parameter search for the models proposed in the paper.  Hyper param budget isn't also reported in the paper. 
- other factors, not the ones associated with the claims in the paper: for example what authors claim is an ablation study was comparing several different models together. It would have been more interesting to see for example the effect of making the # symbols = # roles or removing the orthogonality loss from the roles matrix.

Conclusion: The paper introduces large claims and empirical results that correlate with, however the provided experiments are not done with enough control to attribute gains to the design choices provided in the paper. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a layer on top of BERT which is motivated by a desire to disentangle content (meaning of the tokens) and form (structural roles of the tokens).  Figure 1 shows this clearly. The paper considers two variants of the disentangling layer (TPR), one with LSTMs (figure 2) and the other with attention (figure 3). The aim in both is to obtain a decomposition of the form x(t) = S a_s(v_t) a_r(v_t) R where S and R are shared matrices of parameters and v is the output of BERT. 

The model is well motivated and includes clear reasonable design ideas, including choosing hyper-parameters so that the number of symbols (s) is greater than the number of roles (r), and forcing only the roles to be independent (eqn 6). 

Minor: I would have preferred that figure 1 appeared earlier in page 3. This would help as the authors forgot to define v in eqn 2. One has to wait for the figure. Having said this, the paper is extremely clear in the notation and does an excellent job at defining dimensions for all the quantities of interest.

I read the paper eagerly and with excitement until I got to the results. First, it wasn't clear to me how well motivated is the idea of fine-tuning on intermediate tasks. I understand the authors are just trying to make a point that BERT does worse than their model in this case and that this is not good for transfer, but still I find this to be artificially constructed.
 
 The variations in the numbers seem small and possibly attributable to other factors. For this reason, I feel the authors should have continued showing results for the other baselines from the first experiment. I would also have loved to see some visualizations for a, r, A and R in the appendix. Some visualization and anecdotal results might have helped me see that the motivation is backed up by the results. I hope the authors have the time to do this and consider the extra experiments.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes a fine-tune technique to help BERT models to learn & capture form and content information on textual data (without any form of structural parsing needed). They key addition to the classic BERT model is the introduction of the R and S embeddings. R &S are supposed to learn the information in text that is traditionally represented as the structural positions and the content-bearing  symbols in those positions. 

In order to effectively learn R and S embeddings, the authors propose two possible ways to do so: LSTM (Fig 2) and 1-layer Transformer (Fig 3). The main experiments are based on 1-layer transformer HUBERT b/c from a single test in Table 1, the transformer variant appears to be working better than the LSTM variant.

My main concern regarding this paper is two-fold: limited novelty and insignificant performance gain.
The authors did a great job motivating the need for separating role and filler in the intro. However, in neither implementation of HUBERT, I do not see how the structural information (e.g., a parse tree) is directly incorporated into the learning of HUBERT.

Regarding the performance, it seems HUBERT is gaining very little over the BERT baseline. please refer to my specific question below.


Questions:
What are the numeric values for d_S, d_R, n_S, n_R (defined under Section 3 on page 2) in experiment ? I think d_S, d_R are determined at author's discretion (just like the dimensionality of, say, the LSTM hidden layer). But how are n_S and n_R determined?

Page 7, first paragraph: what is Filler embeddings F? F is not defined in either version the proposed HUBERT( Figure 2 or Figure 3). Did the authors mean S?

Table 2. Why do the first 5 rows and the bottom 5 rows have different baseline Acc. ? Shouldn't we always use the best accuracy as baseline for comparison? If we look at the HUBERT Fine-tuned Acc., in many cases, they are actually worse than the best baseline acc. available. (i.e., QNLI , QQP, and SST).

Other comments:
Typo on page one: “[] To strengthen the generality of ….”
Figure 1 is never referred in main text.

review_title: Official Blind Review #3

Meta-review: The paper introduces additional layers on top BERT type models for disentangling of semantic and positional information.  The paper demonstrates (small) performance gains in transfer learning compared to pure BERT baseline.

Both reviewers and authors have engaged in a constructive discussion of the merits of the proposed method. Although the reviewers appreciate the ideas and parts of the paper the consensus among the reviewers is that the evaluation of the method is not clearcut enough to warrant publication.

Rejection is therefore recommended. Given the good ideas presented in the paper and the promising results the authors are encouraged to take the feedback into account and submit to the next ML conference.   
Decision: Reject