Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method to improve the downstream performance of a pretrained Transformer on NLP tasks. The core idea is to not only use the output of the last Transformer layer for prediction, but let the model decide how to fuse the information from intermediate layers as well. To dynamically decide which intermediate layers to use depending on the input example, the model uses a mechanism conceptually similar to self-attention, which yields a normalized importance score for each layer. The importance-weighted sum then yields a complementary representation to the last layer. Lastly, another network produces a final, integrated representation from the output at the last layer and the complementary representation, which is then used for prediction.
The model is evaluated on the GLUE benchmark.

Strengths:
The paper is well written and the experimental evaluation seems correct. The paper has a nice ablation study which shows that the learned importance scores, the complementary representations, and the fusion network are needed to reach the model's full performance.

Weaknesses:
The main weakness is that the proposed extension to the baseline is relatively complex and rather heavy-weight in terms of new parameters (introducing ~25% more parameters compared to the baseline according to Table 2), yet only achieves a very marginal relative improvement of 0.2 percent over the baseline. It seems likely that this improvement could be achieved through much simpler means, e.g., additional self-attention layers on top of the last pretrained layer. This is supported by the fact that the paper's analysis of the proposed importance score mechanism doesn't show comprehensible patterns (or the paper doesn't talk about it).

I lean slightly towards rejection, because, although the proposed model is reasonable and could have potential, the experiments do not currently demonstrate that potential, and I hence expect it hard for the community to learn from this paper.

Questions:

* You cite several papers which demonstrate that different layers of pretrained Transformers encode different information, which is the motivation for your architecture. However, the cited sources use a feature extracting approach, i.e., they don't fine tune the encoder on their target task. In the finetuning scenario (yours), can't we expect the model to learn to simply forward the relevant information from intermediate layers to the last layer?

* In Section 5 you analyze which intermediate layers are used for which task. You state that we can see that the model learns to prefer different layers for each task, but don't go into detail. Can you relate the nature of the task to the layers it produces? For example, why does QNLI rely on layers 2-4 to some extent despite being a rather high-level understanding task?

* In Figure 2, right side you observe that different examples tend to different layers. Did you qualitatively inspect the examples and try to find a pattern/link to their preferred intermediate layers?

Suggestions:

* RoBERTa already achieves super-human results on the GLUE benchmark, meaning that it is probably already close to some upper limit. This makes it even more difficult for your model to substantially improve upon it. I would consider evaluating it on the SuperGLUE benchmark instead, since there RoBERTa is not so close to human performance yet.

* Your model is designed to make use of representations at lower layers if a task requires it. But all the tasks in your experiments are evaluated on high-level natural language understanding tasks, which typically require representations at higher layers. This makes it more likely that your model does not improve much over the baseline, because the last layer will arguably already contain much of the information needed for the tasks. I think you should consider different tasks in your experiments to increase the chance of significant improvements.

* In the current state, it is unclear whether your improvements are coming from the specific model you chose or merely from the fact that you added a lot of parameters. As a control, you could include an experiment where you simply add several self-attention layers on top of RoBERTa such that you match the number of parameters of your model.
review_title: Interesting model, but the experiments are not convincing.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a new mechanism, called HIRE, to extract more information from the intermediate layers of pre-trained models, which will be further fused with the last layer of pre-trained models. The main contribution of this work is the newly proposed dynamic feature extractor HIRE and the fusion network. Experiments confirmed the effectiveness of the proposed method, and some interesting observations on the importance of different layers for different tasks were given (i.e. Figure 2). 

However, the proposed HIRE and Fusion modules can be viewed as the combination of some widely used deep learning mechanisms (e.g. Bi-GRU, softmax), using Bi-GRE to represent a sequence is a widely adopted choice, so the contribution in modeling is quite limited. Some existing works, like Self-Adaptive Hierarchical Sentence Model https://arxiv.org/pdf/1504.05070.pdf, have already proposed similar ideas to use more information from the intermediate layers of deep models. 

The paper is well-written and organized. I have several concerns:
1. All the models are evaluated on the GLUE dataset, experiments on more challenging tasks like QA (e.g. SQuAD 1.1/2.0) should be added. It would be interesting to visualize the importance scores of each layer for this challenging task. 

2. As illustrated in Figure 2, different downstream task highly depends on different intermediate layers, the authors are suggested to conduct experiments under the multi-task setting, since the HIRE can adaptively select the intermediate layers, the proposed method should have a big advantage for this setting. 
review_title: A hidden representation extractor for improving pre-trained representation

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
### Summary 
In this paper, the authors propose a fine-tuning scheme for pre-trained language models, called Hidden Representation Extractor (HIRE). The proposed method consists of two components.
1) Hidden representation extractor (HIRE): a two-layer bi-GRU model summarizes the hidden representation of the encoder with a weighted-sum scheme.
2) Fusion network: combine the weighed representation with original representation via the bi-GRU model.
The authors apply the proposed approach on top of RoBERTa and evaluated on general benchmark GLUE. The experimental results show that the proposed method could give moderate improvements over RoBERTa baseline system.

The authors' main contribution lies in two components: HIRE and fusion network. However, unfortunately, both components are not novel to me.
1) The idea of HIRE model is very similar to utilizing ELMo in down-stream tasks, where the hidden representations of each layer are weighted.
2) The central part of the fusion network utilizes a concatenation scheme of [A;B;A+B;A*B], which is also a widely used approach, similar to Chen et al. (2017), Seo et al. (2017), Hu et al. (2019), etc.

Besides the novelty of the proposed components, the experiments are also not well-designed. I assume the proposed method could apply to various pre-trained language models (PLMs), as this is an approach that aims to apply on the fine-tuning stage. However, the authors only present the results on top of RoBERTa model, which cannot demonstrate that the proposed model can be generally applied to various PLMs. I would suggest the authors evaluate their approach in a more general way, such as build on top of various PLMs, or apply on PLM in another language. These would have helped to better demonstrate the effectiveness of the proposed method.

Considering the novelty and generalizability of the proposed method, I recommend rejection for this paper.

[1] Chen et al. Enhanced LSTM for Natural Language Inference.  
[2] Seo et al. Bidirectional attention flow for machine comprehension.  
[3] Hu et al. Read+ verify: Machine reading comprehension with unanswerable questions.  


### PROs
1. A possible useful framework for fine-tuning down-stream tasks for pre-trained language models.


### CONs
1. The design of the components is not novel.
2. The experiments are not well-designed, and the results only show marginal improvements over RoBERTa.
3. The generality of the proposed method is not well-studied, as the experiments are only performed on top of RoBERTa.


### Questions
1. Unlike Transformer-based models, RNN models (such as Bi-GRU) are not computationally efficient. How about your training time compared to the baselines? According to Table 2, using HIRE will increase the total parameter from 355 to 437, which will slow down the inference speed.
2. The authors did not cite ELECTRA (Clark et al., 2020) for comparison. After comparing the results, the proposed model is not competitive against ELECTRA, while it would cost more training and inference time, which will weaken the impact of the paper. I strongly recommend the authors also carry out experiments on top of ELECTRA (or ALBERT-xxlarge, etc.) to see if your approach generalizes well on various PLMs.


### Minor Reviews
I did not go through every detail of the writing and only list a few of the issues here.
1. page 1, section 1: CoNNL-2003 -> CoNLL-2003
2. page 1, section 1: (Peter et al., 2018b) points out ... -> Peter et al. (2018b) points out ...
3. page 3, section 2.3: Tansformer -> Transformer; yeild -> yield; elementwise -> element-wise;

review_title: Initial Review

Meta-review: This paper proposes  a new mechanism, called HIRE, to  improve the down-stream performance of a pre-trained Transformer on NLP tasks. Different from directly using the last layer of transformer, the proposed model allows the system to dynamically decide which intermediate layers to use based on the input through some sort of gating. The model is evaluated on GLUE, a benchmark for natural language understanding.  My major concerns are the following 
1.  the gating mechanism on using intermediate sentence representation is not new,  as pointed by some reviewers, although its implementation on transformers is still interesting.
2.  the empirical part is not convincing enough:  a) GLUE data set is relatively simple, the authors should try something more complex, b）the improvement over baseline is rather modest, which could be achieved with simpler modification.

I'd suggest to reject this paper.
Decision: Reject