Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an approach to incorporate span information in pre-trained language models like BERT during fine-tuning. In the proposed approach, the segmentation of a sentence is obtained according to a pre-sampled n-gram dictionary. The fine-grained representation in a same span within the segmentation is aggregated to a span-level representation using a CNN model. These span-level representations are further aggregated using a CNN model to generate sentence-level representation. The experiments show that the proposed model can achieve similar performance gain as other span-based language models which includes span information during pre-training.    

The paper is well written and the proposed method is novel. The novelty of the method is in mainly including span information only during fine-tuning and also the way spans are identified in a sentence. The authors conducted an extensive set of experiments with ablation and results show that the proposed idea is effective and attains similar gains as other span-based language model. 

It is not clear how this model helps in practice since it is achieving similar performance as other models like SpanBERT. It is true that this model doesn’t introduce complexity while pre-training but pre-training is a one-time process and even if pre-training is slower, it does not affect downstream tasks. However, the proposed model introduces complexity in the downstream tasks and may affect training time in the downstream task.
 
The authors claim that the proposed model perform similar to models like SpanBERT but it is not clear if both models are improving on similar aspects. Some fine-grained analysis of results with these models might be insightful.

Another concern I have is that the proposed model can only work in the scenarios where sentence-level representation is required. However, there are tasks like named-entity recognition (NER) where word level representation is needed. 
review_title: An interesting work on incorporating span information in BERT like models during fine-tuning

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces a new mechanism based on span-based representation grouping for better language model finetuning. Specifically, firstly, the paper proposes to utilize an n-gram splitter to segment the text into several segments. Secondly, a hierarchical network is built based on a three-level network (subword level->span level->sentence level). Using these tasks to perform fine-tuning on top of a pre-trained masked language modeling shows improvements than directly fine-tuning on the pre-trained language models. By conducting experiments on the GLUE benchmark with the BERT-base and BERT-large models, the results are improved. The general idea is simple and easy to understand. However, I have several concerns which are list as follows,

1. Combining span-level information has been widely studied in the literature[1] (not cited). Although the tasks between the two papers are different, the novelty of the paper is limited. Besides, it is also worth discussing between Sec 5.2 and [1]. 
2. The experiments are conducted on BERT-base and BERT-large models. However, the two models are weak baselines. I would like to see the empirical results on stronger models, like RoBERTa related models.
3. As the paper focuses on training span representations, it is necessary to perform experiments on entity-based tasks. The experiments setup may refer to the paper [2].
4. The paper proposed to use the n-grams, it is also necessary to have a comparison with the pre-extracted spans using an off-the-shell chunker.


I also have the following questions:
If a pretrained chunker is utilized to extract chunks, what are the results like?
In section 5.2, does the variant of n-grams affect the results?



For these reasons, I do not recommend acceptance of this paper.




[1] Toshniwal et al. A Cross-Task Analysis of Text Span Representations, ACL RepL4NLP-2020
[2] Yamada et al. LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention, EMNLP 2020



review_title: A simple, useful and easy to understand idea. However, the paper lacks novelty and the empirical results may not be convincing.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Previous works reveal that span-level information can enhance the performance of PrLMs if they are used in pre-training. However, the existing methods require enormous resources and lack adaptivity. To this end, the paper proposes a method that combines span-level information into the representations generated by PrLMs during the fine-tuning phase. To combine span-level information, the paper first breaks a sentence into various span components. Then, an accumulated representation with enhanced span-level information is built based on the sub-token-level representation provided by PrLMs. The experimental results on the GLUE benchmark show that the proposed method improves the performance of PrLMs. The main contribution of this paper is the introduction of generating the span components via a pre-sampled dictionary. Overall, the proposed method is not novel since similar methods or ideas have been widely used in NPL.
.

Concerns:

1. Employing span-level information in PrLMs was proposed in previous works such as SpanBERT. The paper presented a way that combines span-level information into the representations generated by PrLMs during the fine-tuning phase. However, as shown in Table 2, the proposed method is not better than SpanBERT. So, what are the advantages of the proposed method.
2. As shown in Table 1, compared with the baselines, the proposed method does not bring remarkable promotion.

Minor comments: 
1. The subscript “j“ should be “1” in the second line of Formula (1).

review_title: enhancing the performance of PrLMs by span-level information

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
 
The paper introduced a fine-tuning approach which adapts the subword level, span level and sentence level to the target tasks. The span segmentation is done by using a pre-trained n-gram statistical model.  Empirical studies on GLUE benchmark show that the proposed approach consistently improves the performance of BERT. 
 
Pros:
 
1. The idea of leveraging a n-gram model for pre-trained language model fine-tuning is interesting.
2. The proposed method is well motivated and easy to understand.
 
Cons:

My concern is the significancy of the results. Since the improvement of the proposed method is marginal (~ 1% avg) and fine-tuning BERT on GLUE tasks might have high variance, authors should report the mean and std over multiple runs.
 
Questions:

If the model is pre-trained with span level information (e.g., spanBERT), will the proposed method outperform normal fine-tuning?

review_title: Is the experimental results significant?

Meta-review: The paper proposes to combine the span-level information into a phrase-level representation in the fine-tuning phrase for pre-trained language models.  The phrases are pre-defined in a dictionary.  Experiments show improvements in various downstream tasks in the GLUE benchmark.  It's a borderline paper.  Various concerns were raised by the reviewers, for example, the relation with the SpanBERT method in pre-training phrase and the significance of the results.  The authors addressed most of the concerns but the reviewers were not fully convinced.  In general, I think it is an interesting paper with good motivation and results.  Hope it can be improved (e.g. more experiments on SpanBERT) and accepted in another conference.
Decision: Reject