Later Span Adaptation for Language Understanding
Authors: Rongzhou Bao
Zhuosheng Zhang
hai zhao
Venue:  (2021)

Abstract: Pre-trained contextualized language models (PrLMs) broadly use fine-grained tokens (words or sub-words) as minimal linguistic unit in pre-training phase. Introducing span-level information in pre-training has shown capable of further enhancing PrLMs. However, such methods require enormous resources and are lack of adaptivity due to huge computational requirement from pre-training. Instead of too early fixing the linguistic unit  input as nearly all previous work did, we propose a novel method that combines span-level information into the representations generated by PrLMs during fine-tuning phase for better flexibility. In this way, the modeling procedure of span-level texts can be more adaptive to different downstream tasks. In detail, we divide the sentence into several spans according to the segmentation generated by a pre-sampled dictionary. Based on the sub-token-level representation provided by PrLMs, we enhance the connection between the tokens in each span and gain a representation with enhanced span-level information. Experiments are conducted on GLUE benchmark and prove that our approach could remarkably enhance the performance of PrLMs in various natural language understanding tasks.

Under review as a conference paper at ICLR 2021
LATER SPAN ADAPTATION FOR LANGUAGE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained contextualized language models (PrLMs) broadly use fineggrined tokens (words sub-words) as minimal linguistic units in the pre-training phase. Introducing span-level information in pre-training has shown capable of further enhancing PrLMs. However, such methods require enormous resources and lack adaptivity due to huge computational requirements from pre-training. Instead of too early fixing the linguistic unit input as nearly all previous work did, we propose a novel method that combines span-level information into the representations generated by PrI.Ms during the fine-tuning phase for better flexibility. way, the modeling procedure of span-level texts can be more adaptive to different downstream tasks. In detail, we divide the sentence into various span components according to the segmentation generated by pre-sampled dictionary. Based on the sub-token-leve representation provided by PrLMs, we bridge the connection between the tokens in each span and yield accumulated representation with enhanced span-level information. Experiments on the GLUE benchmark show that our approach remarkably improves the performance of PrLMs in various natural language understanding tasks.
this
INTRODUCTION
Pre-trained contextualized language models (PrLMs) such as BERT (Devlin et al.] 2018). XLNet Yanng et al.) 2019). ELECTRA (Clark et al., 2020) have led to strong performance gains downstream natural language understanding (NLU) tasks. Such models' impressive power to generate effective contextualized representations is established by using well-designed self- supervised training on a large text corpus. Taking BERT as an example, the model used Masked Language Modeling (MLM) and Nest Sentence Prediction (NSP) as pre-training objects and was trained on a corpus of 3.3 billion words.
PrLMs commonly generate fine-grained representations, i.e., subword-level embeddings, to adapt to broad applications. Different downstream tasks sometimes require representations with different granularity. For example, sentence-level tasks such natural language inference (Bowman al.) 2013 Nangia et al., (017). demand an overall sentence-level analysis to predict the relationships between each sentence. There are also token-level tasks, including question answering and named entity recognition, which require models to generate fine-grained output at the token level (Rajpurkar ctt al. 2016b Sang & De Meulder 2003).. Therefore, the representations provided by PrLMs are fine- grained (word or sub-word), which can be easily recombined to representations at any granularity, and applied various downstream tasks without substantial task-specific modifications.
Besides fine- mee-grrined tokens and sentences, coarse- gaansee span-level language units such as phrases, name entities are also essential for NLU tasks. Previous works indicate that the capability to capture span-level information can be enhanced by altering pre-training objectives. SpanBERT (Joshi et al.) 2019) extends BERT by masking and predicting text spans rather than single token for pre-training. ERNIE models (Sun et al., 2019 Zhang et al. 2019a) employ entity level masking strategy for pre-training. StructBERT (Wang et al. 2019) encourages PrLMs to incorporate span-level structural information by adding trigram de- de-shuffling above show that the incorporation of levn- information in the pre-training phase is effective for various downstream NL.U tasks.
pre- trainiinn objective. The methods mentioned

---

Under review as a conference paper at ICLR 2021
However, since different downstream tasks have different requirements for span-level information, the strategy of incorporating span-level information in pre-training might not be suitable for all downstream tasks. For example, by leveraging entity level masking strategy in pre-training, ERNIE models (Sun et al. 2019 Zhang et al. 2019a) achieve remarkable gain in entity typing and Relation Classification, but when it comes its performance is even worse than BERT. Therefore, incorporating span-level information more flexibly and more universally, is imperatively necessary. The representations generated by PrLMS are supposed to be widely applicable for general cases; meanwhile, they are also expected to be flexibly adapted to various specific downstream tasks. Thus introducing span-level clues in good timing maters lot. In this paper, we propose a novel method, Later Span Adaptapan (LaSA), that would enhance the use of span-level information a task-speeific fine-tuning manner, which is lighter and more adaptive compared to existing methods.
language inference tasks like MNLI (Nangia al.. 2017),
In this work, based on the fine-grained representation generated by BERT, a computationally motivated segmentation is applied to further enhance the utilization of span-level information. Previous work has used semantic role labeling (SRL) (Zhang et al.] 2019b) or dependency parsing (Zhou et al., 2019) as auxiliary segmentation tools. Nevertheless, these methods require extra parsing procedure, which reduces the simplicity of use. In our method, the segmentation obtained according pre-sampled n-gram dictionary. The fine-grained representation in the same span within the segmentation is aggregated to a span-level representation. On this basis, the span-level representations are further integrated to generate sentence-level representation to make the most of both fine-grained and span-level information.
We conduct the experiments and analysis on the GLUE benchmark (Wang et al. 2018), which contain various NLU tasks, including natural language inference, semantic similarity, and text classification. Empirical results show that our method can enhance the performance of PrLMs the same degree as altering the pre- pre-training objectives, but more simply and adaptively. Ablation studies and analysis verify that the introduced method is essential to the further performance improvement.
2 RELATED WORK
2.1 PRE-TRAINED LANGUAGE MODELS
Learning reliable and broadly applicable word representations has long been a prosperous topic for the NLP community. Language modeling objectives are shown effective for generating satisfying distributed representation (Mnih & Hinton] 2009). By leveraging neural network and large text corpus, Mikolov et al.) (2013) and |Pennintonn al. (2014) achieve to train widely applicable word embeddings in an unsupervised manner. ELMo (Peters et al.) 2018) further advances state of the art for various downstream NLU tasks by generating deep contextualized word representations. Equipped with Transformer (Vaswani et al. 2017). GPT (Radford et al., 2018) and BERT (Devlin) jet al. (2018) further explore transfer learning, where models are firstly pre-trained on a large corpus and then applied to downstream tasks in a fine-tuning manner. Recent PrLMs extends BERT in multiple ways, including using permutation language model (Yang et al. 2019), training on a larger corpus and using more efficient parameters (Liu al.. 2019b), leveraging parameter sharing strategy (Lan et al. 2019). employing GtN-style architecture (Clark et al. 2020). T5 (Raffel et al., 2019) further explores the limit of transfer learning by conducting exhaustive experiments.
2.2 COARSE-GRAING PRE- TRAINING METHODS
Previous works indicate that the incorporation of coarse-grained information in pre-training can enhance the performance of the PrLMs. Initially, BERT uses the prediction of single masked tokens as one of the pre-training objectives. Since BERT uses WordPiece embeddings (Wu et al.) 2016), sentences are tokenized into the sub-word level so that the masked token can be sub-word token such as "##ing". Devlin et al. (2018) then points out that instead of masking a single token, using "whole word masking" strategy can further improve BERT's performance. After that, (Sun et al.) 2019 Zhang et al., 2019a) verify that PrLMs can benefit from entity-level masking strategy in pre- training. In Span (Joshi et al.) 2019), the model can better represent and predict spans of text by masking random contiguous spans in pre-training. Recently, by making use of both fine- fine-graine grained and coarse-grained tokenization, AMBERT (Zhang & Li 2020) outperforms its precursor in various
2

---

Under review as a conference paper at ICLR 2021
C
Sentence-level representation
jumping
for
frisbee
the snow
representation
Fine-grained snow representation:
dog
[pad]
is
jumping [pad]
[pad]
#Mls
##bee
Ex
dog
is jumpinn
for
friabee
the snow
PrLM
Pre-sampled r-gram dictionary
(CLS)
is jumping for
#tlls miibee in
the
Subword Tokenization
Sentence Segmentation
dog is jumping for frisbee in the snow
Figure 1: Overview of the framework of our proposed method
NLU tasks. All these works focus on encouraging PrLMs incorporate coarse- coarse-grained information. To the best of our knowledge, incorporating coarse-graiee saasengr information in fine-tuning is still a white space, which makes our work a valuable attempt.
2.3 INTEGRATION OF FINE-GRAINED REPRESENTATION
Different formats of downstream tasks require sentence-leve representations, such natural language inference (Bowman et al.) 2015 Nangia et al.] (017). semantic textual similarity (Cer
al., 2017) and sentiment classification (Socher et al. 2013). Besides directly pre-training the
representation of coarser granularity (L.e & Mikolov] 2014; Logeswaran & Lee 2018). a lot of methods have been explored obtain task-specific sentence-level representation by integrating fine-grained token-level paarrseentatinssConneeeo convolutional neural network (CNN) on top of pre-trained word vectors, can get sentence- level representation that is well adapted to classification tasks. Lin et al. (207)) leverage self- attentive module over hidden states of BiLSTM to generate sentence-leve. representations. [Zhang] al. (2019b) use CNN layer to extract word-level representations form sub-word representations and combine them with word-level semantic role representations. Inspired by these methods, after series of preliminary attempts, we choose hierarchical CNN architecture to recombine fine-grained representations to coarse-gran coarse-grained ones.
et al.) 2017). Kim (2014) shows that by applying
METHODOLOGY
Figure shows the overview of the framework of our method, which is primarily based on BERT and leverages segmentation as an auxiliary tool. We do not exhaustively illustrate the details of BERT, given the ubiquitousness of the architecture. Further information on BERT is available in Devlin et al. (2018). An input sentence is tokenized to the sub-word level and passed to BERT to obtain the fine-grained representation. In the meantime, the segmentation components the very input sentence is generated according pre-sampled n-gram dictionary. Then, incorporate the segmentation into the fine-grained representation provided by BERT and divide the representation into several spans. After passing the spans through hierarchical CNN module, obtain
3

---

Under review as a conference paper at ICLR 2021
dog
is jumping
frisbee
snow
man
drinking
orange juice
restaurant
dog
is jumping
older man is intended to give managers an overview of the acquisitinn process and to help them
for (a)
frisbee the snow
drinking
orange juice
af restaurant
(b)
the model
decrease acquisttion risk
The model intendeed (o] give managers) an overview of ihee acquisition process and to help them decrease (scquisttion risk
(c)
Figure Segmentation Examples
coarse- coarse-grained grained enhanced representation. Eventually, the fine-grained representation of [CLS] token provided by BERT and the coarse-grained information enhanced representation concatenated to form the final representation that makes the most multi- multi-grained information for downstream tasks.
3.1 SENTENCE. SEGMENTATION
Previous works use semantic role labeling (SRL) (Zhang et al. (019b) and dependency parsing Zhou et al.) 2019) as auxiliary segmentation tools. Nevertheless, these methods require extra parsing procedure, which reduces the simplicity of use. To get a reasonable segmentation simpler and more convenient manner, sample meaningful n-grams that occurred in the wikitext- 103 dataset based on frequency and obtain a pre-sampled dictionary.
the pre-sampled dictionary to locate matching n-grams from the start of the sentence. Longer n-grams are prioritized during the matching procedure. Unmatched tokens would be kept they are, by doing so, can obtain segmentation of the sentence. Figure 2 shows several segmentation examples of the sentences from the GLUE dataset.
For given input sentence,
3.2 SENTENCE ENCODER ARCHITECTURE
For a given input sentence X , Fn of length n, the sentence first tokenized to sub-word tokens (with special token [CLS] at the beginning) and transformed to fne-grained representation E {ei...., .mm} (m usually larger than n) based on WordPiece embeddings [Wu et al. 2016). Then the transformer encoder (BERT) captures the contextual information for each token via self-attention and produces a sequence of fine-grained contextual embeddings = (4...., ). where is the contextual representation of special token [CLS]. According to the segmentation generated by the pre-sampled dictionary, the fine-grained contextual representations are grouped to several spans [CI,..., C, with being hyperparameter defining the max span number for all processed sentences. Each C; contains several contextual fine- grained representations extracted from dedoted as (t), (2i,tt..... t2... t/}. the spans. A CNN-Maxpooling module is applied to each C₂ to get span-level representation
is another hyperparameter defining the max token number for all
Pj Ru [1), f4t..... . 1=+x-1] b1)
= M MarPooling(ch ct))
where W, and by are trainable parameters and & is the kernel size. Based on the span-level representations (ci,..., G. another Max-oMapnolinng module is applied to obtain a sentence-level representation with enhanced coarse-grained information:
8 ReLU(W22 (ci, G+1, -etl...... -itt-......... bz)
Max Maonoooiigg(ss...... x))
Finally we concatenate with the contextual fine-grained representation & of special token [CLS] provided by BERT, and generate a sentence-leve representation s" that make the most of both fine- grained and coarse coarse-grained information for downstream tasks: t1.
We have also tried the PMI approach prune dictionary, but the result is competitive.
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
4.1 TASKS AND DATASETS
The evaluation of our method is conducted on nine NLU benchmark datasets, including text classification, natural language inference, and semantic similarity. Among them, eight datasets available from the GLUE benchmark Wang et al. (2018). We also use SNLI Bowman et al. (2015), widely acknowledged natural language inference dataset, one of our datasets.
Two categories of Classification datasets involved in method. The Corpus of Linguistic Acceptability (CoLA) (Warstadt et al. (2018) gives judgment on whether an English sentence is linguistically acceptable not. The Stanford Sentiment Treebank (SST-2) (Socher et al. 2013) provides sentiment labels, i.e., positive or negative, for sentences extracted from movie reviews.
Classification
Natural Language Inference Natural Language Inference aims determining the relationship between a pair of sentences, such as entailment, neutral, and contradiction, based on semantic meanings. Four datasets are involved in our study, including Multi-Genre Natural Language Inference (MNLI) (Nangia et al.] (2017), Question Natural Language Inference (QNLI) (Rajpurkar al. 2016a), Stanford Natural Language Inference (SNL.) (Bowman et al. 2013) and Recognizing Textual Entailment (RTE) (Bentivogli et al. 2009).
Semantic Similarity Semantic similarity tasks target at giving a judgment on whether two sentences are semantically equivalent or not. The current challenge for Semantic Similarity to increase the accuracy of rephrased concepts recognition, negation understanding, and syntactic ambiguity discernment. Three datasets are employed, including Semantic Textual Similarity benchmark (STS-B) (Cer et al. 2017), Quora Question Pairs (QQP) dataset (Chen et al. 2018 and Microsoft Paraphrase corpus (MRPC) (Dolan Brockett (2005).
4.2 PRE-TRAINED LANGUAGE MODEL
We employ the PyTorch implementation of BERT based on HuggingFace's codebase (Wolf et al, 2019).
We use two released BERT versions our pre- pee-traied encoder as well as baselines
BERT Base The pre-trained language model was released by Devlin et al. (0188) with 12 layers, 798 hidden layer size, and up to 108 million parameters.
BERT Large Whole Word Masking The pre-trained language model released by Devlin et al. 2018 with 24 layers, 1048 hidden layer size, and up to 334 million parameters. The BERT Large wwm employs whole word masking strategy in pre-training and achieves a better performance than the initial version of BERT Large. So employ BERT Large whole work masking stronger baseline, and further show that our method can enhance the utilization of span-level information.
4.3
SET UP
We sample all the n-grams, with n S 5, that occur in the wikitext- 103 dataset more than ten times to form dictionary. The dictionary contains more than 400 thousand n-grams and is used segment input sentences. In the process of segmentation, two hyperparameters are introduced: r defining the largest number of spans within an individual sentence, and representing the largest number of tokens contained within a single span. Padding and tail are applied to guarantee the dimensions of features for each sentence to maintain the same. We set r equals to 16, and based on downstream tasks, choose I in {64,128)
We follow the same fine-tuning procedure as BERT. Sentence-level representation (with enhanced span-level information) is directly passed to dense layer to get the class logits score. Training objectives are CrossEntropy for classification tasks and Mean Square Error loss for regression tasks.

---

Under review as a conference paper at ICLR 2021
Method
Classification CoLA (mc)
Natural Language Inference
Semantic Similarity
Avg.
SST-2 (acc)
MNLI
QNLI (acc)
RTE
MRPC
QQP
STS-B
m/mm(acc)
(acc)
(FI)
(FI)
(pc)
In literuture
36.0
90.4
76.4/76.1
79.9
56.8
84.9
64.8
75.1
70.5
GPT
45,4
91.3
82.1/81.4
88.1
56.0
82.3
70.3
82.0
72.8
GPT on STILTs
47.2
93.1
80.8/80.6
87.2
69.1
87.7
70.1
85.3
76,9
MT-DNN
61.5
95.6
86.7/86.0
75.5
90.0
72.4
88.3
82.2
BASE
52.1
93.5
84.6/83.4
66.4
88.9
71.2
87.1
78.3
BERTLARGE
60.5
94.9
86.7/85.9
92.7
70.1
89.3
72.1
87.6
80.5
implementatton
BERTRASE RASE
51.4
92.1
84 44.4/83.5
90.3
67.1
88.3
71.3
85.1
79.3
BERTBASE BASE LaSA
55.1
93.6
84.8/84.3
90.6
69.6
88.7
71.9
86.5
80.4
BERT WwM
61.1
93.6
87.1/86.5
93.9
77.3
90.0
71.9
88.1
83.3
BERT WWM + LaSA
62.9
94.1
87.6/87.0
94.3
81.4
91.1
72.4
89.1
84.4
Test sets performance on GLUE benchmark. All the results are obtained from Liu et al. (019a), Radford al. (2018). We exclude the problematic WNLI set and do not show the accuracy of the datasets have FI scores to save space. me and pc denote the Matthews correlation and Pearson correlation respectively.
Table 1:
We use Adam as our optimizer with an initial learning rate (1e-5,2e-5, 3e-5), a warm-up rate of 0.1, and an 1.2 weight decay of 0.01. The batch size is set in {16, 32, 48). The maximum number of epochs selected transformed to WordPiece embeddings, with maximum length chosen in {128, 256}. The CNN layer's 's output size is the same as the hidden size of PrLM, and the kernel size is set to
(2.3,4.5) depending on downstream tasks. Sentences are tokenized and
4.4 RESULTS
Table shows results the GLUE benchmark datasets, demonstrating our LaSA approach can remarkably improve the performance of PrLMS on downstream tasks. Since LaSA is primarily established on BERT, and evaluated under the same procedure, it is evident that the improvement is entirely contributed by our newly introduced span adaptation.
Compared to previous works that demand large-scale pre-training in the incorporation of span-level information, LaSA is able to achieve the same level of performance improvement, and moreover, require remarkably less time and resources. Table shows the comparison of LaSA and SpanBERT on the GLUE benchmark. reaches one percent gain on average over its baseline (BERT-Iseq). which is a BERT pre-trained without next sentence prediction (NSP) object. Comparatively, LaS improves the performance of our baseline by one point one percent on average. Furthermore, LaSA can improve the performance of PrLM on every downstream task, while underperforms its baseline dataset like MRPC and QQP. Such comparison indicats that our method can be more dynamically adapted to different downstream tasks.
Method
Classification
Natural Language Inference
Semantic Similarity
Avg.
Gain
CoLA
SST-2
MNLI
QNLI
RTE
MRPC
QQP
STS-B
(mc)
(acc)
m/mu(acc)
(acc)
(acc)
(F1)
(F1)
(pc)
BERT-Iseq
94.8
88.0/87.4
93.0
72.1
91.2
72.1
89.0
83.5
1.0
SpanBERT
88. 8/87/87.7
94.3
79.0
71.9
84.5
BERTwwM
61.1
93.6
87.1/86.5
93.9
77.3
90.0
71.9
88.1
83.3
1.1
BERT WERTwwM+ LaSA
94.1
87.6/87.0
81.4
91.1
72.4
84.4
Table 2: Compairison with on test sets of GLUE benchmark
Table shows our approach also improves PrMLs performance on the SNLI benchmark. The result indicates that the performance of our method is comparable published SOTA achieved by SemBERT. Unlike SemBERT that incorporates explicit contextual semantics by leveraging a pre-trained semantic role labeler, LaSA only a pre-sampled dictionary provide the auxiliary segmentation information. Thus, compared to SemBERT, LaSA is more flexible method to achieve performance improvement.
'The baseline BERT pre-trained without sentence prediction object.

---

Under review as a conference paper at ICLR 2021
BASE
BERT LASA
WWRT
BERT SemBERT wWM
dev
90.6
91.3
92.0
92.3
92.2
test
90.7
91.1
91.4
91.7
91.9
Table 3:
Accuracy on dev and test sets of SNLI. SemBERT WWM Zhang et al. 2019b) is the published SOTA on SNLI.
5 ANALYSIS
5.1
ABLATION STUDY
As shown in Table to evaluate the contributions of key factors in LaSA, we perform several ablation studies on the dev sets of eight downstream tasks using BERTBASE as our PrLM. Three sets of ablation studies are conducted. Firstly, in order see the hierarchical CNN structure alone can improve the overall performance, we directly apply hierarchical CNN structure without the segmentation. Secondly, in order to test whether our specific segmentation really helps the model to better incorporate the span-level information, segmented sentences. Thirdly, in order to compare the segmentation generated by our method and the segmentation generated by pre-traaiie pre-trained chunker, leverage pre-trained chunker provided by NLTK (Natural Language Toolkit) to generate the spans and conduct the experiments with the same procedure.
conduct the experiments with randomly
According to the result, both hierarchical CNN structure and random segmentation can improve the performance on various downstream tasks some extent. The result of the third experiments indicate that the segmentation provided by pre-trained chunker is beneficial to our method. However a pre-trained chunker requires extra POS parsing procedure, whereas our method only leverage pre-sampled dictionary which is more time-saving and achieves better results. Our LaSA method improve the performance on downstream tasks significantly, raising the average score from 82.6 of baseline to 84.2. This result demonstrates that the improvement is largely due to the specific sentence segmentation of our method.
Method
Classification CoLA (mc)
Natural Language Inference
Semantic Similarity
Avg.
SST-2 (acc)
MNLI m/mm(ace)
QNLI
RTE
MRPC
QQP STS-B
(acc)
(ace)
(FI)
(acc)
(pc)
BERT RASE
$1.7
92.0
84.0/84.5
91.3
68.7
91.0
90.8
89.0
82.6
BERT BASE CNN
53.4
91.8
84.2/84.4
91.2
65.0
91.6
91.2
89.7
82.5
BART Random LaSA
53.8
91.7
84.5/84.7
91.6
70.4
90.0
91.0
89.7
83.0
BASE
NLTK LaSA
58.2
92.1
84.4/84.8
91.8
70.0
91.0
89.8
83.7
BERTRASE
LaSA
59.1
93.5
84.7/85.1
91.7
71.1
91.2
91.2
90.0
84.2
Table 4: Ablation studied on dev sets of GLUE benchmark
5.2 ENCODER ARCHITECTURE
Conneau 2017)) manifest that different sentence encoder architectures have a significant impact on the performance of models. (Toshiiwll et al.) (2020) also indicate that the choice of span representation has significant impact on many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution.
In order to test the efficacy of LaSA's encoder architecture, we respectively substitute the component of the encoding layer and the overall structure. For the component of the encoding layer, we compare CNN (Kim 2014) with the Self-attentive module (Lin et al. 2017). For the overall structure, two categories are considered: a single layer structure with the max -poolin operation and a hierarchical structure.
*Random LaSA indicate the LaSA method with randomly segmented sentences.
NLTTK LaSA indicate the LaSA method with segmentations generated hy NLTK pre-trained chunker.

---

Under review as a conference paper at ICLR 2021
Mapping each component of the encoding layer with the overall structure, obtain four different Drrention-Maxpolnng Attention-Attent
encoder architectures: CNN-Maxpooling, CNN-CNN,
Experiments conducted SNLI dev and test sets. As shown in table 5 the hierarchical CNN
(CNN-CNN) encoder architecture is most suitable for our situation.
CNN-Max
CNN-CNN
Attention Max
Attention-Attention
dev
90.9
91.3
90.7
90.8
test
90.9
91.1
90.5
90.8
Table 5: The influences of different encoder architecture
5.3 SIZE OF R-GRAM DICTIONARY
Given the fact that we use a pre-sampled dictionary to segment the sentences, different sizes of dictionaries will lead different segmentation. Initially, for sentences without segmentation, every single token regarded as a span. Figure shows how dictionary size and corresponding segmentation influence the average number of spans in the sentences in the CoLA and MRPC datasets. As the dictionary size enlarges, more n-grams are matched and regrouped together, which significantly reduces the average number of spans.
To study the impact of dictionary size on model performance, we conduct experiments on the dev sets of two downstream tasks: CoLA and MRPC. order focus the influence segmentation, and eliminate the effects of fine-grained representations generated by PrLM, we do not apply the concatenation operation in this experiment. Instead, representations with enhanced span-level information are directly passed to a dense layer to obtain the prediction results. As shown in 4 experiments incorporating the segmentation provided by pre-sampled n-gram dictionary achieve better results than the one with random segmentation. Furthermore, the results indicate that middle- sized dictionaries (20k to 500k) usually achieve better performance. This phenomenon is consistent with intuition, since small-sized dictionaries tend to miss meaningful n-grams, while large-sized ones are inclined to group meaningless n-grams together unnecessarily.
Average number spans
Average number spans
26
(average number tokens)
25.1 Javerage number tokens)
22
20
18.7
18.0
6.3
5.7
5.5
5.2
5.1
16
18
17.0
16.2
15.6
14.7
14.0
10k
100k 200K 1000
Size of n-gram dictionary
Size n-gram dictionary
(a) CoLA
(b) MRPC
Figure 3: Influence of dictionary size on the average number of spans in the sentences
58.4
887.0
58
57.4
Accuracy
Accuracy
86.7
96.3
36.5
56
86
96.0
54.8
54.7
Radom
1000k
Radom
1000k
Size n-gram dictionary
Size n-gram dictionary
(a) CoLA
(b) MRPC
Figure 4: Quantitative study on the influence of the size of n-gram dictionary
Attention indicate the Self-attentive module Lin et al. 2017
8

---

Under review as a conference paper at ICLR 2021
5.4 APPLICATION TO STRONGER PRLM
Besides BERT, we want to see if our method can improve the performance of stronger PrLMs such RoBERTa (Liu al. 2019b), which optimizes BERT with improved pretraining procedure, and SpanBERT @Johi et al. 2019). which extends BERT by masking and predicting text spans rather than single token for pre-training
Table 6 shows that our approach can enhance the performance of both RoBERTa and SpanBERT. For RoBERTa, despite RoBERTa is very strong baseline, we remarkably improve the performance of RoBERTa on RTE by four percent. For SpanBERT, since it already incorporated span-level information during the pre-training the results imply that, comparing to SpanBERT, our method utilizes the span-level information and improves the performance of PrLMs in different manner.
Method
Classification Col.A (mc)
Natural Language Inference
Semantic Similarity
Avg.
SST-2 (acc)
MNLI QNLI m/mm(acc)
RTE (acc)
MRPC
QQP
STS-B
(acc)
(F1)
(acc)
(pc)
SpanBERT BASE
55.5
92.4
85.2/85.1
92.1
75.1
91.2
90.9
89.4
84.0
SpanBERT BASE LaSA
57.1
92.7
85.5/85.4
92.6
79.4
92.1
90.9
90.1
85.1
RoBERTa LARGE
68.0
96.4
90.2/90.2
94.7
86.6
90.9
92.2
92.4
89.0
RoBERTa ARGE LaSA
68.9
96.1
90.3/90.2
94.3
90.6
92.8
92.2
92.4
89.8
Table Results on dev sets of GLUE benchmark with stronger baseline
5.5 LASA FOR TOKEN-LEVEL TASK
Our experiments are conducted on the GLUE benchmark which only requires sentence-level representations. However, for natural language understanding, there other tasks such named- entity recognition (NER), where token-level representations are needed. Our method can be applied to token-level tasks with simple modification of encoder architecture (e.g. removing the pooling layer of CNN module). Table shows the results of our approach on the CoNLL-2003 Named Entity Recognition (NER) task (Tong Kim Sang & De Meulder 2003) with BERT our PrLM.
The result indicate that our approach can also improve the performance of PrLM on token-level tasks. As discussed by (Toshniwal et al. 2020), the encoder architecture has remarkable impact on the efficacy of incorporating span-level information. Since we have not explored the encoder architecture for the token-level tasks, we believe that with more adapted encoder architecture the performances on the token-level tasks can be further promoted.
BERTBASE
BARTBASL+LAN
BERT LARGLARGE+LaSM
dev
91.7
92.1
92.3
92.5
test
95.7
96.2
96.5
96.8
Table 7: Fl on dev and test sets of named entity recognition on the CoNLL-2003 dataset.
CONCLUSION
This paper proposes novel method LaSA that takes advantage of flexible span-level information in fine-tuning with fine-grained representations generated by PrLMs. Leveraging reasonable segmentation provided by a pre-sampled n-gram dictionary, LaSA can further enhance the performance of Pr..Ms on various downstream tasks. Recent works focusing the incorporation of span-level information in pre-training by altering pre-training objectives or PrLM structures are time-consuming and require enormous resources, therefore, we hope this work could shed light on future studies on adaptation in fine-tuning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In ACL-PASCAL, 2009.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In EMNLP. 2015.
Daniel Cer, Mona Diab, Eneko Agirre Inigo Jopez-Gazpin and Lucia Specia. Semeval-2017 task
I: Semantic textual similarity muullarty-multilinnua and cross- loos-lingual focused evaluation. arXiv preprint arXiv: 1708. 00055, 2017.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training text encoders discriminators rather than generators. ICLR, 2020.
Alexis Conneau, Douwe Kicla, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv: 170.00/70 2017.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1800008000 2018.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In /WP2005, 2005.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint 1907. /0529, 2019.
Yoon Kim. Convolutional neural networks for sentence classification. arXiv: 5882, 2014.
arXiv preprint
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut, Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.1 11942, 2019.
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference on machine learning, pp. 1188-1196, 2014.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint
17X3v:1703.03310,, 2017.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint 15Xiv:1901.11504, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:19017 1692, 2019b.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803. 02893, 2018
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pp. 1081-1088, 2009.
Nikita Nangia, Adina Williams, Angeliki Lazaridou, and Samuel R Bowman. The repeval 2017 shared task: Multi-genre genre natural language inference with sentence representations. In RepEval, 2017.
10

---

Under review as a conference paper at ICLR 2021
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. Technical report, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.1006 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016a.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016b. Association for Computational Linguisties.
Erik Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. arXiv preprint cx/0360500, 2003.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.009:
2019.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CONLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https:
7/ /ww. acwweb. wrg.or/////o... http:///nhhoooooooooooooooo
0419
Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, and Kevin Gimpel.
A cross-task analysis of text span representations. In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 166-176, Online, July 2020. Association for Computational Linguistics. doi: 10. 11.1183/1112222.rep4lllllp- aclweb. wwweeboorg//tthaeeeo attp////hhhoooy/22000 repp441p-11.2
URL https://www.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 2018 EMNLP Workshop BlackboxNLP, 2018
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv: 1908. 04578, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1805. 12471, 2018
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, 1b0/1111 2019.
11

---

Under review as a conference paper at ICLR 2021
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.080 2016.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019.
Xinsong Zhang and Hang Li. Ambert: A pre-trained language model with multi-grained tokenization, arXiv preprint arXiv:2008. 11669, 2020.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. arXiv preprint arXiv: 1905.07129, 2019a.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. Semantics-aware bert for language understanding. arXiv preprint 1909:29900002200 2019b.
Zhuosheng Zhang, Junjie Yang, and Hai Zhao. Retrospective reader for machine reading comprehension, 2020.
Junru Zhou, Zhuosheng Zhang, and Hai Zhao. Limit-bert: Linguistic informed multi-task bert. arXiv preprint 1910. 2019.
A APPENDIX
Since fine-tuning BERT on GLUE tasks might have high variance, table the average results of the application of LaSA to BART on dev sets of GLUE benchmark. Furthermore, since PrLMs are very strong baselines, the improvements can be marginal, thus a significance test would be beneficial for measuring the difference in model performance.
Follow the procedure of (Zhang et al. 2020), we use the McNemars test to test the statistical significance of our results. This test designed for paired nominal observations, and is appropriate for binary classification tasks.
The p-value is defined as the probability, under the null hypothesis, of obtaining a result equal to or more extreme than what was observed. The smaller the p-value, the higher the significance. A commonly used level of reliability of the result is 95%, written as p = 0.05. As shown table comparing to the baseline, for all the binary classification tasks of GLUE benchmark, our method pass the significance test.
Method
Classification CoLA SST-2 (mc) (acc)
Natural Language Inference
Semantic Similarity
Avg.
MNLI m/mm(acc)
QNLI RTE (acc)
MRPC QQP STS-B
(acc)
(F1)
(acc)
(pc)
BERTBASE
52.9
91.9
84.0/84.4 84.6/85.1
91.3
68.9
91.1
90.8
88.8
82.6
BERTRAST + LaSA
59.5
93.3
91.9
71.7
91.1
91.2
90.3
84.2
Table 8: Results dev sets of GLUE benchmark, each result is median three runs.
CoL.A
SST-2
QNLI
RTE
MRPC
QQP
p-value 0.005
0.012
0.023
0.009
0.008
0.031
Table 9: Results of McNemars tests for binary classification tasks of GLUE benchmark, tests are conducted based on the results of best run.
12


Reviews:
[{"review_id": "k5WqL5d-eHM", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to incorporate span information in pre-trained language models like BERT during fine-tuning. In the proposed approach, the segmentation of a sentence is obtained according to a pre-sampled n-gram dictionary. The fine-grained representation in a same span within the segmentation is aggregated to a span-level representation using a CNN model. These span-level representations are further aggregated using a CNN model to generate sentence-level representation. The experiments show that the proposed model can achieve similar performance gain as other span-based language models which includes span information during pre-training.    \n\nThe paper is well written and the proposed method is novel. The novelty of the method is in mainly including span information only during fine-tuning and also the way spans are identified in a sentence. The authors conducted an extensive set of experiments with ablation and results show that the proposed idea is effective and attains similar gains as other span-based language model. \n\nIt is not clear how this model helps in practice since it is achieving similar performance as other models like SpanBERT. It is true that this model doesn’t introduce complexity while pre-training but pre-training is a one-time process and even if pre-training is slower, it does not affect downstream tasks. However, the proposed model introduces complexity in the downstream tasks and may affect training time in the downstream task.\n \nThe authors claim that the proposed model perform similar to models like SpanBERT but it is not clear if both models are improving on similar aspects. Some fine-grained analysis of results with these models might be insightful.\n\nAnother concern I have is that the proposed model can only work in the scenarios where sentence-level representation is required. However, there are tasks like named-entity recognition (NER) where word level representation is needed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "iBuyelfEcm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new mechanism based on span-based representation grouping for better language model finetuning. Specifically, firstly, the paper proposes to utilize an n-gram splitter to segment the text into several segments. Secondly, a hierarchical network is built based on a three-level network (subword level->span level->sentence level). Using these tasks to perform fine-tuning on top of a pre-trained masked language modeling shows improvements than directly fine-tuning on the pre-trained language models. By conducting experiments on the GLUE benchmark with the BERT-base and BERT-large models, the results are improved. The general idea is simple and easy to understand. However, I have several concerns which are list as follows,\n\n1. Combining span-level information has been widely studied in the literature[1] (not cited). Although the tasks between the two papers are different, the novelty of the paper is limited. Besides, it is also worth discussing between Sec 5.2 and [1]. \n2. The experiments are conducted on BERT-base and BERT-large models. However, the two models are weak baselines. I would like to see the empirical results on stronger models, like RoBERTa related models.\n3. As the paper focuses on training span representations, it is necessary to perform experiments on entity-based tasks. The experiments setup may refer to the paper [2].\n4. The paper proposed to use the n-grams, it is also necessary to have a comparison with the pre-extracted spans using an off-the-shell chunker.\n\n\nI also have the following questions:\nIf a pretrained chunker is utilized to extract chunks, what are the results like?\nIn section 5.2, does the variant of n-grams affect the results?\n\n\n\nFor these reasons, I do not recommend acceptance of this paper.\n\n\n\n\n[1] Toshniwal et al. A Cross-Task Analysis of Text Span Representations, ACL RepL4NLP-2020\n[2] Yamada et al. LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention, EMNLP 2020\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NbG-UyXv4aD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n \nThe paper introduced a fine-tuning approach which adapts the subword level, span level and sentence level to the target tasks. The span segmentation is done by using a pre-trained n-gram statistical model.  Empirical studies on GLUE benchmark show that the proposed approach consistently improves the performance of BERT. \n \nPros:\n \n1. The idea of leveraging a n-gram model for pre-trained language model fine-tuning is interesting.\n2. The proposed method is well motivated and easy to understand.\n \nCons:\n\nMy concern is the significancy of the results. Since the improvement of the proposed method is marginal (~ 1% avg) and fine-tuning BERT on GLUE tasks might have high variance, authors should report the mean and std over multiple runs.\n \nQuestions:\n\nIf the model is pre-trained with span level information (e.g., spanBERT), will the proposed method outperform normal fine-tuning?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2u6trT873YO", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Previous works reveal that span-level information can enhance the performance of PrLMs if they are used in pre-training. However, the existing methods require enormous resources and lack adaptivity. To this end, the paper proposes a method that combines span-level information into the representations generated by PrLMs during the fine-tuning phase. To combine span-level information, the paper first breaks a sentence into various span components. Then, an accumulated representation with enhanced span-level information is built based on the sub-token-level representation provided by PrLMs. The experimental results on the GLUE benchmark show that the proposed method improves the performance of PrLMs. The main contribution of this paper is the introduction of generating the span components via a pre-sampled dictionary. Overall, the proposed method is not novel since similar methods or ideas have been widely used in NPL.\n.\n\nConcerns:\n\n1. Employing span-level information in PrLMs was proposed in previous works such as SpanBERT. The paper presented a way that combines span-level information into the representations generated by PrLMs during the fine-tuning phase. However, as shown in Table 2, the proposed method is not better than SpanBERT. So, what are the advantages of the proposed method.\n2. As shown in Table 1, compared with the baselines, the proposed method does not bring remarkable promotion.\n\nMinor comments: \n1. The subscript “j“ should be “1” in the second line of Formula (1).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to combine the span-level information into a phrase-level representation in the fine-tuning phrase for pre-trained language models.  The phrases are pre-defined in a dictionary.  Experiments show improvements in various downstream tasks in the GLUE benchmark.  It's a borderline paper.  Various concerns were raised by the reviewers, for example, the relation with the SpanBERT method in pre-training phrase and the significance of the results.  The authors addressed most of the concerns but the reviewers were not fully convinced.  In general, I think it is an interesting paper with good motivation and results.  Hope it can be improved (e.g. more experiments on SpanBERT) and accepted in another conference.
Decision: Reject