Improving Span Representation by Efficient Span-Level Attention
Authors: Pengyu Ji
Songlin Yang
Kewei Tu
Venue: EMNLP 2023 Findings (2023)
Keywords: representation learning
efficient methods

Abstract: High-quality span representations are crucial to natural language processing tasks involving span prediction and classification. Most existing methods derive a span representation by aggregation of token representations within the span. In contrast, we aim to improve span representations by considering span-span interactions as well as more comprehensive span-token interactions. Specifically, we introduce layers of span-level attention on top of a normal token-level transformer encoder. Given that attention between all span pairs results in $O(n^4)$ complexity ($n$ being the sentence length) and not all span interactions are intuitively meaningful, we restrict the range of spans that a given span could attend to, thereby reducing overall complexity to $O(n^3)$. We conduct experiments on various span-related tasks and show superior performance of our model surpassing baseline models. Our code is publicly available at \url{https://github.com/jipy0222/Span-Level-Attention}.

Improving Span Representation by Efficient Span-Level Attention
Pengyu Ji, Songlin Yang, Kewei Tu
School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging {jipy2023, yangsl, tukw)@eshangaiitechh. edu.
Abstract
High-quality span representations are crucial natural language processing tasks involv- ing span prediction and classification. Most existing methods derive a span representation by aggregation of token representations within the span. In contrast, we aim to improve span representations by considering span-span inter- actions as well as more comprehensive span- token interactions. Specifically, we introduce layers of span-level attention top of normal loken-level transformer encoder. Given that at- tention between all span pairs results O(n") complexity (n being the sentence length) and not all span interactions are intuitively mean- ingful, we restrict the range of spans that given span could attend to, thereby reducing overall complexity to ) conduct ex- periments various span-related tasks and show superior performance of our model sur- passing haseline models. Our code publicly available at httph:tps::: //gtt jip@2222 Span-Level-Attentiae
X1
3g
X1
Xg
X2
(a) Inside-Toke
(b) Containment
2g
31
25
24
81
32
X4
(c) Adjacency
(d) All-Token
Figure I: Diagrams for attention patterns. Each cell represents span, e.g., the orange cell in each dia- gram represents the span consisting of tokens from J to 223 Orange cells represent target spans and blue cells represent spans they can attend to.
2021). The limitation of these methods lies in: (i) Span representations are dominated by subset tokens, resulting in potential lack of crucial in- formation. (ii) Intuitively, span interactions should play an important role in span encoding. For ex- ample, meanings of spans, especially constituents. can be composed from their sub-spans and disam- biguated by their neighbouring spans. However, such span interactions are completely ignored these methods.
1 Introduction
Many natural language processing tasks involve spans, making it crucial to construct high-quality span representations. In named entity recognition, spans are detected and typed with different labels (Yuan et al., 2022; Zhu et al., 2022); in coreference resolution, mention spans are located and grouped (Lee al.. 2017, 2018; Gandhi al., 2021; Liu al., 2022); constituency parsing, spans are assigned scores for constituent labels, based on which parse tree structure is derived (Stern et al., 2017; Kitaev and Klein, 2018; Kitaev et al., 2019). Most cxisting methods compute span represen- tations by shallowly aggregating token representa- tions. They either pool over tokens within the span (Shen et al., 2021; Hashimoto et al., 2017; Con- neau al., 2017). concatenate the starting and ending tokens (Ouchi et al., 2018; Zhong and Chen,
Inspired by the utilization of self-attention Transformer (Vaswani et al., 2017), we introduce span-level self-attention to capture span interac- tions and improve span representations. However, computing attention scores for all span pairs leads O complexity (n for sequence length). In addition, not all span interactions are intuitively meaningful. Therefore, we design four different span-level patterns to restrict the range of spans that given span could attend to: Inside-Token,
"Coresponding author.

---

Classificr
jection layer obtain final token representations 20, 11
Enhanced version of span representations:
Initial span representations. We follow Toshni- wal al. (2020) initialize span representations from contextualized token representations. During pilot experiments', we observe that among the five pooling methods (max pooling, average pooling, attention pooling, endpoint, diff-sum), max pool- ing performs the best. Therefore, we choose max pooling the default initialization method. Specif- ically, given span (i.j) and the corresponding token representations {-i,..... .Ej. within the span, the initial span representation Sij is computed by selecting the maximum value over each dimension of the token representations.
Soo- So1 502 STT
Span-level Transformer layer
X 4
Initial span representations:
S00: 502 Spp
Span Initialization
Token representations: Xg X
BERT-base-cased
Input text: Wa. Wa..., Wiz
Figure 2: Architecture of our model.
Span-level attention. We enumerate all the spans and input their representations to a Transformer encoder with span-level attention. Note that com- puting attention scores for all span pairs leads to (nn²) time and memory complexity because self- attention has a quadratic complexity and there are total of O(n²) spans. To reduce the complexity as well as encourage more meaningful span inter- actions, we design different attention patterns restrict the range spans that given span could attend to (Fig. 1). We use (el j)) to denote the set of spans that span (i, can attend to.
Containment, Adjacency and All-Token (Fig. 1). Each of them allows only (n) spans for attention, reducing the overall complexity to O(n²).
Many existing studies also aim at improving span representations. Yuan et al. (2022) utilize entity labels to define specific span representations for nested NER. Zhou al. (2022) use syntactic parse trees to enhance span encoding. Zhu al. (2022) improve span representations by stacking multiple span-token attention layers. Wang et al. (2022) introduce intra-span attention to enhance span representations by computing attention be- tween each given span and all other spans. Com- pared to existing works, our method offers unique advantages: (i) We design span representations for a wide range of tasks without relying on external information such as labels and parse trees. (ii) We lay more emphasis on span interactions by in- corporating span-level attention. (iii) We design span-level attention patterns to capture meaningful span interactions and reduce the overall complex- ity to an acceptable level, thereby ensuring both effectiveness and efficiency.
Inside-Tokes Each span attends to tokens within this span. This pattern maintains the connec- tion between spans and their internal tokens.
rel((i.))) (k,, k) =,....... i......)
Containment
Given a span, its super-spans and sub-spans may provide meaningful informa- tion of the span. However, the total number of super-spans and sub-spans of a given span is (n²). Considering the importance of start- ing and ending positions in span encoding, we propose that each span attends to spans that share the same starting or ending position. This pattern takes into account the contain- ment relationship as well as the starting and ending positions of spans, while reducing the number of spans to O
2 Method
Fig. illustrates the architecture of our model, which we describe from bottom up.
Token representations. Given a sentence wo w.,..., Wn. we pass through BERT (Devlin al., 2019) do tokenization and obtain contex- tualized token representations = Co CI....,0 Cr by taking a weighted average of the outputs from all layers. We then feed them into linear pro-
rell ( (1, = ((i, k3) i, .,...,777 TJU
(kk))) k- 0. ......
Resulls can be found in Table 7 Appendix B

---

Pattern
Task
(a)
(b)
(c)
(d)
NEL
REF
SRC
CTL
MED
CTD
Avg.
Max pooling
95.61
95.58
92.85
98.00
97.98
98.19
96.37
Best pooling
95.78
95.78
92.86
98.00
98.10
98.24
96.46
111
Token-level
95.73
95.76
93.32
98.34
98.04 04
98.61
96.63
1v
Pully-connected
95.77
95.63
93.11
98.36
98.12
98.69
96.61
1.
K
X
95.57
95.80
93.62
98.39
98.
98.53
96.67
2.
x
X
X
95.85
95.93
93.46
98.45
98.
98.79 79
96.78
3.
X
X
V
X
95.79
95.63
93.46
98.47
98.24
98.79
96.73
4.
X
X
v
95.85
96.06
93.31
98.35
98.05
98 62
96.71
5.
V
X
X
95.94
96.02
93.52
98.45
98.23
98. 83
96.83
6.
X
V
X
95.78
95.78
93.50
98.50
98,28
98.83
96,
7.
J
X
X
V
95.73
95.98
93.33
98.36
98.02
98.56
96.66
8.
J
V
X
95.76
95.92
93.50
98.47
98.19
98.79
96.77
9.
X
96.08
95.77
93.35
98.43
98.14
98.77
96.76
10.
V
95.95
95.80
93.31
98.44
98.23
98.80
96.75
11.
V
X
95.85
95.68
93.52
98
98.22
98.88
96.76
12.
X
95.87
95.89
93.36
98.44
98.18
98.80
96.76
13.
V <
95.79
95.73
93.29
98.45
98.25
98 80
96.72
14.
V
95.80
95.81
93.33
98.41
98.24
98.77
96.73
15.
V
95.8
95.80
93.38
98.43
98.20
98.76
96.73
Table I: Averaged F1 scores for 6 probing tasks with baselines and different pattern combinations. (a): Inside-Token, (b): Containment, (c): Adjacency, (d): All-Token.
Adjacency Each span attends to spans that share only the starting or ending positions of the span. Intuitively, adjacent spans often have strong correlations.
3 Experiment
3.1 Setup
We use BERT- HERT-base-cased to obtain contextualized token representations and keep it frozen when con- ducting probing tasks. We stack 4 Transformer encoder layers and the number of heads in multi-head attention to 4 to do span-level attention. Dataset details and other hyper-parameten can be found in Table 5 and Table 6 in Appendix A. We conduct all experiments on a single 24GB NVIDIA TITAN RTX and report the micro-averaged FI- scores. All results are averaged over three runs with different random seeds.
rel((i.5) -(i k)]lk .. TUU {(k i) ...
All-Token Each span attends all tokens in the input text. This pattern enables the acquisition of token information beyond span boundaries.
rel( = (kk, k) l- = O, ....
,
It is worth noting that all four patterns ensure that the number spans each span can attend to is Onn), which reduces the overall complexity to One)). Moreover, we can combine these four pat- terns arbitrarily to form new patterns when facing different scenarios.
3.2 Probing tasks results
We conduct probing tasks: named entity labeling (NEL), coreference are prediction (REF), seman- tic role classification (SRC), constituent labeling (CTL), mention detection (MED) and constituent detection (CTD), following Toshniwal et al. (2020). In these 6 tasks, we only need to do classification or prediction on given spans.
Inference and Training. After span-level atten- tion, we obtain an enhanced version of sij for each span. For single span tasks, we feed span rep- resentation into a two-layer MLP classifier. For tasks involving two spans, we concatenate the two span representations and feed them into the MLP classifier. The classifier maps the input into dimensional vector, where q is the size of the label set (including NoneType if necessary). We directly utilize the loss function of downstream tasks to train the model, such as the commonly used bi- nary cross-ent cross-entropy loss and cross-cntropy loss in multi-class classification tasks.
Table shows probing tasks results. We pose (i) pooling, (ii) best performing pooling among five pooling methods mentioned in section 2, (iii) max pooling after four additional layers normal token-level attention, and (iv) fully-connected span- level attention (i.e., the ( full span-level atten- tion without restriction) as four baselines? Overall,
"Stacking four layers of fully connecedde span-level atten- tion can result in the out-of- out-of-memory issue when doing ex-

---

fully-connected span-level attention shows good performance compared to pooling methods, vali-
Entaders&Datasels
BERT-Irozen
BERI-finetune
ACE04
ACE05
Avg.
ACE04
ACE05
Avg.
dating the effectiveness span-level attention. Fur- thermore, applying different attention patterns or pattern combinations not only reduces computa- tional complexity, but also significantly improves performance. This suggests that our proposed atten-
75.54
76.90
76.22
84,12
82.56
83.34
1.
79.18
79.38
79.28
83.92
82.94
83.43
2.
80.00
80.01
80.01
84,23
83,50
83,86
3.
80.61
79.97
80,29
84.38
83.52
83,95
4.
77.39
78.97
78,18
83.71
83.31
83,51
5.
80.37
80.57
80.47
84.33
83.35
83.84
tion patterns effectively capture more meaningful
6. 81.31
80.36
80.83
84.54
83.63
84,08
7.
77.69
78.58
78.14
84.54
82.72
83.63
span interactions than fully-connected span-level attention without restrictions. Our method also outperforms token-level attention with additional
8. 79.65
79.67
79.66
84.21
83,20
83,70
9. 79.54
79.90
79.72
84.25
83,60
83,93
10.
79.49
79.09
79.29
83,83
83.17
83,50
11.
80.23
80.07
80.15
84.57
83,54
84,05
layers, suggesting that the improvement perfor-
12.
79.49
80.23
79.86
84.32
83,49
83,90
mance is not merely due to having more parame- ters.
13.
79.14
79.11
79.12
84.58
83,70
84.14
14.
79.66
80L38
80.02
84.97
83.56
84,26
15.
78.86
79.80
79.33
84.36
83.50
83,93
For specific tasks, the optimal attention patterns vary. For tasks that place emphasis struc- tures, such CTL, CTD and detection task MED, attention patterns inspired by structural span in- teractions (Containment, Adjacency) show better performance. The same applics to pattern combi- nations involving them. This makes sense because grammatical structures are closely related to the structural span interactions within a sentence. For tasks that prioritize textual content, such REF, the Alk-Tooen attention pattern performs better due its attention the entire input text. In SRC, we speculate that the Inside-Token pattern helps us focus specifically on the prefixes or suffixes gener- ated by tokenization, thus improving performance related to semantic roles. In NEL, a combination of the Containment and All-Token patterns strikes a balance between structure and semantics, leading good performance.
Table 2: Averaged Fl scores for nested NER with base- line and different pattern combinations. We use the same index as Tab. in the first column represent the same model.
observed with span-level attention compared max pooling when freezing BERT. Fine-tuning BERT leads further enhancements in overall per- formance. We speculate that combining span-level attention with stronger pretrained language models and carefully-designed decoders will yield even bet- ter results. Specifically, the combination of Inside- Token and Containment/ Adjaceennt////ecccynne performs well when using frozen BERT, which aligns with our observations from probing tasks conducted under similar settings. When BERT is fine-tuned, token representations capture more comprehensive con- textual information, allowing the All-Token pattern to be included in the optimal combination. Table 9 in Appendix B also shows that the improvements brought about by our method are consistent.
In general, as shown in Table 8 Appendix B, our method consistently outperforms the base- line models in all 6 tasks. Moreover, the best per- forming attention pattern is the combination of the mnside-Token and Containment patterns. This pat- tern combination, due to its consideration of both semantics and structure, is reliable choice across different tasks.
3.4 SpanBERT backhone
We also conduct experiments on the REF and SRC tasks with SpanBERT being used as the backbone to analyse the generalizability of our proposed method. As Table 3 shows, span-level attention still brings performance gain after changing the backbone to SpanBERT, no matter fine-tuned or not. It further demonstrates the generalizability of our method. Note that fine-tuned SpanBERT with max pooling is widely favored choice for span- related tasks, and our results show that applying span-level attention to this backbone can still bring performance improvement. More detailed results can be found in Table 10 Appendix B.
3.3
Nested NER results
We conduct nested named entity recognition (nested NER) on the 3CE2004 and ACE20055 datasets (Doddington al., 2004).
As Table 2 shows, significant improvements are
periments 24GB NVIDIA TITAN RTX due its (n,)) complexity. do experiments on 48GB NVIDIA A40 instead.
"https://catalle www.ddd...n.ccoommmmmmmmm www.u..eeddeee wtt..edd/LDD20055520000 "https://catalog: 1dc. upenn. www.uuppee.eenceee2ee wwwpeeC2eeeTLL0000

---

Encoders&Datasse Datasets
posed method and attempt to derive further obser- vations from the experiments. However, there is a lack of solid theoretical explanations and insights for these observations.
SpanBERT-Irozen
ComnBERTTfnntuuu
REF
SRC
Avg.
REF
SRC
Avg.
95.62
92.88
94.25
96.52
93.66
95.09
1.
95.90
93.40
94.65
96.65
93.77
95.21
2.
95.83
93.39
94.61
96.67
93.86
95.27
Moreover, It can be time-consuming to try dif- ferent pattern combinations and pick the optimal tasks. To enhance ef- ficiency, possible approach is propose automated attention pattern combiner based on re- inforcement learning. which can serve as an impor- tant component of the entire model.
3.
95.73
93.43
94.58
96.52
93.81
95.17
4.
95.93
93.22
94.58
96.73
93.77
95.25
5.
95.86
93.45
94.66
96.73
93.78
95.26
one when encountering
6.
95.87
93.51
94.69
96.71
93.87
95.29
7.
96.01
93.12
94.57
96.62
93.80
95.21
8.
95.62
93.47
94.55
96.59
93.84
95.22
9.
95,94
93.25
94.60
96.83
93,79
95.31
10. 95.92
93.19
94.56
96.69
93.81
95.25
11.
95.97
93.41
94.69
96.60
93.85
95.23
12.
96,00
93.28
94.64
96.74
93.78
95.26
13.
95.93
93.21
94.57
96.80
93.87
95.34
Acknowledgements
14.
95.78
93.16
94.47
96.80
93.84
95.32
15.
95.87
93.25
94.56
96.59
93.75
95.17
This work was supported by the National Natural Science Foundation of China (61976139).
Table 3: Averaged Fl scores for REF and SRC with baseline and different pattern combinations when Span- BERT used as the backbone. We the same index as Tab. 1 in the first column to represent the same model.
References
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Nat- ural Language Processing, pages 670-680, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Attention Layer
Avg. F1 improvement
# layer 2 # layer
6.24
6.30
# layer =
5.98
Table 4: Analysis on span-level attention layer.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
3.5 Analysis
We conduct an analysis on the effect of the number of span-level attention layers. We select three tasks, REF, SRC and CTL,, to cover both semantic and structural tasks. To compare the results, we calcu- late the overall improvements of different attention patterns compared to the max pooling baseline and average them across the three tasks. As Table 4 shows, stacking 4 layers slightly outperforms the other two options.
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 04), Lisbon, Portugal. European Language Resources As- sociation (ELRA).
4 Conclusion
We propose to use span-level attention to improve span representations. In order reduce the O(r²) complexity and encourage more meaningful span interactions, we incorporate different attention pat- terns to limit the scope of spans that a particu- lar span can attend to. Experiments on various tasks validate the efficiency and effectiveness of our method.
Nupoor Gandhi, Anjalie Field, and Yulia Tsvetkov. 2021. Improving span representation for domain- adapted coreference resolution. In Proceedings of the Fourth Workshop Computational Models of Reference, Anaphora and Coreference, pages 121- 131, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsu- ruoka, and Richard Socher. 2017. A joint many-task model: Growing neural network for multiple NLP tasks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
Limitations
We conduct an empirical study with extensive ex- periments to validate the effectiveness of our pro-

---

pages 1923-1933, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Shubham Toshniwal. Haoyue Shi. Bowen Shi. Lingyu Gao, Karen Livescu, and Kevin Gimpel. 2020. A cross-task analysis of text span representations. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 166-176, Online. Associa- tion for Computational Linguistics.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multi- lingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Länguistics, pages 3499-3505, Florence, Italy, Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, 2017. Attention is all you need. Advances in neural information processing systems, 30.
Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2676-2686 Melbourne, Australia. Association for Computational Linguistics.
Peiyi Wang, Runxin Xu, Tianyu Liu, Qingyu Zhou, Yunbo Cao, Baobao Chang, and Zhifang Sui. 2022.
An enhanced span-based decomposition method for few-shot sequence labeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5012-5024, Seat- tle, United States. Association for Computational Linguistics.
Kenton Lec, Luheng Hc, Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Zheng Yuan, Chuanqi Tan, Songfang Huang, and Fei Huang. 2022. Fusing heterogeneous factors with triaffine mechanism for nested named entity recog- nition. In Findings of the Association for Compu- lational Linguistics: ACL 2022, pages 3174-3186, Dublin, Ireland. Association for Computational Lin- guistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to- to- fine inference. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Shori Pupers), pages 687-692, New Orleans, Louisiana. Association for Computational Linguistics.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 50-61, Online. Association for Computational Linguistics.
Tianyu Liu, Yuchen Jiang, Ryan Cotterell, and Mrin- maya Sachan. 2022. A structured span selector. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 6629-2641, Seattle, United States, Association for Computational Linguistics.
Hao Zhou, Gongshen Liu, and Kewei Tu. 2022. Improv- ing constituent representation with hypertree neural networks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages (692-1692, Seattle, United States. Association for Computational Linguistics.
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role la- beling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages (630-1642, Brussels, Belgium. Association for Computational Linguisties.
Enwei Zhu, Yiyang Liu, and Jinpeng Li. 2022. Deep span representations for named entity recognition. arXiv preprint 1200:2221.001182.
Yongliang Shen, Xinyin Ma. Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021. Locate and la- bel: A two-stage identifier for nested named entity recognition. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers). pages 2782-2777777 2794, Online. Association for Computa- tional Linguistics.
A Implementation details
We provide dataset statistics and summary in Table 5 and 6. We filter out sentences with length exceeding 40 in probing task datasets and 100 in nested NER datasets. Moreover, we evaluate the model development set every 500 steps while training. If no improvement observed in the previous 5 evaluations, the learning rate is reduced by a factor of 2. The gradient accumulation step is set to 8 for SRC, CTL, CTD and 4 for NEL, REF, MED and nested NER.
Mitchell Stem, Jacob Andreas, and Dan Klein. 2017. minimal span-basee based neural constituency parser. In Proceedings of the 55th Annual Meeting of the A Association for Computational Linguistics ( (Volume Long Papers), pages 818-827, Vancouver, Canada. Association for Computational Linguistics.

---

Task
[C]
Plnsanccs((Prainnaal//eee
NEL
18
103K/ 16K 10K
REF
2
161K / 19K 21K
SRC
62
510K / 52K
CTL
30
1.6M/ 213K/ / 158K
MED
2
718K /66K 90K
CTD
2
2.6M 354K 259K
NER(ACE04)
7
22K / /3K 3K
NER(ACE05)
7
24K / 3K/ / 3K
Table 5: Dataset statistics.
Architecture baraaeeassseters
Span representation dimension
256
Span-level attention head
4
Span-level attention layer
4
Span-level attention FFN dimension
1024
Span-level attention dropout
0.1
Span-level attention layernorm eps
le-5
Classifier hidden dimension
256
Classifier dropout
0.2
Classifier layernorm eps
le-5
Training-related hyper-parametert
Training epoch
20
Batch size
16
BERT learning rate
5e-5
Span-level attention lcarning rate
2c-4
Other learning rate
5e-4
Optimizer
Adam
Table 6: Summary of hyper-parameters.
B Detailed results
We provide detailed experiment results in this sec- tion. Table 7, 8, 9 shows averaged Fl scores along with standard deviations in pilot experiments, prob- ing tasks and nested NER. Table 10 shows REF and SRC results when SpanBERT is used backbone.

---

NEL
REF
SRC
CTL
MED
CTD
Avg.
pooling
95.61+0.009 05.58+0.09
0.09
98.00
97.98+0.05
98.19+0..02
9.007-0.02
average pooling
95.47:10.06
95.21=:0.08
92.00=0004
95.84=0.12
97.76=0..01
96.75:0.14
95.50==0.44
attention pooling
95.78+0.09 95.78+0.005
92.86=0.10
97.34+0.07
97.87+0.01
97.37+0.11
96.17+0.01
endpoint
95.56:10.05
95.30=10.09 92.86=0.02
97.77=0.02
98, 08.09:00.11
98.24:1.011
96.30=:0.02
diff-sum
95.47++0.09 95.39++0.05 92.81=0.06 97.76=0.02
98. 0.000 98.24+0003
96.29+0.02
Table 7: Detailed results for pilot experiments.
Pattern
Task
(a) (b) (c) (d)
NEL
REF
SRC
CTL
MED
CTD
Avg.
Max pooling
95.61-014
+0.58+00.09 92.85-+0.09 98.00+0.001
97.98+0..0
98. 19-0022
96.37+0.02
11
Best pooling
95.78=0.09 95.78:10.05
92.86-1-0110 98.00+4.001
98. 10:00.01
98.24:0.01
96.46:10.01
i1
Token-level
95.73-0.06 95.76++0.05
93.32+00.2 98.34+0.02
98.04+0.02
98.61+0.02
96.63+0.02
Iv
Fully- collectddde
95.77=0.15
95,63-10.03
93.11=00.02
98.36+00.01
98. 12:40.04 98.69+0000
96.61
X
95.57-0.11
95.80-+0.09
93.62++0.02
98.39++0.01
98. 10++0.022 98.53++0.01
96.67+0.02
2.
x
V
95.85=0.02 95.93:10.01 93,46:10.03
98.45:10.03
98. 19:00001
98.79:10.02
96.78:10.01
x
X
95.79-0.06 95.63+00066 93.46+00.03 98.47+0.022
98. 98.24+0.02
98.79+0.03 96.73+00.02
X
X
95.85=0.03 96.06:00.02 93.31=00004 98.35
98.05:+0.04
98 62:1:00:00
96.71110.01
5.
V
95.94-0.12 96.02-+0.1 12 93.52+0.01 98.45+0.055
98.23+00.02
98.83+0000
9..83++0002
X
V
93.78=0.07 95.78-10.15 13 93.50.10.04 98.5011.01
98.28110.04
98.83:10.01
96. 0..04
X
95.73_0.13 90.08 93.33110.04 98.3610..33
98.02.10.03
98.5610.02
96.66110.02
V
93.76-0.09 95.92.10.10 93,50110.33 98.47.10.01
19110.0003
98.79110.03
96.77.0..33
V
96.08 0.09 95.77-10.02 93.35110.01 98.43110.01
98. 14.10.004
98.77-10.03
96. 7..002
10.
X
95.95
03
93,31+00007
98.44+0.02
28.20+0.04
98.80+0.01
96.75+0.04
11.
V
95.85-
15.022
93.52-10.04
98.48.10.01
98.22:10.02
98.81110.04
76.76:10.03
12.
V
95.87-0.06 05.89500.05 93.36++0.05
98.44+0.01
18.00+0.01
98.80+0.01
96.76+00001
13.
X
95.79=0.08 95.73=t0.12 93.29-+0.07 98.45++0.01
98.80=00.03 28.25+0.00
96.72+t0.03
14.
V
95.80-0.03 95.81+0000 93.33+00009
98.41+0011
98.24+0.01
98.77++0011
96.73+0.03
15.
V
95.82
05.80:10.12 93.38:10.08 98.43:10.02 98.00:10.04 98.76110.01
96.73:10.04
Table 8: Detailed results for probing tasks.
Encoders& Datasets
Pattern
BERT-frozen
BERT-finetune
(a) (b) (c) (d)
ACE04
ACE05
Avg.
ACE04
ACE05
Avg.
Max pooling
0.44 76.90+
76.22+
8.012+0.09
82.56++0.12 83.34
1.
/
X
X
x
09 0.33 79.28
83.92:1.113
82.94+1.022 83.43+00.07
2.
X
/
X
x
80.00+0.228 80.01=0.30 80.0110.11
84.23:1.111 83.5010.12 83.86:10.10
3.
X
/
x
80.61+0.27 79.97=0.11 80.29+0.10
84.38+0.20 83.52++0266 83.95++0.10
4.
77.39+0.24
78.97=0.30 78.1810.16
83.71=0.155
83.31:1.30 83.51
80.37+0.05 80.57:0.22 80.47:0.14
84.33=0.16 83.35:10.26 83.844:.211
6.
0.17 80.36:10.32 80.83:10.23
84.54:0.28
83.63:10.38 84.08.10.33
7.
0.26 78.58+0008 78.1440.18
84.54+0.20 82.72++0.24 83.63++0.18
8.
X
0..25 79.67=1.007 79.66:10.13
84.21:10.15 83.20:10.30 83.70-10.19
9 X
X
0.25 79.90-1..2 79.72+0021
84.25+0.133 83.60+10.33 83.93++0.21
10. X
J
79.3 79.09=0.20 19.2940.11
83.8310.28
83.1740.32 83.5010..28
11. / /
V
X
0033 80.07=0.29 80.1510.22
84.57=0.35 83.54:10.09 84.05..0.13
12. /
79.4910.16 80.23-1.23 79.86.10.09
84.32.1.14 83.49.0.34 83.90.10.19
13. /
79.1410.31
79.11=1.26 79.12110.0
84.5810.06
83.70.0.244 84.1440.112
14.
0.
80.3 10.02++0.10
8419
83.56++0.25 0..06
15.
/
7..8 79.80 79.33+10.05
84.36=0.04 83.50+0..1
83.9330..16
Table 9: Detailed results for nested NER.

---

Encoderss&Datseess
Pattern
SpanBERT-frozen
SpanBERT-linetune SRC
(a) (b) (c) (d)
REF
SRC
Avg.
REF
Avg.
Max pooling
0.06
92.8 94.25+00.06
96.52+0021
93.66++0.00
95.09++0.11
1.
/
95.90=10.02 93.40=t0.09 94.65++0.05
96.65+0.005 93.77+00.3 95.21++0.04
2.
X
X
x
95.83=0.05 93.39-1.110 10 94.61=0.07
96.67=10.09 93.86:10.01 95.27=0.005
3.
X
X
/
x
95.73+00.01 93.43-10.04 94.58++0.02
96.52-1.18 93.81+0.01 95.1740.099
4.
X
X
95.93=0.077 93.22-10.03
94.58:10.05
96.73:10.16 93.77=0.02 95. 95.25:10.09
5.
V
95.86+0.05
93.45++0.01
94.66+0.02
96.73++0.77
93.78++0.01 9..26++0.04
6.
95.87=10.33
0.01
94.69:0.02
96.71=10.4 03.1 95.2
V
96.01=0.115
13.12=:0.05
94.57:0.10
96.62=0.12
93.80+1009 95.21
95.62+0.05 93.47=0.01
94.55:10033
96.59=0.30
93.8410.01 95.22:10.15
9.
0.09 93.25-10.07 94.60:10.07
96.83:1.11 93.7940.08 95.31
10. X
0.08 93.19-10033 94.56.10.06
96.69-1.233
93.81+00.55 95.25:10.14
11.
X
95. 93.41=0.13 94.69-10.14
96.60=0.17
93.8510.01 95.233:0.09
12.
X
96.00 0.09 93.28:10.02 94.64:10.06
96.74+10.20
93.78++0.04 95.26:+0.12
13.
V
V
95.9 93.21=0.06 94.57+0.08
96.80=0.08 93.87-1002 95.34
14.
X
/
9.09 93.16=00.08 94.47+-0.09
96.80=0.10 10 93.84+0000 95.32+0005
15.
900 93.25-10.04 94.56+10.07
96.5 93.75 95.17+0.022
Table 10: Detailed results for REF and SRC when SpanBERT is used as backbone.


Reviews:
[{"review_id": "EsisxASjt6", "reviewer": "Reviewer_iZpj", "summary": "", "questions": "A. The mathematical formula of span representations should be provided in the paper even if plain transformers are used.\n\nB. How do the pruning heuristics proposed in the paper perform compared with the simple span pruning method by Lee et al., 2017? I.e., scoring all the spans and taking the top-k, where k is $O(n)$.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reviews and suggestions, and many thanks for pointing out the missing references. We will add them.\n\n## Reasons To Reject\n\n>I'm not sure how many meaningful spans (i.e., spans that are grammatical constituents, see reference 2) can be recalled by the proposed heuristics. It seems the majority of the spans recalled by heuristics (b) and (c) are not syntactic constituents. More discussions on this are needed. I.e., why is it helpful to consider fragment spans that are not syntactic constituents?\n\nFirst of all, we would like to argue that there is no consensus that distituents (i.e., spans that are not constituents) cannot provide useful information. For example, the model may learn distituent representations and propagate them via span-level self-attention to facilitate distinction between distituents vs. constituents.\n\nSecond, there is also no guarantee that grammar-based approaches such as [2] do not output distituents. In fact, since they only output $2n-1$ spans from the predicted optimal parse tree, their recall of constituents can be low if the predicted parse does not align well with the ground-truth parse tree. Our approach, on the other hand, considers a much larger pool of spans and hence is very likely to have a higher recall of constituents. Span-level self-attention is then used to automatically identify useful spans (such as constituents) within the pool. Moreover, our heuristics prohibit crossing spans and thus make it less likely that a constituent span attends to a distituent span.\n\n## Questions For The Authors\n\n### Question A\n\n>The mathematical formula of span representations should be provided in the paper even if plain transformers are used.\n\nThanks for the suggestion. We do not provide detailed formulas because of limited space of a short paper, but we will add them if the paper is accepted and an extra page is given.\n\n### Question B\n\n>How do the pruning heuristics proposed in the paper perform compared with the simple span pruning method by Lee et al., 2017? I.e., scoring all the spans and taking the top-k, where k is $O(n)$\n\nCompared to the pruning method proposed by Lee et al. 2017, our method is more straightforward as it doesn't need an external scoring function. Additionally, our method does not involve discrete pruning decisions and hence circumvents the challenge of backpropagation through hard decisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are extensive and solid.\n2. The proposed heuristics are simple and useful.", "reasons_to_reject": "I’m not sure how many meaningful spans (i.e., spans that are grammatical constituents, see reference 2) can be recalled by the proposed heuristics. It seems the majority of the spans recalled by heuristics (b) and (c) are not syntactic constituents. More discussions on this are needed. I.e., why is it helpful to consider fragment spans that are not syntactic constituents?", "questions_to_authors": "A. The mathematical formula of span representations should be provided in the paper even if plain transformers are used.\n\nB. How do the pruning heuristics proposed in the paper perform compared with the simple span pruning method by Lee et al., 2017? I.e., scoring all the spans and taking the top-k, where k is $O(n)$.", "paper_topic_and_main_contributions": "This paper studies the effect of 4 different span-level attention pruning schemes on downstream span classification tasks. This paper proposes 4 heuristics for span pruning to reduce the $O(n^4)$ complexity of span-level attention to $O(n^3)$. Extensive experiments are conducted on a variety of span classification tasks to show the efficacy of the proposed heuristics in each task. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Improving Span Representation for Domain-adapted Coreference Resolution https://aclanthology.org/2021.crac-1.13.pdf\n2. A Structured Span Selector https://aclanthology.org/2022.naacl-main.189/\n"}, "extra_text": {}}, {"review_id": "JsyKjkcuCr", "reviewer": "Reviewer_T2H8", "summary": "", "questions": "1. It seems that the “inside-token” pattern is a subset of the “all-token” pattern. Then what is the difference between using only “all-token” and using both of them? If they are equivalent, then item 4 and 7 in Table 1 should be the same.\n\n2. Why only REF, SRC and CTL tasks are chosen for layer num analysis in P3.4, rather than including all 6 tasks in Table 1?\n\n3. I think the exact meaning of rel() formula in P2 Method should be clarified. Moreover, the meaning of figure 1 may need extra explanation. For first sight, it’s hard to understand how spans are indicated in this diagram.\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reviews and suggestions, and many thanks for pointing out the writing typos. We will fix them.\n\n## Reasons To Reject\n\n### Reason 1\n\n>This paper shows little novelty, as it is only a marginal improvement of the conventional attention mechanism based on some span-level inductive bias.\n\nThe novelty of our work lies in both the task and the method.\n\n**Task:** The majority of current research is centered around improving representations for fine-grained tokens or coarse-grained sentences and documents. However, spans, which are equally important components of natural language, are often overlooked. We attempt to enhance the quality of span representations by doing span-level attention and capturing span-level interactions, which is currently under-explored.\n\n**Method:** Compared with previous work of span representation, our work has the following novelties: (i) it does not rely on external information such as labels and parse trees. (ii) It explicitly models span interactions via span-level attention. (iii) We design four span-level attention patterns that ensure both effectiveness and efficiency.\n\n### Reason 2\n\n>The performance improvement over simple baselines like max-pooling is also marginal for probing tasks.\n\nProbing tasks are relatively simple, which is why simple baselines like max-pooling can yield impressive results. When facing harder tasks such as nested NER, the performance improvement of our method over simple baselines becomes more pronounced. We will consider conducting more experiments on more complex span prediction tasks such as constituency parsing, relation extraction, etc, where the improvements are expected to be more significant.\n\n### Reason 3\n\n>Although 4 patterns are proposed in this paper, it seems hard to find a unified solution or guideline for how to combine them in different tasks. If we have to try all possible combinations every time, the practicality of this method would be significantly reduced.\n\nIt is natural that the performance of different attention patterns varies on different tasks and thus a unified solution might not be possible.  In our paper, we have discussed principles to choose attention patterns for different tasks.\n\n* Lines 191 to 209: We categorize tasks into semantic and structural types, analyzing the rationales behind the corresponding optimal attention patterns. For tasks that place more emphasis on structures, attention patterns inspired by structural span interactions (Containment, Adjacency) show better performance. For tasks that prioritize textual content, the Inside-Token and All-Token attention patterns perform better due to their attention to the origin input text. This observation can serve as a guidence for combining attention patterns when encountering tasks of different types.\n* Lines 212 to 217: We suggest a reliable choice of attention pattern combination: Inside-Token and Containment.\n* Lines 233 to 238: We provide additional advice that All-Token should be included in combination when using a stronger encoder backbone.\n\n### Reason 4 and 5\n\n>4. The authors validate the effectiveness of the proposed span-level attention only on the backbone of BERT. The paper lacks experiments on other encoder backbones to further demonstrate the generality of the proposed method.\n>5. The authors do not compare their methods with other state-of-the-art methods for span-related tasks, such as SpanBERT, thus lacking some credibthe ility.\n\nBelow we provide the results of experiments when utilizing SpanBERT as the backbone. Due to limited rebuttal time, we only conduct experiments on the REF and SRC tasks. The results of frozen SpanBERT are presented in the first table, while those of fine-tuned SpanBERT are presented in the second table. We use the same index as Table 1 in our paper in column 1 and 4.  As we can see, our proposed method still brings performance gain after changing the backbone to SpanBERT, no matter fine-tuned or not. It further demonstrates the generality of our method (in response to reason 4).\n\nMoreover, Fine-tuned SpanBERT with max pooling is a widely favored choice for span-related tasks, and our results show that applying span-level attention on this backbone can still bring performance improvement, which means our proposed method outperforms \"other state-of-the-art methods\" (in response to reason 5).\n\n|Frozen SpanBERT|REF|SRC|    |REF|SRC|\n|:----|:----|:----|:----|:----|:----|\n| Max pooling |95.62|92.88|  8 |95.62|93.47|\n|           1 |95.90|93.40|  9 |95.94|93.25|\n|           2 |95.83|93.39| 10 |95.92|93.19|\n|           3 |95.73|93.43| 11 |95.97|93.41|\n|           4 |95.93|93.22| 12 |96.00|93.28|\n|           5 |95.86|93.45| 13 |95.93|93.21|\n|           6 |95.87|**93.51**| 14 |95.78|93.16|\n|           7 |**96.01**|93.12| 15 |95.87|93.25|\n\n|Fine-tuned SpanBERT|REF|SRC|    |REF|SRC|\n|:----|:----|:----|:----|:----|:----|\n| Max pooling |96.52|93.66|  8 |96.59|93.84|\n|           1 |96.65|93.77|  9 |**96.83**|93.79|\n|           2 |96.67|93.86| 10 |96.69|93.81|\n|           3 |96.52|93.81| 11 |96.60|93.85|\n|           4 |96.73|93.77| 12 |96.74|93.78|\n|           5 |96.73|93.78| 13 |96.80|**93.87**|\n|           6 |96.71|**93.87**| 14 |96.80|93.84|\n|           7 |96.62|93.80| 15 |96.59|93.75|\n\n### Reason 6\n\n>The writing can be improved. There are some typos and unclear descriptions. Please refer to comments for detail.\n\nThanks for pointing them out. We will fix them.\n\n## Questions For The Authors\n\n### Question 1\n\n>It seems that the “inside-token” pattern is a subset of the “all-token” pattern. Then what is the difference between using only “all-token” and using both of them? If they are equivalent, then item 4 and 7 in Table 1 should be the same.\n\nWhen Inside-Token and All-Token are combined, tokens inside a span is attended to twice by the span, instead of once when only All-Token is used.\n\n### Question 2\n\n>Why only REF, SRC and CTL tasks are chosen for layer num analysis in P3.4, rather than including all 6 tasks in Table 1?\n\nDue to limitations of computational resources, we only choose these three tasks for layer num analysis, which cover both semantic and sturctural tasks, as stated in Lines 242 to 243. The other three tasks are named entity labeling (NEL), mention detection (MED) and constituent detection (CTD). MED is a complement for coreference arc prediction (REF) and shares similar task properities, analogous to how CTD complements constituent labeling (CTL). Therefore, these two tasks are not selected. NEL is not chosen due to its higher performance variance.\n\n### Question 3\n\n>I think the exact meaning of rel() formula in P2 Method should be clarified. Moreover, the meaning of figure 1 may need extra explanation. For first sight, it’s hard to understand how spans are indicated in this diagram.\n\nThanks for the suggestions. We will incorporate additional explanations for formulas and figures if the paper is accepted and an extra page is given.\n\n## Typos Grammar Style And Presentation Improvements\n\n### Improvement 1\n\n>1. The authors do not explicitly state the intuitions of the pattern setting. The authors do not explore other possible patterns or combinations, such as overlapping spans, non-contiguous spans, etc.\n\nThanks for the suggention. We have shown the intuitive advantages of each pattern in Lines 116 to 137, which is also part of our intuitions. We will modify this paragraph to present our intuitions more explicitly. We deliberately avoid patterns such as overlapping spans and non-contiguous spans, because considering these patterns will increase the time complexity to at least $O(n^4)$, which is not practical. We only consider patterns with cubic time complexity for the sake of efficiency.\n\n### Improvement 3-5\n\n>3. As the major goal of this paper is to improve span representation, related works such as [1] [2] conduct experiments on NLU and QA tasks, which are also span-related tasks. Experimental results extended on those tasks would be helpful.\n>4. There are some missing related works such as [1] [2], it would be better to add more relevant discussions.\n>5. Minor comments on writing:\n>(1) Line #039: lies on -> lies in\n>(2) Line #072: Comparing to -> Compared to\n\nThanks for pointing out the missing references and writing typos. We will include more tasks in subsequent updates. We do not see what reference [1] and [2] are in your review. Could you please provide them?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper is well written and easy to follow.\n\n2.\tThe proposed method shows superior performance over the baselines, especially on the nested NER tasks.\n", "reasons_to_reject": "1.\tThis paper shows little novelty, as it is only a marginal improvement of the conventional attention mechanism based on some span-level inductive bias.\n\n2.\tThe performance improvement over simple baselines like max-pooling is also marginal for probing tasks.\n\n3.\tAlthough 4 patterns are proposed in this paper, it seems hard to find a unified solution or guideline for how to combine them in different tasks. If we have to try all possible combinations every time, the practicality of this method would be significantly reduced.\n\n4. The authors validate the effectiveness of the proposed span-level attention only on the backbone of BERT. The paper lacks experiments on other encoder backbones to further demonstrate the generality of the proposed method.\n\n5. The authors do not compare their methods with other state-of-the-art methods for span-related tasks, such as SpanBERT, thus lacking some credibility.\n\n6. The writing can be improved. There are some typos and unclear descriptions. Please refer to comments for detail.\n", "questions_to_authors": "1. It seems that the “inside-token” pattern is a subset of the “all-token” pattern. Then what is the difference between using only “all-token” and using both of them? If they are equivalent, then item 4 and 7 in Table 1 should be the same.\n\n2. Why only REF, SRC and CTL tasks are chosen for layer num analysis in P3.4, rather than including all 6 tasks in Table 1?\n\n3. I think the exact meaning of rel() formula in P2 Method should be clarified. Moreover, the meaning of figure 1 may need extra explanation. For first sight, it’s hard to understand how spans are indicated in this diagram.\n", "paper_topic_and_main_contributions": "This paper proposes a new span-level attention mechanism to enhance the span representation based on a powerful pre-trained language model. It focuses on 4 patterns: inside-token, containment, adjacency and all-token when designing the span-level attention. Experimental results on several span-level tasks show the effectiveness of this method, especially for nested NER tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The authors do not explicitly state the intuitions of the pattern setting. The authors do not explore other possible patterns or combinations, such as overlapping spans, non-contiguous spans, etc. \n\n2. The authors speculate that the proposed attention patterns effectively capture more meaningful span interactions, but they do not provide detailed analysis to verify this statement. Maybe further qualitative analysis is necessary.\n\n3. As the major goal of this paper is to improve span representation, related works such as [1][2] conduct experiments on NLU and QA tasks, which are also span-related tasks. Experimental results extended on those tasks would be helpful.\n\n4. There are some missing related works such as [1][2], it would be better to add more relevant discussions.\n\n5. Minor comments on writing:\n\n(1)\tLine #039: lies on -> lies in\n\n(2)\tLine #072: Comparing to -> Compared to\n"}}, {"review_id": "QRWTPlbkyq", "reviewer": "Reviewer_Qvm2", "summary": "", "questions": "A, how about using 4 token-level encoder layers on top of BERT?\n\nB, is it possible to automatically aggregate the 4 attention patterns?\n\nthanks for the response, I am satisfied with the response for question A.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reviews and suggestions. It helps us improve the quality of our work.\n\n## Reasons To Reject\n\n### Reason A\n\n>the improvements are not large.\n\nProbing tasks are relatively simple, which is why simple baselines like max-pooling can yield impressive results. When facing harder tasks such as nested NER, the performance improvement of our method over simple baselines becomes more pronounced. We will consider conducting more experiments on more complex span prediction tasks such as constituency parsing, relation extraction, etc, where the improvements are expected to be more significant.\n\n### Reason B (also Question A in Questions For The Authors)\n\n>the method has additional costs (4 more encoder layers at span-level in addition to BERT), and adding token-level encoder layers on top of BERT may also improve the performance.\n\nWe evaluate the performance when 4 more token-level encoder layers are used. We will add it into Table 1 as a new baseline. From the table below, it can be seen that employing 4 more token-level layers does lead to performance gains, but the degree of improvement is consistently smaller compared to our method.\n\n|                        |NEL|REF|SRC| CTL   |MED|CTD|\n|:----|:----|:----|:----|:----|:----|:----|\n|           Max pooling | 95.61 | 95.58 | 92.85 | 98.00 | 97.98 | 98.19 |\n|   4 token-level layers | 95.73 | 95.76 | 93.32 | 98.34 | 98.04 | 98.61 |\n| Best attention pattern | 96.08 | 96.06 | 93.62 | 98.50 | 98.28 | 98.83 |\n\n### Reason C\n\n>the paper fails to find a general solution for related problems, using containment, adjacency, and all-token attention patterns at the same time leads to the best performance on average for NER, while using inside-token and containment attention patterns leads to the best performance on average for the other tasks.\n\nIt is natural that the performance of different attention patterns varies on different tasks and thus a general solution might not be possible.  In our paper, we have discussed principles to choose attention patterns for different tasks.\n\n* Lines 191 to 209: We categorize tasks into semantic and structural types, analyzing the rationales behind the corresponding optimal attention patterns. For tasks that place more emphasis on structures, attention patterns inspired by structural span interactions (Containment, Adjacency) show better performance. For tasks that prioritize textual content, the Inside-Token and All-Token attention patterns perform better due to their attention to the origin input text. This observation can serve as a guidence for combining attention patterns when encountering tasks of different types.\n* Lines 212 to 217: We suggest a reliable choice of attention pattern combination: Inside-Token and Containment.\n* Lines 229 to 233: We verify that the combination of Inside-Token and Containment/Adjacency performs well for nested NER when frozen BERT is used as the encoder backbone, aligning with experimental results for probing tasks under similar settings.\n* Lines 233 to 238: We provide additional advice that All-Token should be included in combination when using a stronger encoder backbone.\n\n### Reason D\n\n>the paper lacks in-depth analysis for the co-relation between the attention patterns and specific tasks.\n\nIn Lines 191 to 217, we have discussed the relations between tasks and attention patterns. Given the limited space of a short paper, we find it challenging to conduct more in-depth analysis.\n\n### Reason E\n\n>Table 3 might be wrong, the average F1 score improvements are not consistent with Table 1.\n\nTable 3 is indeed accurate and consistent with Table 1. Note that the corresponding experiments of Table 3 were conducted on three tasks: REF, SRC and CTL, as stated in Lines 241 to 242.\n\n## Questions For The Authors\n\n### Question B\n\n>is it possible to automatically aggregate the 4 attention patterns?\n\nAs we discussed in Lines 265 to 271 in the Limitations section, an automated attention pattern combiner based on reinforcement learning may help choose the optimal pattern combination when encountering new tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents 4 effective attention patterns for the computation of span representations, and experiment results show the effectiveness of the approach.", "reasons_to_reject": "A, the improvements are not large.\n\nB, the method has additional costs (4 more encoder layers at span-level in addition to BERT), and adding token-level encoder layers on top of BERT may also improve the performance.\n\nC, the paper fails to find a general solution for related problems, using containment, adjacency, and all-token attention patterns at the same time leads to the best performance on average for NER, while using inside-token and containment attention patterns leads to the best performance on average for the other tasks.\n\nD, the paper lacks in-depth analysis for the co-relation between the attention patterns and specific tasks.\n\nE, Table 3 might be wrong, the average F1 score improvements are not consistent with Table 1.\n\nthanks for the response, B and E are well addressed.", "questions_to_authors": "A, how about using 4 token-level encoder layers on top of BERT?\n\nB, is it possible to automatically aggregate the 4 attention patterns?\n\nthanks for the response, I am satisfied with the response for question A.", "paper_topic_and_main_contributions": "This paper studies how to perform attention for span representations, and designs 4 attention patterns to reduce the complexity. Experiments on a number of tasks show the effectiveness of the approach.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes and analyzes pruning techniques to efficiently calculate span-level representations. Reviewers have found the proposed approach solid, and the experiments are thorough, whereas the novelty of this paper is somewhat limited.
Decision: Accept-Findings