ScdNER: Span-Based Consistency-Aware Document-Level Named Entity Recognition
Authors: Ying Wei
Qi Li
Venue: EMNLP 2023 Main (2023)
Keywords: named entity recognition
span-based
document-level
consistency-aware

Abstract: Document-level NER approaches use global information via word-based key-value memory for accurate and consistent predictions. However, such global information on word level can introduce noise when the same word appears in different token sequences and has different labels. This work proposes a two-stage document-level NER model, ScdNER, for more accurate and consistent predictions via adaptive span-level global feature fusion. In the first stage, ScdNER trains a binary classifier to predict if a token sequence is an entity with a probability. Via a span-based key-value memory, the probabilities are further used to obtain the entity's global features with reduced impact of non-entity sequences. The second stage predicts the entity types using a gate mechanism to balance its local and global information, leading to adaptive global feature fusion. Experiments on benchmark datasets from scientific, biomedical, and general domains show the effectiveness of the proposed methods.

ScdNER: Span-Based Consistency-Aware Document-Level Named Entity Recognition
Ying Wei Iowa State University / Ames, IA 50011 Iowa State University / Ames, IA 50011 yingwei@iastate.c.
Qi
qli@iastate.edu
Abstract
token-based key-value memory, where the features of the same token are fused produce document- level encoding for token label prediction. However, this token-level feature-sharing strategy may intro- duce noise in global feature fusing, as the same token occurrences in different contexts may not carry the same labels. For instance, labels for token "type" in the entity span "breast cancer type I" and the token sequence "a unique type of cancer" are GENE-I and O, respectively (here, we call an entity span as a token sequence with an entity type). Fus- ing features of these two tokens will bring noise predicting their labels.
Document-level NER approaches use global information via word-based key-value memory for accurate and consistent predictions. How- ever, such global information on word level can introduce noise when the same word ap- pears in different token sequences and has different labels. This work proposes a two- stage document-level NER model, ScdNER, for more accurate and consistent predictions via adaptive span-level global feature fusion. In the first stage, ScdNER trains binary classifier to predict if a token sequence is an entity with a probability. Via a span-based key-value mem- ory, the probabilities the entity's global features with reduced im- pact of non-entity sequences. The second stage predicts the entity types using gate mecha- nism to balance its local and global informa- tion, leading adaptive global feature fusion. Experiments on benchmark datasets from sci- entific, biomedical, and general domains show the effectiveness of the proposed methods.
further used to obtain
To address the problem of token-level fea- ture sharing, this work proposes span-based consistency-aware DL-NER model (ScdNER) enable context feature fusing the span level. In particular, this work proposes a document-level context feature fusion technique for the same en- tity mentions. Given a document, we first extract document-level contextual encoding for each enu- merated token sequence. Based on contextual en- coding, we propose two-stage prediction frame- work. In stage 1, binary classifier is trained to predict if token sequence is entity span with a probability. Stage employs span-based key-value memory to enable context feature fus- ing of the same entity spans using probabilities from Stage to reduce the impact of non-entity spans. The entity span features are updated by fusing global information with gate to balance local and global features. Finally, the entity type is predicted for an entity span.
1 Introduction
Named entity recognition (NER) is an important task for many natural language processing appli- cations (Lample et al., 2016; Li et al., 2020; Yu et al., 2020; Yan et al., 2019). Most NER models process sentences independently, leading to risk of inconsistent prediction for the mentions of the same entity. That is. they can produce different pre- dictions for the same entity mentions different sentences depending on the contextual information for each sentence. Consideration of document-level features in entity representation learning is likely to obtain more consistent and accurate predictions since different occurrences of specific token quence within document likely have the same entity types (Krishnan and Manning. 2006). Based on this idea, some document-level NER
Related Work
DL-NER approaches use document-level contextu- alized information for more accurate and consistent predictions. Qian et al. (2019) use word sequence dependency structures model non-local and non- sequential dependency. Akbik et al. (2019) propose to pool contextualized embeddings of each unique
(DL-NER) approaches (Luo et al., 2020; Gui et al., 2021) proposed to fuse global features of tokens via

---

This
presents
fonmal
unalysii
upproximation
of
the
Formal
analysiis
Ag
Xxx
Al
Encoder
fa
fn
A;
Av
Axx
Span Classification
Key
Value
Candidate Spans
Surface
Span Fizttrees
Probubility
Build Memory
Spann
Biis
By dg
Beclijka) -
fornal analysi
widh
CSpan12
milyis
Ayp
Pia (eelitkus) 0.2
Entity Span Feature Update
CSpun17
fonnal analysi
pir B-lliik,.)
kibh
METHOD
CSpan24
fonnal analysis
Roy mastthu Rig)
B -illiii 0.9
MLP
Figure 1: Illustration of the proposed ScdNER. In this example, "formal analysis" a surface name shared by entity spans. Their features updated via fusing the other's contextual features, resulting "METHOD" prediction.
string. Luo et al. (2020) employ key-value memory to memorize word encoding, and an attention mechanism computes each token's document-level representations. Gui et al. (2021) design two-stage label refinement network via word-based key-value memory network. However, most existing works try to fuse word-level global information, which introduces noise when the same word in different entity spans has different labels. This work proposes to use a span-level key-value memory for global feature fusion.
context as follows. И hh....., hpl Eneoder([z),.... mnl). where æ; E Rdi is the word embedding for w, and h; is its contextual en- coding. We can use pre-trained language model such as BERT (Devlin et al., 2019) or LSTM model (Hochreiter and Schmidhuber, 1997) an encoder. For candidate span sj, its encoding is computed by fusing token encoding:
hs AGG((a hhott)
where AGG(.) is an aggregation function such max-pooling and average In the experi- mental studies, we max-pooling the aggrega- tion function to introduce more non-linearity (Yu et al., 2014).
3 ScdNER
Problem Definition. Given a document = Wn] [wi....,
with wi representing the ith token, we enumerate all contiguous token sequences in each sentence as candidate spans 8, [Wa. ..., Wa+k the starting to-
3.2 Stage 1: Binary Span Classification
with & (0, 1, 1., K)) where ken position in s and K is the maximum sequence length of spans. For each span, we aim to predict its label in {0, 1..... CJ, where indicates it is a non-entity and is the number of entity types. Our Model. ScdNER is two-stage DL-NER model. In the first stage, a binary span classifier is trained to predict if span is entity. The sec- ond stage
In the first stage ScdNER, we train binary clas- sifier to predict if candidate span is an entity span. When building a span-based key-value memory, this binary classifier acts as a filter to reduce the noise in global feature fusion of the same candidate spans. For a candidate span 8;, the prediction is based its encoding hg,:
P: = o ( (hs b1),
(1)
span-based key-value memory to adaptively fuse global features of spans with the same surface names for better span feature repre- sentation learning. Figure illustrates the proposed ScdNER.
where W1 and b are trainable parameters and T1 (.) is sigmoid function. Using threshold 6, candidate span s; is predicted if its probability p: > 0. By span filtering, stage outputs list of entity spans E [e1, .2..... ez). To train this binary span classifier, we create a bi- nary label ye for each span s;. If span 8; not an entity of any type, 2// = O; otherwise, y =
entity span
3.1 Document-Level Span Encoding
To encode candidate spans in the document, we first encode tokens wy with the document as

---

3.3 Stage 2: Adaptive Global Feature Fusion
where de is the width embedding of L. We use a multi-layer perceptron (MLP) to compute the pre- diction values 2; E RC for all entity types, where C' is the number of entity types:
We propose a span-based key-value memory to adaptively fuse features of the same entity spans. Span-based key-value memory. The span-based key-value memory records representations of spans (keys) and their probabilities from Stage I (values), In partieular, we create document-level memory matrix U [u1, U2,.... um], where the jth slot, ui corresponds to span 8; and stores a pair of vec- tors ( Pt). Suppose 8q1 So....... Sq. are spans under the same surface name phj. phy can cor- respond to multiple slots, corresponding to sub- array memory matrix U ph; L Ugz...... Ug-.- Entity span feature update. Based on the span-
2, W 33 WW af, + bs) + b4,
(5)
where W3, 44 b3, b4 are trainable parameters, and Ø3 an element wise activation function. During prediction, span 8; will be predicted non-entity if p; < A. Otherwise, it will be assigned to entity type C: = arg max, zx[j]- It is worth noting that the value of e, can be O, indicating that span initially predicted as an entity span in Stage 1 can ultimately be classified as non-entity.
based key-value memory, each entity span e, up- dates its features fuse global features of the same entity spans. If spans of the same surface name are identified as entities, they are likely to share the same semantic meanings. However, the assumption does not hold for non-entity spans. Thus, we use probabilities from Stage 1 to adjust the impacts of spans in global features to emphasize entity spans. Given entity span e;, its features are updated by fusing features of other spans sharing the same surface name in the memory. To this end, first extract its corresponding sub-array memory matrix U phy = [h,,, Poi ...,, (ha : P- The entity span features are updated as:
4 Experiments
We evaluate the proposed ScdNER model bench- mark NER datasets from various domains. We aim study if span-level global feature fusing im- proves span feature learning and NER results. Experimental setups. We use BioBERT (Lee et al., 2020), SciBERT (cased) (Beltagy et al., 2019), and BERT (cased) (Devlin et al., 2019) based on Huggingface's Transformers (Wolf et al., 2019) document encoder for biomedical, scien- tific, and general domain datasets, respectively. All hyper-parameters are tuned on the validation sets (See Appendix A for details). We report the aver- age over runs for ScdNER. Evaluation metries include Precision, Recall, and Fl scores.
ho =
Σ Pax higa
Σ Pax: 1=1
4.1
Experiments Biomedical Datasets
9i 02(W Waher + b2):
(3)
Dataset. We use three biomedical datasets evalu-
h', gihr, +(1 - (1-g))hh high
(4) ate ScdNER: CDR (Li et al., 2016), NCBI-disease (Doğan et al., 2014), and ChemdNER (Krallinger et al., 2015). The CDR dataset is tagged with 2 en- tity types and contains 500, 500 and 500 abstracts for train, validation, and test, respectively. The NCBI-disease dataset contains 593, 100, and 100 documents for train, validation, and test, respec- tively. It is tagged with entity type: Disease. The ChemdNER dataset, tagged with entity type. consists of 3,500, 3,500, and 3,000 abstracts for training, validation and test, respectively. More statistics can be found in the Appendix A.
where W2, b2 are trainable parameters, and (2(-) is sigmoid function. In Eq. (2), the global features of surface name phj are computed by weighted average of all span features with their probabilities as weights. Thus, the impact of non-entity spans is reduced. Eq. (3) and Eq. (4) use a gate mechanism to balance the span's 's local and global features. If all spans of a surface name are predicted as non-entity, their features are not updated.
3.4 Entity Type Prediction Head
Baseline Models. We compare ScdNER with SOTA biomedical NER models including BiLSTM- CRF (Luo al., 2018), BioBERT, ConNER (Jeong and Kang, 2022), HierNER (Luo et al., 2020), Do- cLNER (Gui et al., 2021), and SpERT-Doc. SpERT- Doc is developed based on SpERT (Eberts and Ulges, 2020) using document-level encoding.
We use entity type prediction head to predict the entity types for entity spans from stage 1. Fol- lowing previous methods (Eberts and Ulges, 2020), we concatenate the entity span encoding and span width embedding, which leads to the final encod- ing for an entity span 6 of width t: f: = [h'as di),

---

Model
CDR
NCBI
ChemdNER 89.4
Model
Level
BILSTM-CRR
84.6
CVI-GloVe
Sent-level
88.88
BiOBERT
89.1
87.7
DocL.NER- NoV-GGoe
Doc-level
88.49
ConNER
89.9
89.1
PooINER-BERT
Sent-level
89.71
DocLNER
90.7
DocLNER BERT
Doc-level
90.28
HierNER
89.5
HierNER-BE BERT
BERTE
Doc-level
90.30
SpERT-DOC
90.1
88.8
92.1
SPERT-BERT
Sent-level
90.24
ScdNER
91.2+00077
90.4+0032
93.1-0.11
SpERT-Doc-BRR BERT
Doc-level
90.34
ScdNER -ENTR-BERT
Doc-level
90.80-10.06
Table 1: Results (FI scores) on biomedical datasets.
Table 3: Results (FI scores) on OntoNotes 5.0 dataset.
Model
R
F1
DyGiE-GloVe
65,20
Model
P
R
F1
#Type INC
DyGlF+++ BERT
67.50
SpERT-Doc
89.7
90.5
90.1
80 (4.6%)
PURE-SciBERT 70.87
68.90
ScdNER
90.6
91.3
90.9
(3.3%)
SpFiR--SciBERT-SN
69.79
70.33
SpERR--ScIBERI-Doc
69.85
70.78
70.31
Table 4: Prediction inconsistency study on the CDR dataset. We report the inconsistent rate of predictions.
Sc-BERTT
69.90
SedNER-SeiBERT
71.60
71.43
7.,44+0.20
Table 2: Results on SciERC dataset from the scientific domain. We report the Precision, Recall, and FI scores.
tions with 18 entity types. It consists of 2,483, 319, and 322 documents for the training, validation, and test sets, respectively.
Main Results are summarized in Table 1. ScdNER achieves consistently better performances than pre- vious SOTA models on all datasets. ScdNER outperforms previous best models by margins of 1. 11, 1.3%, and 1.0% CDR, NCBI-disease and ChemdNER, respectively, which demonstrates the effectiveness of conducting span-level global fea- ture fusion span feature learning.
Baseline Models. We compare with sentence-level NER models: CVT (Clark et al., 2018) and Pool- NER (Akbik et al., 2019), and document-level NER models: HierNER, DocLNER, and SpERT-Doc. Main Results. We summarize the comparison sults in Table 3. The proposed ScdNER outper- forms the existing best model by a margin of 0.5% in terms of F1 score. Although entity repeating in the general domain is not as severe as in the biomedical domain, our method can still boost per- formance by better fusing global information. Error analysis of ScdNER is provided in Section 4.7.
4.2 Experiments on Scientific Datasets
Dataset. We evaluate ScdNER on SciERC (Luan et al.. 2018) dataset, scientific domain dataset constructed from abstracts of 500 Al papers, which are split into 350, 50, and 100 documents for the training, validation and test sets, respectively. The dataset is tagged with scientific entity types. Baseline Models. We compare with previous SOTA span-based joint entity/relation extraction models including DyGIE (Luan et al., 2019), Gy- GIE++ (Wadden et al., 2019), PURE (Zhong and Chen, 2021), SpERT, and PI.-Marker (Ye al., 2022). For fair comparisons, component and loss as SpERT during training. Main Results are summarized in Table 2. From the results, the proposed ScdNER achieves con- sistently better performances than previous SOTA models on all metrics. In particular, ScdNER sig- nificantly outperforms previous models by at least 1.11% in terms of FI.
4.4 Prediction Consistency Study
We study ScdNER can address prediction incon- sistency problem for entities of the same surface same on the CDR dataset. We use SpERT-Doc as baseline, which uses document-level context with- out considering label consistency. The results are summarized in Table 4. Compared to the baseline model, ScdNER effectively reduce the rate of prediction inconsistency for entities of the same surface name, which shows the advantage of con- sidering label consistency.
use the same RE
4.5 Ablation Study of ScdNER
We conduct ablation experiments on the NCBI- discase and SciERC datasets study the contribu- tion of each component in ScdNER. Based on ScdNER model,
remove key-value memory (stage 2) and entity filtering (stage 1) at time. We also remove both to test the overall contribu- tion of the adaptive global feature fusion module.
4.3 Experiments on General Domain Datasets
Dataset. We use the English portion of OntoNotes 5.0 dataset with gtad-sta named entity annota-

---

Model
NCBI
SciERC
Dataset
#Entity
#Kntity REP
#Type INC
ScdNER
90.39
71.44
CDR
4,605
1,854 (40.3%)
(0.0%)
(+)
Key-value memory
89.28
70.36
NCBI Disease
2,982
1,019 (34.2%)
(0.0%)
(-) Entity filter
89.49
70.64
ChemdNER
15,911
5,345 (34.2%)
(0.0%)
Both
88.77
69.84
SciERC
4,959
991 (19,9%)
28 (5.6%)
OntoNotes
58,177
10,766 (18.5%)
173 (1.6%)
Table 5: Ablation study results on NCBI-disease and SciERC datasets. We report Fl (%) scores.
Table 6: Label consistency statistics on benchmark datasets.
The ablation results are summarized in Table 5. We can observe that each component makes signif- icant contribution. In particular, when removing the span-level key-value memory, the performances drop by 1.11% and 1.08% on NCBI-disease and SciERC, which shows the benefit of conducting span-level global feature fusion for better span rep- resentation learning. When removing the entity filter while keeping the key-value memory, the per- formance drops by 0.90% and 0.80% on NCBI- disease and SciERC, which means the non-entity spans can introduce significant noise in adaptively fused global features, and it necessary to reduce the impact of non-entity spans when fusing global information. Removing both leads to 1.52% and 1.60% performance drops on NCBI-disease and SciERC, respectively, demonstrating the overall contribution of the proposed methods.
lapping spans; 4) Wrong Boundary Wrong Type (WW): The model predicts different entity type as that of an entity span in the ground truth with overlapping spans; 5) Correct Boundary Wrong Type (CW): The model predicts different entity type for an entity span in the ground truth.
The errors distribution of the ScdNER is shown in Table 8. Compared to the baseline model, ScdNER can significantly reduce the errors in "Ex- tra" category. The proposed binary span classifier can greatly help with such kind of errors. Most errors of the ScdNER involve incorrect boundaries and incorrect entity type. The errors of incorrect boundaries come from these categories: Missing, Extra, Wrong Boundary Correct Type, and Wrong Boundary Wrong Type. The wrong entity type errors include Wrong Boundary Wrong Type and Correct Boundary Wrong Type. From the error distribution, the errors of incorrect boundaries are major and need to be addressed in the future. In particular, "Missing" and "Extra" contribute the most to the errors on both datasets. Table 10 in the appendix illustrates some examples.
4.6 Label Consistency on Datasets
The proposed methods are based on the assump- tion that the entities sharing the same surface name are highly likely be assigned the same entity types. We examine this assumption on benchmark NER datasets from various domains. The statis- tics are summarized Table 6. We calculate the number and rate of entities repeated in the docu- ment (shown at "#Entity REP" column) and the number and rate of entities with label inconsis- tency (shown "#Type INC" column). We can ob- serve high entity repeat rates and low type inconsis- tency rates, especially on biomedical NER datasets, which strongly supports the proposed methods.
Conclusion
This work proposes a two-stage DL-NER model, ScdNER, to enable more accurate and consistent NER prediction via global feature fusion of the same entity spans. In stage 1, ScdNER trains a binary span classifier to classify span with a prob- ability. Stage 2 uses based key-value mem- ory to adaptively fuse global features of the same entity spans. The probabilities in stage I are reused to reduce the impacts of non-entity spans. Empiri- cal results on document-level NER datasets from various domains demonstrate the effectiveness.
4.7 Error Analysis
To better understand the bottleneck of ScdNER, we analyze the errors that ScdNER makes on the Sci- ERC and OntoNotes datasets. We identify five ma- jor types of errors as follows: 1) Missing (M): The model predicts no entity type for an entity span in the ground truth; 2) Extra (E): The model predicts an entity type for a span that does not exist in the ground truth; 3) Wrong Boundary Correct Type (WC): The model predicts the same entity type as that of an entity span in the ground truth with over-
Acknowledgements
The work is supported in part by NIFA grant no. 2022-67015-36217 from the USDA National Insti- tute of Food and Agriculture, and NSF 2237831 from the National Science Foundation.

---

Limitations
Minbyul Jeong and Jaewoo Kang. 2022. Enhancing label consistency on document-level named entity recognition, arXiv preprint arXiv: 2210. 12949.
In this work, we use SciERC (Luan et al., 2018) one of the datasets to evaluate the proposed model. The kappa score for annotating entities in SciERC only 76.9%, which indicates that only 35%-63% data are reliable (McHugh, 2012). Low- quality dataset can introduce noise and undermine the model performance. There is a chance that the performance gain from ScdNER model is due to over-fitting to the noise in annotation.
Martin Krallinger, Florian Leitner, Obdulia Rabal, Miguel Vazquez. Julen Oyarzabal, and Alfonso Va- lencia. 2015. Chemdner: The drugs and chemical names extraction challenge. Journal of cheminfor- matics, 7(1):1-11.
Vijay Krishnan and Christopher D Manning. 2006. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In Pro- ceedings of the 21st international conference on com- putational linguistics and 44th annual meeting of the association for computational linguistics, pages 1121-1128.
References
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers). pages 724-728.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260-270.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim. Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234 1240.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scib- ert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing, pages 3615-3620
Jiao Li, Yueping Sun, Robin Johnson, Daniela Sci- aky, Chih-HHuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative edr task corpus: A resource for chemical disease relation extraction. Database, 2016.
Kevin Clark, ThanhTThang/ Luong, Christopher D Man- ning, and Quoe Le. 2018. supervisd sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914-1925.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. survey on deep learning for named entity recognition. IÉEE Transactions on Knowledge and Data Engineering, 34(1):50-70.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics, pages 4171 1186.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Yi I Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018 Multi-task identification of entities. relations, and coreference for scientific knowledge graph construction. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 9219-3233
Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Nebi disease corpus: resource for dis- ease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training. In ECAI 2020, pages 2006-2013 IOS Press.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3036-3046.
Tao Gui, Jiacheng Ye, Qi Zhang, Yaqian Zhou, Yeyun Gong, and Xuanjing Huang. 2021. Leveraging document-level label consistency for named entity recogmition. In Proceedings of the Twenty-Ninth In- lernational Conference International Joint Con- ferences on Artificial Intelligence, pages 3976-3982,
Ling Luo. Zhihao Yang, Pei Yang. Yin Zhang, Lei Wang, Hongfei Lin, and Jian Wang. 2018. An attention- based bilstm-crf approach to document- level chemical named entity recognition. Bioinfor- maticy, 1388))1381-1888
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 18): 1780.

---

Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hi- erarchical contextualized representation for named entity recognition. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 8441-8448.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. Graphie: A braph framework for information extraction. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 751-761.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019, Entity, relation, and event extraction with contextualized span representations. In Proceedings the 2019 Conference Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5784-5789.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.0377..
Hang Yan, Bocao Deng, Xiaonan Li. and Xipeng Qiu. 2019. Tener: adapting transformer encoder for named entity recognition. arXiv preprint
arXiv:1911.04474
Deming Ye, Yankai Lin. Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction, In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4904-4917.
Dingjun Yu, Hanli Wang, Peiqiu Chen, and Zhihua Wei. 2014. Mixed pooling for convolutional neural net- works. In Rough Sets and Knowledge Technology: 9th International Conference, RSKT 2014, Shanghai, China, October 24-26, 2014, Proceedings 9, pages 364-375. Springer.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition dependency parsing. arXiv preprint arxiv:2005.07150.
Zexuan Zhong and Danqi Chen. 2021. frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 50-61.

---

P
FT
AUC
SciERC
81.41
82.52
81.96
97.77
OntoNotes 91.32
93.82
92.55
99.71
Table 7: Evaluation of the binary classifier. We report Precision (%), Recall (%), Fl (%), and AUC scores.
A Experimental Setups
The threshold 0 0.5 used the bi- nary span classifier. We use the AdamW opti- mizer (Loshchilov and Hutter, 2018) with a linear warmup. The learning rate starts from 5e-5 and is reduced by linear decay learning rate sched- ule until O. We mostly the hyper-parameters in (Eberts and Ulges, 2020) with minor adjustments better suit each dataset. We provide more hyper- parameters of ScdNER on five benchmark NER datasets in Table 9. We train all models using a Tesla V100 GPU. For the baseline model results we produced, we used the hyper-parametecn reported in the original paper, albeit with slight fine-tuning
B Performance Study of Binary Classifier
We use a binary classifier in the proposed ScdNER model to reduce the impact of noises and conduct first-stage prediction. The errors in this binary clas- sifier may be propagated to the second stage, mak- ing critical impact on the overall performance. We conduct experiments to study its effectiveness the SciERC and OntoNotes 5.0 dataset, with re- sults summarized in Table 7. From the results, observe that the binary classifier achieves 81, 1.96% and 92.55% Fl scores on two datasets, which highly accurate. Thus, using binary classifier as an entity filter has the risk of стror propagation, but can bring more benefits than errors from the results of ablation study. Future work can include soft punishment to avoid the error propagation issue.

---

SciRRC
OntoNotes
ScdNER
SpERT
ScdNER
SpERT
Missing
188
197
350
263
Extra
142
205
473
632
Wrong Boundary Correct Type
120
103
327
322
Wrong Boundary Wrong Type
60
131
128
Correct Boundary Wrong Type
206
228
Table 8: Error distributions on SciERC and OntoNotes datasets.
Dataset
Context
Pretrained
Batch Size
Learning Rate
Training Epoch
Threshold
Max Span
LM
Size K
CDR
Dacument
BioBERT
5e-5
NCBI-disease
Document
BioBERT
5e-5
30
0.5
ChemdNER
Document
BioBERT
5e-5
30
0,5
10
SciERC
Document
ScIBERT
4
3e-5
30
0,5
10
OntoNotes
Document
BERT
5e-5
30
0.5
10
Table 9: Hyper-parameters of the named entity recognition settings.
Category
Document
M
Since we know that Jingguang Bridge is located in the CBD district, well there are many office buildings. ah, lot of them, as well as quite lot of friends who get up early in the morning go to work.
early in the morning: TIME
Taiwan, in recent years the Ministry of Economic Affairs, the National Science Council and the National Health Research Institutes have been strongly pursuing "biochip" research programs.
recent years: DATE
WC
Well, ah, there is an accident, from the time occurs until the time all traffic is cleared, it probably takes about half hour to an hour during peak periods.
about half an hour about half an hour to an hour: TIME
WW
However, southern North China and the Huang Huai area will bid farewell to the snow tonight, ah. Tomorrow during the day, snowfall in the Jiang-Hual area will stop, and there will also be less snow in Northwest China.
Tomorrow:
DATE
Tomorrow during the day: TIME
CW
Judy Miller acknowledges that at Louis Libby's request this the Dick Cheney aide she agreed to refer to him and in stories not as a senior administration official but as a former Hill aide because he had once worked on Capitol Hill.
Capitol Hill: ORG
LOC
Table 10: Case study of NER results the OntoNotes dataset. Here, green texts and blue texts indicate predictions with correct and wrong boundaries, respectively. The predicted entity types in green box and blue box indicate predictions with correct and wrong types, respectively.


Reviews:
[{"review_id": "nhwnujwuzc", "reviewer": "Reviewer_iSfy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons To Reject**\n\n*Q1. Span-based key-value memory is updated by the matching of the surface form, which does not consider the situation of entity coreference.*\n\n**Answer**\n\nThank you for the feedback on the usage of entity coreference in our span-based key-value memory system.\n\nWe agree on the importance of entity coreference. Currently, most NER benchmark datasets do not include coreference annotations to train coreference resolution models. In this work, the SciERC dataset serves as the sole data source containing coreference information. However, it's worth noting that the majority of state-of-the-art models do not leverage this coreference data. To ensure a fair comparison, we opt not to utilize entity coreference information. Instead, we consider entities with identical surface forms as co-referenced entities, thereby minimizing the potential for introducing noise into the model.\n\nFor future work, we want to use the coreference information in SciERC and also establish new benchmark datasets with entity coreference annotations to explore the enhanced versions of the span-based key-value memory that can incorporate coreference resolution. We think the coreference resolution and name entity recognition could be modeled in a multi-task framework and help each other. We appreciate your invaluable feedback and see its potential implications for future advancements.\n\n---\n\n**Missing References**\n\n*You should cite \"Design Challenges and Misconceptions in Named Entity Recognition\" since this paper is the first to suggest global features.*\n\n**Answer**\n\nThanks for your suggestion. We will cite this paper in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written. \nThe performance is good. Evaluating on different datasets from three domains, and still outperforming. \n", "reasons_to_reject": "Span-based key-value memory is updated by the matching of the surface form, which does not consider the situation of entity coreference. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on Document-Level NER, based on the intuition that spans with the same surface form are more likely to have the same entity type. In this paper, the authors propose a 2-stage DL-NER model. In stage 1, they train a binary classifier to classify whether each token is an entity or not. In stage 2, they build a span key-value memory to fuse the features of spans that share the same surface form. They evaluate their model on NER datasets from three domains: biomedical, scientific, and general domain. In these three domains, they show that their model is better than all the baseline models, including other document-level NER models, and also sentence-level NER models. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "You should cite \"Design Challenges and Misconceptions in Named Entity Recognition\" since this paper is the first to suggest global features. "}, "extra_text": {}}, {"review_id": "NrS4wrQKgN", "reviewer": "Reviewer_3NNy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons To Reject**\n\n*The paper does not recognize the computational cost of the span-based approach as opposed to the sequence labeling approach. The complexity of the span-based approach is quadratic. So the binary classifier would need O(n*KC2) predictions. It is a limitation of any span-based approach. But I expected a discussion in the limitations section on this.*\n\n**Answer**\n\nThanks for your feedback. It appears there might be some misunderstanding regarding the computational cost. The number of spans is linear to the length of the sentence. We've set a maximum sequence length for spans, denoted as $K$, as described in the main text (from lines 94 to 100). For a sentence with $n$ words, the total spans enumerated would be $(n + n-K+1) \\times K / 2$, which is $O(nK)$. As $K$ is a constant, the overall computational overhead for the span-based technique is linear $O(n)$. In the experiments conducted in this work, we've employed $k = 8$ or $10$, as shown in Table 5.\n\n---\n\n**Missing references**\n\n*1. \"FLERT: Document-Level Features for Named Entity Recognition”, Schweter and Akbik,*\n\n*2. \"Exploiting Global Contextual Information for Document-level Named Entity Recognition”, Wang et al.*\n\n**Answer**\n\nThank you for your insightful suggestion. The two works you cited focus on encoding document-level features for NER tasks. The two baseline models aim to have each word's encoding contain global or document-level features. Specifically, FLERT explores the advantages of transformer-based document-level features in enhancing NER performance. These features are derived by processing a sentence in the context of its surrounding text to produce word embeddings enriched with document-level characteristics. Meanwhile, GCDoc captures global contextual information at both the word and sentence levels. At the word level, the document is transformed into a graph, and a Graph Neural Network is employed to encode the features of each node. At the sentence level, a specialized cross-sentence module is used to encode relationships between adjacent sentences.\n\nIn contrast, our model enables each entity span's encoding to contain global features from other entity spans that share the same surface name. Our proposed model leverages a span-based key-value memory system to integrate the global features that are unique to each individual entity. We appreciate the relevance of the works you mentioned and will certainly cite them in the final version of our paper.\n\n---\n\n**Typos Grammar Style And Presentation Improvements**\n\n*Section 3.3 is hard to read with so many notations and subscripts. Please try to simplify it.*\n\n**Answer**\n\nThank you for your suggestion. We plan to streamline this section in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study is done across a wide variety of domains and shows consistently better performance across all data sets.\n2. The method is simple and effective.\n", "reasons_to_reject": "1. The paper does not recognize the computational cost of the span-based approach as opposed to the sequence labeling approach. The complexity of the span-based approach is quadratic. So the binary classifier would need O(n*KC2) predictions. It is a limitation of any span-based approach. But I expected a discussion in the limitations section on this.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper deals with the task of document-level named entity recognition. The goal is to ensure the same NER tags are being predicted across all mentions of an entity in a document. To this end, the paper proposed a two-stage approach. In the first stage, the model uses a binary classifier to identify the potential mention spans and stores the surfaces in a key-value memory. In the second stage, the model aggregates the span representations of each unique surface and applies a multi-class classifier MLP to predict the tags. Experiments are done on biomedical, scientific, and general domain NER data sets. The results outperform all baselines considered in this paper.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. “FLERT: Document-Level Features for Named Entity Recognition”, Schweter and Akbik,\n2. “Exploiting Global Contextual Information for Document-level Named Entity Recognition”, Wang et al.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Section 3.3 is hard to read with so many notations and subscripts. Please try to simplify it."}, "extra_text": {}}, {"review_id": "8uvFCdV0VB", "reviewer": "Reviewer_G1FQ", "summary": "", "questions": "- Q1: Concerning stage I of the approach: How is it determined in this context whether a span / string signifies an entity or not? Does it consider only the exact mentions from the gold standard, or does it also include substrings of these? (Lines 143, 144)\n\n- Q2: Concerning state II of the approach: How do you handle prediction conflicts, e.g. if a span is classified as type A and a substring / subspan of this span is classified as type B? \n\n- Q3: Concerning Appendix B: In this analysis, are you solely focusing on exact matches of entity spans, right? Wouldn't it be equally valuable for your research to examine how many entities include tokens that are also labeled as 'O' or as another entity (as exemplified by the \"breast cancer type I\" case from the introduction)? Have you conducted any examination in this regard?\n\n- Q4: Have you ever explored the efficacy of an approach that initially employs a sentence-level classifier, and then in a post-processing step uniformly assigns the same labels to the same surface forms (e.g., if \"breast cancer\" is labeled as a disease in one sentence, label all other instances of \"breast cancer\" as a disease in the document)? This could potentially provide another baseline for your approach.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reasons To Reject**\n\n*Concern 1: Important/major analyses to support the claim/motivation of the work as well as ablation studies for better result interpretation, are only available in the appendix of the paper. Even with an additional page for publication, the appendix' content will not fit into the main text.*\n\n**Answer**\n\nThanks for your feedback. We acknowledge that some parts in the appendix, including Sections B, C, E, and F, would be better to appear in the main text. To this end, we plan to do the following revisions such that these contents can fit into the five-page limit.\n-  We will only use the qualitative analysis part of Section C and leave Table 4 in the appendix.\n- Table 5 will be removed since most settings are shared among datasets with minor differences. We will describe these settings in the text.\n- The 2nd, 3rd, and 4th columns of Table 6 will be removed as this information is already included in the main text. This will streamline Table 6 into a single-column table. Initially, these columns were included in the table because there is no restriction on the page limit of the appendix.\n- We will reformat Table 7 to save space. For example, the text in Table 7 will be abbreviated.\n- Figure 1 will be simplified to save space.\n\n---\n\n*Concern 2: In the main text, only quantitative results are described, which makes it difficult to interpret the quality of the model and to assess whether and to what extent the intent of the approach can be realized.*\n\n**Answer**\n\nThanks for your suggestion. In the final version, we plan to incorporate Section C into the main text to display the qualitative results.\n\n---\n\n*Concern 3: Some details of the approach description are missing, which hampers reproducibility (see questions, improvements)*\n\n**Answer**\n\nThanks for your feedback. We will provide more details and release the source code once the paper is accepted.\n\n---\n\n**Questions For The Authors:**\n\n*Q1: Concerning stage I of the approach: How is it determined in this context whether a span / string signifies an entity or not? Does it consider only the exact mentions from the gold standard, or does it also include substrings of these? (Lines 143, 144)*\n\n**Answer**\n\nThank you for your valuable feedback. Our approach is in line with prior work, such as SpERT, and adheres to a strict matching criterion. In the benchmark NER dataset, each entity is annotated by the starting and ending positions of a contiguous token sequence within the document. In this context, a span is identified as an entity only if its starting and ending positions exactly align with those of a labeled entity. As for substrings, we treat them as separate spans that may share some tokens with the primary span, but their predictions operate independently. Importantly, the concept of a \"subspan\" is not applicable under the span-based NER settings.\n\n---\n\n*Q2: Concerning state II of the approach: How do you handle prediction conflicts, e.g. if a span is classified as type A and a substring / subspan of this span is classified as type B?*\n\n**Answer**\n\nEssentially, the substrings are treated as separate spans that may share some tokens with the primary span. However, their predictions operate independently. For example, the span ``United States'' and its substring ``United'' are considered separate spans, and their predictions are made independently. In some benchmark datasets like the SciERC dataset, there are overlapping entity spans with different entity types and nested entity names. It means that the following cases occur in training data: a span is annotated as type A and a substring / subspan of this span is annotated as type B. And in inference time, the model needs to predict different labels for such cases, too.\n\n---\n\n*Q3: Concerning Appendix B: In this analysis, are you solely focusing on exact matches of entity spans, right? Wouldn't it be equally valuable for your research to examine how many entities include tokens that are also labeled as 'O' or as another entity (as exemplified by the \"breast cancer type I\" case from the introduction)? Have you conducted any examination in this regard?*\n\n**Answer**\n\nThank you for your constructive feedback. Yes, our focus is solely on the exact matching of entity spans. We operate under the assumption that only entity spans with identical surface names will have consistent labels. We do not make the same assumption for non-entity spans. If a span with a matching surface name is not annotated as an entity (and is thus labeled as \"O\"), its features will not be shared with other entity spans bearing the same surface name. \n\nIn general, we don't think that the following assumption holds: if a span A shares the same surface name with an entity span B in the same document, then A must be an entity span with the same type of B. (To our understanding, this is the scenario your question concerns). \nHere is one example. The term \"parsing\" is labeled as \"O\" in this context: \"After parsing the example image into its semantic components,\" yet it is labeled as a \"Task\" in the following context from the same document: \"that utilizes recent advances in unification-based parsing.\"\n\nWe do not want to consider the features of the \"parsing\" from the first context when building the key-value memory for entity \"parsing\" in the second context. In fact, fusing their features will introduce noise for entity spans.\n\nSince we only assume the label consistency of entity spans, analysis in Appendix B, which aims to validate this assumption, only concerns entity spans (i.e., spans that are not \"O\").\n\n---\n\n*Q4: Have you ever explored the efficacy of an approach that initially employs a sentence-level classifier, and then in a post-processing step uniformly assigns the same labels to the same surface forms (e.g., if \"breast cancer\" is labeled as a disease in one sentence, label all other instances of \"breast cancer\" as a disease in the document)? This could potentially provide another baseline for your approach.*\n\n**Answer**\n\nThank you for your thoughtful feedback. We have implemented the method you suggested, building upon SpERT, one of our baseline models. For predicted entities that share the same surface name, we assign them a uniform label based on the majority entity type. This method was evaluated using the SciERC dataset, and the comparative results are summarized in the subsequent table.\n\n| Model | Precision | Recall | F1 |\n| :---        |     :----:    |     :----:    |     :----:    | \n| SpERT | 70.87 | 69.79 | 70.33 |\n| SpERT + post | 69.89 | 71.19 | 70.53 |\n| **ScdNER** | 71.60 | 71.43 | **71.44** |\n\n\nBased on the results, it is shown that post-processing yields a marginal performance improvement, specifically a 0.2\\% increase. We recognize the value of this as a solid baseline model and plan to include these findings in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe work proposes (to the best of my knowledge) a novel approach for document-level named entity recognition.\n-\tThe experimental results highlight the suitability of the proposed approach.\n-\tThe article is linguistically essentially well written.", "reasons_to_reject": "-\tImportant / major analyses to support the claim/motivation of the work as well as ablation studies for better result interpretation are only available in the appendix of the paper. Even with an additional page for publication the appendix’ content will not fit into the main text.\n-\tIn the main text, only quantitative results are described, which makes it difficult to interpret the quality of the model and to assess whether and to what extent the intent of the approach can be realized.\n-\tSome details of the approach description are missing, which hampers reproducibility (see questions, improvements)", "questions_to_authors": "- Q1: Concerning stage I of the approach: How is it determined in this context whether a span / string signifies an entity or not? Does it consider only the exact mentions from the gold standard, or does it also include substrings of these? (Lines 143, 144)\n\n- Q2: Concerning state II of the approach: How do you handle prediction conflicts, e.g. if a span is classified as type A and a substring / subspan of this span is classified as type B? \n\n- Q3: Concerning Appendix B: In this analysis, are you solely focusing on exact matches of entity spans, right? Wouldn't it be equally valuable for your research to examine how many entities include tokens that are also labeled as 'O' or as another entity (as exemplified by the \"breast cancer type I\" case from the introduction)? Have you conducted any examination in this regard?\n\n- Q4: Have you ever explored the efficacy of an approach that initially employs a sentence-level classifier, and then in a post-processing step uniformly assigns the same labels to the same surface forms (e.g., if \"breast cancer\" is labeled as a disease in one sentence, label all other instances of \"breast cancer\" as a disease in the document)? This could potentially provide another baseline for your approach.", "paper_topic_and_main_contributions": "The paper presents an approach for document-level NER, which aims to achieve consistent predictions for entities within a document. The approach is divided into two stages: first, a binary classifier identifies spans that mention entities. In the second stage, a key-value store is used to store and learn span information on a global, document-level information. Both local (the individual mention) and global representations are then used for prediction. Experiments on data sets from three different research areas (i.e., biomedical, scientific, and general domain) show superior results compared to competitor models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tThe analyses in Appendix B and (at least one or two of) D, E, F should be included in the main text to strengthen argumentation. In my opinion, the work could have been made into a solid long paper.\n\n-\tIt would also be interesting to see the results of the approach on biomedical datasets that have more than 2 entities (e.g. BioRED or BioNLP2013-ST-CG). Due to the higher number of entity types, there are also higher probabilities for inconsistent annotations."}}]

Author rebuttal:

Meta-review: The paper presents a method for generating consistent NER labels across documents using the intuition that entities that share the same surface form are very likely to have the same NER label.

All three reviewers recognise the soundness of the approach and highlight the evaluation on multiple domains as an important contribution. Equally, all three reviewers are satisfied that the experiments demonstrate the increased performance of the proposed solution. 

While any remaining technical issues raised by Reviewers 3NNy and G1FQ seem to have been addressed by the authors, the general consensus in the discussion was that the paper was rather incremental in nature which kept the reviewers from being more excited about it.  As the discussion with Reviewer G1FQ suggests, including some of the material currently found in the appendices would make this paper this paper more robust and the authors have proposed a plan to do so already.
Decision: Accept-Main