Context-aware Attention Model for Coreference Resolution
Authors: Yufei Li
Xiangyu Zhou
Jie Ma
Yu Long
Xuan Wang
Chen Li
Venue:  (2020)
Keywords: Coreference resolution
Feature Attention

Abstract: Coreference resolution is an important task for gaining more complete understanding about texts by artificial intelligence. The state-of-the-art end-to-end neural coreference model considers all spans in a document as potential mentions and learns to link an antecedent with each possible mention. However, for the verbatim same mentions, the model tends to get similar or even identical representations based on the features, and this leads to wrongful predictions. In this paper, we propose to improve the end-to-end system by building an attention model to reweigh features around different contexts. The proposed model substantially outperforms the state-of-the-art on the English dataset of the CoNLL 2012 Shared Task with 73.45% F1 score on development data and 72.84% F1 score on test data.

Under review as a conference paper at ICLR 2020
CONTEXT-AWARE ATTENTION MODEL FOR COREFER- ENCE RESOLUTION
Anonymous authors Paper under double-blind review
ABSTRACT
Coreference resolution is important task for gaining more complete under- standing about texts by artificial intelligence. state-of-the-art end-to-end neu- ral coreference model considers all spans in document potential mentions and learns to link an antecedent with each possible mention. However, for the verba- tim same mentions, the model tends get similar or even identical representations based on the features, and this leads to wrongful predictions. In this paper, we pro- pose to improve the end-to-ent system by building an attention model reweigh features around different contexts. The proposed model substantially outperforms the state-of-the-ae on the English dataset the CoNLL 2012 Shared Task with 73.45% F1 score on development data and 22.84% F1 score on test data.
INTRODUCTION
Coreference is one of the most frequent phenomena in English and the other languages. Coreference resolution crucial task before artificial intelligent systems capable fully understanding the human language. Supervised methods, especially the models using neural-network-geneeeeee
word
representations, achieve outstanding performances Clark Manning (016): Lee et al. (2017) 2018). However, for similar or identical text units, the problem of wrongfully getting the same coreferences still puzzling. For example, the following conversation has the sentences, and
A: Yeah,it's not far. Through the S-bahn I mean it's like twenty minutes.
B: Or something And so,if do it,I'd love to have you join Ie. a fancy wedding too.
The pronoun "it's" in the sentence A and the "it's" in the sentence B are obviously referring to the different things. As they are likely to get similar or even the same expression, a false link between them is often predicted. A similar case in sentence B. Due to different lemma and lexeme, the model would not predicate that "T" in A and "you" in B are coreferential. On the opposite, a false coreference between "T" in A and "T" in B would be predicted.
In coreference resolution tasks, words referring each other are called mentions, while a mention could be common noun, proper noun pronoun. Taking the above example, coreference system partitions the mentions a sentence into one coreference chain-("it's", "it's"), and single- ton: "the S-bahn" for speakerl. One coreference chain-("T' "md","m""), and singleton: "It's" for speaker2 and one coreference chain-("T", "you"") between two speakers.
In recent years, several supervised approaches have been proposed to coreference resolution. The work can be categorized into three classes. 1) mention-pair models: A mention pair model a bi- nary classifier that determines whether a pair of mentions co-referring or not McCarthy Lehnert (1993). One of the common limitation of the mention-palt model is that it cannot capture infor- mation beyond the mention pair. The information that can be obtained from the two markables to determine their coreferential status is very limited.; 2) mntit-mmnnion models: The entity- mentton model aims to classify whether an NP (Noun phrase) is coreferent with a preceding cluster Yangy et al. (2004); Culotta et al. (2007): >: Daumé III & Marcu (2005). This strategy considers the candi- dates independently. relative to the other candidates; 3) ranking models: Ranking models allows candidate antecedents of a mention to be ranked simultancously lida et al. (2003): Denis & Baldridge (2008); Durrett &
cannot measure how likely a candidate is the antecedent for a given anaphor,
Klein (2013).

---

Under review as a conference paper at ICLR 2020
In the above three classes, the ranking models recently obtained the ofate-o-the--il performance Wiseman et al. (2015 2016): Clark & Manning (016)) More recently, Lee et al. (2018) proposed the first state-of-thee potential mentions and learn distributions over possible antecedents for each. In addition, they use a fully differentiable approximation higher-order inference to iteratively refine span representations, and the model only uses a minimal set of hund-engineered features (speaker ID, document genre, span distance, span width). This leads to the problem that identical mentions tend to get similar or even identical representations, and further misled coreference resolutions to make mistakes.
end-to-end neural coreference resolution system. They consider all spans as
In this paper, we demonstrate a novel method utilizing attention mechanism to adaptively exploit features to represent identical mentions with different contexts. Inspired by the recent success of attention mechanism, we focus on this issue and develop general attention mechanism that learns the importance/weight of each feature based on mention's 's contexts and then add this information to the end-to-end neural model. The entire model could be trained end-to-end with gradient descent.
The proposed model is evaluated the CoNLL 2012 Shared Task Pradhan et al. (2012). The results show that the method outperforms the baselines. Meanwhile, we made statistic of the different features' weights in the attention mechanism. The statistic shows that the feature attention algorithm does help to distinguish the features for identical mentions based on different contexts.
2
THE MODEL
2.1 TASK
In an end-to-end coreference resolution, the input is a document D with T words along with meta- data, and the output is of mention clusters. Let N be the number of possible text spans in D. We consider all possible spans up to predefined maximum width. Denote the start and end indices of span in D respectively by ST A AR)(1 and EN D(i). For each span i the system needs sign an antecedent a, ( (e-1....i- from all preceding spans or dummy antecedent The dummy antecedent represents two cases: (1) the span is not an entity mention, or (2) the span i is an entity mention but not coreferential with any previous span. Finally, all spans, that are connected by set of antecedent predictions, are grouped.
2.2 METHODOLOGY
The section elaborates the model. We adopt similar span representation approach it in Lee et al. (2018) using bidirectional LSTMs and a headfinding attention. In our approach, we investi- gate grammatical numbers and use general attention mechanism reweighing span features based on mention's 's contexts to generate the new feature representation. Thereafter, the scores of how likely spans could be entity mentions are generated by a feed forward network. We also propose an attention model of antecedent scoring to reweigh pair-wise features and produce the new feature vectors.
Span Representation Illustrated in Figure 1, we assume vector representation of sentence with I. words as {z1,52...,TL while I, denotes the concatenation fixed pretrained word embeddings Siaeedddngg/Snttss Santos & Zadrozny (2014) for t-th word. Then Bidirectional LSTMs Hochreiter & Schmidhuber (1997) are used to encode each zy:
and CNN character embeddings
hy = LST Mback (fe41 xz))
(1)
h LSTM forward (h441, 23))
x [he, Tee)
Then, the model learns a spec-fpe notion of headedness using the attention mechanism Bahdanau et al. (214)) over words in each span:
2

---

Under review as a conference paper at ICLR 2020
peimary primary stele three three secondary steles
steles und two
two iascribed inscribed steles
Score (tx)
Span Represcutaiin
Span Feature
Attention (*(1)
Span Hcad(x)
BI-LSTM outputs (c)
Wurd& Chacnncler embedddngg O3)
primary
serondary
steles
Inseribed
steles
Figure
The model computing the span embedding representations.
0; = W. :.= (FFNN(2)
(4)
exp(as)
0i,
Σ END(O
R STATTO) exp(akk
(5)
END() Σ
8;
Oitt Tt
(6)
1- ST ART
Where is weighted sum of word vectors in span i. FFNN is feed forward network. Then the final representation yi of span was produced by:
I (ST) (f))
Where in Lee et al. 20188) (p()) is 20-dimensional feature vector encoding only span width infor- mation. In our model, we incorporate one more feature, grammatical numbers. The grammatical numbers reinforced with the feature attention method by reweighing contextual features. A new feature vector yp" is generated by the method.
Feature Attention Shown in Figure portance/weight of each feature based on contexts. Suppose the initial feature vectors yp = [91, 22: ..... (v). where (P; E R20 indicates the i-th feature and F,, is the contexts vectors gen- erated by Bi-LSTM. Then the model learns the weight of each feature based on the contexts, and generate the feature vectors sp":
general attention mechanism that leamn the im-
dg = Wa FFFN (F:NN(P; exp(d))
(8)
dy EV exp(a() 6- 1 49j
(9)
(10)
Where @ the concatenate operation and is linear function map Iu to the same dimension with the feature vector. s² is the new reweighed feature vectors and az is the weight of each feature based on the contexts.
Mention Scoring The new span representation is encoded to a FFNN to compute the mention score measuring if it is entity mention, where FA is the feature attention algorithm:

---

Under review as a conference paper at ICLR 2020
O
Contexts Vectors (Xx)
Feature Vectors Φ
Figure 2: The Feature Attention model. The model learns to weigh each feature based on contexts.
(i) FA( 9. (RTT) I -END()) Ee, (p" (i)
(11)
(12)
8m Wm FFNN
(13)
Antecedent Scoring For antecedent scoring, we use the same features (speaker, genre, distance) as Lee 2018). And incorporate our feature attention algorithm to the features (i(,
= FA(.))=F((( (g. 91 gn. (i.,))
(14)
sa (1. Wa FFN
(15)
Coreference Score The final coreference score of span i and j indicates that (1) whether span is mention, (2) whether span j a mention, and (3) whether an antecedent if
0,
==
si, = mm + smj + sa (i.j),
(16)
j +
Loss Regression we use the same loss regression as Lee et al. (0018) where GOLD(i) the set of spans in the gold cluster containing span
N
L = log II
Σ
p(v')
(17)
3 EXPERIMENTS AND RESULTS
3.1 EXPERIMENTAL SETUP
Dataset and baseline. The experiments are conducted on the English subset of the CONLL2012 Shared Task data Pradhan et al. (2012) and evaluated three standard metries: MUC Wiseman et al. (2016), B3 Bagga & Baldwin (1998), and CEAF Luo (2005). We report precision, recall, Fl for each metric and the average Fl the final CoNLL score. We the baseline below:

---

Under review as a conference paper at ICLR 2020
Table 1: Results CONLL 2012 English development set.
MUC
System
P
R
F
P
R
F
P
R
F
Avg Fl
Lee et al. (2018)
82.06 77.85 79.90 74.15 68.86 71.41 69.27 66.47
67.84 73.05
+pair-wise FA
81.53 78.54 80.01 73.06 70.03 71.77 69.29 67.27 68.26 73.36
+grammatical numbers
81.65 78 43 80.00 73.97 69.60
71.72 69.05
67.60
68.32 73.35
+span FA
82.07 78.30 80.10 74.20 69.63
71.84
69.54
67.23
68.37 73.45
Björkelund & Kuhn 2014 The system obtains significant improvements over the baseline by modifying LaSO to delay updates until the end of each instance.
Wiseman et al. 2015 The system learns distinct feature representations for anaphoricity detec-
tion and antecedent ranking, encouraged by pre-training on a pair of corresponding subtasks.
Wiseman et al. 2016 The system proposes use recurrent neural networks (RNNs) learn latent, global representations of entity clusters directly from their mentions.
Clark & Manning 2016 The system presents neural network based coreference system that produces high-dimensional vector representations for pairs of coreference clusters to learn when combining clusters is desirable.
Lee et al. 2017 The first state-of-the-art end-to-end neural coreference resolution system and only uses a minimal set of hand-craft features.
Lee et al. 20188 The improved model for Lee et al. (2017) using the antecedent distribution from span-ranking architecture an attention mechanism iteratively refine span representations.
Hyperparameters. we follow the same hyperparameters as in Lee et al. (2018): Using Glo word embeddings |Pennington et al. (2014) with window size of 2 for the head word embeddings and a window size of 10 for the LSTM inputs and embedding representations from a language model Peters al. (2018) as the input to the LSTMs (ELMo in the results). The maximum span width was 30 words and only consider 50 antecedents. À = 0.4 is used for span pruning. The model is trained up to 150 epoch with early stopping.
3.2 RESULTS AND DISCUSSION
Contrast Experiments We first run the Lee2018 system[ on the development data and test data of CONLL2012 Shared Task. To train our model, we employ the same Lee al. (018). Then the model trained and compared with the Lee2018 system. We compare the model with the baseline system with each improvement (1) Pair-wise FA (Feature Attention): incorporating feature attention algorithm to the pair-wise features (i(i..)): (2) Grammatical numbers
reported in
(GN): incorporating grammatical numbers to span features yoi); (3) Span FA(Feature Attention):
using feature attention algorithm to encode span feature cpi).
Comparison and Analysis To investigate the importance of each component in the proposed model, we report the results of each part Table and Table 2, comparing with Lee2018 system.
Pair-wise Feature Attention: The performance increases by 0.31 Fl dev dataset and 0.17 on test with the Pair-wiss FA. The improvement of the precision shows that the false-positee links for the identical or similar mentions (not coreferent) are decreased. In the baseline model, when mention is predicated "not coreferent" to the mention u, the other mentions identical to / will be likely to be predicated "not coreferent" the mention the recall indicates that such false negative links are substantially decreased by the model.
But the significant improvements from
Grammatical numbers: The addition of new span feature, grammatical numbers, helps little with the system. This indicates that the grammatical information extraction of the whole system is relatively sufficient. So the GN does not help much.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Results the CONLL 2012 English test set.
MUC
B
System
P
R
P
Avg F1
Lee al. (2018)
81.8 78.54 80.14 73.2
68.15 15 70.58 68.51 65.94 67.20 72.64
+pair-wise F A
81.71 78.84 80.24 72.82
68.60 70.67 68.82 66.26 67.51 72.81
+grammatical numbers
81.30
79.13 80.20 72.16 69.11 70.60 68.69 66.36 67.51 72.77
+span F A
81.42 79.20 80.29
72.44
68.94 70.65 69.12 66.08
67.57 72.84
Table 3: Overall performance on CoNLL 2012 English test set.
MUC
B
C
System
P
R
F
Avg
B&K (2014)
74.30 67.46 70.72 62.71 54.96
58.58
59.40
52.27
55.61 61.63
Wiseman et al. (2015)
76.23 69.31 72.6 66.07 55.83 60.52 59.41 54.88
57.05 63.39
Wiseman al. (2016)
77.49 69.75 73.42 66.83 56.95 61.50 62.14 53.85 57.70 64.21
Clark & Manning (2016)
79.91 69.30 74.23 71.01 56.53 62.95 63.84 54.33 58.70 70 65.29
Lee et al. (2017)
78.40 73.40 75.80 68.60 61.80 65.00 62.70 59.00 60.80 67.20
Lee et al. (2018)
81.80 78.54 80.14 73.20 68.15 70.58 68.51 65.94 67.20 72.64
This work
81.42 79.20 80.29 72.44 68.94 70.65 69.12 66.08 67.57 72.84
Span Feature Attention: Finally, we consider the span feature attention. We see a contribution of 0.1 F1 on dev dataset and 0.13 on test. The results show that the span FA does help to improve the system, especially in the mention detection subtask. The contribution in mention detection subtask further discussed Section Mention Detection Subtask.
Overall Performance Overall performance comparing with the other state-of-the-ard systems are shown in Table 3. We could see significant improvement average Fl scores over the previous work, and the highest Fl scores on all three metrics especially in CEAFg4 metric. And the most significant gains come fron improvements of recall. Such improvements indicate that the applying of Feature Attention algorithm does help to distinguish the features for identical spans and men- tions based different contexts. That means the weights of different features useful for making coreference decisions.
Mention Detection Subtask To further understand the utility of Feature Attention algorithm for mention detection subtask, we list the mention detection performance in Table 4. Compared wirh Lec2018, the performance increases by 0.33 Fl on dev dataset and 0.31 on test. And the results show that the model indeed performs better in the recall scores. In the baseline model, when there is a span not predicated to be mention, the other identical spans will unlikely to be detected as mentions, either. However, in our model, such false negative links are decreased, being benefit from the span FA algorithm that reweighs the features distinguish identical spans with different representations based on different contexts.
3.3 FEATURE WEIGHTS
To gain a further insight about how identical terms' representations could distinguished by the attention mechanism, the feature attention weights are investigated. For example, span feature at- tention weights are shown in Figure For the three "it's" the figure, we can observe that the first two lines of "it's" that are coreferent gain similar weights for the two features: higher weight in span width than grammatical number, but the third "it's" not coreferent gains the opposite results. This indicates that the identical spans which have the same features will have the different span feature attention weights depending on their contexts through the span feature attention and finally improves the performance.

---

Under review as a conference paper at ICLR 2020
Table 4: Mention Detection Results CONLLL 2012
English data
CoNLL--2012 Dev Set CoNIL--2012 Test Set
System
P
F
R
Lee et al. (2018)
86.71 82.49
84.55
86.62 83.22
84.89
Our work
86.78 83.05
84.88
86.64 83.80
85.20
A: AeYeah,it's not fax Through the S S-balm here.I mean it's like twenty minutes.
B:Or something.eo And so,ifII I do it love to have you join me It's fancy wedding loo.
NPs
span width
grammatical number
0.533
0.477
it's
0.525
0.475
It's
0.498
0.502
Figure 3: Example of span feature attention weights in different features. The rows show the span feature attention weights features of each "it's".
CONCLUSION
Identical mentions impose difficulties on the current methods coreference resolution as they tend get similar olution to wrong predictions. In the paper, we focus on this issue and develop an attention model named Feature Attention adaptively exploit features order represent identical mentions with consideration of different contexts. The results show that our model with the Feature Attention algo- rithm performed reasonably well in coreference resolution, which is evaluated on the CoNLL-2012 Shared Task in English.
identical representations. The problem may directly lead coreference res-
ACKNOWLEDGMENTS
This work has been supported by the National Key Research and Development Program of China 2018YFC09100044); National Natural Science Foundation of China (61772409); Ministry of AluccationResearree Foundation of China Mobile Communication Corp (MCM20160404); the sulting research project of the Chinese Academy of Engineering (The Online and Offline Mixed Educational Service System for "The Belt and Road" Training in MOOC China); Project of China Knowledge Centre for Engineering Science and Technology: the innovation team from the Min- istry of Education(IRT 17R86); and the Innovative Research Group of the National Natural Science Foundation of China (61721002), Professor Chen Li' Recruitment Program for Young Profession- als of "he Thousand Talents Plan"
REFERENCES
Amit Bagga and Breck Baldwin. Algorithms for scoring coreference chains. The first inter- national conference on language resources and evaluation workshop on linguistics coreference, volume 1, pp. 563-566. Granada, 1998.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv:/409.00 409.04433 2014.
7

---

Under review as a conference paper at ICLR 2020
Anders Björkelund and Jonas Kuhn. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 47-57, 2014.
Kevin Clark and Christopher D Manning. Improving coreference resolution by learning entity-level distributed representations. arXiv preprint arXiv:1606.01325 2016.
Aron Culotta, Michael Wick, and Andrew McCallum. First-order probabilistic models for corefer- ence resolution. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp. 81 31-88,220 2007.
Hal Daumé III and Daniel Mareu. large-scale exploration of effective global features for a joint entity detection and tracking model. In Proceedings of the conference on Human Language Tech- nology and Empirical Methods Natural Language Processing, pp. 97-104. Association for Computational Linguistics, 2005.
Pascal Denis and Jason Baldridge. Specialized models and ranking for coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 660- 669. Association for Computational Linguistics, 2008.
Greg Durrett and Dan Klein. Easy victories and uphill battles in coreference resolution. In Pro- ceedings of the 2013 Conference on Empirical Methods Natural Language Processing, pp- 1971-1982, 2013.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ryu lida, Kentaro Inui, Hiroya Takamura, and Yuji Matsumoto. Incorporating contextual cues in trainable models for coreference resolution. In Proceedings of the EACL Workshop on The Computational Treatment of Anaphora, pp. 23-30. Citeseer, 2003.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference reso- 2017.
lution. arXiv preprint
Kenton Lee, Luheng He, and Luke Zettlemoyer. Higher-order coreference resolution with coarse- to-fine inference. arXiv preprint arXiv:1804.05392. 2018.
Xiaoqiang Luo. On coreference resolution performance metrics. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pp. 25-32. Association for Computational Linguisties, 2005.
Joseph F McCarthy and Wendy G Lehnert. Using decision trees for coreference resolution. arXiv preprint cmp-lg/9505043, 1995.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint 1rXiv:1802.053666 2018.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Confer- ence on EMNLP and CoNLL-Shared Task, pp. 1-40. Association for Computational Linguistics, 2012.
Cicero D Santos and Bianca Zadrozny. Learning character-level representations for pfrt-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 1818-1826, 2014.
Sam Wiseman, Alexander M Rush, and Stuart M Shieber. Learning global features for coreference resolution. arXiv preprint arXiv: 1604, 03035, 2016.
8

---

Under review as a conference paper at ICLR 2020
Sam Joshua Wiseman, Alexander Matthew Rush, Stuart Merrill Shieber, and Jason Weston. Learn- ing anaphoricity and antecedent ranking features for coreference resolution. Association for Com- putational Linguistics, 2015.
Xiaofeng Yangy, Jian Su, Guodong Zhou, and Chew Lim Tan. An np-cluster based approach to coreference resolution. In Proceedings of the 20th international conference on Computational Linguistics, pp. 226. Association for Computational Linguistics. 2004.


Reviews:
[{"review_id": "rkgHD5LntS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use an extra feature (grammatical number) for context-aware coreference resolution and an attention-based weighting mechanism. The approach proposed is built on top of a recent well performing model by Lee et al. The improvement is rather minor in my view: 72.64 to 72.84 in the test set. \n\nThere is not much in the paper to review. I don't think the one extra feature warrants a paper at a top conference. The weighting mechanism over the features is also unclear to me why it benefits from attention. Couldn't we just learn the weights using another layer? It could be context dependent if desired.\n\nIt is also incorrect to criticise Lee et al. (2018) that they would give the same representation to the same mention every time. Their model is context dependent as they use a BiLSTM over the sentence. Of course the same mentions are likely to get similar representations, but this is desirable.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "Skgu_zsaKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper unfortunately violates the blind-review policy: its acknowledgement exposes the authors. I thus support desk rejection. \n                                                                                                                                                                                                                                                                                                                                                                                ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgdQY86KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the neural coreference resolution model in Lee et al. (2018) by 1) introducing an additional mention-level feature (grammatical numbers), and 2) letting the mention/pair scoring functions attend over multiple mention-level features. The proposed model achieves marginal improvement (0.2 avg. F1 points) over Lee et al., 2018, on the CoNLL 2012 English test set.\n\nI recommend rejection for this paper due to the following reasons:\n- The technical contribution is very incremental (introducing one more features, and adding an attention layer over the feature vectors). \n- The experiment results aren't strong enough. And the experiments are done on only one dataset.\n- I am not convinced that adding the grammatical numbers features and the attention mechanism makes the model more context-aware.\n\nOther suggestions:\n- The citation format seems weird through out the paper. \n- Table 1 and 3 look like ablation results. It might be less confusing if it's presented as \"Full system: xx%; without pairwise FA: yy%; without grammatical numbers zz% ...\".\n- Equation 8 - 10 are quite confusing. What is f(x)? How large is V? What is u? etc.\n- Please define/explain the \"grammatical numbers\" feature when it's introduced in Section 2.2.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #2 summarizes it well:

This paper extends the neural coreference resolution model in Lee et al. (2018) by 1) introducing an additional mention-level feature (grammatical numbers), and 2) letting the mention/pair scoring functions attend over multiple mention-level features. The proposed model achieves marginal improvement (0.2 avg. F1 points) over Lee et al., 2018, on the CoNLL 2012 English test set.

--

Discussion:

All reviewers rejected.

--

Recommendation and justification:

The paper must be rejected due to its violation of blind submission (the authors reveal themselves in the Acknowledgments).

For information, blind review #2 also summarized well the following justifications for rejection:

I recommend rejection for this paper due to the following reasons:
- The technical contribution is very incremental (introducing one more features, and adding an attention layer over the feature vectors). 
- The experiment results aren't strong enough. And the experiments are done on only one dataset.
- I am not convinced that adding the grammatical numbers features and the attention mechanism makes the model more context-aware.
Decision: Reject