MULTI-SPAN QUESTION ANSWERING USING SPAN-IMAGE NETWORK
Authors: Tarik Arici
Hayreddin Ceker
Ismail Baha Tutar
Venue:  (2021)
Keywords: BERT
deep learning
multi-span answer
question-answering
SQuAD
transformers

Abstract: Question-answering (QA) models aim to find an answer given a question and con- text. Language models like BERT are used to associate question and context to find an answer span. Prior art on QA focuses on finding the best answer. There is a need for multi-span QA models to output the top-K likely answers to questions such as "Which companies Elon Musk started?" or "What factors cause global warming?" In this work, we introduce Span-Image architecture that can learn to identify multiple answers in a context for a given question. This architecture can incorporate prior information about the span length distribution or valid span patterns (e.g., end index has to be larger than start index), thus eliminating the need for post-processing. Span-Image architecture outperforms the state-of-the-art in top-K answer accuracy on SQuAD dataset and in multi-span answer accuracy on an Amazon internal dataset.


Under review as a conference paper at ICLR 2020
MULTI-SPAN QUESTION ANSWERING USING SPAN- IMAGE NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Question-answering (QA) models aim to find answer given question and con- text. Language models like BERT are used to associate question and context to find an answer span. Prior on QA focuses on finding the best answer. There is a need for multi-span QA models to output the top-K K likely answers to questions such as "Which companies Elon Musk started?" or "What factors cause global warming?" In this work, we introduce Span- -ma-lmmge architecture that can learn to identify multiple answers in a context for a given question. This architecture can incorporate prior information about the span length distribution or valid span pat- terns (e.g., end index has to be larger than start index), thus climinating the need for post-processing. Span-Image architecture outperforms the state-of-the-arr in top-K answer accuracy on SQuAD dataset and multi-span answer accuracy on Amazon internal dataset.
INTRODUCTION
Answering questions posted as text to search engines or spoken to virtual assistants like Alexa has become key feature in information retrieval systems. Publicly available reading comprehension datasets including WikiQA (Yang et al./ 2015). TriviaQA (Joshi et al. 2017), NewsQA (Trischler et al. 2016). and SQuAD (Rajpurkar et al. 2016 have fostered research in QA models. SQuAD one of the most widely-used reading comprehension benchmarks that has an active leaderboard with many participants. Even though there are models that beat human-level accuracy SQuAD, these QA systems can do well by learning only context and type-matching heuristics (Weissenborn al.) 2017] but may still be far from true language understanding since they do not offer robustness to adversarial sentences (Jia & Liang] 2017). To better measure performance, SQuAD v2.0 Rajpurkar ct al. (2018) extends vI. 1..1 by allowing questions that have no explicit answers in a given paragraph.
QA can be modeled as a task to predict the span (i.e., start and end indices) of answer given question and an input paragraph. To find the answer span, language representation models such as BERT can be used associate question with given paragraph Devlin et al. (2019). BERT is pre-trained on unsupervised tasks using large corpora. Its input representation permits pair, which well suited for having question and passage input. By fine-tuning BERT on SQuAD, QA model can be obtained. Questions without an answer are treated as having a span that begins and ends with the special BERT token: [CLS]] In this way, BERT-based QA model can offer an actual answer or "no-answer" to all questions in SQuAD v1.1 and v2.0 datasets.
Prior work on QA assumes presence of single answer or lack of any answer (Soo al. (016)). Devlin et al. (201)). Furthermore, they assume separable probability distribution function (pdf) for start and end indices of answer span, which leads separable loss function. This approach has two major disadvantages: 1) It prevents the QA model from predicting multiple spans without post- processing. 2) Since separable pdf is used, the QA model can not learn to evaluate compatibility of start and end indices, thus suffering from performance degradation. |ang et al.) (2019) consider a hierarchical answer span by sorting the product of start and end probabilities to support multiple spans. However, they still assume separable pdf for start and end indices. To the best of our knowledge, a multi-span QA architecture has not been proposed.
We introduce Span-Image architecture to enable multi-span answers (or multiple answers) given question and a paragraph. Each pixel (i.j) in the span-image corresponds to a span starting at ith position and ending at jth. Typical image processing networks like 2D) convolutional network layers

---

Under review as a conference paper at ICLR 2020
used. Span-Image architecture enables the model couple start and end indices to check for their compatibility. Constraints such as "the end index has to be bigger than the start index", can be automatically embedded into the model. Moreover, other span characteristics such as "shorter answers are more likely to occur" (see Figure] can be learned by the model, thus eliminating the need for post-processing regularization.
Span Length Distribution of SQUAD
20000
10000
10
20
50
Figure 1: Span length histogram on SQuAD shows that shorter answers are more likely to occur. Span-Image architecture can incorporate this prior information since each output pixel predicts span whose length is known by the model.
Our contributions are summarized as below:
We present Span-Iman Image architecture, novel method that enables multi-span pre- diction.
Specially designed image channels in the proposed architecture can help the QA model capture span-characteristics and eliminate the need for post-processing.
Span-Image network is modular and can be added to most DNN-based QA models without requiring changes to the previous layers.
MULTI-SPAN PREDICTION
In this section we first highlight BERT QA architecture, and present our span-image architecture that consumes BERT outputs.
2.1 BERT QA ARCHITECTURE
The QA task in BERT uses separable pdf: ,) p(s5)(())))))) x p(sB) where 8 and _ denotes one-hot variables of length N for start and end indices for a paragraph of length N, respectively. Therefore, BERT QA architecture assumes start and end index probabilities to be independent from each other. Given predicted probabilities PBERT(sS) and PBERT (s as outputs BERT, a ques- tion g of length M and passage of length N, the QA loss fuction for fine-tuning BERT is then given by
Loss(g.p.1$ 1,,5,,1))) -) H(S (PBERT(SS)) + H(t -PBERT(F))
(1)
where H the cross-entropy function, and is target span with start and end indices S and t.. respectively.
BERT has two separate outputs for start and end indices, which makes it impossible for the model check for compatibility of s and E or utilize information such span length (ie., F - in its predictions. Figure Sigure shows BERT QA architecture.
'One can claim attention heads in the transformer network will correlate tokens hut this does not happen explicitly architecture, where probability for each possible span computed jointly.
2

---

Under review as a conference paper at ICLR 2020
Softmmax
Softmax
Affine Transformation
Affine Transformation
BERT
Expr
Exps
[CLS]
Toke
Tollwnt
Questiion
Paragraph
Figure 2: BERT QA architecture predicts start and end index probability distributions separately.
2.2 SPAN-IMAGE NETWORK
Span-image network does not dictate separable pdf for start and end indices, hence p(s sB) + p(sS) ) x p)s Given a question and paragraph BERT outputs D dimensional vector sequence BERT (q, g) of length M + N + 2 (see Figure 2p. Let's denote the last N vectors in the sequence, which corresponds to paragraph
with BERT g). Using two affine transformations denoted by 8 and W., each of which has units, we create vector sequences W (BERT" (q.9)) and W (BERT g)) of length N. A pixel at location (i,j) has D) channels and is given by
span_imig S (BERT" (g. (q.9)); W (BERT* (q. ().g));
where o denotes element-wise multiplication of D-dimensional vectors in 7th and jth locations of (BERTT (q,9)) 9)) and WE(BETT' g)), respectively. Hence, span-image span_im shown in Figure is 3-dimensional tensor of depth D and of height and width N. This enables us borrow techniques such as 2-dimensional convolutional filtering, max-pooling, and ReLU from convolutional neural network (CNN) architectures for image classification. The output of the span- image network an N x N logit-image, logit im, with single channel (i.e., logit for each possible pixel/span).
Each channel in spon_im matrix of rank Therefore, each channel separable and has limited potential beyond the separable approach described in Section [.1] However, applying two- dimensional convolutional layers improves performance and makes logit_im non-separable, thus eliminating the independence assumption start and end indices.
The probability of each span can computed by applying sigmoid function on each pixel or soft- max in logit_im. Using sigmoid makes assumption number of spans, while using softmax assumes a single span in every paragraph. The best function to use depends on the QA dataset. For example, in experiments, using softmax gave us best results for fine-tuning BERT SQuAD while sigmoid performed better on our internal multi-span datasets. Denoting p(s = i.js = j) by pr.j for simplicity, span probabilities for single-span and multi-span datasets can be computed by
sigmoid sigmoid logit logit_im..)). Piá
if training datasets can have multi-span answers
softmas softmar (logit.im) 1.3"
if training datasets only have single-span answers
Target image, target_im, is a binary image with zeros at every pixel except those corresponding to target spans. (i.e., target.im(i.j) j) 1 for any target span in g with start index and end index j). Given logit im and target im, the loss function using sigmoid given below
Loss(g, P. target.im)
=
Σ H Carreet.im,, ims.s. sigmoid) /N22 b
(4)
3

---

Under review as a conference paper at ICLR 2020
IN 1!
Loggt Image (ogii im))
(span ch)
Span Image span_im)
Ni
Element-wise Multiplication
D
[NxDx1]
Affine Transformation (W9)
Affine Transformation (W9)
Figure 3: Span-image network implements convolution layers as typically used in image classifica- tion tasks.
where H is the cross-entrope function. The loss function for softmax is given by
Loss(q. p. tom) = target_im, softmaz)
(5)
Note that torget.im and proftmar are joint pdfs 8 and F while Pi,j variable indicating if (i.j) is answer span not.
sigmoid is pdf for binary
2.2.1 INCORPORATING SPAN CHARACTERISTICS
Typical post-processing on BERT QA output involves checking for valid spans and sorting them with the multiplication of start/end index probabilities. As shown in Figure [ priori information about the span length distribution probabilities. In BERT QA, this can only be achieved by implementing post-processing technique that penalizes spans based on their lengths. Our span-image architecture, however, has inherent capability to learn and incorporate such patterns. We simply create new channel span_ch, and our model learns how utilize this channel to capture cpan-characteristies during training.
be used to break ties prefer between spans that have close
--1 ifi<j
span ch
j
ifj
(6)
5
ifj-iss. 5.
span.ch is concatenated to span_im increasing its depth to D + 1.
3 EXPERIMENTS
3.1
IMPLEMENTATION DETAILS
In multi-span QA tasks, we compare the standard BERT-QA model by Devlin et al. (2019). which has separate output layer for start and end index prediction, against the following variants of the span-image architecture, which described in Section 2.2
bert-qa: The BERT base model that is available from Transformers (Wlf et al. 2019 library bert-base-uncased.
4

---

Under review as a conference paper at ICLR 2020
bert-ms-sigmoid: Consists of the BERT base model augmented by the span-image net- work, which involves two 2D convolution layers with 100 and 50 filters, respectively. Both layers use 3x3 filters. The output layer has sigmoid activation span-image pixels to enable multiple span predictions.
base-ms-softmax: Replaces the sigmoid activation of the bert-ms-sigmoid with softmax activation. Softmax serves useful regularization when task dictates answer (no- answer counts as "null" answer).
bert-ms-sigmoid-sl. bert-ms-sigmoid model with span-length indicator channel concate- nated to the span-image.
bert-ms-softmax--e bert-ms-softmax model with span-length indicator channel concate- nated to the span-image.
3.2 SQUAD
The Stanford Question Answering Dataset (SQuAD v1.1) is collection 100,000+ crowd-sourced question/answer pairs (Rajpurkar et al. 2016). The SQuAD v2.0 task extends the SQuAD 1.1 prob- lem with question/answer pairs, in which there may be no answer to the question. We use SQuAD v2.0 in all our experiments. This makes the QA task more realistic and challenging.
3.3 INTERNAL AMAZON MULTI-ANSWER DATASET
Consumers use the total quantity information of product to compare its value offer against similar products. To provide Amazon customers with accurate quantity information, we formulated the quantity extraction as a multi-span question answering problem where the question is "what the total quantity of this item in terms of its unit of measure?". ?", and the context is the textual item description provided by sellers. The unit of measure can volume (e.g, liquid detergent), weight (e.g., powddr detergent), or count (e.g., number of loads) depending on the product type. To compute the total quantity correctly, text and multiplied. For example, if the product title is "Original Roast Ground Coffee K Cups, Caffeinated, 36 ct 12.4 oz Box, Pack of 2",, then the applicable question is "what is the count?". and the total quantity is "36 2 72". Our dataset consists of manual labels accumulated in time. We split 80,000 out of 450,000 labels into a test set. The average number of answer spans in our dataset 1.8.
relevant quantities need to be extracted from the seller-provided
3.4 RESULTS
The performance metrics fine-tuning BERT with SQuAD v2.0 dataset given in Table All models perform similarly with the exception of bert-ms-sigmoid. This is expected since SQUAD 2.0 dataset has only answer, the softmax activation exploits this information by its inherent normalization, which can be seen as projection to the single-answer constraint.
Table 1: SQuAD v2.0 results for top answer prediction HasAns-Fl HasAms-total NoAns-EMt
Model Name
EM
F1
total
HasAns-EM
NoAns-F1
NoAns-total
bert-qa
63.6
67.2
11873
63.2
70.2
5928
64.1
64.1
5945
ben--ms-oftmaa
64.0
66.8
11873
63.3
5928
64.7
5945
her-mss-oftmaax-
64,9
67.4
11873
62.5
67.5
5928
67.3
67.3
5945
bert-ms sigmoidd
59.7
62.3
11873
59.4
64.6
5928
60.0
60.0
5945
Since span-image architecture enables BERT to generate multi-span answers, we use top-K accuracy to compare bert-qa with merttms--offmax-s/ for K=1,3,5, and 10. As shown in Table 2 both models perform similarly for K=1, but bert-ms-softmax-s/ model significantly performs better in all metrics for
3.4.1 AMAZON INTERNAL MULTI-SPAN ANSWER DATASET
Using top-K accuracy is a proxy to capture answer quality on SQuAD when multiple answers are output by the models. Our internal quantity dataset is true multi-span answer dataset that
5

---

Under review as a conference paper at ICLR 2020
Table 2: SQuAD results using top-K answers
Row Labels
EM
F1
HasAns-FM
HasAns-F1
NoAns-FM
NoAns-F1
K=1
bert-qa
63.24=00.23
66.71=00.25
63.60:+..25
70.54=0.20
62.89=0.7
62.89=0.7
bert-ms-softmax-sl
64.12:10.37
66.71-10.33
62.90:10.48
68.07=0.488
65.35-10.49
65.35-10.49
K=3
bert-qa
80.44-+0.14
83.08-00.18
83.05-+0.00
88.34+0.009
77.83+0.27
77.83+00.27
bert-ms-softmmx-sl
88.42.10.28
90.53:10.19
86.04:0.49
90.25::0.27
90.80:10.39
90.80:10.39
K=5
bert-qa
85,39110.18
87.5320.18
87.23
91.51=0.07
83.56-10.29
83.56.10.29
bert-ms-softmax-sl
93.44=0.16
94.16+0.15
90.74+0.14
0.08
9..13+0.224
96.13+0.244
K=10
beri-qa
90.45=10.11
92.09=10.07
90.93:1.114
94.23-00.06
89.96:10.08
89.96=10.08
96.81=0.110
97.83=10.06
94.63:1.188
96.66=0.112
98.98-0.09
98.98-10.99
fine-tune BERT with. We measure performances multi-span answer prediction (i.e., whether all relevant information to compute the total quantity has been extracted or not). On this task, since the span count be any number, compare bert-ms-sigmant and bert-ms- sigmoid-sl with bert-qa. While both of the span-image models perform better, porr-ms-sigmoid-1 performs best. Making the QA model aware of span-length leads to small improvements.
Table Question-answer pairs in the multi-answer dataset
Paragraph
Question
Answers [5]
Blackstrap Molasses Yummmy 5 Lbs, Kosher Certified, BPA free container, All Natural, Unsulfured Sale by weight
What is the weight?
Sky Organics Grapeseed Oil 100% Pure, Natural & Cold-Pressed Ideal for Massage, Cooking and Aromatherapy- Rich in Vitamin A, E and K- Helps Reduce Wrinkles, 8 oz (Pack 2) (Packaging may vary)
What is the volume?
[8, 2]
XUAN YUAN Fabric Shaver- Wool Clothes Pilling Trim Rechargeable Shaving Hair Removal Artifact Cleaner Tick Suction Household Ball Machine Fabric razor (Size Spare
How many there?
cutter head x1)
Table Multi-Answer Dataset Results HasAns-total
Model Name
EM
total
HasAns-EM EM HasAns-Fi
NoAns-EM
NoAns-FI
NoAns-total
bert-qa
79.2
85.0
35728
64.4
76.3
17510
93.5
93.5
18218
bert ms sigmoid
88.9
90.3
35728
84.9
87.6
17510
92.8
92.8
18218
hirt-ms-sigmoid-sl
89.1
W1.4
35728
85.0
87.6
17510
93.0
93.0
18218
To improve bert-qa's multi-span prediction performance, we introduce a post-processing step where weighted penalty term for the span length is added to start/end index probabilities. Results for different span-length penalty weights (A's) are given in Table 5 Span-length penalization improves the performance of berl-qa, but it still performs worse than span-image models when answer is present in the input text.
4 CONCLUSION
QA models can answer complex questions by predicting a span in given paragraph context. To associate a question and a context, large language-eprrseenatto models, which can be trained on big corpora, are utilized. As the performance of QA models improves, the need for more realistic scenarios grows. In this work, we propose span-image network to predict multiple spans of an answer. We measure its performance using top-K accuracy SQuAD and exact match of all spans on an internal Amazon multi-span QA dataset. While performing similarly on top accuracy, span- image network significantly outperforms separable prediction for K >

---

Under review as a conference paper at ICLR 2020
Table 5: Multi-answer dataset results using penalty weight À
Model
EM
HasAns-
HasAms-F1
HusAns-total
EM
NoAns-FI
NoAms-total
cer-hhae-uncased
79.2
76.3
93.5
bert aase-uncased
5
81.9
86.8
35728
69.8
79.9
17510
93.5
93.5
18218
bert-has-uncased
87.3
89.7
85.8
17510
93.5
93.5
18218
bert base uncased
20
88.5
90.1
35728
83.3
86.5
17510
93.5
93.5
18218
bert-haseuncased
25
87.5
89.2
84.7
17510
93.5
93.5
18218
bert base uncased
30
86.8
88.6
35728
79.8
83.4
17510
93.5
93.5
18218
REFERENCES
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACI. HLT 2019 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the Conference, 14117-41866 2019.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP 2017 Conference Empirical Methods in Natural Language Processing, Proceedings, pp. 2021 1-311. Association for Computational Linguistics (ACL), 2017. ISBN 9781945626838. doi: 10.18653/v1/117-122
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 1wo.ii::70..0335 03.55, 2017.
Liang Pang. Yanyan Lan, Jiafeng Guo, Jun Xu, Lixin Su, and Xueqi Cheng. HAS-QA: Hierar- chical Answer Spans Model for Open-Domain Question Answering. Proceedings of the AAAI Conference Artificial Intelligence, 33(01))6875-688) jul 2019. ISSN 2159-5399. doi:
10011699aaaai.33311.333.111 w33.03.331111117.777 URL www. aaai. org
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuad: 100,000+ questions for machine comprehension of text. EMNLP 2016 Conference Empirical Methods in Natural Language Processing, Proceedings, (ii):2333-23 2392,, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. ACL 2018 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), 2:784-789, 2018. doi: 10. 10.18653/v1/p 1/182/1214/11114
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. CoRR, abs/ 161 1.01603, 2016. URL /ttpatt//axxxi..rrr
abs 1611 01603
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. Technical report, 2016. URL nuba. com/ NewsQA.
https::/dataset:
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. Making Neural QA as Simple as Possible but not Simpler. CoNLL 2017 21st Conference Computational Natural Language Learning, Proceedings, pp. 271-280, mar 2017. URLhttt:://axiv. org. abs//1033 04816
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, abs/1910.03771, 2019,
Yi Yang, Wen-tau Yih, and Christopher Meek. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 2013-2018, 2015.
7


Reviews:
[{"review_id": "rO6dl5aZwid", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural model called Span-Image Network that allows the model to output multiple spans as answers (existing BERT-based QA models usually only output two endpoints of the answer span). The model is evaluated on SQuAD (top-K answer prediction) and an internal Amazon dataset.\n\nUnfortunately, it is clear that this paper doesn’t meet the standard of the ICLR publications. I can’t recommend the acceptance of the paper.\n\nFirst of all, it is mentioned in the paper that “To the best of our knowledge, a multi-span QA architecture has not been proposed.” This is certainly incorrect. There have been many models proposed recently to tackle the multi-span QA problemm.  Some examples include:\n\n- (Hu et al., EMNLP 2019): A Multi-type Multi-span Network for Reading Comprehension that Requires Discrete Reasoning\n- (Segal et al, EMNLP 2020): A Simple and Effective Model for Answering Multi-span Questions\n- (Andor et al., EMNLP 2019): Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension => This one is not straightforward but they support merging single spans at the end\n\nIt is very natural to cast a multi-span QA problem as a sequence tagging task and this paper doesn’t have such a baseline to compare with.\n\nSecond, I think the model is not motivated well and the writing of the paper needs to be improved. Why is it called a span-image network? Is it because there is a CNN layer applied? What is the rationale behind that? It is really not an image of pixels. It is just an N by N matrix, and each entry is an elementwise-multiplication of the BERT hidden vectors.\n\nThird, the evaluation of the paper also should be improved. SQuAD is a single-span QA dataset and using top-K prediction looks quite artificial and unnatural. Also, the performance on SQuAD 2 is also lower than expected. For a BERT-base-uncased model, it is expected to achieve at least ~75 F1 on SQuAD 2 so even the baselines don’t seem to be right. The internal Amazon dataset lacks details so I can’t comment much on that.  There are indeed several multi-span QA datasets (e.g., DROP, Quoref, Natural Questions) and I think the paper should experiment with those datasets and compare to previous approaches. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fEzTo8YBIkR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework to inherently break the independence assumption of start-end decision making in extractive question answering models. The general idea is to expand the current point-to-point product to a 3-D convolution. Since each point absorbs information within the sliding window, the probability (i, j) does not solely depend on $h_i, h_j$. The basic motivation makes sense to me, but I still have the following concerns about the paper:\n1) the proposed convolutional network though incorporates more information, it's constrained to be the nearest information in its sliding window, let's say 2 or 3, and the probability of p(i, j) thus depends on $h_{i-2:i+2}, h_{j-2:j+2}$. However, I don't think this modification will help the model make a better decision since the nearest surrounding words' information is already well encoded in the lower-layers of the transformer. The errors the model made are mainly due to a lack of global reasoning, the more further-away context is actually a critical issue for breaking the bottleneck. The proposed span-image framework doesn't seem to touch on the core problem in the current QA datasets.\n2) the proposed method obtains similar or even worse performance on top-answer selection, this point kind of reflects my previous concern. Adding more neighboring word context does not help the model make a wiser decision, and thus the final scores remain similar to the standard BERT QA model.\n3) the results in in-house datasets are not quite convincing. I think you can add some heuristics to the BERT QA answer span selection to improve its top-k results like adding diversity, prevent overlapping, etc. I'm not sure whether the gap will still remain as significant as reported in the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S3YsLI8JsaC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new QA model based on BERT, which is called Span-Image Network. The paper first points out that previous span extraction models model independent probability of the start and the end of the span, making the extension to multi-span extraction harder. Span-Image Network model the joint probability of the start and the end by deploying a 2-D convolution, enabling multi-span extraction.\n\nI think the paper is not ready for publication for a few reasons.\n\nFirst, the limitation of independent modeling of the start and the end of the span has been reported in the literature, and there have been a few works that use joint probability instead. I will only list some of them: work in QA [1] [2] as well as QA baselines for pretrained LMs [3][4][5]; [6], which is concurrent work to thsi work, includes detailed survey. Such literature has not been mentioned or discussed in this manuscript.\n\nSecond, although “multi” span extraction is the main motivation for this work, as described in the Introduction, there is no evaluation on public dataset. Experiment on Amazon internal data is included, however, as the detailed description or the data statistic is missing, it cannot be considered as academic empirical evaluation. The only experiment on public dataset is SQuAD which does not require multi-answer extraction and the proposed model does not show superior results compared to the baseline. (The top-k experiment in the paper is not convincing - it is synthetic and does not align with the goal of predicting multiple answers.) If the authors want to include public datasets for multi-span extraction, they can consider multi-answer portions of Natural Questions dataset [7] or AmbigQA dataset [8].\n\nThird, as mentioned above, calculating alignment of the start and end position has been already incorporated in the previous work (such as [2]). The convolutional component is the one I have not seen in the previous literature - however, as there is no ablation with and without convolution, its effect is not shown in the paper.\n\nFourth, even with Amazon internal dataset, the gains are not significant compared to bert baseline with postprocessing. Based on Table 5, the baseline’s best EM is 88.5, whereas the best performance of the proposed model in Table 4 is 89.1. In fact, the baseline number in Table 4 is not the best number of the baseline, which will mislead the audience to believe that the gap is significant - another major issue of this paper.\n\nLastly, this is a minor point, but I believe “Span-Image” in the name of the model is largely misleading. There is no “image” involved in the model architecture or training. Something like “2D” or “convolution” might be a better term.\n\n\n[1] Lee et al. Learning recurrent span extractions for extractive question answering. 2016. \n[2] Seo et al. Real-time open-domain question answering with dense-sparse phrase index. 2019.\n[3] Yang et al. Xlnet: Generalized autoregressive pretraining for language understanding. 2019.\n[4] Lan et al. Albert: a lite bert for self-supervised learning of language representation. 2019.\n[5] Clark et al. Electra: pre-training text encoders as discriminators rather than generators. 2020.\n[6] Fajcik et al. Rethinking the objectives of extractive question answering. 2020.\n[7] Kwiatkowski et al. Natural questions: a benchmark for question answering research. 2019.\n[8] Min et al. Ambigqa: answering ambiguous open-domain questions. 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IAU0cnsAFB1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThe paper proposes a novel method for predicting multiple answer spans in question-answering (QA) tasks. When the Span-Image technique is applied to a base BERT model, the authors show performance gains on a single-span dataset (SQuAD) and substantial improvements on a multi-span dataset (an internal Amazon dataset). The authors propose that the method is both model-agnostic and can eliminate common post-processing steps via built-in architecture design choices.\n#### Positives\n- The authors show significant improvement in top-K accuracy on the SQuAD dataset, highlighting their technique’s ability to hone in on multiple relevant spans.\n- The authors show that the method works (either does not degrade performance or indeed improves performance) for both single-span and multi-span datasets.\n- The technique is not too complex and is model agnostic (does not require BERT models to function)\n- The paper is well-written, with clear descriptions of baseline architectures and their new Span-Image architecture and appropriate accompanying diagrams and results tables.\n\n#### Negatives\n- The authors only compare to one baseline, BERT-QA, and do not fully motivate how it is state-of-the-art on the respective datasets considered; a few other observations call into question the generalizability of the method:\n    - The authors claim that the technique is modular and can be applied to other architectures, however only one architecture is considered.\n    - The authors also claim that the method is amenable to specifically designed channels to eliminate post-processing, however only one such channel is considered.\n- A central claim is that the method works better for multi-span question answering, however the authors only consider one true multi-span dataset (an internal dataset). There are other multi-span “reading comprehension” datasets in the literature (DROP[1], Quoref[2]) that would probably have been appropriate datasets to which the authors could have applied their technique.\n- The significant performance gains of the authors’ technique on the multi-span dataset are mitigated almost entirely by adding an additional length penalty to the base BERT-QA model, raising the question about whether the authors’ technique itself is what effects better performance (rather than, say, the explicit length penalty channel in the architecture)\n\n#### Decision\nI think the paper is marginally below the acceptance threshold. The method is novel and explainable, and appears to improve performance on the two datasets over the BERT QA baseline. However, the method is not compared to other QA architectures (is BERT QA SOTA on SQuAD?), the gains do not appear to be entirely significant (especially on the multi-span dataset, where the performance gap is reduced significantly by adding a length penalty to the baseline model), and various proposed contributions are not expanded upon (e.g. the modularity of the architecture is not demonstrated). Ultimately, the paper could be improved by further exploration of the method, whether via application to new datasets or more in depth analysis of the generalizability of the method.\n\n#### Questions\nMostly curious about the author’s responses to the questions posited above, and in addition:\n1. Did the authors consider the RC datasets specified above (DROP, Quoref), or is there a reason they were omitted.\n2. Did the authors try applying their method to other DNN architectures?\n3. Did the authors consider other image channels to eliminate post-processing?\n4. Did the authors try adding a specific span length penalty term to the Span-Image network as well? \n\n#### Cites\n[1] Dua, M. (2019). DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 2368–2378). Association for Computational Linguistics.\n[2] Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, & Matt Gardner (2019). Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning. In Proc. of EMNLP-IJCNLP.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is not ready for a publication at ICLR, as agreed unanimously by the reviewers. 

There are three main reasons for that:
1. Novelty: it is mentioned in the paper that “To the best of our knowledge, a multi-span QA architecture has not been proposed", which is certainly incorrect. See the multiple references provided by the reviewers.
2. Evaluation: there is no evaluation in multi-span setting on a public dataset. SQuAD being single span, As stated by R1, "Experiment on Amazon internal data is included, however, as the detailed description or the data statistic is missing, it cannot be considered as academic empirical evaluation." Several public datasets could be used like DROP, Quoref, or Natural Questions.
3. Motivation: the reviewers also note that the clarity and motivation behind the work could be improved. Some choices of the architecture or the model should be more clearly justified.

We encourage the authors to look into the multiple comments in the reviews in order to improve the paper and the research project overall.




Decision: Reject