Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering
Authors: Xuanyu Zhang
Qing Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational Question Answering
Machine Reading Comprehension
Attention

Abstract: Conversational question answering (CQA) is a more complicated task than traditional single-turn machine reading comprehension (MRC). Different from large language models (LLMs) like ChatGPT, the models of CQA need to extract answers from given contents to answer follow-up questions according to conversation history. In this paper, we propose a novel architecture, i.e., Gated Recurrent Memory Reader (GRMR), which integrates traditional extractive MRC models into a generalized sequence-to-sequence framework. After the passage is encoded, the decoder will generate the extractive answers turn by turn. Different from previous models that concatenate the previous questions and answers as context superficially and redundantly, our model can use less storage space and consider historical memory deeply and selectively. Experiments on the Conversational Question Answering (CoQA) dataset show that our model achieves comparable results to most models with the least space occupancy.

Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question Answering
Xuanyu Zhang
Qing Yang
Du Xiaoman Financial
Du Xiaoman Financial
Beijing, China
Beijing, China
Zhangxuanyu@dduxiaomann com
yangqingeduxiaaomn.. com
Abstract
carrying something?
What?
the door?
Passage:
Conversational question answering (CQA) is more complicated task than traditional single- turn machine reading comprehension (MRC). Different from large language models (LLMs) like ChatGPT, the models of CQA need to extract answers from given contents answer follow-up questions according to conversation history. In this paper, we propose novel architecture, i.e., Gated Recurrent Memory Reader (GRMR), which integrates traditional extractive MRC models into a generalized seneenccet--squenee framework. After the passage encoded, the decoder will generate the extractive answers turn by turn. Different from previous models that concatenate the previous questions and answers as context superficially and redundantly, our model can use less storage space and consider historical memory deeply and selectively. Experiments the Conversational Question Answering (CoQA) dataset show that our model achieves comparable results to most models with least space occupancy.
My rings. On the step, find elderly lady, small and slight, holding hand little boy. other hand, she holds cerrier beg .....
+C*
the elderly lody
Yes
paper carrier bag
Figure 1: An example from CoQA.
(Reddy et al., 2018), QuAC (Choi et al., 2018) and QBLink (Elgohary et al.. 2018), are proposed. Generally, a dialogue contains long passage and some short questions about this passage. The current question may rely previous questions answers. Here is an example from CoQA dataset in Figure 1. Only if know the conversation history can we understand that the third question "What?" represents "What is she carrying?".
However, most previous approaches view this task as a traditional single-turn MRC task by concatenating previous questions or answers as conversation history superficially, such as BiDAF++ (Yatskar, 2018), DrQA+PGNet (Reddy et al., 2018), SDNet (Zhu et al., 2018) and so on. They can not grasp and understand the representation of history profoundly. And multiple historical questions and answers in one sentence may confuse the model.
1 Introduction
Recently, large language models (LLMs) like ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) have revolutionized the question-answeriig and conversation domains, pushing them to new heights. Different from ChatGPT-style conversations, the task of conversational question answering (CQA) requires models to answer follow-up questions using extracted answers based on given passages and conversation history. It can be regarded as an expansion of traditional single- turn machine reading comprehension (MRC) to multi-turn conversations. However, the follow- up questions usually have more complicated phenomena, such as co-reference, ellipsis and so on. is necessary to consider historical memory in a conversation. To enable machines to answer such questions, many CQA datasets, such as CoQA
Besides, these methods occupy lot of storage space precious and limited graphics memory because of duplicated questions and passages. Although, FlowQA (Huang al., 2018) proposes a flow mechanism without concatenating previous questions, the hidden states of the passage are still duplicated many times to obtain the question-aware passage for each question. Meanwhile, it can not utilize dialogue history selectively.
To address these issues, we propose our new

---

architecture, i.e., Gated Recurrent Memory Reader (GRMR). integrates traditional MRC models into a generalized sequence-to-qque (Sutskever et al., 2014) framework to generate extractive answers. For one thing, the deeper representation of historical conversation can be utilized selectively by our recurrent structure and gated mechanism. For another thing, our model uses original questions and passages without redundancy, which saves lots of storage and memory. Experiments on the Conversational Question Answering (CoQA) dataset show that our model achieves comparable results to most models with the least space.
However, regardless of the method used, the previous dialogue history needs be encoded and interacted with each turn of the conversation. It is not possible to save the previous dialogue states and directly use them.
(seq2seq)
3 Approaches
3.1 Task Formulation
In this section, we will illustrate our model from encoder to decoder. The task of the CQA can be formulated as follows. Suppose we are given a conversation, which contains passage with n tokens P = {wf | and multi-turn questions with question answering turns Q = [Q+)i=1- The r-th questions with m tokens is Qr {w))) )m And the model requires to give the corresponding answer Ar. A conversation in the dataset can be considered tuple (P, (.Q.A))
2 Related Work
In the field of generative conversations, large language models have become the prevailing approach. These models (OpenAI, 2022, 2023; Zhang and Yang, 2023b) are typically pretrained using vast amounts of unsupervised text data and subsequently fine-tuned using supervised instruction data. This supervised instruction data is often obtained through human annotations or distilled from existing large-scale models (Zhang and Yang, 2023a).
3.2 Gated Recurrent Memory Reader
As shown in Figure 2, we use a generalized seq2seq framework to solve the task of CQA. As know, the traditional seq2seq structure is used for many tasks, such machine translation, semantic parsing, and so on. Given sentence, the encoder will transform the input to the intermediate representation, which used by the decoder to generate new sentence word by word.
In the field of extractive conversations, the answers are typically extracted directly from the original passages by the model. This task is often seen as an extension of single-turn machine reading comprehension to multi-turn conversations. Prior to the era of pretraining, attention mechanisms were commonly employed in various subdomains of question answering, such as BiDAF (Seo al., 2017) and Rception (Zhang and Wang, 2020) in classic single-turn machine reading comprehension, BiDAF++ (Yatskar, 2018) in conversational machine reading comprehension, and other methods based on multi-modal or structured knowledge question answering (Zhang, 2020; Zhang and Yang, 2021b).
Similar to the seq2seq framework, our model also consists two modules: passage encoder and question decoder. the passage encoder module designed to generate the hidden representation according to the passage. And question decoder module is for generating extractive answers turn by turn according to the result of the encoder.
There are also many differences between our model and seq2seq. First, the basic unit of model is sentences rather than words. The intermediate representation (generated by the passage encoder) of our model contains the hidden states tokens in the passage. The input and the output of the decoder also sentences. And the parallelism of our model is between conversations rather than sentences in batch. Second, the output length of the question decoder depends on the turns of questions in conversation. There is no start flag "(GO)" end flag "(EOS)" in the decoder. Third, the output of the decoder, i.e., the answer, is not fed the input of the next turn. Because the hidden states of the answer in the memory our model.
Subsequently, pre-trained models based on Transformer (Vaswani et al., 2017) like ELMo (Peters et al., 2018) or BERT (Devlin et al., 2018) are employed in a wide range of natural language processing tasks (Zhang and Yang, 2021a; Zhang et al., 2023). In question answering tasks, these pre-trained models are either combined with designed attention structures through embedding- style (Zhu et al., 2018; Zhang, 2019) or directly fine-tuned using concatenated historical dialogues.

---

Our model can be formulated in Eq. 1. The decoder takes the result of the encoder P Senc( and generates extractive answers according to questions.
information between the origin passage ef and the memory of the passage cf- inspired by memory network(Kumar et al., 2016). The memory ci-1 is obtained from the previous question turn. The r-th history-aware passage of can be obtained as follows:
P, Q) =
II
Po mondel A5|Qss,
j=1
sum = [ 1
poum, Q
II j=1
Po Dinaader (A,1Quj: P)
9r o (w2 T tamh (wg ouum)
(3)
& = g- + gr
where Qij [Q1, -,.... (1.2..... s). And P, Q, is the embedding result of P. P.Q, A, respectively, For one conversation with question answering turns, P Rlxxxx Q C R and A E R RaxmxA, where denotes the dimension of the embedding.
where c- is also obtained from the previous question turn. They will be interpreted in the next layer.
Specially, for the first question turn of the conversation, directly use the intermediate representation, which is generated by the passage encoder, i.e., cp- =
3.3 Passage Encoder Module
Interaction Layer The passage and the r-th question will interact this layer. First, we obtain the exact match feature ematch, which indicate whether the token wf appears in the question Qr Q ( ( in the form of prototype, lowercase and lemma. Then another match feature ematehh can be obtained by attention mechanism:
This module aims encode the words of the passage into latent semantic representation, which will used in the question decoder module. First, we obtain the embedding of the passage GIV by the pre-trained Glove (Pennington et al., 2014). The part-of-speech (POS) and named entity recognition (NER) tags of each word are also transformed to the embedding vectors ef .OS and e; e,NE, respectively. They are learned during training. Then we concatenate them to a vector and feed it into the bi-directional recurrent neural network (RNN) to generate an intermediate representation of the passage ef" BiR L.Te, :e eePOS,NERR)) NERR).
m
smotch Crt i=1
a Crs
att x eaple Q cf) (4)
We concatenate the information above for the passage and fed them bi-directional RNN generate the question- aware passage representation &ht = BiRNN ( [cAs amatch, amptch)).
Then another attention is used to fuse the question and the passage as follows:
3.4 Question Decoder Module
This module is the core of our model. We take a turn one conversation as an example to illustrate this module. And "turn" in our model is similar to "step" in RNN.
att Fet
a
Q Cra:
Q @ a a explc E.))
(5)
i=1
After that, we refine the representation of the passage by bi-directional RNN, i.e., 2., BiRNN(E, 1: 1.
Question Input Layer Suppose the current question is the r-th question, we encode the embedding of the question (cf)) into ( with bi-directional RNN. We can obtain vector of the question by weighted sum of tokens in Eq. 2. And w; represents different trainable weights in this section.
Meanwhile, self- attention is also used to enhance the representation in Eq. 6. Then a bi-directional RNN integrates the representation above and generate the new cf in Eq. 7, which will be used by next turn in Eq. 3.
LiteP ar Cr.lt
(6)
gsarn
o x cxp(wiTe.) (2)
atd exp((hott, L.: S.. of [hott, cfi
Gated Memory Layer (Passage) In this layer, we use a gated mechanism to leverage the
P = ( sels n 2 (7)

---

Model
Embedding
Passage text
Seq2seq
27.5
RNN
Passage Encoder
PGNet
45.4
Intermediate
2-rd Question Text
DrQA
54.7
SDNet (w/o BERT lock weights)
64.2
Passage
DrQA+PGNet
66.2
Gated Memory
Embedding
GRMR
66.3
RNN
RNN
FlowQA (with ELMo)
76.2
Attention
RNN
Question Input
SDNet (with BERT)
77.7
Self-Attention
MC² (with BERT)
81.3
Question
Gated Memory
Table The performance on CoQA dev set.
Answer Layer
1-st
2-rd question
Question Decoder
Configuration
F1
A Fl
GRMR
66.3
Figure 2: Overview of our model. (Best see colors) (The red solid line refers to the memory flow of the passage. The red dashed line refers to the memory flow of the questions.)
w/o gated memory (question)
66.1
-0.2
w/o gated mechanism (passage)
64.2
-2.1
w/o gated memory (passage)
61.8
-4.5
w/o gated memory (all)
58.8
-7.5
Lastly, we can get the representation of the passage G by weighted sum of tokens like Eq. 2. It will be used both on gated memory layer in the next turn (in Eq. 3) and on gated memory layer for the current question (in Eq. 8). And the answer layer also uses the representation. Gated Memory Layer (Question) As shown in Eq. 8, another gated memory is used to leverage the information of the current question agaum and the previous question memory h, where agsum is processed by RNN cell, i.e.,
Table Ablation studies on the CoQA dev set.
by another bilincar function. We then concatenate the sentence-level representation of the passage and fed it to the linear classification T wig pown, Aagum for unanswerable questions.
Experiments
4.1 Dataset and Metric
ogoum (h ) Specialy, for
We the CoQA (Reddy al., 2018) evaluation dataset. It a large-scale conversational question answering dataset notated by people. It contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The text passages in the dataset are collected from seven diverse domains. And use the F1 as the metric the official evaluation.
the first question turn, use the representation of
the current question as the question memory, i.e., 1 gsuu
ef gsum Then 1 iu will be used in the answer layer and the next turn.
Gr cum = (c Aauum
9r Now]l[too tanh wa dr ))
(8)
gasum gr 295am + (1 gr) 9r) g
Answer Layer This layer is the top layer of the question decoder module. Following pointer network (Vinyals et al., 2015) and DrQA (Chen al., 2017), we use the bilinear function [xx y) = xWy to compute the probabilities of each token being start and end.
4.2 Implementation Details
We use the Adamax (Kingma and Ba, 2014) as our optimizer. The initial learning rate is 0.004, and is halved after 10 epochs. To prevent overfitting, set dropout to 0.4. The dimension of word embedding 300, which is fixed during training. And the embedding dimension of POS and NER are set to 12 and separately. we use LSTM our recurrent neural network. The hidden size LSTM 128 throughout our model.
pi x exppffalee (sum))
As shown in Eq. 9, we can obtain the start probability of each token of in passage. And the end probability of also can be obtained like this

---

O-ctx
1-ctx
2-ctx
3-ctx
References
(Kai
47854
183853
188960
193675
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia answer open- domain questions. arXiv preprint arXiv: 704,00051
multiple
1x
3.84x
3.95x
4.05x
dev(KB)
8878
13008
13379
13723
multiple
1x
1.47x
1.51x
1.55x
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quae: Question answering context. arXiv preprint 1 1808.070366
Table 3: Space occupancy on the CoQA.
(N-ctx refers to using previous QA pairs)
4.3 Result
Jacob Devlin, Ming-We Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810004805.
As shown in Table 1, we compare our method with other baseline models: PGNet(Seq2Seq with copy mechanism)(See et al., 2017), DrQA (Chen et al., 2017), DrQA+PGNet (Reddy et al., 2018), BiDAF++ (Yatskar, 2018), FlowQA (Huang et al.. 2018), SDNet (Zhu et al., 2018) and MC² (Zhang, 2019). GRMR achieves comparable results to most models on the dev set, except FlowQA and SDNet. These two models all use the contextualized embedding, ELMo (Peters et al., 2018) or BERT (Devlin et al., 2018). However, our model still outperforms SDNet without BERT weights.
Ahmed Elgohary, Chen Zhao, and Jordan Boyd-Graber 2018. dataset and baselines for sequential open- domain question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 1077-1083.
Hsin- Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2018. Flowqa: Grasping flow in history for conversational machine comprehension. arXiv preprint arXiv: 1810.0668,
Diederik P Kingma and Jimmy Ba. 2014, Adam: A method for stochastic optimization. arXiv preprint
We also conduct ablation studies for our model Table 2. We can find both the gated mechanism in memory and gated memory are crucial to our architecture. The score drops a lot without all gated memories. Different from the passage, the effect of the gated mechanism in question memory is not included, because model can only use the current question without gate.
wrXiv:1412.69800
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit lyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference Machine Learning, pages 1378-1387.
OpenAl. 2022. Chatgpt.
Lastly, we compare the storage space occupancy between our model and others on the CoQA dataset in Table 3. Our model takes up the least space with 0-ctx. Other models usually append 2 or historical conversation turns to the current question. For the words in the training dataset, we can observe that the space they used is about four times larger than ours. And the difference will be larger when words are converted to vectors.
OpenAl. 2023. Gpt-4 technical report.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for
word representation, In Proceedings of the 2014 conference on empiricul methods in natural language processing (EMNLP), pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint 180. 033655.
5 Conclusion
Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint 1808v:1808.00 07042.
We propose a novel structure, GRMR, for conversational question answering. Traditional extractive MRC models are integrated into a generalized sequence-to- to-suueee--- framework. Gated mechanism and recurrent memory enable the model consider the latent semanties of conversation history selectively and deeply with less space. The experiments show that this is a successful attempt to integrate extraction and generation in conversational question answering.
Abigail See, Peter Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint 1704.:1704.04436
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. International Conference on Learning Representations.

---

Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
32nd ACM International Conference on Information and Knowledge Management, CIKM '23, page 4435-4439, New York, NY, USA. Association for Computing Machinery.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812..35593
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692-2700.
Mark Yatskar. 2018. A qualitative comparison of coqa, squad 2.0 and quac. arXiv preprint
arXiv:1809. 10735.
Xuanyu Zhang. 2019. MC²2: Multi Multi-perspective convolutional cube for conversational machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6185-6190, Florence, Italy. Association for Computational Linguistics.
Xuanyu Zhang. 2020. Cigan: Cross flow graph neural
networks for question answering complex tables. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9596- 9603.
Xuanyu Zhang. Zhepeng Lv, and Qing Yang. 2023. Adaptive attention for sparse-based sequene transformer. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 8602-8610, Toronto, Canada. Association for Computational Linguistics.
Xuanyu Zhang and Zhichun Wang. 2020. Reeption: Wide and deep interaction networks for machine reading comprehension (student abstract), Proceedings of the AAAI Conference on Artificial Intelligence, 34(10):1987713998
Xuanyu Zhang and Qing Yang. 2021a. Dml: Dynamic multi craltiiraaularityy learning for bert-based document reranking. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, page 3642-366 -366,, New York, NY, USA. Association for Computing Machinery.
Xuanyu Zhang and Qing Yang. 2021b. Position- augmented transformers with entity-aligned mesh for textvqa. Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 2519-2528, New York, NY, USA. Association for Computing Machinery.
Xuanyu Zhang and Qing Yang. 2023a. Self-qa: Unsupervised knowledge guided language model alignment.
Xuanyu Zhang and Qing Yang. 2023b. Xuanyuan 2.0: large chinese financial chat model with hundreds Proceedings of the of billions parameters.


Reviews:
[{"review_id": "DVFOJvJa6W", "reviewer": "Reviewer_G5eK", "summary": "", "questions": "A - In line 115, the authors mentioned \"the output of the decoder, i.e.,the answer, is not fed to the input of the next turn\". Won't this lead to problems if the subsequent question is referring to the preceding answer string?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Achieves 10 points less F1 than the standard BERT model architecture. I am assuming that the performance gap will be even higher when using more recent and better-pretrained models such as RoBERTa, GPTs, LLAMAs etc. Lacking memory overhead comparisons between baselines.\n\nThank you for your comments. Incorporating large-scale pretrained models can indeed improve model performance. However, the main contributions and innovations of our paper revolve around integrating extractive models into generative models, achieving reduced space occupancy in multi-turn conversations. Our goal was not to simply integrate and rely on existing LLMs for achieving state-of-the-art performance.\n\nMoreover, due to structural differences, many LLMs cannot be directly integrated and would require retraining with massive amounts of data. Our proposed approach offers a powerful and influential idea that can be extended to other models and tasks.\nAdditionally, it's worth noting that the memory usage of integrating pretrained language models is significantly higher compared to our model (approximately 50x-30000x higher).\n\n> In line 115, the authors mentioned \"the output of the decoder, i.e.,the answer, is not fed to the input of the next turn\". Won't this lead to problems if the subsequent question is referring to the preceding answer string?\n\nThis is a great question. We have actually conducted experiments on whether to include the output of each round as input for the next round, and overall, the results show that not including the output yields better performance. Intuitively, this can be seen as a question of explicit versus implicit information transfer. Dense intermediate vectors can effectively carry forward the implicit states from previous rounds.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- new memory efficient architecture for Conversational QA task", "reasons_to_reject": "- Achieves 10 points less F1 than the standard BERT model architecture. I am assuming that the performance gap will be even higher when using more recent and better-pretrained models such as RoBERTa, GPTs, LLAMAs etc.\n- Lacking memory overhead comparisons between baselines (there should be a table in the appendix at least)", "questions_to_authors": "A - In line 115, the authors mentioned \"the output of the decoder, i.e.,the answer, is not fed to the input of the next turn\". Won't this lead to problems if the subsequent question is referring to the preceding answer string?", "paper_topic_and_main_contributions": "The authors propose a new RNN architecture for Conversational Question Answering tasks. Their model comprises 2 modules, 1) Passage encoder, 2) Question decoder. The question decoder contains a question-passage attention, self-attention, and Gated Memory Layer to encoder sequences of conversational questions in a memory-efficient way. Experiments on CoQA dataset show that their model beats most of the baselines except for models using contextualized embeddings such as Elmo and BERT.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rWsKkyGjcx", "reviewer": "Reviewer_wcpV", "summary": "", "questions": "* Could you please update Table-3 to make it more granular, for example can you add separate rows for FlowQA and SDNet.\n* A plot comparing trade-off in space vs F1 of GRMR vs best performing models would be very informative.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The F1 Score obtained compared to other models, such as FlowQA and SDNet, on CoQA dataset has a wide gap.\n\nThank you for your question. In Table 1, we included FlowQA and SDNet as references to avoid missing important comparisons. However, it is not a fair comparison as FlowQA and SDNet are both based on existing pretrained models. Our proposed GRMR model has a significantly smaller volume compared to these models and was not pretrained on massive corpora.\nThe main contributions and innovations of our paper lie in integrating extractive MRC models with a gated recurrent memory mechanism and achieving better space occupancy in the context of conversational question answering. Our goal was not to achieve state-of-the-art performance by leveraging existing LLMs and surpassing other models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Use of Gated Recurrent Memory Reader (GRMR) method for CQA seems to save space but w/ drop in F1 compared to best performing methods. ", "reasons_to_reject": "* The F1 Score obtained compared to other models, such as FlowQA and SDNet, on CoQA dataset has a wide gap.\n* The improvement in space with 0-ctx doesn't seem proportionate to trade-off in F1.", "questions_to_authors": "* Could you please update Table-3 to make it more granular, for example can you add separate rows for FlowQA and SDNet.\n* A plot comparing trade-off in space vs F1 of GRMR vs best performing models would be very informative.", "paper_topic_and_main_contributions": "Authors present Gated Recurrent Memory Reader (GRMR) which which integrates traditional extractive Machine Reading models into a generalized sequence-to-sequence framework. Authors show they achieve better space usage compared to several other models on CoQA dataset, but reports a wide gap with best performing models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following reference/s also contain passage and answer with multi-turn questions.\n* Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534."}}, {"review_id": "LuKErfR4Em", "reviewer": "Reviewer_3NnP", "summary": "", "questions": "Why not choose to use LLMs as base models?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> (1) The baselines involved in Table 1 are generally traditional non-LLM baselines before 2020, which are kind of out of date. (2) Two LLM-equipped baselines in Table 1 are FlowQA (with ELMo) and SDNet (with BERT), which show quite better performance gaps beyond proposed GRMR model. What's more, the authors do not have a clear explanation on the improvement of space cost.\n\n> Why not choose to use LLMs as base models?\n\nThank you for your question. The main focus of our paper was to propose a novel approach that integrates extractive MRC models with a gated recurrent memory mechanism for conversational question answering. Our goal was not to achieve state-of-the-art performance by leveraging existing LLMs, but rather to address the challenge of reducing space occupancy while maintaining competitive performance.\n\nDifferent from models based on the Transformer framework, including encoder (BERT), decoder (GPT), encoder-decoder (T5), our model adopts a structure similar to RNNs as the main framework. Training such a novel model architecture requires significant computational resources, extensive training data, and a substantial amount of time. For individual researchers, it is often not feasible to undertake such large-scale pretraining, which is necessary for LLMs. For example, a concurrent work, RWKV, requires extensive resources and retraining based on RNN-inspired architectures. Therefore, in Table 1, we included some results from LLM-based models to avoid missing important references. However, it is not a fair comparison as our model has significantly smaller scale (1/50~1/30000) and was not pretrained on massive corpora. \n\nIn summary, the main contributions of our paper lie in the integration of extractive MRC models with a gated recurrent memory mechanism in the context of conversational question answering. We acknowledge that LLMs have become prominent in the field, and if we have the necessary resources in the future, we plan to explore their potential and conduct large-scale pretraining. We hope this explanation clarifies our approach and improves your evaluation of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A space-efficient gated memory mechanism to reduce space occupancy.", "reasons_to_reject": "(1) The baselines involved in Table 1 are generally traditional non-LLM baselines before 2020, which are kind of out of date.\n(2) Two LLM-equipped baselines in Table 1 are FlowQA (with ELMo) and SDNet (with BERT), which show quite better performance gaps beyond proposed GRMR model. What's more, the authors do not have a clear explanation on the improvement of space cost.", "questions_to_authors": "Why not choose to use LLMs as base models?", "paper_topic_and_main_contributions": "The paper prosed to incorporate Gated Recurrent Memory with traditional extractive MRC models to deal with conversational question answering task. Experiments on CoQA dataset proved the effectiveness and better space occupancy of proposed GRMR model, compared with traditional baselines such as DrQA, SDNet and PGNet.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new architecture that provides direct access to passage rep and question history for conversational QA, through a gated memory layer. This provides inference-time space savings that become increasingly significant as the conversation grows.

The new approach achieves significantly worse performance than recent architectures that use contextualized embeddings. However, those also require much more memory to run. Compared to similar base architectures, the new approach performs well although it should be noted that the closest comparison from Reddy et.al 2018 achieves essentially the same performance, in terms of accuracy.

Multiple reviewers raised questions about the significant gap in performance to more recent, Transformer-based architectures and whether the accuracy / space trade-off is reasonable, especially for the 0-ctx case. It is clear that the current focus of the paper is not just increasing the accuracy metrics but the work would benefit from a deeper discussion of these tradeoffs and / or discussion of how they could be overcome.
Decision: Accept-Findings