SlowGuess/ACL_2020_OCR huggingface dataset:
Souvik Kundu, Qian Lin, and Hwee Tou Ng
souvik@u.nus.edu, qlin@u.nus.edu, nght@comp.nus.edu.sg
Despite recent progress in conversational question answering, most prior work does not focus on follow-up questions. Practical conversational question answering systems often receive follow-up questions in an ongoing conversation, and it is crucial for a system to be able to determine whether a question is a follow-up question of the current conversation, for more effective answer finding subsequently. In this paper, we introduce a new follow-up question identification task. We propose a three-way attentive pooling network that determines the suitability of a follow-up question by capturing pair-wise interactions between the associated passage, the conversation history, and a candidate follow-up question. It enables the model to capture topic continuity and topic shift while scoring a particular candidate follow-up question. Experiments show that our proposed three-way attentive pooling network outperforms all baseline systems by significant margins.
Conversational question answering (QA) mimics the process of natural human-to-human conversation. Recently, conversational QA has gained much attention, where a system needs to answer a series of interrelated questions from an associated text passage or a structured knowledge graph (Choi et al., 2018; Reddy et al., 2019; Saha et al., 2018). However, most conversational QA tasks do not explicitly focus on requiring a model to identify the follow-up questions. A practical conversational QA system must possess the ability to understand the conversation history well, and to identify whether the current question is a follow-up of that particular conversation. Consider a user who is trying to have a conversation with a machine (e.g., Siri, Google Home, Alexa, Cortana, etc). First, the user asks a question and the machine answers it. When
Passage: ... script for Verhoeven's first American film, Flesh and Blood (1985), which starred Rutger Hauer and Jennifer Jason Leigh. Verhoeven moved to Hollywood for a wider range of opportunities in filmmaking. Working in the U.S. he made a serious change in style, directing big-budget, very violent, special-effects-heavy smashes RoboCop and Total Recall. RoboCop, for ... Verhoeven followed those successes with the equally intense and provocative Basic Instinct (1992)
received two Academy Awards nominations, for Editing and for Original Music ...
Q: What was the first film Verhoeven did in the US?
A: Flesh and Blood
Q: What genre of films did he make?
A: big-budget, very violent, special-effects-heavy smashes
Candidate follow-up question examples:
What year did his first film debut? - V
Did he make any films during his final years? - Invalid
What did she do after her debut film? – Invalid
Figure 1: Examples illustrating the follow-up question identification task.
the user asks the second question, it is very important for the machine to understand whether it is a follow-up of the first question and its answer. Further, this needs to be determined for every question posed by the user in that ongoing conversation. By identifying whether the question is a follow-up question, a machine determines whether the conversation history is relevant to the question. Based on this decision, it is expected to use a suitable answer finding strategy for answering the question. Additionally, a QA system first retrieves some relevant documents using an information retrieval (IR) engine to answer a question. If a follow-up question identifier predicts the question as an invalid follow-up question given the retrieved documents, it can communicate to the IR engine to retrieve additional supporting documents.
A few example instances are given in Figure 1 to illustrate the follow-up question identification task in a conversational reading comprehension setting.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 959-968 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We present a new dataset for learning to identify follow-up questions, namely LIF. Given a text passage as knowledge and a series of question-answer pairs as conversation history, it requires a model to identify whether a candidate follow-up question is valid or invalid. The proposed dataset requires a model to understand both topic continuity and topic shift to correctly identify a follow-up question. For instance, in the first example given in Figure 1, a model needs to capture the topic continuity from the first question-answer pair (i.e., first film is Flesh and Blood) and the topic shift from the second question-answer pair (i.e., genre of films) of the conversation history. The candidate follow-up question in the second example is invalid since the associated passage does not provide any information about his final years. The last follow-up question example is invalid since Verhoeven is a he, not she.
There has been some research in the past which focuses on identifying what part of the conversation history is important for processing follow-up questions (Bertomeu et al., 2006; Kirschner and Bernardi, 2007). However, the recently proposed neural network-based models for conversational QA have not explicitly focused on follow-up questions. In this paper, we propose a three-way attentive pooling network for follow-up question identification in a conversational reading comprehension setting. It evaluates each candidate follow-up question based on two perspectives – topic shift and topic continuity. The proposed model makes use of two attention matrices, which are conditioned over the associated passage, to capture topic shift in a follow-up question. It also relies on another attention matrix to capture topic continuity, directly from the previous question-answer pairs in the conversation history. For comparison, we have developed several strong baseline systems for follow-up question identification.
1. We propose a new task for follow-up question identification in a conversational reading comprehension setting which supports automatic evaluation.
2. We present a new dataset, namely LIF, which is derived from the recently released conversational QA dataset QuAC (Choi et al., 2018).
3. We propose a three-way attentive pooling network which aims to capture topic shift and
topic continuity for follow-up question identification. The proposed model significantly outperforms all the baseline systems.
2 Task Overview
Given a passage, a sequence of question-answer pairs in a conversation history, and a candidate follow-up question, the task is to identify whether or not the candidate follow-up question is a valid follow-up question. We denote the passage as  $\mathcal{P}$  which consists of  $T$  tokens. Let the sequence of previous questions and their corresponding answers be denoted as  $\{\mathcal{Q}_1,\mathcal{Q}_2,\ldots ,\mathcal{Q}_M\}$  and  $\{\mathcal{A}_1,\mathcal{A}_2,\dots ,\mathcal{A}_M\}$ , where  $M$  is the number of previous question-answer pairs in the conversation history. The candidate follow-up question is denoted as  $\mathcal{C}$ . We formulate this task as a binary classification task, which is to classify  $\mathcal{C}$  as valid or invalid. In the remainder of this paper, we denote the length of the candidate follow-up question as  $V$ . In our model, we concatenate all previous questions and their answers with special separator tokens as follows:  $\mathcal{Q}_1\mid \mathcal{A}_1\mid \mathcal{Q}_2\mid \mathcal{A}_2\mid \ldots \mid \mathcal{Q}_M\mid \mathcal{A}_M$ . The combined length of the previous question-answer pairs in the conversation history is denoted as  $U$ .
3 LIF Dataset
In this section, we describe how we prepared the LIF dataset, followed by an analysis of the dataset.
3.1 Data Preparation
We rely on the QuAC dataset (Choi et al., 2018) to prepare the LIF dataset. Each question in the QuAC dataset is assigned one of three categories: should ask, could ask, or should not ask a follow-up question. We construct the valid instances of the dataset using the should ask follow-up question instances. Since the test set of QuAC is hidden, we split the QuAC development set into two halves to generate the development set and the test set of LIF. The split is done at the passage level to ensure that there is no overlap in the passages used in the development and test set.
To create each instance in LIF from QuAC, we take the associated passage, the previous question-answer pairs till it says should ask a follow-up question, and the next question as the gold valid candidate follow-up question. For each instance, we sample invalid follow-up questions from two sources:
1. Questions from other conversations in QuAC which can serve as potential distractors, and
2. Non-follow-up questions from the same conversation in QuAC which occurs after the gold valid follow-up question.
The sampling from the first source involves a two-step filtering process. We first compare the cosine similarity between the associated passage and all the questions from the other conversations by using embeddings generated by InferSent (Conneau et al., 2017). We take the top 200 questions based on higher similarity scores. In the second step, we concatenate the gold valid candidate follow-up question with the question-answer pairs in the conversation history to form an augmented follow-up question. Then, we calculate the token overlap count between each ranked question obtained in the first step and the augmented follow-up question. We normalize the token overlap count by dividing it by the length of the ranked question (after removing stop words). For each valid instance, we fix a threshold and take at least one but up to two questions with the highest normalized token overlap count as invalid candidate follow-up questions.
We also introduce potential distractors from the same conversation in QuAC. We check through the remaining question-answer pairs which occur after the valid follow-up question. We tag a question as an invalid candidate if the question appears just before it is labeled with should not ask a follow-up question. Throughout the invalid question sampling process, we exclude generic follow-up questions containing keywords such as what else, any other, interesting aspects and so on, to avoid selecting follow-up questions which can be potentially valid (e.g., Any other interesting aspects about this article?).
For the training and the development sets, we combine all candidate follow-up questions from both other conversations and the same conversation. We keep three test sets with candidates from different sources: from both other conversations and the same conversation (Test-I), from other conversations only (Test-II), and from the same conversation only (Test-III). The overall dataset statistics are given in Table 1. We randomly sampled 100 invalid follow-up questions from Test-I set, and manually checked them. We verified that  $97\%$  of them are truly invalid.
LIF	Train/Dev/Test-I/Test-II/Test-III
#Instances	126,632/5,861/5,992/5,247/2,685
Avg #prev QA	3.6/3.7/3.7/3.9/3.5
Avg passage len	447.4/521.9/533.2/533.7/532.0
Avg question len	7.2/7.3/7.3/7.3/7.3
Avg answer len	16.3/15.8/15.6/15.7/15.6
Avg FUQ†len	8.8/8.4/8.4/8.6/7.4
Table 1: LIF dataset statistics. †follow-up question
3.2 Challenges of the Dataset
To identify whether a question is a valid follow-up question, a model needs the ability to capture its relevance to the associated passage and the conversation history. The model is required to identify whether the subject of the question is the same as in the associated passage or in the conversation history, which is often distracted by the introduction of pronouns (e.g., I, he, she) and possessive pronouns (e.g., my, his, her). Such resolution of pronouns is a critical aspect while determining the validity of a follow-up question. It also needs to examine whether the actions and the characteristics of the subject described in the candidate follow-up question can be logically inferred from the associated passage or the conversation history. Moreover, capturing topic continuity and topic shift is necessary to determine the validity of a follow-up question. The subjects and their actions or characteristics in the invalid follow-up questions are often mentioned in the passages, but associated with different topics.
We randomly sampled 100 invalid instances from the Test-I set, and manually analyzed them based on different properties as given in Table 2. We found that  $35\%$  of the invalid questions have identical topics as the associated passages,  $42\%$  of the questions require pronoun resolution,  $11\%$  of the questions have the same subject entity as the gold follow-up question, and  $5\%$  of the questions have the same subject entity as the last question in the conversation history. Pronouns in  $8\%$  of the invalid questions match the pronouns in the corresponding valid follow-up questions, and match the last question in the conversation history for another  $8\%$  of the cases. For  $7\%$  of the cases, the question types are the same as the valid questions, and for  $6\%$  of the cases they are the same as the last question in the conversation history. We also observed that  $4\%$  of the invalid questions mention the same actions as in the corresponding valid ones, and they are the same as the last question in the conversation
Properties	%	Example
Identical topic	35	P: ... the band released their second album ...
Q: Is “A Rush of Blood to the Head” their album name?
Pronoun resolution	42	Q: Was he the wealthiest person?
Q: Did she go to college?
Entity match (gold)	11	G: What was in the song that caused a feud?
Q: What was some of the songs on this album?
Entity match (last)	5	L: Did her writing win any awards?
Q: Did he win any awards?
Pronoun match (gold)	8	G: How many goals did he make?
Q: Was he married for many years?
Pronoun match (last)	8	L: Where was he born?
Q: Why did he live in Exile?
Q-type match (gold)	7	G: In what year did this happen?
Q: What year did he enact the reproductive health act?
Q-type match (last)	6	L: What happened after that fight?
Q: What happened in this episode?
Action match (gold)	4	G: Did he go on any tours?
Q: When did Mr Brando go to New York?
Action match (last)	3	L: When did he release?
Q: When was their next album released?
Table 2: An analysis of the LIF dataset. The percentages do not add up to  $100\%$  since many examples consist of multiple properties.  $(\tilde{\mathcal{Q}} -\text{invalid follow-up question};\mathcal{P} - \text{associated passage};\mathcal{G} - \text{gold valid follow-up question};\mathcal{L} - \text{last question in the conversation history.})$
history for  $3\%$  of the cases. The distribution of these properties shows the challenges in tackling this task.
4 Three-Way Attentive Pooling Network
In this section, we describe our proposed three-way attentive pooling network<sup>1</sup>. First, we apply an embedding layer to the associated passage, the conversation history, and the candidate follow-up question. Further, they are encoded to derive sequence-level encoding vectors. Then the proposed three-way attentive pooling network is applied to score each candidate follow-up question.
4.1 Embedding and Encoding
We use both character and word embeddings. Similar to Kim (2014), we obtain the character-level
embedding using convolutional neural networks (CNN). First, characters are embedded as vectors using a character-based lookup table, which are fed to a CNN, and whose size is the input channel size of the CNN. Then the CNN outputs are max-pooled over the entire width to obtain a fixed-size vector for each token. We use pre-trained vectors from GloVe (Pennington et al., 2014) to obtain a fixed-length word embedding vector for each token. Finally, both word and character embeddings are concatenated to obtain the final embeddings.
For encoding the conversation history and the candidate follow-up question, we use bidirectional LSTMs (Hochreiter and Schmidhuber, 1997). We represent the sequence-level encoding of the conversation history and the candidate follow-up question as  $\mathbf{Q} \in \mathbb{R}^{U \times H}$  and  $\mathbf{C} \in \mathbb{R}^{V \times H}$ , respectively, where  $H$  is the number of hidden units. Similarly, we compute the sequence-level passage encoding, resulting in  $\mathbf{D} \in \mathbb{R}^{T \times H}$ . Then a similarity matrix  $\mathbf{A} \in \mathbb{R}^{T \times U}$  is derived, where  $\mathbf{A} = \mathbf{D} \mathbf{Q}^{\top}$ .
4.1.1 Joint Encoding
We then jointly encode the passage and the conversation history. We apply a row-wise softmax function on  $\mathbf{A}$  to obtain  $\mathbf{R} \in \mathbb{R}^{T \times U}$ . Now, for all the passage words, the aggregated representation of the conversation history is given as  $\mathbf{G} = \mathbf{RQ} \in \mathbb{R}^{T \times H}$ . The aggregated vectors corresponding to the passage words in  $\mathbf{G}$  are then concatenated with the passage vectors in  $\mathbf{D}$ , followed by another BiLSTM to obtain a joint representation  $\mathbf{V} \in \mathbb{R}^{T \times H}$ .
4.1.2 Multi-Factor Attention
In addition, multi-factor self-attentive encoding (Kundu and Ng, 2018) is applied on the joint representation. If  $m$  represents the number of factors, multi-factor attention  $\mathbf{F}^{[1:m]} \in \mathbb{R}^{T \times m \times T}$  is formulated as:
$$
\mathbf {F} ^ {[ 1: m ]} = \mathbf {V} \mathbf {W} _ {f} ^ {[ 1: m ]} \mathbf {V} ^ {\top} \tag {1}
$$
where  $\mathbf{W}_f^{[1:m]}\in \mathbb{R}^{H\times m\times H}$  is a 3-way tensor. A max-pooling operation is performed on  $\mathbf{F}^{[1:m]}$ , over the number of factors, resulting in the self-attention matrix  $\mathbf{F}\in \mathbb{R}^{T\times T}$ . We normalize  $\mathbf{F}$  by applying a row-wise softmax function, resulting in  $\tilde{\mathbf{F}}\in \mathbb{R}^{T\times T}$ . Now the self-attentive encoding can be given as  $\mathbf{M} = \tilde{\mathbf{F}}\mathbf{V}\in \mathbb{R}^{T\times H}$ . The self-attentive encoding vectors are then concatenated with the joint encoding vectors, and a feed-forward
1The source code and data are released at https:// github.com/nusnlp/LIF 2We also experimented with ELMO and BERT but did not observe any consistent improvement.
neural network-based gating is applied to control the overall impact, resulting in  $\mathbf{Y} \in \mathbb{R}^{T \times 2H}$ . The final passage encoding  $\mathbf{P} \in \mathbb{R}^{T \times H}$  is obtained by applying another BiLSTM layer on  $\mathbf{Y}$ .
4.2 Three-Way Attentive Pooling
Now, we use our proposed three-way attentive pooling network to score every candidate follow-up question. The architecture of the network is depicted in Figure 2.
Attentive pooling (AP) was first proposed by dos Santos et al. (2016) and successfully used for the answer sentence selection task. AP is essentially an attention mechanism that enables joint learning of the representations of a pair of inputs as well as their similarity measurement. The primary idea is to project the paired inputs into a common representation space to compare them more plausibly even if both inputs are not semantically comparable, such as a question-answer pair. In this paper, we extend the idea of attentive pooling network to the proposed three-way attentive pooling network for the follow-up question identification task, where the model needs to capture the suitability of a candidate follow-up question by comparing with the conversation history and the associated passage. In particular, the proposed model aims to capture topic shift and topic continuation in the follow-up question. dos Santos et al. (2016) used a single attention matrix to compare a pair of inputs. In contrast, our proposed model relies on three attention matrices, where the two additional attention matrices make use of the associated passage. Moreover, our proposed model is developed to deal with a more complex follow-up question identification task, in contrast to the proposed model in dos Santos et al. (2016). We score each candidate follow-up question based on its relevance to the conversation history in two different perspectives: (1) considering the associated passage (i.e., knowledge) and (2) without considering the passage.
Attention Matrix Computation
In this step, we compute three different attention matrices for capturing the similarity between the conversation history and the candidate follow-up question - two matrices when the associated passage is taken into consideration, and another one when the passage is not considered. The attention matrix  $\mathbf{A}_{q,p} \in \mathbb{R}^{T \times U}$ , which captures the token-wise contextual similarity between the conversation
history and the passage, is given as:
$$
\mathbf {A} _ {q, p} = \operatorname {f} _ {\text {a t t n}} (\mathbf {Q}, \mathbf {P}), \tag {2}
$$
where the  $\mathrm{f_{attn}(.)}$  function can be written as  $\mathbf{f}_{\mathrm{attn}}(\mathbf{Q},\mathbf{P}) = \mathbf{P}\mathbf{Q}^{\top}$ . Intuitively,  $A_{q,p}(i,j)$  captures the contextual similarity score between the  $i$ -th token in the passage (i.e.,  $i$ -th row of  $\mathbf{P}$ ) and the  $j$ -th token in the conversation history (i.e.,  $j$ -th row of  $\mathbf{Q}$ ). Similarly, the attention matrix  $\mathbf{A}_{c,p} \in \mathbb{R}^{T \times V}$ , which captures the contextual similarity of a candidate follow-up question and the associated passage, is given as:
$$
\mathbf {A} _ {c, p} = \mathrm {f} _ {\text {a t t n}} (\mathbf {C}, \mathbf {P}) \tag {3}
$$
Note that,  $\mathbf{A}_{q,p}$  and  $\mathbf{A}_{c,p}$  will be used jointly to capture the similarity between  $\mathbf{Q}$  and  $\mathbf{C}$ , given  $\mathbf{P}$ .
The attention matrix  $\mathbf{A}_{c,q}\in \mathbb{R}^{U\times V}$ , which captures the similarity between a candidate follow-up question and the conversation history without considering the associated passage, is given as:
$$
\mathbf {A} _ {c, q} = \mathrm {f} _ {\text {a t t n}} (\mathbf {C}, \mathbf {Q}) \tag {4}
$$
Attention Pooling
After obtaining the attention matrices, we apply column-wise or row-wise max-pooling. When the associated passage is considered to capture the similarity between the conversation history and the candidate follow-up question, we perform columnwise max-pooling over  $\mathbf{A}_{q,p}$  and  $\mathbf{A}_{c,p}$ , followed by normalization with softmax, resulting in  $\mathbf{r}_{qp} \in \mathbb{R}^U$  and  $\mathbf{r}_{cp} \in \mathbb{R}^V$ , respectively. For instance,  $\mathbf{r}_{qp}$  is given as  $(1 \leq i \leq U)$ :
$$
\mathbf {r} _ {q p} = \operatorname {s o f t m a x} (\dots , \max  _ {1 \leq j \leq T} [ A _ {q, p} (j, i) ], \dots) \tag {5}
$$
Intuitively, the  $i$ -th element in  $\mathbf{r}_{qp}$  represents the relative importance score of the contextual encoding of the  $i$ -th token in the conversation history with respect to the passage encoding vectors. Every element of  $\mathbf{r}_{cp}$  can be interpreted in the same fashion. When the associated passage encoding is not considered, we perform both row-wise and column-wise max-pooling over  $\mathbf{A}_{c,q}$  to generate  $\mathbf{r}_{qc} \in \mathbb{R}^U$  and  $\mathbf{r}_{cq} \in \mathbb{R}^V$ , respectively.
Candidate Scoring
In this step, we score each candidate follow-up question. Each candidate  $\mathcal{C}$  is scored based on two perspectives - with and without consideration of
Figure 2: Architecture of the three-way attentive pooling network.
the associated passage encoding  $\mathbf{P}$ :
$$
\begin{array}{l} \operatorname {s c o r e} (\mathcal {C}) = s _ {1} + s _ {2} \\ = \mathrm {f} _ {\text {s i m}} (\mathbf {C}, \mathbf {Q} \mid \mathbf {P}) + \mathrm {f} _ {\text {s i m}} (\mathbf {C}, \mathbf {Q}), \tag {6} \\ \end{array}
$$
where  $\mathbf{C}$  is the encoding of  $\mathcal{C}$ . The similarity function  $\mathrm{f_{sim}}(\mathbf{C},\mathbf{Q}|\mathbf{P}) = \mathbf{xy}^{\top}$ , where  $\mathbf{x} = \mathbf{r}_{qp}\mathbf{Q}\in$ $\mathbb{R}^H$  and  $\mathbf{y} = \mathbf{r}_{cp}\mathbf{C}\in \mathbb{R}^{H}$ . The other similarity function  $\mathrm{f_{sim}}(\mathbf{C},\mathbf{Q}) = \mathbf{m}\mathbf{n}^{\top}$ , where  $\mathbf{m} = \mathbf{r}_{qc}\mathbf{Q}\in \mathbb{R}^{H}$  and  $\mathbf{n} = \mathbf{r}_{cq}\mathbf{C}\in \mathbb{R}^{H}$ .