SlowGuess/ACL_2020_OCR huggingface dataset:
We use binary cross entropy loss for training the model. For prediction, we find a threshold to maximize the scores on the development set. For the test instances, we use the threshold to predict whether a follow-up question is valid or invalid.
We develop several rule-based, statistical machine learning, and neural baseline models. For all the models, a threshold is determined based on the best performance on the development set.
5.1 Rule-Based Models
We develop two models based on word overlap counts - between the candidate follow-up ques
tion and the passage, and between the candidate follow-up question and the conversation history. We normalize the count values based on the length of the candidate follow-up question.
Next, we develop two models based on the contextual similarity scores using InferSent sentence embeddings (Conneau et al., 2017). The two models compare the candidate follow-up question with the associated passage and the conversation history, respectively. The similarity scores are computed based on vector cosine similarity.
We also develop another rule-based model using tfidf weighted token overlap scores. We prepend the last question from the conversation history to the candidate follow-up question and add the tfidf of overlapping words between the concatenated context and the passage.
5.2 Statistical Machine Learning Models
We handcraft two sets of features for the statistical machine learning models. One set of features consists of tfidf weighted GloVe vectors. Since we adopt 300 dimensional GloVe vectors in our experiments, these features are of dimension 300.
Models	Dev
V-P/-R/-F1/Macro F1	Test-I
V-P/-R/-F1/Macro F1	Test-II
V-P/-R/-F1/Macro F1	Test-III
V-P/-R/-F1/Macro F1
Norm. overlap (Psg)	34.4/46.4/39.5/50.9	36.0/52.1/42.6/52.5	40.5/60.3/48.5/52.3	71.5/65.4/68.3/48.6
Norm. overlap (Hist)	34.1/40.7/37.1/51.0	33.8/43.1/37.9/50.8	40.6/33.2/36.6/52.2	78.6/67.4/72.6/58.3
InferSent (Psg)	28.4/42.7/34.1/44.2	28.5/47.0/35.5/43.6	30.0/40.6/34.5/42.0	72.3/71.4/71.9/50.1
InferSent (Hist)	22.0/11.9/15.5/43.5	25.2/12.7/16.9/45.1	26.6/10.5/15.1/42.8	72.1/69.6/70.8/49.7
Tfidf + Overlap	32.6/61.7/42.7/45.5	32.5/66.3/43.7/44.6	37.5/66.3/47.9/46.7	72.1/85.6/78.2/48.2
Logistic Regression
Tfidf + GloVe	58.4/67.5/62.6/71.1	53.5/61.1/57.0/67.1	63.7/66.0/64.8/71.8	73.5/95.1/82.9/50.1
Overlap count	41.0/61.9/49.4/57.1	39.7/58.4/47.3/56.1	49.1/57.6/53.0/60.7	73.1/98.5/83.9/47.0
CNN-Maxpool	61.6/67.3/64.3/72.9	58.0/62.3/60.1/69.9	69.0/62.4/65.5/73.4	78.4/62.2/69.4/56.5
CNN-Attnpool	52.8/56.9/54.8/65.7	48.3/54.0/51.0/62.7	56.2/54.3/55.2/64.9	77.6/53.7/63.5/53.0
LSTM-MaxPool	75.1/70.0/72.4/79.9	72.9/66.1/69.3/77.8	89.9/66.1/76.2/82.5	79.3/66.1/72.1/58.7
LSTM-AttnPool	72.6/65.7/69.0/77.5	72.1/66.2/66.8/76.2	89.2/62.2/73.3/80.6	79.0/62.2/69.6/57.1
BERT	74.2/76.4/75.3/81.5	72.4/76.1/74.2/80.7	88.5/76.1/81.8/86.2	79.9/76.1/78.0/62.6
Three-way AP	76.2/77.3/76.8/82.7	74.4/75.7/75.0/81.4	89.0/75.7/81.8/86.2	81.9/75.7/78.7/65.0
Table 3: Comparison results for the follow-up question identification task. We compare the performance of three-way attentive pooling network with several rule-based, statistical machine learning, and neural models (V - Valid, P - Precision, R - Recall, Psg - Passage, Hist - Conversation history).
Another set of features consists of word overlap counts. We compute the pairwise word overlap counts among the candidate follow-up question, the associated passage, and the conversation history. The overlap count-based features are of dimension 3. We experiment with logistic regression using the derived features.
5.3 Neural Models
We also develop several neural baseline models. We first concatenate the associated passage, the conversation history, and the candidate follow-up question, followed by embedding (the same as described earlier). Then, we apply sequence-level encoding with either BiLSTM or CNN. For CNN, we use equal numbers of unigram, bigram, and tri-gram filters, and the outputs are concatenated to obtain the final encoding. Next, we apply either global max-pooling or attentive pooling to obtain an aggregated vector representation, followed by a feed-forward layer to score the candidate follow-up question. Let the sequence encoding of the concatenated text be  $\mathbf{E} \in \mathbb{R}^{L \times H}$ , and  $\mathbf{e}_t$  be the  $t$ th row of  $\mathbf{E}$ . The aggregated vector  $\tilde{\mathbf{e}} \in \mathbb{R}^H$  for attentive-pooling can be obtained as:
$$
a _ {t} \propto \exp \left(\mathbf {e} _ {t} \mathbf {w} ^ {\top}\right); \tilde {\mathbf {e}} = \mathbf {a E}, \tag {7}
$$
where  $\mathbf{w} \in \mathbb{R}^H$  is a learnable vector. We also develop a baseline model using BERT (Devlin et al., 2019). We first concatenate all the inputs and then apply BERT to derive the contextual vectors. Next, we aggregate them into a single vector using attention. Then a feed-forward layer is used to score each candidate follow-up question.
In this section, we present the experimental settings, results, and performance analysis.
We do not update the GloVe vectors during training. We use 100-dimension character-level embedding vectors. The number of hidden units in all the LSTMs is  $150(H = 300)$ . We use dropout (Srivastava et al., 2014) with probability 0.3. Following Kundu and Ng (2018), we set the number of factors as 4 in multi-factor attentive encoding. We use the Adam optimizer (Kingma and Ba, 2015) with learning rate 0.001 and clipnorm 5. Following Choi et al. (2018), we consider at most 3 previous question-answer pairs in the conversation history. This being a binary classification task, we use precision, recall, F1, and macro F1 as evaluation metrics. All scores reported in this paper are in  $\%$
Table 3 shows that our proposed model outperforms the competing baseline models by significant margins across all test sets. We perform statistical significance tests using paired t-test and bootstrap resampling. Performance of our proposed model is significantly better  $(p < 0.01)$  than the best baseline system which provides the highest Macro-F1 score on Test-I. The LSTM-based neural baselines perform better than the rule-based and statistical machine learning models in most cases. On Test-III, the statistical models tend to predict valid, and the number of valid instances is much higher than the invalid instances (about  $75\%$ :  $25\%$ ), resulting
Model	V-P	V-R	V-F1	Macro F1
- History	72.7	67.0	69.7	77.9
- Knowledge	75.8	73.8	74.8	81.4
- Ac,q	71.8	75.8	73.7	80.2
- Multi-factor Attn	75.6	76.4	76.0	82.1
- Joint encoding	75.3	76.6	76.0	82.1
- Char embedding	74.2	72.3	73.2	80.2
Three-way AP	76.2	77.3	76.8	82.7
Table 4: An ablation study on the development set.
in high Valid F1 scores. These baseline systems (while performing well on valid questions) perform poorly when evaluated using Macro F1 which measures performance on both valid and invalid follow up questions. Macro F1 is the overall evaluation metric used to compare all systems. Overall, identifying follow-up questions from the same conversation (Test-III) is harder compared to other conversations (Test-II).
We perform an ablation study as shown in Table 4. The proposed model performs worst when we do not consider the conversation history. This is because the question-answer pairs in the conversation history help to determine topic continuity while identifying a valid follow-up question. The performance also drops when we do not consider the associated passage (i.e., knowledge) because it helps to capture topic shift. The performance also degrades when we remove  $\mathbf{A}_{c,q}$ . It performs better than the model where we do not consider the conversation history at all, as the conversation history is taken into consideration in passage encoding. The performance also degrades when we remove other components such as multi-factor attentive encoding, joint encoding, and character embedding.
6.3 Qualitative Analysis
The proposed model aims to capture topic continuity and topic shift by using a three-way attentive pooling network. Attention pooling on  $\mathbf{A}_{q,p}$  and  $\mathbf{A}_{c,p}$  aims to capture topic shift in the follow-up question for a given conversation history. Consider the first example in Table 5. When we do not consider the passage, it could not identify the follow-up question correctly while our proposed model correctly identifies the topic shift to the duration of the riot by validating with the passage words after four days and restore order and take back the prison on September 13. In the second example, while our model could correctly identify topic continuity through Schuur, the model without history fails to identify the follow-up question.
We performed an error analysis where our proposed model failed to identify the follow-up questions. We randomly sampled 50 such instances (25 valid and 25 invalid) from the development set. We found that  $32\%$  of them require pronoun resolution for the subject in the follow-up questions.  $38\%$  of the instances require validation of the actions/characteristics of the subjects (e.g., did they have any children? vs. gave birth to her daughter).  $14\%$  of the errors occur when it requires matching objects or predicates which occur in different forms (e.g., hatred vs hate, television vs TV). For the remaining  $16\%$  of the cases, it could not correctly capture the topic shift.
Many data-driven machine learning methods have been shown to be effective for tasks relevant for dialog such as dialog policy learning (Young et al., 2013), dialog state tracking (Henderson et al., 2013; Williams et al., 2013; Kim et al., 2016), and natural language generation (Sordoni et al., 2015; Li et al., 2016; Bordes et al., 2017). Most of the recent dialog systems are either not goal oriented (e.g., simple chit-chat bots), or domain-specific if they are goal oriented (e.g., IT help desk). In the last few years, there has been a surge of interest in conversational question answering. Saha et al. (2018) released a Complex Sequential Question Answering (CSQA) dataset for learning conversations through a series of interrelated QA pairs by inferencing over a knowledge graph. Choi et al. (2018) released a large-scale conversational QA dataset, namely question answering in context (QuAC), which mimics a student-teacher interactive scenario. Reddy et al. (2019) released the CoQA dataset and many systems were evaluated on it. Zhu et al. (2018) proposed SDNet to fuse context into traditional reading comprehension models. Huang et al. (2019) proposed a "Flow" mechanism that can incorporate intermediate representations generated during the process of answering previous questions, through an alternating parallel processing structure. In a conversation setting, given the previous QA pairs as conversation history, while these models focus on answering the next question, our work is focused on identifying follow-up questions. Recently, Saeidi et al. (2018) proposed a dataset for regulatory texts that requires a model to ask follow-up clarification questions. However, the answers are limited to yes or no, which makes the task rather
Passage: On September 9, 1971, prisoners at the state penitentiary at Attica, NY, took control of a cell block and seized thirty-nine correctional officers as hostages. After four days of negotiations, Department of Correctional Services Commissioner Russell Oswald agreed to most of the inmates' demands for various reforms but refused to grant complete amnesty to the rioters, with passage out of the country and removal of the prison's superintendent. When negotiations stalled and the hostages appeared to be in imminent danger, Rockefeller ordered New York State Police and national guard troops to restore order and take back the prison on September 13. ...
Q: Where was the Attica Prison? A: On September 9, 1971, prisoners at the state penitentiary at Attica, NY, took ...
Q: Why did they riot? A: the inmates' demands for various reforms but refused to grant complete amnesty to the rioters, with passage out of the country and removal of the prison's superintendent.
Candidate follow-up question: How long did the riot last? - Valid
Passage: In 1975, at age 22, Schuur auditioned for drummer/bandleader Ed Shaughnessy. Escorted by her twin brother, she went backstage to ... singing the blues. ... He hired her to be the vocalist in his orchestra, "Energy Force". Jazz trumpeter Dizzy ...
Q: When was Schuur discovered? A: In 1975, at age 22, Schuur auditioned for drummer/bandleader Ed Shaughnessy. . .
Candidate follow-up question: Was she hired by Ed Shaughnessy? - Valid
Table 5: Examples taken from the LIF development set where our model correctly identified a valid follow-up question.
restrictive. Moreover, while Saeidi et al. (2018) focuses on generating a clarification question in response to a question of a conversation, we focus on identifying whether a question is a follow-up question of a conversation.
In this paper, we present a new follow-up question identification task in a conversational setting. We developed a dataset, namely LIF, which is derived from the previously released QuAC dataset. Notably, the proposed dataset supports automatic evaluation. We proposed a novel three-way attentive pooling network which identifies whether a follow-up question is valid or invalid by considering the associated knowledge in a passage and the conversation history. Additionally, we developed several strong baseline systems, and showed that our proposed three-way attentive pooling network outperforms all the baseline systems. Incorporating our three-way attentive pooling network into open domain conversational QA systems will be interesting future work.
This research is supported by the National Research Foundation Singapore under its AI Singapore Programme (Award Number: AISG-RP-2018-007).
Núria Bertomeu, Hans Uszkoreit, Anette Frank, Hans-Ulrich Krieger, and Brigitte Jörg. 2006. Contextual phenomena and thematic relations in database QA dialogues: Results from a Wizard-of-Oz experiment.
In Proceedings of the Interactive Question Answering Workshop at HLT-NAACL.
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In Proceedings of ICLR.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. QuAC: Question answering in context. In Proceedings of EMNLP.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL.
Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. In Proceedings of SIGDIAL.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Hsin-Yuan Huang, Eunsol Choi, and Wen tau Yih. 2019. FlowQA: Grasping flow in history for conversational machine comprehension. In Proceedings of ICLR.
Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Banchs, Jason Williams, and Matthew Henderson. 2016. The fourth dialog state tracking challenge. In Proceedings of IWSDS.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP.
Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Manuel Kirschner and Raffaella Bernardi. 2007. An empirical view on IQA follow-up questions. In Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue.
Souvik Kundu and Hwee Tou Ng. 2018. A question-focused multi-factor attention network for question answering. In Proceedings of AAAI.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of EMNLP.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the ACL.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Roktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of EMNLP.
Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In Proceedings of AAAI.
Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016. Attentive pooling networks. arXiv preprint arXiv:1602.03609.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of NAACL.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1958.
Jason Williams, Antoine Raux, Deepak Ramachadran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of SIGDIAL.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. POMDP-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5).
Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. SDNet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593.