SlowGuess/ACL_2020_OCR huggingface dataset:
Tag Collection We collect natural language tag annotations for the FAQ documents. First, we use domain experts to define the set of possible freeform tags. The tags are not restricted to a predefined ontology and can be a phrase or a single word describing the topic of the document. We remove duplicate tags to finalize the set. Experts combine some binary tags to categorical tags. For example, tags 'IOS', 'Android operating system' and 'Windows operating system' are combined to the categorical tag 'phone operating system'. We use a small set of deterministic, heuristically-designed templates to convert tags into questions. For example, the tag 'international roaming' is converted into a binary question 'Is it about international roaming?'; the categorical tag 'phone operating system' is converted into a multi-choice question 'What is your phone operating system?' Finally, we use non-experts to collect user responses to the questions by associating tags with FAQ targets. For binary questions, we ask workers to associate their tags to the FAQ target if they would respond 'yes' to the question. We show the workers a list of ten tags for a given target as well as a 'none of the above' option. Annotating all possible target-tag combinations is still expensive and most pairings are negative. We rank the tags based on the relevance against the target using  $S(\cdot)$  trained only
on the initial queries and show only the current top-50 to the workers. Later, we re-train  $S(\cdot)$  on the complete data. For multi-choice questions, we show the workers a list of possible answers to a tag-generated question for a given FAQ. The workers need to choose one answer that they think best applies. They also have the option of choosing 'not applicable'. The workers do not engage in a multi-round interactive process. This allows for cheap and scalable collection.
Task I: FAQ Suggestion We use the FAQ dataset from Shah et al. (2018). The dataset contains 517 troubleshooting documents from Sprint's technical website. We collect 3,831 initial queries and 118,640 tag annotations using the setup described in Section 5. We split the data into 310/103/104 documents as training, development, and test sets. Only the queries and tag annotations of the 310 training documents are used for pre-training and learning the policy controller, leaving the queries and tag annotations in the development and test splits for evaluation only.
Task II: Bird Identification We use the Caltech-UCSD Birds dataset (CUB-200; Wah et al., 2011). The dataset contains 11,788 bird images for 200 different bird species. Each bird image is annotated with a subset of 27 visual attributes and 312 attribute values pertaining to the color or shape of a particular part of the bird. We create categorical questions from attributes with less five possible values, providing eight categorical questions in total. The remaining 279 attributes are converted to binary questions. Each image is annotated with 10 image captions describing the bird in the image (Reed et al., 2016). We use the image captions as initial user queries and bird species as labels. Since each caption contains only partial information about the bird species, the data is naturally noisy and provides challenging user interactions. We do not use the images from the dataset for model training. The images are only provided for grounding during human evaluation.
Baselines We compare with four methods:
- No Interaction: the classification label is predicted using only the initial query. We consider four implementations: (1) BM25: a common keyword-based scoring model for retrieval methods (Robertson and Zaragoza,
<sup>3</sup>https://www.mturk.com/
2009); (2) RoBERTa<sub>BASE</sub>: we use a fine-tuned RoBERTa<sub>BASE</sub> model (Liu et al., 2019) as text encoder; (3) RNN: we use a recurrent neural network (RNN) with simple recurrent unit recurrence (SRU; Lei et al., 2018) as text encoder, together with a fastText word embedding layer (Bojanowski et al., 2017); and (4) RNN + self-attn: the same RNN neural model with a multi-head self-attention layer (Lin et al., 2017; Vaswani et al., 2017).
- Random Interaction: at each turn, the system randomly selects a question to present the user. After  $T$  turns, the classification label is chosen according to the belief  $p(y|X^T)$ .
- No Initial Query Interaction: the system selects questions without conditioning on the initial user query using maximum information criterion. This is equivalent to using a static decision tree to pick the question, always asking the same first question (Utgoff, 1989; Ling et al., 2004).
- Variants of Our Approach: we consider several variants of our full model. First, we replace the policy controller with two termination strategies: (1) end the interaction when  $\max p(y|X^t)$  passes a threshold, or (2) end the interaction after a fixed number of turns. Second, we disable the parameterized estimator  $\tilde{p}(r_k|q_j, y_i)$  by setting  $\lambda = 1$ .
Evaluation We use human evaluation, and further analyze performance using our simulator. For human evaluation, users interact with our systems and baseline models using a web-based interactive interface. Each interaction starts with a user scenario:4 a bird image or a device-troubleshooting scenario described in text. The user types an initial query and answers follow-up questions selected by the system. Once the system returns its prediction, we measure its accuracy, and the user is asked to rate the whole interaction according to rationality and naturalness.5 The user does not know the correct target label. We use a five-points Likert score for the followup questions. For FAQ Suggestion, we consider two evaluation setups: (1) assuming the model has access to tags in the development and test set for interaction, and (2) using only tags in the
training set annotation. The former is equivalent to adding tags for new documents not seen during training time. The latter zero-shot evaluation setup allows us to investigate the model's performance on unseen targets with no additional tags associated with them. Appendix A.4 provides further details of the human evaluation setup. We do further analysis with the user simulator. We evaluate classification performance using Accuracy@  $k$ , which is the percentage of time the correct target appears among the top-  $k$  predictions of the model.
Implementation Details We use the same encoder to encode initial queries, question-answer pairs and FAQ documents in the FAQ suggestion task. In the bird identification task, where the structure of bird names differs from the other texts, we use one encoder for user initial queries and question-answer pairs and a second encoder for bird names. The policy controller receives a reward of 20 for returning the correct target label, a negative reward of -10 for the wrong target, and a turn penalty of -0.5 for each question asked. For our simulated analysis, we report the averaged results as well as the standard derivation from three independent runs for each model variant and baseline. Appendix A.2 provides more implementation and training details.
Our simulated analysis shows that the SRU RNN text encoder performs better or similar to the other encoders. This encoder is also the most lightweight. Therefore, we use it for the majority of our experiments.
Human Evaluation Figure 2 and Table 1 show the human evaluation results of our full model and three baselines: our approach with a fixed number of turns (four for FAQ and five for Bird), our approach without access to the initial query (No Init. Query) and our approach without interaction (No Int. (RNN)). Naturalness and rationality measure the quality of the interaction, so we show the results of the user survey in Figure 2 only for interactive systems. Because we do not ask users to fill the end-of-interaction survey for the no interaction baseline, we simply compute its numbers following the first query when evaluating our full approach. Our approach balances between accuracy and the user-centric measures, including naturalness and rationality, achieving stronger performance across
Each scenario is related to a single groundtruth label and serves to ground user interactions.
We also surveyed users for perceived correctness, but observed it is interpreted identically to rationality. Therefore, we omit this measure.
Figure 2: Human evaluation Gantt charts showing user ratings. We show the mean rating for each measure and system on the right of each bar.
FAQ	FAQ (zero-shot)	Bird
Our Approach	57%	52%	45%
Our Approach w/fixed turn	53%	47%	37%
No Init. Query	43%	41%	28%
No Int. (RNN)	30%	26%	20%
Table 1: Human evaluation classification accuracy.
the board. All three models improve the classification performance with the addition of interaction. Qualitatively, the users rate our full approach better than the two other interaction variants. This demonstrates that our model handles effectively real user interaction despite being trained with only non-interactive data. We include additional details in Appendix A.4.
Analysis with Simulated Interactions Table 2 shows performance using the user simulator. We use these results to evaluate different choices beyond what is possible with human studies. We observe interaction is critical; removing the ability to interact decreases performance significantly. The Random Interaction and the No Initial Query Interaction baselines both barely improve the performance over the No Interaction RNN baseline, illustrating the importance of guiding the interaction and considering the initial query. Our full model achieves an Accuracy@1 of  $79\%$  for FAQ Suggestion and  $49\%$  for Bird Identification using less than five turns, outperforming the No Interaction RNN baseline by  $41\%$  and  $26\%$ . When having no access to questions and answers in the development and test set during evaluation, the full model performance drops only slightly to  $75\%$ , highlighting the model's ability to generalize to unseen tags. The two baselines with alternative termination strategies underperform the full model, indicating the effectiveness of the policy controller. The relatively low performance of the  $\lambda = 1$  variant, which effectively has fewer probability components leveraging natural language than
our full model, and No Initial Query Interaction confirm the importance of the learned natural language embedding encoder. Appendix A.3 includes further details on how different text encoders impact performance.
Figure 3 shows the trade-off between classification accuracy and the number of turns. Each point on the plots is computed by varying the reward turn penalty for our model, the prediction threshold and the predefined number of turns  $T$ . Our model with the policy controller or the threshold strategy does not explicitly bound the number of turns, so we report the average number of turns across multiple runs for these two models. We achieve a relative accuracy boost of  $40\%$  for FAQ and  $65\%$  for Birds over no-interaction baselines with only one clarification question. This highlights the value of leveraging human feedback to improve model accuracy in classification tasks.
Figure 4 shows the learning curves of our model with the policy controller trained with different turn penalties  $r_a \in \{-0.5, -1, -3\}$ . We observe the models explore during the first 1,000 training episodes in the middle and the right plots. The models achieve relatively stable accuracy after the early exploration stage. The three runs end up using different numbers of expected turns because of the different  $r_a$  values.
We propose an approach for interactive classification, where the system can inquire missing information through a sequence of simple binary or multi-choice questions when users provide underspecified natural language queries. Our expert-guided, incremental design of questions and answers enables easy extension to add new classes, striking the balance between simplicity and extendability. Our modeling choices enable the system to perform zero-shot generalization to unseen classification targets and questions. Our method uses information gain to select the best question to ask
FAQ Suggestion	Bird Identification
Acc@1	Acc@3	Acc@1	Acc@3
No Interaction (BM25)	26%	31%	N.A.	N.A.
No Interaction (RoBERTaBASE)	30 ± 0.5%	45 ± 0.6%	17 ± 0.3%	29 ± 0.3%
No Interaction (RNN)	38 ± 0.5%	61 ± 0.3%	23 ± 0.1%	41 ± 0.2%
No Interaction (RNN + self-attn)	39 ± 0.5%	63 ± 0.4%	23 ± 0.1%	41 ± 0.1%
Random Interaction	39 ± 0.3% (38 ± 0.1%)	62 ± 0.4% (63 ± 0.2%)	25 ± 0.1%	44 ± 0.1%
No Initial Query Interaction	46 ± 0.5% (46 ± 0.1%)	66 ± 0.6% (67 ± 0.3%)	29 ± 0.2%	50 ± 0.3%
Our Approach	79 ± 0.7% (75 ± 0.4%)	86 ± 0.8% (83 ± 0.4%)	49 ± 0.3%	69 ± 0.5%
w/ threshold	73 ± 0.6% (69 ± 0.6%)	82 ± 0.7% (81 ± 0.6%)	41 ± 0.3%	59 ± 0.4%
w/ fixed turn	71 ± 1.0% (68 ± 0.4%)	81 ± 0.9% (81 ± 0.6%)	39 ± 0.2%	56 ± 0.4%
w/ λ = 1	66 ± 0.8% (64 ± 0.2%)	71 ± 1.0% (73 ± 0.2%)	40 ± 0.1%	56 ± 0.2%
Table 2: Performance with simulated interactions. We evaluate our approach and several baselines using Accuracy@{1, 3}. Best performance numbers are in bold. We report the averaged results as well as the standard deviations from three independent runs for each model variant and baseline. For FAQ Suggestion, in parentheses, we provide zero-shot results, where the system has access to tags only for training questions.
Figure 3: Accuracy@1 (y-axis) against turns of interactions (x-axis) for FAQ (left) and Birds (right) tasks.
Figure 4: Learning curves of our full model. We show accumulative reward (left), interaction turns (middle), and Accuracy@1 (right) on the test set, where x-axis is the number of episodes (400 trials per episode). The results are compared on different turn penalty  $r_a$ .
at every turn, and a lightweight policy to efficiently control the interaction. We demonstrate that the system can be bootstrapped without any interaction data and show effectiveness on two tasks. A potential future research direction is to bridge the gap between this simple bootstrapping paradigm and the incorporation of user-free-form responses to allow the system to handle free-text responses. We hope our work will encourage more research on different possibilities of building interactive systems that do not necessarily require handling full-fledged dialogue, but still benefit from user interaction.
We thank Derek Chen, Alex Lin, Nicholas Matthews, Jeremy Wohlwend, Yi Yang and the anonymous reviewers for providing valuable feedback on the paper. We would also like to thank Michael Griffiths, Anna Folinsky and the ASAP annotation team for their help on setting up and performing the human evaluation. Finally, we thank Hugh Perkins, Ivan Itzcovich, and Brendan Callahan for their support on the experimental environment setup.
Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics.
Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, and Devi Parikh. 2017. Evaluating visual conversational agents via cooperative human-ai games. In Fifth AAAI Conference on Human Computation and Crowdsourcing.
Cen Chen, Chilin Fu, Xu Hu, Xiaolu Zhang, Jun Zhou, Xiaolong Li, and Forrest Sheng Bao. 2019. Reinforcement learning for user intent prediction in customer service bots. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Yihong Chen, Bei Chen, Xuguang Duan, Jian-Guang Lou, Yue Wang, Wenwu Zhu, and Yong Cao. 2018. Learning-to-ask: Knowledge acquisition via 20 questions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. Quac : Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4299-4307.
Pei-Hung Chung, Kuan Tung, Ching-Lun Tai, and Hung yi Lee. 2018. Joint learning of interactive spoken content retrieval and trainable user simulator. In INTERSPEECH.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE international conference on computer vision.
Marin Ferecatu and Donald Geman. 2007. Interactive search for image categories by mental matching. In 2007 IEEE 11th International Conference on Computer Vision.
Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. 2018. Dialog-based interactive image retrieval. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 678-688. Curran Associates, Inc.
Izzeddin Gur, Semih Yavuz, Yu Su, and Xifeng Yan. 2018. Dialsql: Dialogue based structured query generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Huang Hu, Xianchao Wu, Bingfeng Luo, Chongyang Tao, Can Xu, Wei Wu, and Zhan Chen. 2018. Playing 20 question game with policy-based reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
In-Ho Kang and GilChang Kim. 2003. Query type classification for web document retrieval. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval.
J. F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications.
Adriana Kovashka and Kristen Grauman. 2013. Attribute pivots for guiding relevance feedback in image search. In Proceedings of the IEEE International Conference on Computer Vision.
Sang-Woo Lee, Tong Gao, Sohee Yang, Jaejun Yoo, and Jung-Woo Ha. 2019. Large-scale answerer in questioner's mind for visual dialog question generation. In International Conference on Learning Representations.
Sang-Woo Lee, Yu-Jung Heo, and Byoung-Tak Zhang. 2018. Answerer in questioner's mind: Information theoretic approach to goal-oriented visual dialog. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances
in Neural Information Processing Systems 31. Curran Associates, Inc.
Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Charles X. Ling, Qiang Yang, Jianning Wang, and Shichao Zhang. 2004. Decision trees with minimal costs. In Proceedings of the Twenty-first International Conference on Machine Learning.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R. Trippas, and Minghui Qiu. 2019. User intent prediction in information-seeking conversations. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval.
Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, (4).
Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the 13th international conference on World Wide Web.
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018. Adversarial domain adaptation for duplicate question detection. In