SlowGuess/ACL_2020_OCR huggingface dataset:
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Pushkar Shukla, Carlos E. L. Elmadjian, Richika Sharan, Vivek Kulkarni, Matthew Turk, and William Yang Wang. 2019. What should I ask? using conversationally informative rewards for goal-oriented visual dialog. CoRR.
Paul E. Utgoff. 1989. Incremental induction of decision trees. Machine Learning, 4.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Harm de Vries, Florian Strub, A. P. Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Xianchao Wu, Huang Hu, Momo Klyen, Kyohei Tomita, and Zhan Chen. 2018. Q20: Rinna riddles your mind by asking 20 questions. Japan NLP.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified
framework and a text-to-SQL case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
We collect two types of data for the FAQ task. For the bird identification task we re-purpose existing data (Section 6).
Initial Query Collection Qualification One main challenge for the data collection process is familiarizing the workers with the set of target documents. We set up a two-stage process to ensure the quality of the initial queries. The first stage is to write paraphrases of a given target, which is often a question in the FAQ task. We first allow the full pool of Amazon Mechanical Turk workers to perform the task. After that, we manually inspect the written queries and pick the ones that are good paraphrases of the FAQs. We selected 50 workers that showed good understanding of the FAQs. In the second stage, workers are asked to provide initial queries with possibly insufficient information to identify the target. Out of the first 50 workers, we manually selected 25 based on the quality of the queries such as naturalness and whether they contain ambiguity or incompleteness by design. We used this pool of workers to collect 3,831 initial queries for our experiments.
Tag Association Qualification The goal of this annotation task is to associate tags with classification labels. We train a model on the collected initial queries to rank tags for each classification target. We pick out the highest ranked tags as positives and the lowest ranked tags as negatives for each target. The worker sees in total ten tags without knowing which ones are the negatives. To pass the qualification task, the workers need to complete annotation on three targets without selecting any of the negative tags.
Tag Association Task Details After the qualification task, we take the top 50 possible tags for each target and split them into five non-overlapping lists (i.e., ten tags for each list) to show to the workers. Each of the lists is assigned to four separate workers to annotate. We observe that showing only the top-50 tags out of 813 is sufficient. Figure A.1 illustrates this: after showing the top-50 tags, the
curve plateaus and no new tags are assigned to a target label. Table A.1 shows annotator agreement using Cohen's  $\kappa$  score.
Figure A.1: Accumulated number of tags assigned to the targets (y-axis) by the workers against tag ranking (x-axis). The ranking indicates the relevance of the target-tag pairs from the pre-trained model. The curve plateaued at rank 50 suggesting that the lower ranked tags are less likely to be assigned to the target by the crowdsourcing workers.
Tag Ranks
1-10	11-20	21-30	31-40	41-50
Mean # tags	3.31	1.45	0.98	0.61	0.48
N.A. (%)	1.9	30.7	43.6	62.1	65.2
Mean κ	0.62	0.54	0.53	0.61	0.61
Table A.1: Target-tag annotation statistics. We show five sets of tags to the annotators. The higher ranked ones are more likely to be related to the given target. The row mean # tags is the mean number of tags that are annotated to a target, N.A. is the percentage of the tasks that are annotated as "none of the above", and mean  $\kappa$  is the mean pairwise Cohen's  $\kappa$  score.
We use a single-layer bidirectional Simple Recurrent Unit (SRU) as the encoder for the FAQ suggestion task and two layer bidirectional SRU for bird identification task. The encoder uses pre-trained fastText (Bojanowski et al., 2017) word embeddings of size 300, hidden size 150, batch size 200, and dropout rate 0.1. The fastText embeddings remain fixed during training. We use the Noam learning rate scheduler (Vaswani et al., 2017) with initial learning rate  $1e-3$ , warm-up step 4,000 and a scaling factor of 2.0. For the self-attention model, we use a multi-head self-attention layer with 16 heads and a hidden size of 64 for each head. The same dropout rate used for the text encoder is applied to the self-attention layer. For the no interaction model with the RoBERTa encoder, we use the RoBERTaBASE model implemented by HuggHING Face (Wolf et al., 2019). The RoBERTaBASE model is fine-tuned with learning rate of  $1e-5$ , warmup step of 1,000, weight decay of 0.1, batch size of 16
and gradient accumulation step of 10. The policy controller is a two layer feed-forward network with a hidden layer of size 32 and ReLU activations. The network takes the current turn and the top- $k$  values of the belief probabilities as input. We choose  $k = 20$  and allow a maximum of 10 interaction turns.
We use the user simulator for further analysis of our system performance and alternative configurations.
Text Encoder Training Table A.2 shows the breakdown analysis of different ways to train the text encoder. We use initial queries as well as paraphrase queries to train the encoder, which has around 16K target-query examples. To analyze the effectiveness of tags in addition to initial queries, we generate pseudo-queries by combining existing queries with sampled subset of tags from the targets. This augmentation strategy is useful to improve the classification performance. We also observe that using the set of tags instead of initial queries as text inputs for a specific target label improves classification performance, indicating that the designed tags can capture the target label well. Finally, when we concatenate user initial queries and tags and use that as text input to the classifier, we achieve Accuracy@1 of  $76\%$ . In our full model, we achieve  $79\%$  with only querying about five tags.
Performances of Different Encoders Table A.3 show our system performance with different text encoders for both tasks.
A.4 Human Evaluation
Each interaction session starts with presenting a user scenario (e.g., a bird image or a phone issue). The user types an initial natural language query and answers follow-up questions selected by the system.
FAQ Suggestion We design a user scenario for each target to present to the worker. At the end of each interaction, the predicted FAQ and the ground truth are presented to the user, as shown in the top right panel in Figure A.2. The user answers the following questions: 'how natural is the interaction?' and 'do you feel understood by the system during the interactions?' on the scale of 1 (strongly disagree) to 5 (strongly agree), which we record as naturalness and rationality in Figure 2 and Table 1.
Our full model performs best on Accuracy@1, naturalness and rationality. We show human evaluation examples in Table A.4.
Bird Identification The interface for bird identification task is similar to the FAQ suggestion task. Instead of presenting a scenario, we show a bird image to the user. The user needs to describe the bird to find out its category, which is analogous to writing an initial query. When answering system questions about attributes, we allow the user to reply 'not visible' if part of the bird is hidden or occluded. Given this reply, the system stops asking binary questions from the same label group. For example, if a user replies 'not visible' to a question 'does the bird has a black tail?', then questions such as 'does the bird has yellow tail?' and 'does the bird has red tail?' will be skipped for the rest of the interaction. At the end of the interaction, the predicted and ground-truth bird images along with their categories are presented to the user as illustrated at the bottom right panel in Figure A.2. The user fills out a questionnaire as in FAQ domain. The bird identification task is very challenging because of its fine-grained categories, where many bird images look almost identical while belonging to different classes. Our full system improves classification accuracy from  $20\%$  to  $45\%$  against non-interactive baselines after less than three turns of interaction. To better understand the task and the model behavior, we show the confusion matrix of the final model prediction after interaction in Figure A.3. Of the 200 bird classes, there are 21 different kinds of sparrows and 25 different warbler. Most of the model errors are due to mistakes between these fine-grained bird classes. Figure A.4 shows how the confusion matrix changes when adding the interaction. The model makes improvement in distinct and also similar bird types.
Text Input	Init Query	Init Query + Tags	Init + Paraphrase Query	Full Data
init query	tags	Acc@1	Acc@3	Acc@1	Acc@3	Acc@1	Acc@3	Acc@1	Acc@3
✓	✘	0.28	0.47	0.32	0.51	0.35	0.60	0.38	0.61
✘	✓	0.31	0.50	0.57	0.79	0.56	0.74	0.70	0.87
✓	✓	0.36	0.58	0.55	0.79	0.63	0.81	0.76	0.91
Table A.2: Comparison of text encoders trained on different textual inputs and evaluated on three different prediction tasks. The model uses (a) the initial queries, (b) all attribute tags, or (c) both initial queries and tags as text inputs to predict the target. Each model is evaluated using Accuracy@{1, 3}.
FAQ Suggestion	Bird Identification
RNN	RNN + attention	RoBERTaBASE	RNN	RNN + attention	RoBERTaBASE
No Interaction	38%	39%	30%	23%	23%	17%
Random Interaction	39%	38%	31%	25%	24%	17%
No Initial Query Interaction	46%	41%	37%	29%	29%	21%
Our Approach w/ threshold	72%	73%	54%	41%	38%	33%
Our Approach w/ fixed turn	71%	68%	47%	39%	37%	32%
Our Approach w/ λ = 1	66%	67%	52%	40%	37%	32%
Table A.3: Accuracy@1 of our system with three different encoders. For all experiments, maximal number of turns is set to five.
U:	How can I text my friends house phone?
S:	Text message to landline?
U:	Yes.
S:	Do you want to send or receive text?
U:	Yes.
S:	Please read: How do I send a text message to landline?
U:	I'm in sprint's one up and want to upgrade my phone.
S:	Do you want annual upgrade option?
U:	Yes.
S:	Do you want to exercise an option?
U:	Yes.
S:	Please read: I am enrolled in Sprint One Up. How do I exercise my annual upgrade option?
U:	Battery ran out immediately.
S:	How can I help you with your phone device?
App or feature information; Fee and charge; Troubleshooting; Not applicable
U:	Troubleshooting.
S:	Are you using mobile wifi hotspot?
U:	Yes.
S:	Please read: Does using my phone as a mobile Wi-Fi hotspot drain my battery?
Table A.4: Three examples of user interactions from the FAQ suggestion human evaluation.
Scenario You would like to use your phone as hotspot for your laptop for some urgent work, but you are worried it gonna cost you lots of money.
Model Predicted FAQ
Scenario
- You would like to use your phone as hotspot for your laptop for some urgent work, but you are worried it gonna cost you lots of money.
Model Predicted FAQ  
FAQs related to Wi-Fi hotspots on your samsung, Does it cost more to use my samsung as a mobile Wi-Fi hotspot?
Groundtruth  
FAQs related to Wi-Fi hotspots on your samsung, Does it cost more to use my samsung as a mobile Wi-Fi hotspot?
Figure A.2: The user interface for FAQ Suggestion (top) and Bird Identification (bottom) tasks. The left panel shows the interface at the beginning of the interaction and the right panel shows the interface at the end of the interaction.
Model Predicted Bird Type nashville warbler
Correct Bird Type nashville warbler
Figure A.3: Confusion matrix of our final output for bird identification task.
Figure A.4: Confusion matrix difference between the initial query with and without the interactions. High values along the diagonal and low values elsewhere are good.