SlowGuess/ACL_2020_OCR huggingface dataset:
Lili Yu $^{1}$ , Howard Chen $^{1}$ , Sida I. Wang $^{1,2}$ , Tao Lei $^{1}$  and Yoav Artzi $^{1,3}$ $^{1}$ ASAPP Inc., New York, USA  
 $^{2}$ Princeton University, New Jersey, USA  
 $^{3}$ Cornell University, New York, USA  
{liliyu, hchen, tao}@asapp.com  
sidaw@cs.princeton.edu yoav@cs.cornell.edu
We study the potential for interaction in natural language classification. We add a limited form of interaction for intent classification, where users provide an initial query using natural language, and the system asks for additional information using binary or multi-choice questions. At each turn, our system decides between asking the most informative question or making the final classification prediction. The simplicity of the model allows for bootstrapping of the system without interaction data, instead relying on simple crowdsourcing tasks. We evaluate our approach on two domains, showing the benefit of interaction and the advantage of learning to balance between asking additional questions and making the final prediction.
Responding to natural language queries through simple, single-step classification has been studied extensively in many applications, including user intent prediction (Chen et al., 2019; Qu et al., 2019), and information retrieval (Kang and Kim, 2003; Rose and Levinson, 2004). Typical methods rely on a single user input to produce an output, and do not interact with the user to reduce ambiguity and improve the final prediction. For example, users may under-specify a request due to incomplete understanding of the domain; or the system may fail to correctly interpret the nuances of the input query. In both cases, a low quality decision could be mitigated by further interaction with the user.
In this paper we take a low-overhead approach to add limited interaction to intent classification. Our goal is two-fold: (a) study the effect of interaction on the system performance, and (b) avoid the cost and complexities of interactive data collection. We build an interactive system that poses a sequence of binary and multiple choice questions follow
Figure 1: Two examples of interactive classification systems: providing a trouble-shooting FAQ suggestion (left) and helping identifying bird species from a descriptive text query (right). The top parts show example classification labels: FAQ documents or bird species. The ground truth label of each interaction example is shaded. The lower parts show user interactions with the systems. The user starts with an initial natural language query. At each step, the system asks a clarification question. The interaction ends when the system returns an output label.
ing the initial user natural language query. Figure 1 illustrates such interactions in two domains, showcasing the opportunity for clarification while avoiding much of the complexity involved in unrestricted natural language interactions. We design our approach not to rely on user interaction during learning, which requires users to handle low quality systems or costly Wizard of Oz experiments.
We adopt a Bayesian decomposition of the posterior distributions over intent labels and user responses through the interaction process. We use the posteriors to compute question expected information gain, which allows us to efficiently select the next question at each interaction turn. We bal
<sup>1</sup>The images are for illustration only. Our approach does not use images.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2664-2680 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ance between the potential increase in accuracy and the cost of asking additional questions with a learned policy controller that decides whether to ask additional questions or return the final prediction. We estimate each distribution in our posterior decomposition independently by crowdsourcing initial queries and keywords annotation. We use non-interactive annotation tasks that do not require Wizard-of-Oz style dialog annotations (Kelley, 1984; Wen et al., 2017). During training, we train a shared text encoder to compare natural language queries, clarification questions, user answers and classification targets in the same embedding space. This enables us to bootstrap to unseen clarification targets and clarification questions, further alleviating the need of expensive annotation.
We evaluate our method on two public tasks: FAQ suggestion (Shah et al., 2018) and bird identification using the text and attribute annotations of the Caltech-UCSD Birds dataset (Wah et al., 2011). The first task represents a virtual assistant application in a trouble-shooting domain, while the second task provides well-defined multiple-choice question annotations and naturally noisy language inputs. We evaluate with both a simulator and human users. Our experiments show that adding user interaction significantly increases the classification accuracy. Given at most five turns of interaction, our approach improves the accuracy of a no-interaction baseline by over  $100\%$  on both tasks for simulated evaluation and over  $90\%$  for human evaluation. Even a single clarification question provides significant accuracy improvements,  $40\%$  for FAQ suggestion and  $65\%$  for bird identification in our simulated analysis. Our code and data are available at https://github.com/asappresearch/interactive-classification.
Our goal is to classify a natural language query to a label through an interaction.
Notation We treat the classification label  $y$ , interaction question  $q$  and the user response  $r$  as random variables. We denote an assignment of a random variable using subscripts, such as  $y = y_{i}$  and  $q = q_{j}$ . We use superscripts for the observed value of the random variable at a given time step, for example,  $q^{t}$  is a question asked at time step  $t$ . When clear from the context, we write  $y_{i}$  instead of  $y = y_{i}$ . For example,  $p(r|q_j,y_i)$  denotes the conditional distribution of  $r$  given  $y = y_{i}$  and  $q = q_{j}$ ,
and  $p(r_k|q_j, y_i)$  further specifies the corresponding probability when  $r = r_k$ .
An interaction starts with the user providing an initial user query  $x$ . At each turn  $t$ , the system selects a question  $q^t$ , to which the user responds with  $r^t$ , or returns a label  $y$  to conclude the interaction. We consider two types of questions: binary and multiple choice questions. The predefined set of possible answers for a question  $q^t$  is  $\mathcal{R}(q^t)$ , where  $\mathcal{R}(q^t) = \{\text{yes, no}\}$  for binary questions, or a predefined set of question-specific values for multiple choice questions. We denote an interaction up to time  $t$  as  $X^t = (x, \langle (q^1, r^1), \ldots, (q^t, r^t) \rangle)$ , and the set of possible class labels as  $\mathcal{Y} = \{y_1, \ldots, y_N\}$ . Figure 1 shows example interactions in our two evaluation domains.
Model We model the interactive process using a parameterized distribution over class labels that is conditioned on the observed interaction (Section 4.1), a question selection criterion (Section 4.2), and a parameterized policy controller (Section 4.5). At each time step  $t$ , we compute the belief of each  $y_{i} \in \mathcal{V}$  conditioned on  $X^{t-1}$ . The trained policy controller decides between two actions: to return the current best possible label or to obtain additional information by asking a question. The model selects the question with the maximal information gain. Given a user response, the model updates the belief over the classification labels.
Learning We use crowdsourced data to bootstrap model learning. The crowdsourcing data collection includes two non-interactive tasks. First, we obtain a set of user initial queries  $\mathcal{X}_i$  for each label  $y_i$ . For example, for an FAQ, 'How do I sign up for Spring Global Roaming', an annotated potential initial query is 'Travel out of country'. Second, we ask annotators to assign text tags to each  $y_i$ , and heuristically convert these tags into a set of question-answer pairs  $\mathcal{A}_i = \{(q_m, r_m)\}_{m=1}^{M_i}$  where  $q_m$  denotes a templatd question and  $r_m$  denotes the answer. For example, the question 'What is your phone operating system?' can pair with one of the following answers: 'IOS', 'Android operating system', 'Windows operating system' or 'Not applicable'. We denote this dataset as  $\{(y_i, \mathcal{X}_i, \mathcal{A}_i)\}_{i=1}^N$ . We describe the data collection process in Section 5. We use this data to train our text embedding model (Section 4.3), to create a user simulator (Section 4.4), and to train the policy controller (Section 4.5).
Evaluation We report classification the model accuracy, and study the trade-off between accuracy and the number of turns that the system takes. We evaluate with both a user simulator and real human users. When performing human evaluation, we additionally collect qualitative ratings.
Human feedback has been leveraged to train natural language processing models, including for dialogue (Li et al., 2016), semantic parsing (Artzi and Zettlemoyer, 2011; Wang et al., 2016; Iyer et al., 2017) and text classification (Hancock et al., 2018). These methods collect user feedback after the model-predicting stage and treat user feedback as additional offline training data to improve the model. In contrast, our model leverages user interaction to increase prediction performance. Human feedback has been incorporated in reinforcement learning as well, for example to learn a reward function from language as reflecting human preferences (Christiano et al., 2017).
Language-based interaction has been studied in the context of visual question answering (de Vries et al., 2017; Lee et al., 2018; Chattopadhyay et al., 2017; Das et al., 2017; Lee et al., 2019; Shukla et al., 2019), SQL generation (Gur et al., 2018; Yao et al., 2019), information retrieval (Chung et al., 2018; Aliannejadi et al., 2019) and multi-turn text-based question answering (Rao and Daumé III, 2018; Reddy et al., 2019; Choi et al., 2018). Most methods require learning from recorded dialogues (Wu et al., 2018; Hu et al., 2018; Lee et al., 2018; Rao and Daumé III, 2018) or conducting Wizard-of-Oz dialog annotations (Kelley, 1984; Wen et al., 2017). Instead, we limit the interaction to multiple-choice and binary questions. This simplification allows us to reduce the complexity of data annotation while still achieving effective interaction. Our task can be viewed as an instance of the popular 20-question game (20Q), which has been applied to a celebrities knowledge base (Chen et al., 2018; Hu et al., 2018). Our approach differs in using natural language descriptions of classification targets, questions and answers to compute our distributions, instead of treating them as categorical or structural data.
Our question selection method is related to several existing methods. Kovashka and Grauman (2013) refine image search by asking to compare visual qualities against selected reference images,
and Lee et al. (2018) perform object identification in an image by posing binary questions about the object or its location. Both methods, as well as ours use an entropy reduction criterion to select the best next question. We use a Bayesian decomposition of the joint distribution, which can be easily extended to other model-driven selection methods. Rao and Daumé III (2018) propose a learning-to-ask approach by modeling the expected utility of asking question. Our selection method can be considered as a special case when entropy is used as the utility. In contrast to Rao and Daumé III (2018), we model the entire interaction history instead of a single turn of follow-up questioning. Our model is trained using crowdsourced annotations, while Rao and Daumé III (2018) uses real user-user interaction data. Alternatively to asking questions, Ferecatu and Geman (2007) and Guo et al. (2018) present to the user the most likely image in an image retrieval scenario. The user compares it with the ground-truth image and provides feedback using relevance score or natural language describing the discrepancy between them.
We maintain a probability distribution  $p(y|X^t)$  over the set of labels  $\mathcal{V}$ . At each interaction step, we first update this belief, decide if to ask a question or return the classification output using a policy controller and, if needed, select a question to ask using information gain.
4.1 Belief Probability Decomposition
We decompose the conditional probability  $p(y = y_i \mid X^t)$  using Bayes rule:
$$
\begin{array}{l} p \left(y _ {i} \mid X ^ {t}\right) = p \left(y _ {i} \mid X ^ {t - 1}, q ^ {t}, r ^ {t}\right) \\ \propto p (r ^ {t}, q ^ {t}, y _ {i} | X ^ {t - 1}) \\ = p \left(q ^ {t} \mid y _ {i}, X ^ {t - 1}\right) p \left(y _ {i} \mid X ^ {t - 1}\right) \\ p (r ^ {t} | q ^ {t}, y _ {i}, X ^ {t - 1}). \\ \end{array}
$$
We make two simplifying assumptions as modeling choices. First, the user response depends only on the question  $q^t$  and the underlying target label  $y_i$ , and is independent of past interactions. While this independence assumption is unlikely to reflect the course of interactions, it allows to simplify  $p(r^t | q^t, y_i, X^{t-1})$  to  $p(r^t | q^t, y_i)$ . Second, the selection of the next question  $q^t$  is deterministic given the interaction history  $X^{t-1}$ . Therefore,
$p(q = q^t |y_i,X^{t - 1}) = 1$  , or zero for  $q\neq q^t$  . Section 4.2 describes this process. We rewrite the decomposition as:
$$
\begin{array}{l} p \left(y _ {i} \mid X ^ {t}\right) \propto p \left(r ^ {t} \mid q ^ {t}, y _ {i}\right) \cdot 1 \cdot p \left(y _ {i} \mid X ^ {t - 1}\right) \\ = p \left(y _ {i} \mid x\right) \prod_ {\tau = 1} ^ {t} p \left(r ^ {\tau} \mid q ^ {\tau}, y _ {i}\right). \tag {1} \\ \end{array}
$$
Predicting the classification label given the observed interaction  $X^t$  is reduced to modeling  $p(y_i|x)$  and  $p(r_k|q_j,y_i)$ , the label  $y_i$  probability given the initial query  $x$  only and the probability of user response  $r_k$  conditioned on the chosen question  $q_j$  and class label  $y_i$ . This factorization enables leveraging separate annotations to learn the two components directly, alleviating the need for collecting costly recordings of user interactions.
4.2 Information Gain Question Selection
The system selects the question  $q^t$  to ask at turn  $t$  to maximize the efficiency of the interaction. We use a maximum information gain criterion. Given  $X^{t - 1}$ , we compute the information gain on classification label  $y$  as the decrease on entropy by observing possible answers to question  $q$ :
$$
I G (y; q | X ^ {t - 1}) = H (y | X ^ {t - 1}) - H (y | X ^ {t - 1}, q),
$$
where  $H(\cdot|\cdot)$  denotes the conditional entropy. Intuitively, the information gain measures the amount of information obtained about the variable  $y$  by observing the value of another variable  $q$ . Because the first entropy term  $H(y|X^{t-1})$  is a constant regardless of the choice of  $q$ , the selection of  $q^t$  is equivalent to  $q^t = \arg \min_{q_j} H(y|X^{t-1}, q_j)$ , where
$$
H (y | X ^ {t - 1}, q _ {j}) = \sum_ {r _ {k} \in \mathcal {R} (q _ {j})} p (r _ {k} | X ^ {t - 1}, q _ {j})
$$
$$
H (y | X ^ {t - 1}, q _ {j}, r _ {k})
$$
$$
H (y | X ^ {t - 1}, q _ {j}, r _ {k}) = \sum_ {y _ {i} \in \mathcal {Y}} p (y _ {i} | X ^ {t - 1}, q _ {j}, r _ {k})
$$
$$
\log p (y _ {i} \mid X ^ {t - 1}, q _ {j}, r _ {k})
$$
$$
\begin{array}{l} p \left(r _ {k} \mid X ^ {t - 1}, q _ {j}\right) = \sum_ {y _ {i} \in \mathcal {Y}} p \left(r _ {k}, y _ {i} \mid X ^ {t - 1}, q _ {j}\right) \\ = \sum_ {y _ {i} \in \mathcal {Y}} p \left(r _ {k} \mid q _ {j}, y _ {i}\right) \\ p (y _ {i} \mid X ^ {t - 1}). \\ \end{array}
$$
We use the independence assumption (Section 4.1) to calculate  $p(r_k|X^{t - 1},q_j)$ . Both  $p(r_k|X^{t - 1},q_j)$  and  $p(y_i|X^{t - 1},q_j,r_k)$  can be iteratively updated
using  $p(y_i|x)$  and  $p(r_k|q_j,y_i)$  as the interaction progresses (Equation 1) to efficiently compute the information gain.
4.3 Modeling the Distributions
We model  $p(y_{i} | x)$  and  $p(r_{k} | q_{j}, y_{i})$  by encoding the natural language descriptions of questions, answers and classification labels. In our domains, the text representation of a label is the FAQ document or the bird name. We do not simply treat the labels, questions and answers as categorical variables. Instead, we leverage their natural language content to estimate their correlation. This reduces the need for heavy annotation and improves our model in low-resource scenarios. We use a shared neural encoder  $\mathsf{enc}(\cdot)$  parameterized by  $\psi$  to encode all texts. Both probability distributions are computed using the dot-product score:  $S(u, v) = \mathsf{enc}(u)^{\top} \mathsf{enc}(v)$ , where  $u$  and  $v$  are two pieces of text. The probability of predicting the label  $y_{i}$  given an initial query  $x$  is:
$$
p (y _ {i} | x) = \frac {\exp (S (y _ {i} , x))}{\sum_ {y _ {j} \in \mathcal {Y}} \exp (S (y _ {j} , x))}.
$$
The probability of an answer  $r_k$  given a question  $q_j$  and label  $y_i$  is a linear combination of the observed empirical distribution  $\hat{p}(r_k | q_j, y_i)$  and a parameterized estimation  $\tilde{p}(r_k | q_j, y_i)$ :
$$
p (r _ {k} | q _ {j}, y _ {i}) = \lambda \hat {p} (r _ {k} | q _ {j}, y _ {i}) + (1 - \lambda) \tilde {p} (r _ {k} | q _ {j}, y _ {i}),
$$
where  $\lambda \in [0,1]$  is a hyper-parameter. We use the question-answer annotations  $\mathcal{A}_i$  for each label  $y_{i}$  to estimate  $\hat{p} (r_k|q_j,y_i)$  using empirical counts. For example, in the FAQ suggestion task, we collect multiple user responses for each question and class label, and average across annotators to estimate  $\hat{p}$  (Section 5). The second term  $\tilde{p} (r_k|q_j,y_i)$  is computed using the text encoder:
$$
\begin{array}{l} \tilde {p} \left(r _ {k} \mid q _ {j}, y _ {i}\right) \\ = \frac {\exp (w \cdot S (q _ {j} \# r _ {k} , y _ {i}) + b)}{\sum_ {r _ {l} \in \mathcal {R} (q _ {j})} \exp (w \cdot S (q _ {j} \# r _ {l} , y _ {i}) + b)}, \\ \end{array}
$$
where  $w, b \in \mathbb{R}$  are scalar parameters and  $q_j \# r_k$  is a concatenation of the question  $q_j$  and the answer  $r_k$ . Because we do not collect complete annotations to cover every label-question pair,  $\tilde{p}$  provides
'For example, for a templated question 'What is your phone operating system?' and an answer 'IOS',  $q_{m} =$  'phone operating system' and  $r_{m} =$  'IOS', therefore,  $q_{m} \# r_{m} =$  'phone operating system IOS'.
a smoothing of the partially observed counts using the learned encoding  $S(\cdot)$ .
We estimate the parameters  $\psi$  of  $\mathbf{enc}(\cdot)$  by pretraining using a dataset  $\{(y_i, \mathcal{X}_i, \mathcal{A}_i)\}_{i=1}^N$ , where  $y_i$  is a label,  $\mathcal{X}_i$  is a set of initial queries and  $\mathcal{A}_i$  is a set of question-answer pairs. We create from this data a set of text pairs  $(u, v)$  to train the scoring function  $S(\cdot)$ . For each label  $y_i$ , we create pairs  $(x, y_i)$  for each initial query  $x \in \mathcal{X}_i$ . We also create  $(q_m \# r_m, y_i)$  for each question-answer pair  $(q_m, r_m) \in \mathcal{A}_i$ . We minimize the cross-entropy loss using gradient descent:
$$
\mathcal {L} (\psi) = - S (u, v) + \log \sum_ {v ^ {\prime}} \exp \left(S (u, v ^ {\prime})\right).
$$
The second term requires summation over all  $v'$ , which are all the labels in  $\mathcal{V}$ . We approximate this sum using negative sampling that replaces the full set  $\mathcal{V}$  with a sampled subset in each training batch. The parameters  $\psi, w$  and  $b$  are fine-tuned using reinforcement learning during training of the policy controller (Section 4.5).
4.4 User Simulator
We use a held-out dataset to build a simple simulator. We use the simulator to train the policy controller (Section 4.5) and for performance analysis, in addition to human evaluation. The user simulator provides initial queries to the system and responds to the system initiated clarification questions. The dataset includes  $N$  examples  $\{(y_i,\mathcal{X}_i',\mathcal{A}_i')\}_{i = 1}^N$ , where  $y_{i}$  is a goal,  $\mathcal{X}_i^\prime$  is a set of initial queries and  $\mathcal{A}_i' = \{(q_m,r_m)\}_{m = 1}^{M_i'}$  is a set of question-answer pairs. While this data is identical in form to our training data, we keep it separated from the data used to estimate  $S(\cdot)$ ,  $p(y_i|x)$  and  $p(r_k|q_j,y_i)$  (Section 4.3). We estimate the simulator question response distribution  $p'(r_k|q_j,y_i)$  using smoothed empirical counts from the data.
At the beginning of a simulated interaction, we sample a target label  $\hat{y}$ , and sample a query  $x$  from the associated query set  $\mathcal{X}'$  to start the interaction. Given a system clarification question  $q^t$  at turn  $t$ , the simulator responds with an answer  $r^t \in \mathcal{R}(q^t)$  by sampling from  $p'(r|q^t, \hat{y})$ . Sampling provides natural noise to the interaction, and our model has no knowledge of  $p'$ . The interaction ends when the system returns a label, which we can then evaluate, for example to compute a reward in Section 4.5. This setup is flexible in that the user simulator can be easily replaced or extended by a real human, and
the system can be further trained with a human-in-the-loop setup.
4.5 Policy Controller
The policy controller decides at each turn  $t$  to either select another question to query the user or to conclude the interaction. This provides a tradeoff between exploration by asking questions and exploitation by returning the most probable classification label. The policy controller  $f(\cdot, \cdot; \theta)$  is a feed-forward network parameterized by  $\theta$  that takes the top-  $k$  probability values and current turn  $t$  as input. It generates one of two actions: STOP or ASK. When selecting ASK, a question is selected to maximize the information gain. For STOP, the label  $y_i$  with highest probability is returned using  $\arg \max_{y_i \in \mathcal{Y}} p(y_i | X^{t-1})$  and the interaction ends.
4.6 Training Procedure
Algorithm 1 describes the complete training process. First, we estimate  $p(y|x)$  and  $p(r|q,y)$ . We use randomly initialized and fixed  $w$  and  $b$  parameters. We also estimate  $p'(r|q,y)$  for the user simulator (Section 4.4). We then learn the policy controller using the user simulator with a policy gradient method. We use the REINFORCE algorithm (Williams, 1992). The reward function provides a positive reward for predicting the correct target at the end of the interaction, a negative reward for predicting the wrong target, and a small negative reward for every question asked. We learn the policy controller  $f(\cdot ,\cdot ;\theta)$ , and estimate  $w$  and  $b$  in  $p(r_k|q_j,y_i)$  by back-propagating through the
policy gradient. We keep the  $\mathbf{enc}(\cdot)$  parameters fixed during policy gradient.
5 Data Collection
We design a crowdsourcing process to collect data for the FAQ task using Amazon Mechanical Turk. For the Birds domain, we re-purpose an existing dataset. We collect initial queries and tags for each FAQ document. Appendix A.1 describes the worker training process.
Initial Query Collection We ask workers to consider the scenario of searching for an FAQ document using an interactive system. Given a target FAQ, we ask for an initial query that they would provide to such a system. The set of initial queries that is collected for each document  $y_{i}$  is  $\mathcal{X}_i$ . We encourage workers to provide incomplete information and avoid writing a simple paraphrase of the FAQ. This process provides realistic and diverse utterances because users have limited knowledge of the system and the domain.