SlowGuess/ACL_2020_OCR huggingface dataset:
Substituting this in Eqn 1 and using Jensen's inequality, any distribution  $q$  over logical forms provides a lower-bound on the log-likelihood:
$$
\begin{array}{l} \mathcal {L} (\theta) \geq \sum_ {l} q (l) \log \frac {p (x | l) \mathbb {I} _ {\text {V a l i d} (l)}}{q (l)} \tag {3} \\ = \sum_ {l} q (l) \left(\log p (x | l) + \log \mathbb {I} _ {\text {V a l i d} (l)}\right) + \mathcal {H} _ {q} \\ \end{array}
$$
where  $\mathcal{H}_q$  is the entropy for distribution  $q$ . In Sec 4.1, we represent  $l$  as a tuple of variables. Next, we make a mean field approximation by assuming the distribution  $q(l)$  decomposes as:
$$
q (l) = q \left(e _ {0}, a, t\right) \prod_ {i} q _ {f _ {0 i}} q _ {e _ {1}} q _ {r} \prod_ {i} q _ {f _ {1 i}} \prod_ {j} q _ {f _ {t j}} \tag {4}
$$
Focusing on the unobserved variables (given a demonstration), we have  $q(l) = q_{f_0}q_{e_1}q_rq_{f_1}q_{f_t}$ .<sup>5</sup>
Parsing model: We assume that the probability of an explanation decomposes into the probability of individual words as  $\log p_{\theta}(x|l) = \sum_{w\in x}\log p(w|f_0,r,f_1,f_t,a)$ . Further, we assume that individual words are generated from features, relations and actions in the logical form as:
$$
\begin{array}{l} \log p(w|f_{0},r,f_{1},f_{t},a) = \log \frac{1}{C}\sum_{k}^{\substack{\{f_{0},r,f_{1},\\ f_{t},a\}}}p(w|k)^{z_{kw}}p(z_{kw}) \\ \geq \sum_ {k} ^ {\left\{f _ {0}, r, f _ {1}, \right.} b _ {k w} \left(\log p (w | k, z _ {k w}) + \log p \left(z _ {k w}\right)\right) + \mathcal {H} _ {b _ {k w}} \tag {5} \\ \end{array}
$$
Here,  $k$  is an index over values of  $f_0, r, f_1, f_t$  and  $a$ .  $z_{kw}$  denotes an alignment between a particular value of a feature, relation or action  $(k)$  and word  $w$  in the explanation, in which case the word is generated from the distribution  $p(w|k)$ . The presence of a summation inside of a logarithm makes maximizing this objective hard. We again use Jensen's inequality to get a bound by introducing variational distributions  $b_{kw}$  over alignments  $z_{kw}$ .  $b_{kw}$  can be thought of as representing the proportions of an explanation word contributed by specific feature values, relations or actions  $k$  in the logical form. Each  $p(w|k)$  is parameterized as a multinomial distribution,  $\theta_{kw}$ , over the vocabulary.
Training and Inference: Our model training follows a variational EM approach, where in the E-step, we perform inference for the latent logical form variables and alignment proportions, keeping the model parameters as fixed. In the M-step, we update the parameters,  $\theta_{kw}$ , taking the variational distributions and alignments as fixed. Combining Eqn 2, Eqn 3 and Eqn 5, we get:
$$
\begin{array}{l} \mathcal {L} (\theta) \geq \sum_ {l} q _ {f _ {0}} q _ {e _ {1}} q _ {r} q _ {f _ {1}} q _ {f _ {t}} \left(\left(\sum_ {w} \sum_ {k} b _ {k w} [ \log \theta_ {k w} ^ {z _ {k w}} \right. \right. \\ \left. + \log p \left(z _ {k w}\right) \right] + \mathcal {H} _ {b _ {k w}}) + \log \mathbb {I} _ {\text {V a l i d} (l, d, c)}\bigg) \tag {6} \\ + \mathcal {H} _ {f _ {0}} + \mathcal {H} _ {e _ {1}} + \mathcal {H} _ {r} + \mathcal {H} _ {f _ {1}} + \mathcal {H} _ {f _ {t}} \\ \end{array}
$$
Maximizing this objective w.r.t. the variational
- DOM elements or strings on the webpage – to search over. Compare this with an operation in arithmetic, e.g., add(int, int), which might require a search over infinite co-domains.
Using  $q_{f_0}$  as shorthand notation for the product of variational distributions  $\prod_i q_{f_{0i}}$ , and so on.
distributions yields the following E-step updates:
$$
q_{f_{0i}}(v_{f_{0i}})\propto \exp \Big(\sum_{w}b_{v_{f_{0i}}w}\log (\theta_{v_{f_{0i}}w}) + \log \mathbb{I}_{\substack{Valid(v_{f_{0i}},\\ e_{0},c)}}\Big)
$$
$$
\begin{array}{l} q _ {e _ {1}} (v _ {e _ {1}}) \propto \exp \left(\left(\sum_ {f _ {1 i}} \sum_ {w} b _ {f _ {1 i} w} \sum_ {v _ {f _ {1 i}}} q _ {f _ {1 i}} (v _ {f _ {1 i}}) \log \theta_ {v _ {f 1 i w}} \right. \right. \\ + \log \mathbb {I} _ {\left(v _ {f _ {1 i}}, e _ {1}, c\right)}) + \left(\sum_ {v _ {r}} q _ {r} (v _ {r}) \sum_ {w} b _ {v _ {r} w} \log \theta_ {v _ {r} w} \right. \\ \left. + \log \mathbb {I} _ {\text {V a l i d} (e _ {1}, v _ {r}, e _ {0}, c)}\right) \\ q _ {r} (v _ {r}) \propto \exp \left(\sum_ {v _ {e _ {1}}} q _ {e _ {1}} (v _ {e _ {1}}) \left(\sum_ {w} b _ {v _ {r} w} \log \theta_ {v _ {r} w}\right) \right. \\ \left. + \log \mathbb {I} _ {\text {V a l i d} (e _ {1}, v _ {r}, e _ {0}, c)\right) \\ \end{array}
$$
$$
\begin{array}{l} q _ {f _ {1 i}} \left(v _ {f _ {1 i}}\right) \propto \exp \left(\sum_ {v _ {e _ {1}}} q _ {e _ {1}} \left(v _ {e _ {1}}\right) \left(\sum_ {w} b _ {v _ {f _ {1 i} w}} \log \theta_ {v _ {f _ {1 i} w}}\right) \right. \\ \left. + \log \mathbb {I} _ {V a l i d (v _ {f _ {1 i}}, e _ {1}, c)\right)} \\ \end{array}
$$
$$
\begin{array}{l} q _ {f _ {t j}} \left(v _ {f _ {t j}}\right) \propto \exp \left(\sum_ {w} b _ {v _ {f _ {t j}} w} \log \left(\theta_ {v _ {f _ {t j}} w}\right) \right. \\ \left. + \log \mathbb {I} _ {\text {V a l i d} \left(v _ {f _ {t j}}, t, c\right)}\right) \tag {7} \\ \end{array}
$$
Similarly, the updates for the alignment proportions (taking  $p(z_{kw})$  in Eqn 6 to be uniform) are:
$$
b _ {k w} \propto \exp \left(\sum_ {k} q _ {k} (k) \log \theta_ {k w}\right) \tag {8}
$$
LED(+syntax): The above approach allows for arbitrary alignments between words and features, relations or actions in the grounded logical form  $(k)$ , essentially representing  $x$  as a bag-of-words. We also explore a variant that models  $x$  as a sequence of tokens by introducing a prior over joint alignments  $\mathbf{z}_{kx} = z_{k_1w_1}\ldots z_{k_tw_T}$  in a sentence  $x\coloneqq w_{1}\ldots w_{T}$  (in Eqn 5). This is done by simply modeling  $p(\mathbf{z}_{kx})$  with pairwise transition probabilities as  $p(\mathbf{z}_{kx})\coloneqq \prod_n p(z_{k_t}|z_{k_{t - 1}}) = \prod_n\mathcal{T}_{k_t,k_{t - 1}}$ . In this case, updates for alignment proportions (Eqn. 8) correspond with emission probabilities in a HMM (which we omit here for brevity).
Since the updates in Eqn 7 and Eqn 8 are cyclic, in each E-step, we make 20 iterations of updates to the variational distributions and alignment proportions in a round-robin schedule. We note that consistency is enforced during training by the log-of-indicator-variable terms in Eqn 7. This is because any inconsistent assignments get a score of  $\log (0)$  which tends to negative infinity. However, to ensure smooth training (and alleviate modeling issues from our mean field approximation), we leverage an annealing based strategy, where we incremen
tally increase the penalty for  $\log (0)$  terms during training as  $-N / 2$  for the  $N$ 'th EM iteration (for large  $N$ , this also is a prohibitive penalty). In our experiments, this was seen to improve training.
In the M-step, we maximize the objective w.r.t.
$$
\theta_ {k}:
$$
$$
\theta_ {k} (w) \propto \exp \left(\sum_ {n} \sum_ {w \in x _ {n}} b _ {k w} ^ {n} q _ {k}\right) \tag {9}
$$
The one exception is a special copy mechanism for string-valued features. For these,  $\theta_{kw}$  is not learned, but simply corresponds to an indicator function denoting if  $w$  matches the value of the feature. e.g.,  $\theta_{\mathrm{HasText}(\mathrm{submit})}, \mathrm{submit}' = 1$ .
We next discuss LED's empirical performance.
5.1 Procedure Learning performance
First, we evaluate the method for completion rates on tasks from the MiniWoB framework. Following Liu et al. (2018), we filtered 40 tasks from the MiniWoB framework (Shi et al., 2017) that require only clicking and typing actions. During training of the LED model, we sample an explained demonstration for each of the 40 tasks, and models are trained on the aggregate of these (the model sees one explanation-demonstration pair for a task). For testing, models are evaluated on a new instance of a task, where the model greedily computes the demonstration  $d$  (specifying a click or typing action on a web element in the current DOM) that would maximize  $p(x|d,c)$  (see Eqn 1) and executes the corresponding actions. The method then moves to the next explanations. This requires an enumeration of all possibly clicking and typing actions that can be performed in a context  $c$  at every step.<sup>8</sup> Since the number of actions in a demonstration can be different from the number of steps in the explanation, we heuristically align the sequence of actions in demonstrations to the sequence of sentences in the explanations in our dataset based on a small manually defined list of trigger words.
A direct comparison of LED with other approaches is not possible, since they differ considerably in the type of supervision and resources used. Nonetheless, here we compare LED's performance with the following two methods to get a coarse sense of its effectiveness:
This is possible since the set of actionable elements on a webpage, and the set of candidate strings that can be typed (up to two length tokens from task description) are not large.
The optimal value for the concave problem  $\sum_{j}x_{j}\log \frac{y_{j}}{x_{j}}$  s.t.  $\sum_{j}x_{j} = 1$  is achieved when  $x_{j}^{*}\propto y_{j}$
E.g., this won't differentiate between "click the URL below the button" and "click the button below the URL".
Figure 5: Task-completion rates for MiniWoB tasks with varying difficulty. Rates are calculated over 100 new instances of each task
1. SemParse: This is a supervised semantic parsing baseline, trained on a manually annotated dataset of around 300 explanations labeled with their DSL logical forms (covering roughly one annotated explanation sequence for every task). The model is based on a sequence-to-sequence neural semantic parser from Jia and Liang (2016). During testing, the method parses the sequence of explanations to logical forms, and sequentially (attempts to) executes the predicted logical forms. In contrast, LED requires no logical form annotations. However, it leverages the inverse semantics of the DSL operators, which may not be feasible for every DSL.
2.  $BC + RL$ : This is the original approach from Shi et al. (2017), who proposed the MiniWoB framework and consists of behavior cloning and exploration. This learns a task by supervised learning on about 200 demonstrations, followed by exploration via reinforcement learning to fine-tune the learned policies. In comparison, LED requires no exploration of the environment but leverages additional supervision in the form of natural language explanations. Multiple methods have since explored other RL-based approaches, resulting in much improved performance (Liu et al., 2018; Luo, 2019; Jia et al., 2019). In particular, Liu et al. (2018) leverage a constraint language similar to our DSL to train a RL policy to get large gains in performance. However, all these methods require multiple demonstrations and exploration of the environment.
Figure 5 shows task completion performance for different methods on a subset of tasks from the MiniWoB framework. We compute task comple
Approach	Action-prediction accuracy
LED (+Syntax)	0.45
LED	0.43
SemParse	0.39
Random	0.28
Table 2: Semantic parsing performance (predicted action match) for interpreting individual explanations in a context
tion rates over 100 randomly selected test instances of each task. The differences between instances involves different arguments for a task and differences in the state of the environment. Firstly, we note that the LED approaches consistently outperforms SemParse across all tasks. This is a strong result, since LED does not have access to logical form annotations for explanations as SemParse does. This strongly indicates that knowledge of the pragmatic context is important for language interpretation in this domain, since our approach which roots logical forms in observed demonstrations performs better or as well for all but one task.
We note that there is a large variance among tasks in terms of amenability to learning from explanations or exploration. For tasks like tic-tactoe, explanation-based methods perform poorly as expected, since learning the game involves reasoning that is hard to explain through step-wise explanation of a demonstration, but can be more naturally learned from exploration. On the other hand, explanation-based methods perform well on tasks that are easily expressed through language. On the whole, the LED approaches and are roughly competitive with  $BC + RL$ , while requiring no exploration and only a single demonstration. Note that unlike exploration-based methods, LED and SemParse can potentially generalize to new tasks during testing (where no demonstration is seen during training) from explanations and context only.
We also note that  $LED(+Syntax)$  generally outperforms vanilla  $LED$ , although the effect size is not large. However, this trend is statistically significant (binomial test,  $p < 0.1$ ).
5.2 Language Interpretation performance
Next, we quantitatively evaluate the parsing performance of our method at the level of individual explanations (rather than task completion rate). For this, we evaluate the trained models on explanations from a set of 80 demonstrations from the dataset (unseen during training), where we calculate the match between the predicted action from an explanation in the context, and the actual action in the logged demonstration (accuracy of pre
Figure 6: Heatmap showing learned values of  $\theta_{kw}$  for 20 frequent words  $w$  in and representative values of  $k$ . Darker shades correspond to higher probability values.
dicted action in a context). Table 2 summarizes this performance, which shows a similar trend as Section 5.1. Both LED methods perform substantially better than SemParse, and all three methods perform much better than randomly choosing the next executable action in the context (Random). We note again that LED's involves no logical form annotations, and is driven purely by grounding explanations in observed demonstrations.
5.3 Visualizing learned language
Figure 6 depicts the learned lexicon by visualizing a representative subset of learned  $\theta_{kw}$  values for LED (+Syntax) (from Sec 4.2) as a heatmap. We note that the model correctly induces mappings between words and DSL operators. The rows and columns are manually ordered to emphasize the block diagonal structure.
Table 3 shows the learned transition probabilities,  $\mathcal{T}_{k_1,k_2}$ , for LED (+Syntax). To reduce model size, we share parameters for values of  $k$  corresponding to types  $f_{0},r,f_{1},f_{t}$  and  $a$ . A common template about the general structure of user explanations is reflected from the parameter values. Most explanations start with the description of the action  $a$ , followed by mentioning features that identify the relevant element  $f_{0}$ . In fact,  $f_{0}$  distributions generate the majority of words in most explanations. Relation mentions, when present, usually follow this, in turn followed by features corresponding to  $f_{0}$ , reflective of a VSO word order in most explanations. Diagonal values are substantially higher, indicating that words describing specific objects and actions tend to cluster together, as would be expected from the semantics of natural language.
5.4 Common Errors
From a qualitative error analysis, we note that most errors in task learning come from three sources. Firstly, although the method learns reasonable mappings between words and semantic operators, the method often misaligns attributes of different ele
a	f0	r	f1	ft
a	0.12	0.70	0.07	0.04	0.07
f0	0.05	0.82	0.08	0.04	0.01
r	0.01	0.12	0.57	0.26	0.04
f1	0.03	0.07	0.22	0.63	0.04
ft	0.08	0.30	0.09	0.06	0.47
Table 3: Learned transition probabilities between latent variable categories for LED (+Syntax). These reflect a prominence of VSO sentence structures in user explanations.
ments, even with the  $LED(+Syntax)$  model. This is likely because the training data is not adequate to learn these constraints, and methods that enforce these through informed priors maybe more effective. Another common error is due to challenges with anaphora resolution and discourse referents. Finally, a large number of explanations are not explicit in describing the sequence of actions required to perform a task, and some needed actions remain unmentioned. While this would be expected in realistic computer-human interactions, fixing these errors is beyond the scope of the current method.
Our work here is a step in the direction of teachable AI agents that can learn new behavior from conversational interactions with ordinary users. In terms of technique, our bottom-up approach to generating logical forms ensures consistency between interpretations and the ambient context during search. Conversely, this would be complicated in domains with rich composition and nesting in logical forms, which go beyond simple features and relations. e.g., "click the third email from Jeanette", and where modeling inverse semantics is infeasible.
Here, we posed the learning of web-based tasks as similar to instruction-following problem, with no aspect of interactivity or exploration of the environment. In future work, the possibility of learning from a mix of explanations, exploration and a limited budget of interaction with the environment can be explored. Also, language grounding models that incorporate richer alignments between explanations and demonstrations can lead to more effective learning. Since LED only requires tokenization as pre-processing, it can possibly extend to low resource scenarios. In terms of problem framing, interactive use-cases that enable the agent to ask questions when it is confused may also be realistic. Future work can also explore curriculum learning in this domain, by first learning simpler tasks, which can be compositionally invoked in explanations for complex tasks.
7660
Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2166-2179. Association for Computational Linguistics.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 39-48. IEEE Computer Society.
Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 421-432, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 82-90, Suntec, Singapore. Association for Computational Linguistics.
James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. Driving semantic parsing from the world's response. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 18–27, Uppsala, Sweden. Association for Computational Linguistics.
Michael C. Frank and Noah D. Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998-998.
Dan Goldwasser and Dan Roth. 2014. Learning from natural instructions. *Mach. Learn.*, 94(2):205-232.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Sheng Jia, Jamie Kiros, and Jimmy Ba. 2019. DOM-Q-NET: grounded RL on structured language. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jayant Krishnamurthy and Tom Mitchell. 2012. Weakly supervised training of semantic parsers. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 754-765, Jeju Island, Korea. Association for Computational Linguistics.
Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1223-1233, Cambridge, MA. Association for Computational Linguistics.
John E. Laird, Kevin A. Gluck, John R. Anderson, Kenneth D. Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario D. Salvucci, Matthias Scheutz, Andrea Thomaz, J. Gregory Trafton, Robert E. Wray, Shiwali Mohan, and James R. Kirk. 2017. Interactive task learning. IEEE Intell. Syst., 32(4):6-21.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456-1465, Berlin, Germany. Association for Computational Linguistics.
Katie Luo. 2019. Goal-induced inverse reinforcement learning. Technical Report EECS-2019-81, University of California, Berkeley.
7661