SlowGuess/NAACL_2022_OCR huggingface dataset:
Mutual Information between random variables  $X, Y$  is defined as in Equation (2). We consider a weighted Mutual Information metric as defined as in Equation (3) similar to Li and Jurafsky (2016) and introduce the hyperparameter  $\lambda$ . Identifying  $Y$  that maximizes the weighted Mutual Information is equivalent to maximizing the expression in Equation (4). We use this metric to rank hypotheses generated by the language model.
$$
\operatorname {M I} (X, Y) = \log \frac {p (x , y)}{p (x) p (y)} \tag {2}
$$
$$
\operatorname {w M I} (X, Y) = \log \frac {p (x , y)}{p (x) p (y) ^ {\lambda}} \tag {3}
$$
$$
\begin{array}{l} \operatorname * {a r g m a x} _ {y} \mathrm {w M I} (X, Y) \\ = \operatorname {a r g m a x} _ {y} \log \frac {p (x , y)}{p (x) p (y) ^ {\lambda}} \\ = \operatorname * {a r g m a x} _ {y} \log \left(\frac {p (x , y)}{p (x)}\right) ^ {1 - \lambda} \left(\frac {p (x , y)}{p (y)}\right) ^ {\lambda} \frac {1}{p (x) ^ {\lambda}} \\ = \operatorname * {a r g m a x} _ {y} (1 - \lambda) \log p (y | x) + \lambda \log p (x | y) - \lambda \log p (x) \\ = \underset {y} {\operatorname {a r g m a x}} (1 - \lambda) \log p (y | x) + \lambda \log p (x | y) \tag {4} \\ \end{array}
$$
B Subgoal representation
Table 6 shows the subgoal representation we use in this work.
Subgoal	Representation
(Pickup, X)	pick up X
(Put, X)	put in X
(Heat, X)	heat in X
(Cool, X)	cool in X
(Clean, X)	clean in X
(Slice, X)	slice X
(ToggleOn, X)	turn on X
Table 6: Subgoals and corresponding text representation. X represents an object argument.
C Ranking model: Architecture
State embedding HLSM represents the agent state as a semantic voxel representation  $s \in [0,1]^{X \times Y \times Z \times C}$  where the value  $s(x,y,z,c)$  represents if there is an object of type  $c$  at position  $(x,y,z)$  of the room layout. We pool across the spatial dimensions of the representation and project it using a linear mapping to obtain  $f^{\mathrm{state}}(s)$ . We use this encoding as the state embedding.
Instruction and subgoal sequence encoding The instruction  $\tau$  and a candidate subgoal sequence  $g$  are jointly processed using a BERT encoder and the CLS representation is used as a representation vector  $\mathrm{BERT}_{\mathrm{CLS}}(\tau ,g)$  .The ranking model is represented as  $f(g,\tau ,s;\theta) =$ $\theta^T\mathrm{concat}(f^{\mathrm{state}}(s),\mathrm{BERT}_{\mathrm{CLS}}(\tau ,g))$  where  $\theta$  is a parameter vector.
D Ranking Model: Training and Inference
Formally, the learning problem is a MDP  $(\mathcal{S},\mathcal{G},\mathcal{L},\mathcal{R},\mathcal{T})$ , where  $s_t \in \mathcal{S}$  is the agent state,  $g^{(t)} \in \mathcal{G}$  is a subgoal,  $\tau \in \mathcal{L}$  is a text instruction,  $\mathcal{R}(\tau, s_t)$  is a reward function that provides success/failure feedback for completing a given instruction,  $\mathcal{T}: (s_t, g^{(t)}) \to s_{t+1}$  is a state transition function where  $s_{t+1}$  is computed by a low-level policy  $\pi_L$  pre-trained to execute a given subgoal  $g$ . Our goal is to train a high-level policy  $\pi_H(g^{(t)}|\tau, s_t, g^{(< t)})$  where  $s_t$  is the agent state and  $g^{(< t)}$  is the sequence of subgoals completed so far at high-level time-step  $t$ .
Algorithm 1 describes how we collect training data to train the ranking model. Algorithm 2 shows how the ranking model is used during inference.
E Analysis of model errors
Figure 4 shows the confusion matrix for object type prediction. Predictions are from top-1 subgoal sequences predicted by GPT-J.
Figure 4: Object type prediction confusion matrix.