SlowGuess/NAACL_2022_OCR huggingface dataset:
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation.
Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould. 2020. Language and visual entity relationship graph for agent navigation. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS).
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Shuhei Kurita and Kyunghyun Cho. 2021. Generative language-grounded policy in vision-and-language navigation with bayes' rule. In Proceedings of the International Conference on Learning Representations (ICLR).
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A Smith, and Yejin Choi. 2019. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Chong Liu, Fengda Zhu, Xiaojun Chang, Xiaodan Liang, Zongyuan Ge, and Yi-Dong Shen. 2021. Vision-language navigation with random environmental mixup. In Proceedings of the International Conference on Computer Vision (ICCV).
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. In Proceedings of the International Conference on Learning Representations (ICLR).
Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. Self-monitoring navigation agent via auxiliary progress estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
Gabriel Magalhães, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. 2019. General evaluation for instruction conditioned navigation using dynamic time warping. In Workshop on Visually Grounded Interaction and Language (ViGIL).
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. 2020. Improving vision-and-language navigation with imag-text pairs from the web. In Proceedings of the European Conference on Computer Vision (ECCV).
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt: A tool for holistic comparison of language generation systems. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) Demonstrations.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. 2021a. Meta pseudo labels. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Hieu Pham, Xinyi Wang, Yiming Yang, and Graham Neubig. 2021b. Meta back-translation. In Proceedings of the International Conference on Learning Representations (ICLR).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML).
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022. How much can clip benefit vision-and-language tasks? In Proceedings of the International Conference on Learning Representations (ICLR).
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, and Jianbing Shen. 2021. Structured scene memory for vision-language navigation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In Proceedings of the European Conference on Computer Vision (ECCV).
Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. 2020. Optimizing data usage via differentiable rewards. In Proceedings of the International Conference on Machine Learning (ICML).
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations (ICLR).
Ming Zhao, Peter Anderson, Vihan Jain, Su Wang, Alexander Ku, Jason Baldridge, and Eugene Ie. 2021. On the evaluation of vision-and-language navigation instructions. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL).
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. 2020a. Vision-language navigation with self-supervised auxiliary reasoning tasks. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR).
Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. 2020b. Babywalk: Going farther in vision-and-language navigation by taking baby steps. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
A Derivation of the Speaker Gradient
As shown in Section 2.2, at training step  $t$ , we update the follower according to:
$$
\theta_ {F} ^ {t} = \theta_ {F} ^ {t - 1} - \eta_ {F} \nabla_ {\theta_ {F}} L _ {u} \left(\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}\right). \tag {7}
$$
Model	Train	Val-Seen	Val-Unseen
BLEU
EnvDrop	38.16	32.42	31.13
FOAM	39.66	33.11	31.10
BERTScore
EnvDrop	91.64	91.08	91.04
FOAM	91.79	91.08	91.10
We then derive the speaker gradient following previous work (Pham et al., 2021b,a). We define the expected parameters of the follower as  $\bar{\theta}_F^t$ :
$$
\bar {\theta} _ {F} ^ {t} = \mathbb {E} _ {\hat {\mathbf {r}} _ {k} \sim \mathcal {E}, \hat {\mathbf {i}} _ {k} \sim P (\mathbf {i} | \hat {\mathbf {r}} _ {k}; \theta_ {S} ^ {t - 1})} [ \theta_ {F} ^ {t - 1} - \eta_ {F} \nabla_ {\theta_ {F}} L _ {u} \left(\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}\right) ]. \tag {8}
$$
Then, using the chain rule, we can obtain
$$
\nabla_ {\theta_ {S}} L _ {l} = \frac {\partial L _ {l}}{\partial \bar {\theta} _ {F} ^ {t}} \frac {\partial \bar {\theta} _ {F} ^ {t}}{\partial \theta_ {S}}, \tag {9}
$$
where the first term can be approximated with  $\frac{\partial L_l}{\partial\theta_F^t}$ . Then, for the second term, we have
$$
\frac {\partial \bar {\theta} _ {F} ^ {t}}{\partial \theta_ {S}} = \frac {\partial}{\partial \theta_ {S}} \mathbb {E} _ {\hat {\mathbf {r}} _ {k} \sim \mathcal {E}, \hat {\mathbf {i}} _ {k} \sim P (\mathbf {i} | \hat {\mathbf {r}} _ {k})} \left[ \theta_ {F} ^ {t - 1} - \eta_ {F} \nabla_ {\theta_ {F}} L _ {u} \left(\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}\right) \right]. \tag {10}
$$
We can assume that  $\theta_F^{t - 1}$  does not depend on  $\theta_S$  with Markov assumption (Pham et al., 2021a), and apply the REINFORCE (Williams, 1992) equation on the second term:
$$
\begin{array}{l} \frac {\partial \bar {\theta} _ {F} ^ {t}}{\partial \theta_ {S}} = \frac {\partial}{\partial \theta_ {S}} \mathbb {E} _ {\hat {\mathbf {r}} _ {k} \sim \mathcal {E}, \hat {\mathbf {i}} _ {k} \sim P (\mathbf {i} | \hat {\mathbf {r}} _ {k})} [ - \eta_ {F} \nabla_ {\theta_ {F}} L _ {u} (\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}) ] \\ = - \eta_ {F} \mathbb {E} _ {\hat {\mathbf {r}} _ {k} \sim \varepsilon , \hat {\mathbf {i}} _ {k} \sim P (\mathbf {i} | \hat {\mathbf {r}} _ {k})} \left[ \nabla_ {\theta_ {F}} L _ {u} \left(\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}\right) \frac {\partial}{\partial \theta_ {S}} \log P \left(\hat {\mathbf {i}} _ {k} | \hat {\mathbf {r}} _ {k}; \theta_ {S} ^ {t - 1}\right) \right], \tag {11} \\ \end{array}
$$
Using Monte Carlo approximation to approximate terms in Equation 11 using a batch of samples and substituting the result into Equation 9, we can get
$$
\nabla_ {\theta_ {S}} L _ {l} = - \eta_ {F} \left[ \nabla_ {\theta_ {F}} L _ {l} \left(\theta_ {F} ^ {t}\right) ^ {\mathrm {T}} \nabla_ {\theta_ {F}} L _ {u} \left(\theta_ {F} ^ {t - 1}, \theta_ {S} ^ {t - 1}\right) \right] \nabla_ {\theta_ {S}} \log P \left(\hat {\mathbf {i}} _ {k} \mid \hat {\mathbf {r}} _ {k}; \theta_ {S} ^ {t - 1}\right)
$$
Note that here  $\eta_F$  is a hyper-parameter and can be incorporated into the learning rate of the speaker  $\eta_S$ , thus we remove this term in Section 2.2 and our derivation is complete.
B Evaluations of the Generated Instructions
Automatic Evaluations. As in Table 5, we measure the quality of the generated instructions in BLEU (Papineni et al., 2002) and
BERTScore (Zhang et al., 2020). We find that our speaker can generate instructions of higher qualities according to the two metrics.
Table 5: Automatic evaluations of the generated instructions. The instructions generated by our model can obtain higher BLEU and BERTScore than the baseline.
Method	Instruction
Reference	walk downstairs and outside . stop in the out-house through the door on the right .
EnvDrop	go down the stairs and turn right . go down the hallway and stop in front of the door .
FOAM	go down the stairs and turn right . go down the hallway and go through the door on the right .
Reference	turn left and take a right at the table . take a left at the painting and then take your first right . wait next to the exercise equipment .
EnvDrop	walk past the dining room table and chairs and turn left . walk past the table and chairs and turn right . walk into the room and stop .
FOAM	walk past the dining room table and chairs and turn left . walk past the table and chairs and turn right . walk into the room and turn right . stop in front of the exercise bike .
Table 6: Examples of the generated instructions. Our generated instructions are generally longer and more accurate compared with the baseline.
Qualitative Examples. As in Table 6, we also find that after training the speaker using our method, the generated instructions are generally longer than the baseline and are more accurate compared with the references.
We evaluate our models on the Room-to-Room (R2R) (Anderson et al., 2018b) and Room-across-Room (RxR) (Ku et al., 2020) datasets based on Matterport3D (Chang et al., 2017). The datasets are released under the Matterport3D Terms of Use. The datasets do not contain any information that names or uniquely identifies individual people or offensive content. Our code is based on EnvDrop that is released under the MIT license. We use the datasets and code for research purposes, which is consistent with their intended use.
D Limitations and Potential Risks
As in the experiments, our models may not work well when the instructions are long and it is hard to train a reasonable speaker model. Also, our model requires fine-tuning the speaker during training based on the feedback of the follower, which
$^{2}$ http://dovahkiin.stanford.edu/matterport/public/MP_TOS.pdf