SlowGuess/ACL_2020_OCR huggingface dataset:
In Q-Learning (Watkins and Dayan, 1992; Mnih et al., 2015), given an interactive environment, an agent takes an action  $a_{t}$  in state  $s_{t}$  by consulting a state-action value estimator  $Q(s,a)$ ; this value estimator estimates the action's expected long-term reward. Q-Learning helps the agent to learn an optimal value estimator. An agent starts from performing randomly and gradually updates its value estimator by interacting with the environment and propagating reward information. In our case, the estimated Q-value at game step  $t$  is simply the sum of Q-values of the action token and QUERY token as introduced in Eqn. 1:
$$
Q _ {t} = Q _ {t, \text {a c t i o n}} + Q _ {t, \text {q u e r y}}. \tag {5}
$$
In this work, we adopt the Rainbow algorithm (Hessel et al., 2017), which is a deep Q-network boosted by several extensions such as a prioritized replay buffer (Schaul et al., 2016). Rainbow exhibits state-of-the-art performance on several RL benchmark tasks (e.g., Atari games).
4.3.3 Negative Log-likelihood (NLL)
During information gathering phase, we use another replay buffer to store question answering transitions (observation string when interaction stops, question string, ground-truth answer) whenever the terminal observation string contains the ground-truth answer. We randomly sample mini-batches of such transitions to train the question answerer to minimize the negative log-likelihood loss.
In this study, we focus on four main aspects:
1. difficulty levels (easy | hard mode);
2. strategies for generating QUERY (from question | question and observation | vocabulary);
3. sizes of the memory queue  $(1\mid 3\mid 5)$
4. RL algorithms for the information gathering phase (A2C | DQN)
Regarding the four aspects, we report the baseline agent's training performance followed by its generalization performance on test data. We use DQN and A2C to refer to our baseline agent trained with DQN and A2C, respectively.
We set the maximum number of episodes (data points) to be 1 million, this is approximately 10 epochs in supervised learning tasks given the size of datasets. The agent may further improve after 1 million episodes, however we believe some meaningful and interesting trends can already be observed from the results. Besides, we hope to keep the wall clock time of the task reasonable<sup>4</sup> to encourage the community to work on this direction.
5.1 Mastering Training Games
It remains difficult for RL agents to master multiple games at the same time. In our case, each document-question pair can be considered a unique "game," and there are hundreds of thousands of
4Basic experiment setting (e.g., QUERY from question, single slot memory) take about a day on a single NVIDIA P100 GPU.
Figure 2: Training  $\mathrm{F}_1$  scores in easy mode with different QUERY types and memory sizes. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 3: Training  $\mathrm{F}_1$  scores in hard mode with different QUERY types and memory sizes. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
them. Therefore, as it is common practice in the RL literature, we study an agent's training curves.
Figure 2 and Figure 3 show the agent's training performance (in terms of  $\mathrm{F}_1$  score) in easy and hard mode, respectively. Due to the space limitations, we select several representative settings to discuss in this section. We provide the agent's training and validation curves for all experiments, and its sufficient information rewards (as defined in § 4.2.2) in Appendix A.
It is clear that our agent performs better on easy mode consistently across both datasets and all training strategies. This may due to the fact that the previous and next commands provide the agent an inefficient but guaranteed way to stumble on the sought-after sentence no matter the game. The Ctrl+F command matches human behavior more closely, but it is arguably more challenging (and interesting) for an RL agent to learn this behavior. RL agents may require extra effort and time to reach a desired state since they rely heavily on ran-
dom exploration, and the Ctrl+F command leads to much larger action space to explore compared to commands such as next.
Related to action space size, we observe that the agent performs best when pointing to the QUERY tokens from the question, whereas it performs worst when generating QUERY tokens from the entire vocabulary. This is particularly clear in hard mode, where agents are forced to use the Ctrl+F command. As shown in Table 2, both datasets have a vocabulary size of more than  $100\mathrm{k}$ , whereas the average length of questions is around 10 tokens. This indicates the action space for generating QUERY from entire vocabulary is much larger. This again suggests that for moving toward a more realistic problem setting where action spaces are huge, methods with better sample efficiency are needed.
Experiments show that a larger memory queue almost always helps. Intuitively, with a memory mechanism (either explicit as in this work, or implicit as with a recurrent network aggregating rep
Dataset	iSQuAD	iNewsQA
Easy Mode
QUERY	Agent	Mem=1	=3	=5	Mem=1	=3	=5
Question	A2C	0.245 (0.493)	0.357 (0.480)	0.386 (0.478)	0.210 (0.554)	0.316 (0.532)	0.333 (0.490)
DQN	0.575 (0.770)	0.637 (0.738)	0.666 (0.716)	0.330 (0.708)	0.326 (0.619)	0.360 (0.620)
Question+Memory	A2C	0.221 (0.479)	0.484 (0.590)	0.409 (0.492)	0.199 (0.595)	0.233 (0.448)	0.253 (0.459)
DQN	0.579 (0.784)	0.651 (0.734)	0.656 (0.706)	0.336 (0.715)	0.334 (0.626)	0.347 (0.596)
Vocabulary	A2C	0.223 (0.486)	0.314 (0.448)	0.309 (0.391)	0.192 (0.551)	0.224 (0.440)	0.224 (0.403)
DQN	0.583 (0.774)	0.624 (0.738)	0.661 (0.731)	0.326 (0.715)	0.323 (0.590)	0.316 (0.593)
Hard Mode
Question	A2C	0.147 (0.404)	0.162 (0.446)	0.158 (0.435)	0.166 (0.529)	0.160 (0.508)	0.164 (0.520)
DQN	0.524 (0.766)	0.524 (0.740)	0.551 (0.739)	0.352 (0.716)	0.367 (0.632)	0.353 (0.613)
Question+Memory	A2C	0.160 (0.441)	0.150 (0.413)	0.156 (0.429)	0.163 (0.520)	0.160 (0.508)	0.164 (0.520)
DQN	0.357 (0.749)	0.362 (0.729)	0.364 (0.733)	0.260 (0.692)	0.264 (0.645)	0.269 (0.620)
Vocabulary	A2C	0.161 (0.444)	0.163 (0.448)	0.160 (0.441)	0.160 (0.510)	0.167 (0.532)	0.162 (0.516)
DQN	0.264 (0.728)	0.261 (0.719)	0.218 (0.713)	0.326 (0.694)	0.214 (0.680)	0.214 (0.680)
Table 3: Test  $\mathrm{F}_{1}$  scores in black and  $\mathrm{F}_{1\text{info}}$  scores (i.e.,an agent's  $\mathrm{F}_{1}$  score iff sufficient information is in its observation when it terminates information gathering phase) in blue.
resentations over game steps), an agent renders the environment closer to fully observed by exploring and storing observations. Presumably, a larger memory could further improve an agent's performance; considering the average number of sentences in each iSQuAD game is 5, a memory with more than 5 slots defeats the purpose of our study of partially observable text environments.
We observe that DQN generally performs better on iSQuAD whereas A2C sometimes works better on the harder iNewsQA task. However, we observe huge gap between them on generalization performance, which we discuss in a later subsection.
Not surprisingly, our agent performs better in general on iSQuAD than on iNewsQA. As shown in Table 2, the average number of sentences per document in iNewsQA is about 6 times more than in iSQuAD. This is analogous to partially observable games with larger maps in the RL literature. We believe a better exploration (in our case, jumping) strategy that can decide where to explore next conditioned on what has already been seen may help agents to master such harder games.
5.2 Generalizing to Test Set
To study an agent's ability to generalize, we select the best performing checkpoint in each experimental setting on the validation set and report their test performance, as shown in Table 3. In addition, to support our claim that the more challenging part of iMRC tasks is information gathering rather than answering questions given sufficient information, we report the agents'  $\mathrm{F}_1$  scores when they have reached the piece of text that contains the answer,
which we denote as  $\mathrm{F}_{\mathrm{linfo}}$
From Table 3 (and validation curves provided in Appendix A) we observe trends that match with training curves. Due to the different sizes of action space, the baseline agents consistently performs better on the easy mode. For the same reason, the agent learns more efficiently when the QUERY token is extracted from the question. The best  $\mathrm{F}_1$  score on hard mode is comparable to and even slightly higher than in easy mode on iNewsQA, which suggests our baseline agent learns some relatively general trajectories in solving training games that generalize to unseen games.
It is also clear that during evaluation, a memory that stores experienced observations helps, since the agent almost always performs better with a memory size of 3 or 5 (when memory size is 1, each new observation overwrites the memory).
While performing comparably with DQN during training, the agent trained with A2C generalizes noticeably worse. We suspect this is caused by the fundamental difference between the ways DQN and A2C explore during training. Specifically, DQN relies on either  $\epsilon$ -greedy or Noisy Net (Fortunato et al., 2017), both of which explicitly force an agent to experience different actions during training. In A2C, exploration is performed implicitly by sampling from a probability distribution over the action space; although entropy regularization is applied, good exploration is still not guaranteed (if there are peaks in the probability distribution). This again suggests the importance of a good exploration strategy in the iMRC tasks, as in all RL tasks.
Finally, we observe  $\mathrm{F}_{1\text{info}}$  scores are consistently
higher than the overall  $\mathrm{F_1}$  scores, and they have less variance across different settings. This supports our hypothesis that information gathering plays an important role in solving iMRC tasks, whereas question answering given necessary information is relatively straightforward.
In this work, we propose and explore the direction of converting MRC datasets into interactive, partially observable environments. We believe information-seeking behavior is desirable for neural MRC systems when knowledge sources are partially observable and/or too large to encode in their entirety, where knowledge is by design easily accessible to humans through interaction. Our idea for reformulating existing MRC datasets as partially observable and interactive environments is straightforward and general. It is complementary to existing MRC dataset and models, meaning almost all MRC datasets can be used to study interactive, information-seeking behavior through similar modifications. We hypothesize that such behavior can, in turn, help in solving real-world MRC problems involving search. As a concrete example, in real world environments such as the Internet, different pieces of knowledge are interconnected by hyperlinks. We could equip the agent with an action to "click" a hyperlink, which returns another webpage as new observations, thus allowing it to navigate through a large number of web information to answer difficult questions.
iMRC is difficult and cannot yet be solved, however it clearly matches a human's information-seeking behavior compared to most static and fully-observable laboratory MRC benchmarks. It lies at the intersection of NLP and RL, which is arguably less studied in existing literature. For our baseline, we adopted off-the-shelf, top-performing MRC and RL methods, and applied a memory mechanism which serves as an inductive bias. Despite being necessary, our preliminary experiments do not seem sufficient. We encourage work on this task to determine what inductive biases, architectural components, or pretraining recipes are necessary or sufficient for MRC based on information-seeking.
Our proposed setup presently uses only a single word as QUERY in the Ctrl+F command in an abstractive manner. However, a host of other options could be considered in future work. For example, a multi-word QUERY with fuzzy matching is more
realistic. It would also be interesting for an agent to generate a vector representation of the QUERY in some latent space and modify it during the dynamic reasoning process. This could further be used to retrieve different contents by comparing with pre-computed document representations (e.g., in an open-domain QA dataset), with such behavior tantamount to learning to do IR. This extends traditional query reformulation for open-domain QA by allowing to drastically change the queries without strictly keeping the semantic meaning of the original queries.
The authors thank Mehdi Fatemi, Peter Potash, Matthew Hausknecht, and Philip Bachman for insightful ideas and discussions. We also thank the anonymous ACL reviewers for their helpful feedback and suggestions.
Philip Bachman, Alessandro Sordoni, and Adam Trischler. 2016. Towards information-seeking agents. arXiv preprint arXiv:1612.02605.
Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 209-220.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. 2017. Noisy networks for exploration. CoRR, abs/1706.10295.
Mor Geva and Jonathan Berant. 2018. Learning to search in long documents using document structure. arXiv preprint arXiv:1806.03529.
Moonsu Han, Minki Kang, Hyunwoo Jung, and Sung Ju Hwang. 2019. Episodic memory reader: Learning what to remember for question answering from streaming data. arXiv preprint arXiv:1903.06164.
Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. 2019. Neural speed reading with structural-jump-lstm. arXiv preprint arXiv:1904.00761.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar,
and David Silver. 2017. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01783.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533.
Karthik Narasimhan, Adam Yala, and Regina Barzilay. 2016. Improving information extraction by acquiring external evidence with reinforcement learning. CoRR, abs/1603.07954.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268.
Rodrigo Nogueira and Kyunghyun Cho. 2016. Webnav: A new large-scale task for natural language based sequential decision making. CoRR, abs/1602.02261.
Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. CoRR, abs/1704.04572.
Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. Answering complex open-domain questions through iterative query generation. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2018. Coqa: A conversational question answering challenge. CoRR, abs/1808.07042.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2016. Prioritized experience replay. In International Conference on Learning Representations, Puerto Rico.
Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Neural speed reading via skim-rnn. arXiv preprint arXiv:1711.02085.
Lei Sha, Feng Qian, and Zhifang Sui. 2017. Will repeated reading benefit natural language understanding? In National CCF Conference on Natural Language Processing and Chinese Computing, pages 366-379. Springer.
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1047-1055. ACM.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? CoRR, abs/1808.09384.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. CoRR, abs/1611.09830.
Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Machine Learning, 8(3):279-292.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. CoRR, abs/1809.09600.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. *Qanet: Combining local convolution with global self-attention for reading comprehension*. CoRR, abs/1804.09541.
Adams Wei Yu, Hongrae Lee, and Quoc V Le. 2017. Learning to skim text. arXiv preprint arXiv:1704.06877.
Xingdi Yuan, Marc-Alexandre Côté, Jie Fu, Zhouhan Lin, Christopher Pal, Yoshua Bengio, and Adam Trischler. 2019. Interactive language learning by question answering.
We show our experimental results (training and validation curves) in Figure 4,5,6,7,8,9,10,11.
In all experiments, we use Adam (Kingma and Ba, 2014) as the step rule for optimization, with the learning rate set to 0.00025. We clip gradient norm at 5.0. We initialize all word embeddings by the 300-dimensional fastText (Mikolov et al., 2018) word vectors trained on Common Crawl (600B tokens), they are fixed during training. We randomly initialize character embeddings by 200-dimensional vectors. In all transformer blocks, block size is 96.
Dimensionality of  $\mathrm{MLP}_{\mathrm{shared}}$  in Eqn. 1 is  $\mathbb{R}^{96\times 150}$ ; dimensionality of  $\mathrm{MLP}_{\mathrm{action}}$  is  $\mathbb{R}^{150\times 4}$  and  $\mathbb{R}^{150\times 2}$  in easy mode (4 actions are available) and hard mode (only 2 actions are available), respectively; dimensionality of  $\mathrm{MLP}_{\mathrm{query}}$  is  $\mathbb{R}^{150\times V}$  where  $V$  denotes vocabulary size of the dataset, as listed in Table 2.
Dimensionalities of  $\mathrm{MLP}_0$  and  $\mathrm{MLP}_1$  in Eqn. 2 are both  $\mathbb{R}^{192\times 150}$ ; dimensionalities of  $\mathrm{MLP}_2$  and  $\mathrm{MLP}_3$  in Eqn. 3 are both  $\mathbb{R}^{150\times 1}$ .
During A2C training, we set the value loss coefficient to be 0.5, we use an entropy regularizer with coefficient of 0.01. We use a discount  $\gamma$  of 0.9 and mini-batch size of 20.
During DQN training, we use a mini-batch of size 20 and push all transitions (observation string, question string, generated command, reward) into a prioritized replay buffer of size 500,000. We do not compute losses directly using these transitions. After every 5 game steps, we sample a mini-batch of 64 transitions from the replay buffer, compute loss, and update the network. We use a discount  $\gamma$  of 0.9. For noisy nets, we use a  $\sigma_0$  of 0.5. We update target network per 1000 episodes. For multi-step returns, we sample  $n\sim \mathrm{Uniform}[1,2,3]$ .
When our agent terminates information gathering phase, we push the question answering transitions (observation string at this time, question string, ground-truth answer) into a question answering replay buffer. After every 5 game steps, we randomly sample a mini-batch of 64 such transitions from the question answering replay buffer and train the model using NLL loss.
For more detail please refer to our open-sourced code.
Figure 4: Training performance on iSQuAD, easy mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 5: Validation performance on iSQuAD, easy mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 6: Training performance on iSQuAD, hard mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 7: Validation performance on iSQuAD, hard mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 8: Training performance on iNewsQA, easy mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 9: Validation performance on iNewsQA, easy mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 10: Training performance on iNewsQA, hard mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.
Figure 11: Validation performance on iNewsQA, hard mode. Solid line: DQN, dashed line: A2C; number of memory slots: 1, 3, 5.