SlowGuess/ACL_2020_OCR huggingface dataset:
Xingdi Yuan†* Jie Fu‡* Marc-Alexandre Côté† Yi Tay* Christopher Pal‡* Adam Trischler†
†Microsoft Research, Montréal ‡Polytechnique Montréal
$\diamond$  Nanyang Technological University  $\diamond$  Canada CIFAR AI Chair eric.yuan@microsoft.com jie.fu@polymtl.ca
Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA). We argue that this stems from the nature of MRC datasets: most of these are static environments wherein the supporting documents and all necessary information are fully observed. In this paper, we propose a simple method that reframes existing MRC datasets as interactive, partially observable environments. Specifically, we "occlude" the majority of a document's text and add context-sensitive commands that reveal "glimpses" of the hidden text to a model. We repurpose SQuAD and NewsQA as an initial case study, and then show how the interactive corpora can be used to train a model that seeks relevant information through sequential decision making. We believe that this setting can contribute in scaling models to web-level QA scenarios. $^{1}$
Many machine reading comprehension (MRC) datasets have been released in recent years (Rajpurkar et al., 2016; Trischler et al., 2016; Nguyen et al., 2016; Reddy et al., 2018; Yang et al., 2018) to benchmark a system's ability to understand and reason over natural language. Typically, these datasets require an MRC model to read through a document to answer a question about information contained therein.
The supporting document is, more often than not, static and fully observable. This raises concerns, since models may find answers simply through shallow pattern matching; e.g., syntactic similarity between the words in questions and documents. As
Question: What was the Harvard endowment total in 2011?
o1 | Harvard has the largest university endowment in the world.
next | a1
o2 | At the end of June 2009, it was worth $25.7 billion, about 30% less than at the same time in 2008.
Ctrl+F Harvard | a2
o3 | In December 2008, Harvard announced that its endowment had lost 22% from July to October 2008, necessitating budget cuts.
Ctrl+F 2011 | a3
o4 | As of September 2011, it had nearly regained the loss suffered during the 2008 recession.
Ctrl+F 2011 | a4
o5 | It was worth $32 billion in 2011, up from $28 billion in September 2010 and $26 billion in 2009.
stop | a5
Prediction: $32 billion
Table 1: Example of the interactive machine reading comprehension behavior.
pointed out by Sugawara et al. (2018), for questions starting with when, models tend to predict the only date/time answer in the supporting document. Such behavior limits the generality and usefulness of MRC models, and suggests that they do not learn a proper 'understanding' of the intended task. In this paper, to address this problem, we shift the focus of MRC data away from 'spoon-feeding' models with sufficient information in fully observable, static documents. Instead, we propose interactive versions of existing MRC tasks, whereby the information needed to answer a question must be gathered sequentially.
The key idea behind our proposed interactive MRC (iMRC) is to restrict the document context that a model observes at one time. Concretely, we split a supporting document into its component sentences and withhold these sentences from the model. Given a question, the model must issue commands to observe sentences in the withheld set; we equip models with actions such as Ctrl+F
* Equal contribution.  
<sup>1</sup>The dataset and implementation of our baseline projects are publicly available at https://github.com/longdi-eric-yuan/imrc_public.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2325-2338 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
to search for matches to a QUERY within partially observed documents. A model searches iteratively, conditioning each command on the input question and the sentences it has observed previously. Thus, our task requires models to 'feed themselves' rather than spoon-feeding them with information. This casts MRC as a sequential decision-making problem amenable to reinforcement learning (RL).
Our proposed approach lies outside of traditional QA work, the idea can be applied to almost all existing MRC datasets and models to study interactive information-seeking behavior. As a case study in this paper, we re-purpose two well known, related corpora with different difficulty levels for our iMRC task: SQuAD and NewsQA. Table 1 shows an example of a model performing interactive MRC on these datasets. Naturally, our reframing makes the MRC problem harder; however, we believe the added demands of iMRC more closely match web-level QA and may lead to deeper comprehension of documents' content.
1. We describe a method to make MRC datasets interactive and formulate the new task as an RL problem.
2. We develop a baseline agent that combines a top performing MRC model and two state-of-the-art RL optimization algorithms and test it on iMRC tasks.
3. We conduct experiments on several variants of iMRC and discuss the significant challenges posed by our setting.
Skip-reading (Yu et al., 2017; Seo et al., 2017; Choi et al., 2017) is an existing setting in which MRC models read partial documents. Concretely, these methods assume that not all tokens in the input sequence are equally useful, and therefore learn to skip irrelevant tokens. Since skipping decisions are discrete, the models are often optimized by the REINFORCE algorithm (Williams, 1992). For example, the structural-jump-LSTM (Hansen et al., 2019) learns to skip and jump over chunks of text, whereas Han et al. (2019) designed a QA task where the model reads streaming data without knowing when the question will be provided. Skip-reading approaches are limited in that they only
consider jumping forward over a few consecutive tokens. Based on the assumption that a single pass of reading may not provide sufficient information, multi-pass reading methods have also been studied (Sha et al., 2017; Shen et al., 2017).
Compared to skip-reading and multi-pass reading, our work enables an agent to jump through a document in a more dynamic manner, in some sense combining aspects of skip-reading and rereading. Specifically, an agent can choose to read forward, backward, or to jump to an arbitrary position depending on the query. This also distinguishes the model we develop in this work from ReasoNet (Shen et al., 2017), a model that decides when to stop forward reading.
Recently, there has been various work on and around interactive environments. For instance, Nogueira and Cho (2016) proposed WebNav, a tool that automatically transforms a website into a goaldriven web navigation task. They train a neural agent to follow traces using supervised learning. Qi et al. (2019) proposed GoldEn Retriever, an iterative retrieve-and-read system that answers complex open-domain questions, which is also trained with supervised learning. Although an effective training strategy, supervised learning requires either human labeled or heuristically generated trajectories. However, there often exist multiple trajectories to solve each question, many of which may not be observed in the supervised data since it is difficult to exhaust all valid trajectories. Generalization can be limited when an agent is trained on such data.
Bachman et al. (2016) introduced a collection of synthetic tasks to train and test information-seeking capabilities in neural models. Narasimhan et al. (2016) proposed an information extraction system that acquires and incorporates external evidence to improve extraction accuracy in domains with limited data. Geva and Berant (2018) proposed a DQN-based agent that leverages the (tree) structure of documents and navigates across sentences and paragraphs. iMRC is distinct from this body of literature in that it does not depend on extra meta information to build tree structures, it is partially-observable, and its action space is as large as 200,000 (much larger than, e.g., the 5 query templates in (Narasimhan et al., 2016) and tree search in (Geva and Berant, 2018)). Our work is also inspired directly by QAit (Yuan et al., 2019), a set of interactive question answering tasks developed on text-based games. However, QAit is based on
Figure 1: A demonstration of the proposed iMRC pipeline, in which the agent is illustrated as a shaded area. At a game step  $t$ , it encodes the question and text observation into hidden representations  $M_{t}$ . An action generator takes  $M_{t}$  as input to generate commands to interact with the environment. If the agent generates stop at this game step,  $M_{t}$  is used to answer question by a question answerer. Otherwise, the iMRC environment will provide new text observation in response to the generated action.
synthetic and populated language which might not require strong language understanding components. This work extends the principle of interactivity to the natural language setting, by leveraging existing MRC tasks already written in natural language.
Broadly speaking, our work is also linked to the query reformulation (QR) task in information retrieval literature (Nogueira and Cho, 2017). Specifically, QR aims to automatically rewrite a query so that it becomes more likely to retrieve relevant documents. Our task shares the spirit of iterative interaction between an agent (reformulator in QR) and an environment. However, the rewritten queries in QR tasks keep the semantic meaning of the original queries, whereas in our task, actions and queries across different game steps can change drastically — since our task requires an agent to learn a reasoning path (trajectory) towards answering a question, rather than to search the same concept repeatedly.
3 iMRC: Making MRC Interactive
The iSQuAD and iNewsQA datasets are based on SQuAD v1.1 $^2$  (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016). Both original datasets share similar properties. Specifically, each data-point consists of a tuple,  $\{p,q,a\}$ , where  $p$  represents a paragraph,  $q$  a question, and  $a$  is the answer. The answer is a word span defined by head and tail positions in  $p$ . NewsQA is more chal
lenging because it has a larger vocabulary, more difficult questions, and longer source documents.
Every paragraph  $p$  is split into a list of sentences  $S = \{s_1, s_2, \dots, s_n\}$ , where  $n$  stands for number of sentences in  $p$ . At the start of a question answering episode, an agent observes the question  $q$ , but rather than observing the entire paragraph  $p$ , it sees only the first sentence  $s_1$  while the rest is withheld. The agent must issue commands to reveal the hidden sentences progressively and thereby gather the information needed to answer  $q$ .
The agent should decide when to stop interacting and output an answer, but the number of interaction steps is limited. Once the agent has exhausted its step budget, it is forced to answer the question.
3.1 Interactive MRC as a POMDP
As described in the previous section, we convert MRC tasks into sequential decision-making problems (which we will refer to as games). These can be described naturally within the reinforcement learning (RL) framework. Formally, tasks in iMRC are partially observable Markov decision processes (POMDP) (Kaelbling et al., 1998). An iMRC data-point is a discrete-time POMDP defined by  $(S,T,A,\Omega ,O,R,\gamma)$ , where  $\gamma \in [0,1]$  is the discount factor and the other elements are described in detail below.
Environment States  $(S)$ : The environment state at game step  $t$  in the game is  $s_t \in S$ . It contains the environment's underlying conditions (e.g., the semantics and information contained in the document, which part of the document has been revealed so far), much of which is hidden from an agent, the agent can only estimate the state from its partial observations. When the agent issues an action  $a_t$ , the environment transitions to state  $s_{t+1}$  with probability  $T(s_{t+1} | s_t, a_t)$ . In this work, transition probabilities are either 0 or 1 (i.e., deterministic environment).
Actions  $(A)$ : At each game step  $t$ , the agent issues an action  $a_{t} \in A$ . We will elaborate on the action space of iMRC in §3.2 and §3.3.
Observations  $(\Omega)$ : The text information perceived by the agent at a given game step  $t$  is the agent's observation,  $o_t \in \Omega$ , which depends on the environment state and the previous action with
3We use 20 as the maximum number of steps, because information revealed by 20 interactions can cover a large portion of the text in either an iSQuAD or iNewsQA paragraph. A reasonable step budget also speeds up training.
2We choose SQuAD v1.1 because in this preliminary study, we focus on extractive question answering.
probability  $O\left( {{o}_{t} \mid  {s}_{t}}\right)$  . Again,observation proba- lities are either 0 or 1 (i.e.,noiseless observation).
Reward Function  $(R)$ : Based on its actions, the agent receives rewards  $r_t = R(s_t, a_t)$ . Its objective is to maximize the expected discounted sum of rewards  $E\left[\sum_t \gamma^t r_t\right]$ .
3.2 Easy and Hard Modes
As a question answering dataset, we adopt the standard output format of extractive MRC tasks, where a system is required to point to a span within a given paragraph  $p$  as its prediction. However, we define two difficulty levels in iMRC, which are based on different action spaces and dynamics during the interactive information gathering phase.
Easy Mode: At a step  $t$ , an agent can issue one of the following four actions to interact with the (partially observable) paragraph  $p$ , where  $p$  consists of  $n$  sentences. Assume the agent's observation  $o_{t}$  corresponds to sentence  $s_{k}$ , where  $1 \leq k \leq n$ .
previous: jump to  $\left\{ \begin{array}{ll}s_n & \text{if} k = 1,\\ s_{k - 1} & \text{otherwise;} \end{array} \right.$
- next: jump to  $\left\{ \begin{array}{ll}s_1 & \text{if } k = n,\\ s_{k + 1} & \text{otherwise;} \end{array} \right.$
- Ctrl+F QUERY: jump to the sentence that contains the next occurrence of QUERY;
- stop: terminate information gathering phase and ready to answer question.
Hard Mode: Only the Ctrl+F and stop commands are available (i.e., an agent is forced to generate QUERY to navigate the partially observable paragraph  $p$ ).
3.3 QUERY Types
Given an objective (e.g., a question to answer), humans search by using both extractive and abstractive queries. For instance, when searching information about the actor "Dwayne Johnson", one may either type his name or "The Rock" in a search engine. We believe abstractive query searching requires a deeper understanding of the question, and some background knowledge (one cannot refer to "Dwayne Johnson" as the "The Rock" if they know nothing about his wrestling career).
Inspired by this observation, we study the following three settings, where in each, the QUERY is generated from different sources:
Dataset	iSQuAD	iNewsQA
#Training Games	82,441	92,550
Vocabulary Size	109,689	200,000
Avg. #Sentence / Document	5.1	29.5
Avg. Sentence Length	26.1	22.2
Avg. Question Length	11.3	7.6
Table 2: Statistics of iSQuAD and iNewsQA.
1. One token from the question: extractive QUERY generation with a relatively small action space.
2. One token from the union of the question and the current observation: still extractive QUERY generation, although in an intermediate level where the action space is larger.
3. One token from the dataset vocabulary: abstractive QUERY generation where the action space is huge (see Table 2 for statistics of iSQuAD and iNewsQA).
3.4 Evaluation Metric
Since iMRC involves both MRC and RL, we adopt evaluation metrics from both settings. First, as a question answering task, we use  $\mathrm{F}_1$  score to compare predicted answers against ground-truth, as in previous work. When there exist multiple ground-truth answers, we report the max  $\mathrm{F}_1$  score.
Second, mastering multiple games remains quite challenging for RL agents. Therefore, we evaluate an agent's performance during both its training and testing phases. Specifically, we report training curves and test results based on the best validation  $\mathrm{F}_1$  scores.
4 Baseline Agent
As a baseline agent, we adopt QA-DQN (Yuan et al., 2019), we modify it to enable extractive QUERY generation and question answering.
As illustrated in Figure 1, the baseline agent consists of three components: an encoder, an action generator, and a question answerer. More precisely, at a step  $t$  during the information gathering phase, the encoder reads observation string  $o_t$  and question string  $q$  to generate the attention aggregated hidden representations  $M_t$ . Using  $M_t$ , the action generator outputs commands (depending on the mode, as defined in § 3.2) to interact with iMRC. The information-gathering phase terminates whenever the generated command is stop or the agent
has used up its move budget. The question answerer takes the hidden representation at the terminating step to generate head and tail pointers as its answer prediction.
4.1 Model Structure
In this section, we only describe the difference between the model our baseline agent uses and the original QA-DQN. We refer readers to (Yuan et al., 2019) for detailed information.
In the following subsections, we use "game step  $t$ " to denote the  $t$ th round of interaction between an agent with the iMRC environment.
4.1.1 Action Generator
Let  $M_t \in \mathbb{R}^{L \times H}$  denote the output of the encoder, where  $L$  is the length of observation string and  $H$  is hidden size of the encoder representations.
The action generator takes  $M_{t}$  as input and generates rankings for all possible actions. As described in the previous section, a Ctrl+F command is composed of two tokens (the token "Ctrl+F" and the QUERY token). Therefore, the action generator consists of three multilayer perceptrs (MLPs):
$$
R _ {t} = \operatorname {R e L U} \left(\operatorname {M L P} _ {\text {s h a r e d}} \left(\operatorname {m e a n} \left(M _ {t}\right)\right)\right),
$$
$$
Q _ {t, \text {a c t i o n}} = \operatorname {M L P} _ {\text {a c t i o n}} \left(R _ {t}\right) \cdot M _ {\text {m o d e}}, \tag {1}
$$
$$
Q _ {t, \text {q u e r y}} = \mathrm {M L P} _ {\text {q u e r y}} (R _ {t}) \cdot M _ {\text {t y p e}}.
$$
In which,  $Q_{t,\text{action}}$  and  $Q_{t,\text{query}}$  are Q-values of action token and QUERY token (when action token is "Ctrl+F"), respectively.  $M_{\text{mode}}$  is a mask, which masks the previous and next tokens in hard mode;  $M_{\text{type}}$  is another mask which depends on the current QUERY type (e.g., when QUERY is extracted from the question  $q$ , all tokens absent from  $q$  are masked out). Probability distributions of tokens are further computed by applying softmax on  $Q_{t,\text{action}}$  and  $Q_{t,\text{query}}$ , respectively.
4.1.2 Question Answerer
Following QANet (Yu et al., 2018), we append two extra stacks of transformer blocks on top of the encoder to compute head and tail positions:
$$
h _ {\text {h e a d}} = \operatorname {R e L U} \left(\operatorname {M L P} _ {0} ([ M _ {t}; M _ {\text {h e a d}} ])\right), \tag {2}
$$
$$
h _ {\text {t a i l}} = \operatorname {R e L U} \left(\operatorname {M L P} _ {1} \left([ M _ {t}; M _ {\text {t a i l}} ]\right)\right).
$$
In which,  $[\cdot ;\cdot ]$  denotes vector concatenation,  $M_{\mathrm{head}}\in \mathbb{R}^{L\times H}$  and  $M_{\mathrm{tail}}\in \mathbb{R}^{L\times H}$  are the outputs of the two extra transformer stacks.
Similarly, probability distributions of head and tail pointers over observation string  $o_t$  can be computed by:
$$
p _ {\text {h e a d}} = \operatorname {s o f t m a x} \left(\mathrm {M L P} _ {2} \left(h _ {\text {h e a d}}\right)\right), \tag {3}
$$
$$
p _ {\text {t a i l}} = \operatorname {s o f t m a x} \left(\mathrm {M L P} _ {3} \left(h _ {\text {t a i l}}\right)\right).
$$
4.2 Memory and Reward Shaping
4.2.1 Memory
In iMRC tasks, some questions may not be easily answerable by observing a single sentence. To overcome this limitation, we provide an explicit memory mechanism to our baseline agent to serve as an inductive bias. Specifically, we use a queue to store strings that have been observed recently. The queue has a limited number of slots (we use queues of size [1, 3, 5] in this work). This prevents the agent from issuing next commands until the environment is observed fully in memory, in which case our task degenerates to the standard MRC setting. We reset the memory slots episodically.
4.2.2 Reward Shaping
Because the question answerer in our agent is a pointing model, its performance relies heavily on whether the agent can find and stop at the sentence that contains the answer. In the same spirit as (Yuan et al., 2019), we also design a heuristic reward to guide agents to learn this behavior.
In particular, we assign a reward if the agent halts at game step  $k$  and the answer is a sub-string of  $o_k$  (if larger memory slots are used, we assign this reward if the answer is a sub-string of the memory at game step  $k$ ). We denote this reward as the sufficient information reward, since, if an agent sees the answer, it should have a good chance of having gathered sufficient information for the question (although this is not guaranteed).
Note this sufficient information reward is part of the design of the baseline agent, whereas the question answering score is the only metric used to evaluate an agent's performance on the iMRC task.
4.3 Training Strategy
Since iMRC games are interactive environments and we have formulated the tasks as POMDPs (in § 3.1), it is natural to use RL algorithms to train the information gathering components of our agent. In this work, we study the performance of two widely used RL algorithms, one based on Q-Learning (DQN) and the other on Policy Gradients (A2C). When an agent has reached a sentence
2329
that contains sufficient information to answer the question, the task becomes a standard extractive QA setting, where an agent learns to point to a span from its observation. When this condition is met, it is also natural to adopt standard supervised learning methods to train the question answering component of our agent.
In this section, we describe the 3 training strategies mentioned above. We provide implementation details in Appendix B.
4.3.1 Advantage Actor-Critic (A2C)
Advantage actor-critic (A2C) was first proposed by Mnih et al. (2016). Compared to policy gradient computation in REINFORCE (Williams, 1992),
$$
\nabla_ {\theta} J (\theta) = \mathbb {E} _ {\pi} \left[ \sum_ {t = 1} ^ {T} \nabla_ {\theta} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) G _ {t} \right], \tag {4}
$$
where the policy gradient  $\nabla_{\theta}J(\theta)$  is updated by measuring the discounted future reward  $G_{t}$  from real sample trajectories, A2C utilizes the lower variance advantage function  $A(s_{t},a_{t}) = Q(s_{t},a_{t}) - V(s_{t})$  in place of  $G_{t}$ . The advantage  $A(s_{t},a_{t})$  of taking action  $a_{t}$  at state  $s_t$  is defined as the value  $Q(s_{t},a_{t})$  of taking  $a_{t}$  minus the average value  $V(s_{t})$  of all possible actions in state  $s_t$ .
In the agent, a critic updates the state-value function  $V(s)$ , whereas an actor updates the policy parameter  $\theta$  for  $\pi_{\theta}(a|s)$ , in the direction suggested by the critic. Following common practice, we share parameters between actor and critic networks. Specifically, all parameters other than  $\mathrm{MLP}_{\mathrm{action}}$  and  $\mathrm{MLP}_{\mathrm{query}}$  (both defined in Eqn. 1) are shared between actor and critic.
4.3.2 Deep Q-Networks (DQN)