Related Works	Skip-reading , ,  is an existing setting in which MRC models read partial documents. Concretely, these methods assume that not all tokens in the input sequence are useful, and therefore learn to skip irrelevant tokens based on the current input and their internal memory. Since skipping decisions are discrete, the models are often optimized by the REINFORCE algorithm . For example, the structural-jump-LSTM proposed in  learns to skip and jump over chunks of text. In a similar vein,  designed a QA task where the model reads streaming data unidirectionally, without knowing when the question will be provided. Skip-reading approaches are limited in that they only consider jumping over a few consecutive tokens and the skipping operations are usually unidirectional. Based on the assumption that a single pass of reading may not provide sufficient information, multi-pass reading methods have also been studied , . Compared to skip-reading and multi-turn reading, our work enables an agent to jump through a document in a more dynamic manner, in some sense combining aspects of skip-reading and re-reading. For example, it can jump forward, backward, or to an arbitrary position, depending on the query.

Interactive Machine Comprehension with Information Seeking Agents	Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA). We argue that this stems from the nature of MRC datasets: most of these are static environments wherein the supporting documents and all necessary information are fully observed. In this paper, we propose a simple method that reframes existing MRC datasets as interactive, partially observable environments. Specifically, we "occlude" the majority of a document's text and add context-sensitive commands that reveal "glimpses" of the hidden text to a model. We repurpose SQuAD and NewsQA as an initial case study, and then show how the interactive corpora can be used to train a model that seeks relevant information through sequential decision making. We believe that this setting can contribute in scaling models to web-level QA scenarios.

Baseline Agent ::: Training Strategy ::: Question Answering	Similarly, we use another replay buffer to store question answering transitions (observation string when interaction stops, question string, ground-truth answer). Because both iSQuAD and iNewsQA are converted from datasets that provide ground-truth answer positions, we can leverage this information and train the question answerer with supervised learning. Specifically, we only push question answering transitions when the ground-truth answer is in the observation string. For each transition, we convert the ground-truth answer head- and tail-positions from the SQuAD and NewsQA datasets to positions in the current observation string. After every 5 game steps, we randomly sample a mini-batch of 64 transitions from the replay buffer and train the question answerer using the Negative Log-Likelihood (NLL) loss. We use a dropout rate of 0.1.

Experimental Results ::: Mastering Training Games	It remains difficult for RL agents to master multiple games at the same time. In our case, each document-question pair can be considered a unique game, and there are hundred of thousands of them. Therefore, as is common practice in the RL literature, we study an agent's training curves. Due to the space limitations, we select several representative settings to discuss in this section and provide QA-DQN's training and evaluation curves for all experimental settings in the Appendix. We provide the agent's sufficient information rewards (i.e., if the agent stopped at a state where the observation contains the answer) during training in Appendix as well. Figure FIGREF36 shows QA-DQN's training performance ($\text{F}_1$ score) when next and previous actions are available. Figure FIGREF40 shows QA-DQN's training performance ($\text{F}_1$ score) when next and previous actions are disabled. Note that all training curves are averaged over 3 runs with different random seeds and all evaluation curves show the one run with max validation performance among the three.

iMRC: Making MRC Interactive ::: Evaluation Metric	Since iMRC involves both MRC and RL, we adopt evaluation metrics from both settings. First, as a question answering task, we use $\text{F}_1$ score to compare predicted answers against ground-truth, as in previous works. When there exist multiple ground-truth answers, we report the max $\text{F}_1$ score. Second, mastering multiple games remains quite challenging for RL agents. Therefore, we evaluate an agent's performance during both its training and testing phases. During training, we report training curves averaged over 3 random seeds. During test, we follow common practice in supervised learning tasks where we report the agent's test performance corresponding to its best validation performance .

Baseline Agent ::: Model Structure ::: Encoder	The encoder consists of an embedding layer, two stacks of transformer blocks (denoted as encoder transformer blocks and aggregation transformer blocks), and an attention layer. In the embedding layer, we aggregate both word- and character-level embeddings. Word embeddings are initialized by the 300-dimension fastText  vectors trained on Common Crawl (600B tokens), and are fixed during training. Character embeddings are initialized by 200-dimension random vectors. A convolutional layer with 96 kernels of size 5 is used to aggregate the sequence of characters. We use a max pooling layer on the character dimension, then a multi-layer perceptron (MLP) of size 96 is used to aggregate the concatenation of word- and character-level representations. A highway network  is used on top of this MLP. The resulting vectors are used as input to the encoding transformer blocks. Each encoding transformer block consists of four convolutional layers (with shared weights), a self-attention layer, and an MLP. Each convolutional layer has 96 filters, each kernel's size is 7. In the self-attention layer, we use a block hidden size of 96 and a single head attention mechanism.

Baseline Agent ::: Training Strategy ::: Action Generation	iMRC games are interactive environments. We use an RL training algorithm to train the interactive information-gathering behavior of QA-DQN. We adopt the Rainbow algorithm proposed by , which integrates several extensions to the original Deep Q-Learning algorithm . Rainbox exhibits state-of-the-art performance on several RL benchmark tasks (e.g., Atari games). During game playing, we use a mini-batch of size 10 and push all transitions (observation string, question string, generated command, reward) into a replay buffer of size 500,000. We do not compute losses directly using these transitions. After every 5 game steps, we randomly sample a mini-batch of 64 transitions from the replay buffer, compute loss, and update the network. Detailed hyper-parameter settings for action generation are shown in Table TABREF38.

Introduction	Many machine reading comprehension (MRC) datasets have been released in recent years , , , ,  to benchmark a system's ability to understand and reason over natural language. Typically, these datasets require an MRC model to read through a document to answer a question about information contained therein. The supporting document is, more often than not, static and fully observable. This raises concerns, since models may find answers simply through shallow pattern matching; e.g., syntactic similarity between the words in questions and documents. As pointed out by , for questions starting with when, models tend to predict the only date/time answer in the supporting document. Such behavior limits the generality and usefulness of MRC models, and suggests that they do not learn a proper `understanding' of the intended task. In this paper, to address this problem, we shift the focus of MRC data away from `spoon-feeding' models with sufficient information in fully observable, static documents. Instead, we propose interactive versions of existing MRC tasks, whereby the information needed to answer a question must be gathered sequentially.

The key idea behind our proposed interactive MRC (iMRC) is to restrict the document context that a model observes at one time. Concretely, we split a supporting document into its component sentences and withhold these sentences from the model. Given a question, the model must issue commands to observe sentences in the withheld set; we equip models with actions such as Ctrl+F (search for token) and stop for searching through partially observed documents. A model searches iteratively, conditioning each command on the input question and the sentences it has observed previously. Thus, our task requires models to `feed themselves' rather than spoon-feeding them with information. This casts MRC as a sequential decision-making problem amenable to reinforcement learning (RL). As an initial case study, we repurpose two well known, related corpora with different difficulty levels for our interactive MRC task: SQuAD and NewsQA. Table TABREF2 shows some examples of a model performing interactive MRC on these datasets. Naturally, our reframing makes the MRC problem harder; however, we believe the added demands of iMRC more closely match web-level QA and may lead to deeper comprehension of documents' content.

iMRC: Making MRC Interactive ::: Interactive MRC as a POMDP	As described in the previous section, we convert MRC tasks into sequential decision-making problems (which we will refer to as games). These can be described naturally within the reinforcement learning (RL) framework. Formally, tasks in iMRC are partially observable Markov decision processes (POMDP) . An iMRC data-point is a discrete-time POMDP defined by $(S, T, A, \Omega , O, R, \gamma )$, where $\gamma \in [0, 1]$ is the discount factor and the other elements are described in detail below. Environment States ($S$): The environment state at turn $t$ in the game is $s_t \in S$. It contains the complete internal information of the game, much of which is hidden from the agent. When an agent issues an action $a_t$, the environment transitions to state $s_{t+1}$ with probability $T(s_{t+1} | s_t, a_t)$). In this work, transition probabilities are either 0 or 1 (i.e., deterministic environment).

Broadly speaking, our approach is also linked to the optimal stopping problem in the literature Markov decision processes (MDP) , where at each time-step the agent either continues or stops and accumulates reward. Here, we reformulate conventional QA tasks through the lens of optimal stopping, in hopes of improving over the shallow matching behaviors exhibited by many MRC systems.

This vector could then be compared with precomputed document representations (e.g., in an open domain QA dataset) to determine what text to observe next, with such behavior tantamount to learning to do IR. As mentioned, our idea for reformulating existing MRC datasets as partially observable and interactive environments is straightforward and general. Almost all MRC datasets can be used to study interactive, information-seeking behavior through similar modifications. We hypothesize that such behavior can, in turn, help in solving real-world MRC problems involving search.

Experimental Results ::: Generalizing to Test Set	To study QA-DQN's ability to generalize, we select the best performing agent in each experimental setting on the validation set and report their performance on the test set. The agent's test performance is reported in Table TABREF41. In addition, to support our claim that the challenging part of iMRC tasks is information seeking rather than answering questions given sufficient information, we also report the $\text{F}_1$ score of an agent when it has reached the piece of text that contains the answer, which we denote as $\text{F}_{1\text{info}}$. From Table TABREF41 (and validation curves provided in appendix) we can observe that QA-DQN's performance during evaluation matches its training performance in most settings. $\text{F}_{1\text{info}}$ scores are consistently higher than the overall $\text{F}_1$ scores, and they have much less variance across different settings. This supports our hypothesis that information seeking play an important role in solving iMRC tasks, whereas question answering given necessary information is relatively straightforward. This also suggests that an interactive agent that can better navigate to important sentences is very likely to achieve better performance on iMRC tasks.

Baseline Agent ::: Training Strategy	In this section, we describe our training strategy. We split the training pipeline into two parts for easy comprehension. We use Adam  as the step rule for optimization in both parts, with the learning rate set to 0.00025.

This also distinguishes the model we develop in this work from ReasoNet , where an agent decides when to stop unidirectional reading. Recently,  propose DocQN, which is a DQN-based agent that leverages the (tree) structure of documents and navigates across sentences and paragraphs. The proposed method has been shown to outperform vanilla DQN and IR baselines on TriviaQA dataset. The main differences between our work and DocQA include: iMRC does not depend on extra meta information of documents (e.g., title, paragraph title) for building document trees as in DocQN; our proposed environment is partially-observable, and thus an agent is required to explore and memorize the environment via interaction; the action space in our setting (especially for the Ctrl+F command as defined in later section) is arguably larger than the tree sampling action space in DocQN. Closely related to iMRC is work by , in which the authors introduce a collection of synthetic tasks to train and test information-seeking capabilities in neural models. We extend that work by developing a realistic and challenging text-based task.

Baseline Agent ::: Memory and Reward Shaping ::: Memory	In iMRC, some questions may not be easily answerable based only on observation of a single sentence. To overcome this limitation, we provide an explicit memory mechanism to QA-DQN. Specifically, we use a queue to store strings that have been observed recently. The queue has a limited size of slots (we use queues of size [1, 3, 5] in this work). This prevents the agent from issuing next commands until the environment has been observed fully, in which case our task would degenerate to the standard MRC setting. The memory slots are reset episodically.

Baseline Agent	As a baseline, we propose QA-DQN, an agent that adopts components from QANet  and adds an extra command generation module inspired by LSTM-DQN . As illustrated in Figure FIGREF6, the agent consists of three components: an encoder, an action generator, and a question answerer. More precisely, at a game step $t$, the encoder reads observation string $o_t$ and question string $q$ to generate attention aggregated hidden representations $M_t$. Using $M_t$, the action generator outputs commands (defined in previous sections) to interact with iMRC. If the generated command is stop or the agent is forced to stop, the question answerer takes the current information at game step $t$ to generate head and tail pointers for answering the question; otherwise, the information gathering procedure continues. In this section, we describe the high-level model structure and training strategies of QA-DQN. We refer readers to  for detailed information. We will release datasets and code in the near future.

Experimental Results	In this study, we focus on three factors and their effects on iMRC and the performance of the QA-DQN agent: different Ctrl+F strategies, as described in the action space section; enabled vs. disabled next and previous actions; different memory slot sizes. Below we report the baseline agent's training performance followed by its generalization performance on test data.

Baseline Agent ::: Model Structure	In this section, we use game step $t$ to denote one round of interaction between an agent with the iMRC environment. We use $o_t$ to denote text observation at game step $t$ and $q$ to denote question text. We use $L$ to refer to a linear transformation. $[\cdot ;\cdot ]$ denotes vector concatenation.

Baseline Agent ::: Model Structure ::: Question Answerer	Following , we append two extra stacks of aggregation transformer blocks on top of the encoder to compute head and tail positions: Here, $M_{head}$ and $M_{tail}$ are outputs of the two extra transformer stacks, $L_0$, $L_1$, $L_2$ and $L_3$ are trainable parameters with output size 150, 150, 1 and 1, respectively.

The main contributions of this work are as follows: We describe a method to make MRC datasets interactive and formulate the new task as an RL problem. We develop a baseline agent that combines a top performing MRC model and a state-of-the-art RL optimization algorithm and test it on our iMRC tasks. We conduct experiments on several variants of iMRC and discuss the significant challenges posed by our setting.

Discussion and Future Work	In this work, we propose and explore the direction of converting MRC datasets into interactive environments. We believe interactive, information-seeking behavior is desirable for neural MRC systems when knowledge sources are partially observable and/or too large to encode in their entirety — for instance, when searching for information on the internet, where knowledge is by design easily accessible to humans through interaction. Despite being restricted, our proposed task presents major challenges to existing techniques. iMRC lies at the intersection of NLP and RL, which is arguably less studied in existing literature. We hope to encourage researchers from both NLP and RL communities to work toward solving this task. For our baseline, we adopted an off-the-shelf, top-performing MRC model and RL method. Either component can be replaced straightforwardly with other methods (e.g., to utilize a large-scale pretrained language model). Our proposed setup and baseline agent presently use only a single word with the query command. However, a host of other options should be considered in future work. For example, multi-word queries with fuzzy matching are more realistic. It would also be interesting for an agent to generate a vector representation of the query in some latent space.

Baseline Agent ::: Model Structure ::: Action Generator	The action generator takes $M_t$ as input and estimates Q-values for all possible actions. As described in previous section, when an action is a Ctrl+F command, it is composed of two tokens (the token “Ctrl+F” and the query token). Therefore, the action generator consists of three MLPs: Here, the size of $L_{shared} \in \mathbb {R}^{95 \times 150}$; $L_{action}$ has an output size of 4 or 2 depending on the number of actions available; the size of $L_{ctrlf}$ is the same as the size of a dataset's vocabulary size (depending on different query type settings, we mask out words in the vocabulary that are not query candidates). The overall Q-value is simply the sum of the two components:

Baseline Agent ::: Memory and Reward Shaping ::: Reward Shaping	Because the question answerer in QA-DQN is a pointing model, its performance relies heavily on whether the agent can find and stop at the sentence that contains the answer. We design a heuristic reward to encourage and guide this behavior. In particular, we assign a reward if the agent halts at game step $k$ and the answer is a sub-string of $o_k$ (if larger memory slots are used, we assign this reward if the answer is a sub-string of the memory at game step $k$). We denote this reward as the sufficient information reward, since, if an agent sees the answer, it should have a good chance of having gathered sufficient information for the question (although this is not guaranteed). Note this sufficient information reward is part of the design of QA-DQN, whereas the question answering score is the only metric used to evaluate an agent's performance on the iMRC task.

From Figure FIGREF36, we can see that the three Ctrl+F strategies show similar difficulty levels when next and previous are available, although QA-DQN works slightly better when selecting a word from the question as query (especially on iNewsQA). However, from Figure FIGREF40 we observe that when next and previous are disabled, QA-DQN shows significant advantage when selecting a word from the question as query. This may due to the fact that when an agent must use Ctrl+F to navigate within documents, the set of question words is a much smaller action space in contrast to the other two settings. In the 4-action setting, an agent can rely on issuing next and previous actions to reach any sentence in a document. The effect of action space size on model performance is particularly clear when using a datasets' entire vocabulary as query candidates in the 2-action setting. From Figure FIGREF40 (and figures with sufficient information rewards in the Appendix) we see QA-DQN has a hard time learning in this setting. As shown in Table TABREF16, both datasets have a vocabulary size of more than 100k. This is much larger than in the other two settings, where on average the length of questions is around 10.

An agent decides when to stop interacting and output an answer, but the number of interaction steps is limited. Once an agent has exhausted its step budget, it is forced to answer the question.

This suggests that the methods with better sample efficiency are needed to act in more realistic problem settings with huge action spaces. Experiments also show that a larger memory slot size always helps. Intuitively, with a memory mechanism (either implicit or explicit), an agent could make the environment closer to fully observed by exploring and memorizing observations. Presumably, a larger memory may further improve QA-DQN's performance, but considering the average number of sentences in each iSQuAD game is 5, a memory with more than 5 slots will defeat the purpose of our study of partially observable text environments. Not surprisingly, QA-DQN performs worse in general on iNewsQA, in all experiments. As shown in Table TABREF16, the average number of sentences per document in iNewsQA is about 6 times more than in iSQuAD. This is analogous to games with larger maps in the RL literature, where the environment is partially observable. A better exploration (in our case, jumping) strategy may help QA-DQN to master such harder games.

iMRC: Making MRC Interactive ::: Action Space	To better describe the action space of iMRC, we split an agent's actions into two phases: information gathering and question answering. During the information gathering phase, the agent interacts with the environment to collect knowledge. It answers questions with its accumulated knowledge in the question answering phase. Information Gathering: At step $t$ of the information gathering phase, the agent can issue one of the following four actions to interact with the paragraph $p$, where $p$ consists of $n$ sentences and where the current observation corresponds to sentence $s_k,~1 \le k \le n$: previous: jump to $ \small {\left\lbrace \begin{array}{ll} s_n & \text{if $k = 1$,}\\ s_{k-1} & \text{otherwise;} \end{array}\right.} $ next: jump to $ \small {\left\lbrace \begin{array}{ll} s_1 & \text{if $k = n$,}\\ s_{k+1} & \text{otherwise;} \end{array}\right.}

Baseline Agent ::: Memory and Reward Shaping ::: Ctrl+F Only Mode	As mentioned above, an agent might bypass Ctrl+F actions and explore an iMRC game only via next commands. We study this possibility in an ablation study, where we limit the agent to the Ctrl+F and stop commands. In this setting, an agent is forced to explore by means of search a queries.

Actions ($A$): At each game turn $t$, the agent issues an action $a_t \in A$. We will elaborate on the action space of iMRC in the action space section. Observations ($\Omega $): The text information perceived by the agent at a given game turn $t$ is the agent's observation, $o_t \in \Omega $, which depends on the environment state and the previous action with probability $O(o_t|s_t)$. In this work, observation probabilities are either 0 or 1 (i.e., noiseless observation). Reward Function ($R$): Based on its actions, the agent receives rewards $r_t = R(s_t, a_t)$. Its objective is to maximize the expected discounted sum of rewards $E \left[\sum _t \gamma ^t r_t \right]$.

$ Ctrl+F $<$query$>$: jump to the sentence that contains the next occurrence of “query”; stop: terminate information gathering phase. Question Answering: We follow the output format of both SQuAD and NewsQA, where an agent is required to point to the head and tail positions of an answer span within $p$. Assume that at step $t$ the agent stops interacting and the observation $o_t$ is $s_k$. The agent points to a head-tail position pair in $s_k$.

iMRC: Making MRC Interactive ::: Query Types	Given the question “When is the deadline of AAAI?”, as a human, one might try searching “AAAI” on a search engine, follow the link to the official AAAI website, then search for keywords “deadline” or “due date” on the website to jump to a specific paragraph. Humans have a deep understanding of questions because of their significant background knowledge. As a result, the keywords they use to search are not limited to what appears in the question. Inspired by this observation, we study 3 query types for the Ctrl+F $<$query$>$ command. One token from the question: the setting with smallest action space. Because iMRC deals with Ctrl+F commands by exact string matching, there is no guarantee that all sentences are accessible from question tokens only. One token from the union of the question and the current observation: an intermediate level where the action space is larger. One token from the dataset vocabulary: the action space is huge (see Table TABREF16 for statistics of SQuAD and NewsQA). It is guaranteed that all sentences in all documents are accessible through these tokens.

Layer normalization and dropout are applied after each component inside the block. We add positional encoding into each block's input. We use one layer of such an encoding block. At a game step $t$, the encoder processes text observation $o_t$ and question $q$ to generate context-aware encodings $h_{o_t} \in \mathbb {R}^{L^{o_t} \times H_1}$ and $h_q \in \mathbb {R}^{L^{q} \times H_1}$, where $L^{o_t}$ and $L^{q}$ denote length of $o_t$ and $q$ respectively, $H_1$ is 96. Following , we use a context-query attention layer to aggregate the two representations $h_{o_t}$ and $h_q$.

Specifically, the attention layer first uses two MLPs to map $h_{o_t}$ and $h_q$ into the same space, with the resulting representations denoted as $h_{o_t}^{\prime } \in \mathbb {R}^{L^{o_t} \times H_2}$ and $h_q^{\prime } \in \mathbb {R}^{L^{q} \times H_2}$, in which, $H_2$ is 96. Then, a tri-linear similarity function is used to compute the similarities between each pair of $h_{o_t}^{\prime }$ and $h_q^{\prime }$ items: where $\odot $ indicates element-wise multiplication and $w$ is trainable parameter vector of size 96. We apply softmax to the resulting similarity matrix $S$ along both dimensions, producing $S^A$ and $S^B$. Information in the two representations are then aggregated as where $h_{oq}$ is aggregated observation representation. On top of the attention layer, a stack of aggregation transformer blocks is used to further map the observation representations to action representations and answer representations.

The configuration parameters are the same as the encoder transformer blocks, except there are two convolution layers (with shared weights), and the number of blocks is 7. Let $M_t \in \mathbb {R}^{L^{o_t} \times H_3}$ denote the output of the stack of aggregation transformer blocks, in which $H_3$ is 96.

iMRC: Making MRC Interactive	We build the iSQuAD and iNewsQA datasets based on SQuAD v1.1  and NewsQA . Both original datasets share similar properties. Specifically, every data-point consists of a tuple, $\lbrace p, q, a\rbrace $, where $p$ represents a paragraph, $q$ a question, and $a$ is the answer. The answer is a word span defined by head and tail positions in $p$. NewsQA is more difficult than SQuAD because it has a larger vocabulary, more difficult questions, and longer source documents. We first split every paragraph $p$ into a list of sentences $\mathcal {S} = \lbrace s_1, s_2, ..., s_n\rbrace $, where $n$ stands for number of sentences in $p$. Given a question $q$, rather than showing the entire paragraph $p$, we only show an agent the first sentence $s_1$ and withhold the rest. The agent must issue commands to reveal the hidden sentences progressively and thereby gather the information needed to answer question $q$.

Question: Do they provide decision sequences as supervision while training models?
Answer: No