Data Augmentation and Multi-Task Learning in QA	There have been several works addressing the paucity of annotated data for QA. Data noisily annotated with short answer spans has been generated automatically through distant supervision and shown to be useful  . Unlabeled text and data augmentation through machine translation have been used to improve model quality  ,  ,  . min-seo-hajishirzi:2017:Short used short-answer annotations in SQuAD  to improve paragraph-level question answering for WikiQA  . To the best of our knowledge, there has been no prior work using QA data annotated at the paragraph level to improve models for short question answering.

Note that none of the models have seen this gold distribution during training and have thus not been trained to minimize these distances (the PD latent variable models are trained to minimize distance with respect to projected model distributions given coarse passage labels $z$ ). We can see that the predictive distribution of the best method PD( $err^2$ ) is closest to the gold labels. The maximum marginal likelihood method MML comes second in approaching the gold distribution. The multi-task approach lags behind others in distance to the fine-grained gold labels, but comes first in the measurement in the last column, Passage-MRR. That column indicates the mean reciprocal rank of the correct gold passage according to the model. Here passages are ranked by the score of the highest-scoring short answer span within the passage. This measurement indicates that the multi-task model is able to learn to rank passages correctly from the coarse-grained passage-level annotation, but has a harder time to transfer this improvement to the task of picking fine-grained short answers within the passages.

Task Definitions	The fine-grained short question answering task asks to select an answer span in a document containing multiple paragraphs. In the left example in Figure 1, the short answer to the question What was Nikola Tesla's ethnicity? is the phrase Serbian in the first paragraph in the document. The coarse-grained labels indicate the relevance of document paragraphs. In the right example in Figure 1, the labels indicate whether or not the paragraphs in a given document contain the answers for the given question What was Martin Luther's nationality? without specifying the answer spans. The goal of our paper is to design methods to learn from both fine-grained and coarse-grained labeled data, to improve systems for fine-grained QA.

Text-based Question Answering	In span-based reading comprehension, a system must be able to extract a plausible text-span answer for a given question from a context document or paragraph  ,  ,  . Most work has focused on selecting short answers given relevant paragraphs, but datasets and works considering the more realistic task of selection from full documents are starting to appear  . Sentence selection or paragraph selection datasets test whether a system can correctly rank texts that are relevant for answering a question higher than texts that do not. Wang2007EMNLP constructed the QASent dataset based on questions from TREC 8-13 QA tracks. WikiQA  associates questions from Bing search query log with all the sentences in the Wikipedia summary paragraph which is then labeled by crowd workers. Most state-of-the-art models for both types of tasks make use of neural network modules to construct and compare representations for a question and the possible answers. We build on a near state-of-the-art baseline model and evaluate on a document-level short question answering task.

Results	We report results evaluating the impact of using coarsely annotated data in the two dataset conditions in Figure 3 . There are two groups of rows corresponding to the two data sizes: in the smaller setting, only 5% of the original fine-grained data is used, and in the medium setting, 20% of the fine-grained data is used. The first row in each group indicates the performance when using only finely labeled fully supervised data. The column Fine-F1 indicates the performance metric of interest – the test set performance on document-level short answer selection. The next rows indicate the performance of a multi-task and the best latent variable method when using the finely labeled data plus the additional coarsely annotated datasets. The ceiling performance in each group shows the oracle achieved by a model also looking at the gold fine-grained labels for the data that the rest of the models see with only coarse paragraph-level annotation. The column Gain indicates the relative error reduction of each model compared to the supervised-only baseline with respect to the ceiling upper bound. As we can see all models benefit from coarsely labeled data and achieve at least 20% error reduction.

To gain more insight into the behavior of the different methods using coarsely annotated data, we measured properties of the predictive distributions $P(y|x,\theta )$ for the three methods on the dataset used with coarse labels in training $D_{70coarse}$ . The results are shown in Figure 5 . For models MTL, MML, PD( $xent$ ), and PD( $err^2$ ), trained on finely labeled $D_{20fine}$ and coarsely labeled $D_{70coarse}$ , we study the predictive distributions $P(y|x,\theta ^M)$ for the four model types $M$ . We measure the properties of these distributions on the dataset $D_{70fine}$ , which is the finely labeled version of the same (question, document)-pairs $D_{70}$ as $D_{70coarse}$0 . Note that none of the models see the fine-grained short answer labels for $D_{70coarse}$1 in training since they only observe paragraph-level relevance annotations.

Mixed supervision data	We focus on the document-level variant of the SQuAD dataset  , as defined by docqa, where given a question and document, the task is to determine the relevant passage and answer span within the passage $(a_p, a_{\mathit {start}}, a_{\mathit {end}})$ . We define finely annotated subsets $D_{y}$ with two different sizes: 5% and 20% of the original dataset. These are paired with non-overlapping subsets of coarsely annotated data $D_{z}$ with sizes 20% and 70% of the original training set, respectively. Both of these settings represent the regime where coarsely annotated data is available in higher volume, because such data can be obtained faster and at lower cost. For both dataset settings, we derive $D_{y}$ and $D_{z}$ from the SQuAD training set, by allocating whole documents with all their corresponding questions to a given subset. In both settings, we also reserve a finely annotated non-overlapping set $\mbox{Dev}_{y}$ , which is used to select optimal hyperparameters for each method.

min-seo-hajishirzi:2017:Short recently studied a related problem of transferring knowledge from a fine-grained QA model to a coarse-grained model via multi-task learning and showed that finely annotated data can help improve performance on the coarse-grained task. We investigate the opposite and arguably much more challenging direction: improving fine-grained models using coarse-grained data. We explore alternatives to the standard approach of multi-task learning via representation sharing  by leveraging the known correspondences between the coarse and fine-grained tasks. In the standard representation sharing approach, the dependencies between the fine-grained and coarse-grained tasks are modeled implicitly. The model must learn representations that are useful for all tasks without knowing how they relate to each other. However, in the scenario of learning from both fine and coarse supervision, the dependencies between the tasks can be modeled explicitly. For example, if a paragraph answers a question, we know that there exists a fine-grained answer span in the paragraph, providing strong constraints on the possible fine-grained answers for the question. We evaluate a multi-task approach and three algorithms that explicitly model the task dependencies. We perform experiments on document-level variants of the SQuAD dataset  .

Nevertheless, the models can assign a probability distribution over fine-grained labels in the documents, and we can measure the peakiness (entropy) of this distribution, as well as see how it compares to the gold hidden label distribution. The first column in the table reports the entropies of the predictive distributions for the four trained models (using the fine task model for the multi-task method MTL). We can see that multi-task method MTL and PD( $xent$ ) (which is equivalent to generalized EM) have lowest entropy, and are most confident about their short answer predictions. MML marginalizes over possible fine answers, resulting in flatter predictive distributions which spread mass among multiple plausible answer positions. The best-performing method PD( $err^2$ ) is somewhere in between and maintains more uncertainty. The next two columns in the Table look at the cross-entropy ( $xent$ ) and squared error ( $err^2$ ) distances of the predictive distributions with respect to the gold one. The gold label distribution has mass of one on a single point indicating the correct fine answer positions.

Improving Span-based Question Answering Systems with Coarsely Labeled Data	We study approaches to improve fine-grained short answer Question Answering models by integrating coarse-grained data annotated for paragraph-level relevance and show that coarsely annotated data can bring significant performance gains. Experiments demonstrate that the standard multi-task learning approach of sharing representations is not the most effective way to leverage coarse-grained annotations. Instead, we can explicitly model the latent fine-grained short answer variables and optimize the marginal log-likelihood directly or use a newly proposed \emph{posterior distillation} learning objective. Since these latent-variable methods have explicit access to the relationship between the fine and coarse tasks, they result in significantly larger improvements from coarse supervision.

Experiments	We present experiments on question answering using the multi-task and latent variable methods introduced in the prior section.

QA model	We build on the state-of-the-art publicly available question answering system by docqa. The system extends BiDAF  with self-attention and performs well on document-level QA. We reuse all hyperparameters from docqa with the exception of number of paragraphs sampled in training: 8 instead of 4. Using more negative examples was important when learning from both fine and coarse annotations. The model uses character embeddings with dimension 50, pre-trained Glove embeddings, and hidden units for bi-directional GRU encoders with size 100. Adadelta is used for optimization for all methods. We tune two hyperparameters separately for each condition based on the held-out set: (1) $\alpha \in \lbrace .01, .1, .5, 1, 5, 10, 100 \rbrace $ , the weight of the coarse loss, and (2) the number of steps for early stopping. The training time for all methods using both coarse and fine supervision is comparable. We use Adadelta for optimization for all methods.

Latent Variable Methods for MixedQA	We study two types of latent variable methods that capture the dependencies between the fine and coarse tasks explicitly. Unlike the multitask learning algorithm described above, both eliminate the need for parameters specifically for the coarse task $\theta _z$ , since we treat the fine labels as a latent variable in the coarsely annotated data. The dependencies between the coarse and fine supervision labels can be captured by the following consistency constraints implied by our task definition: $
\begin{split}
P(y, z|x) = 0, & \forall y \notin \mathcal {Y}(z, x), \text{ and } \\
P(z|y,x) = 1, & \forall y \in \mathcal {Y}(z, x).
\end{split}
$

The best latent variable method (Posterior Distillation with squared error distance) significantly outperforms the multi-task approach, achieving up to 41% relative gain. Figure 4 compares the performance of the three different optimization methods using latent fine-grained answer variables for coarsely annotated data. Here we inlcude an additional last column reporting performance on an easier task where the correct answer paragraph is given at test time, and the model only needs to pick out the short answer within the given paragraph. We include this measurement to observe whether models are improving just by picking out relevant paragraphs or also by selecting the finer-grained short answers within them. Since EM and MML are known to optimize the same function, it is unsurprising that MML and PD with cross-entropy (equivalent to EM) perform similarly. For posterior distillation, we observe substantially better performance with the squared error as the distance function, particularly in the second setting, where there is more coarsely annotated data.

Introduction	Question answering (QA) systems can provide most value for users by showing them a fine-grained short answer (answer span) in a context that supports the answer (paragraph in a document). However, fine-grained short answer annotations for question answering are costly to obtain, whereas non-expert annotators can annotate coarse-grained passages or documents faster and with higher accuracy. In addition, coarse-grained annotations are often freely available from community forums such as Quora. Therefore, methods that can learn to select short answers based on more abundant coarsely annotated paragraph-level data can potentially bring significant improvements. As an example of the two types of annotation, Figure 1 shows on the left a question with corresponding short answer annotation (underlined short answer) in a document, and on the right a question with a document annotated at the coarse-grained paragraph relevance level. In this work we study methods for learning short answer models from small amounts of data annotated at the short answer level and larger amounts of data annotated at the paragraph level.

Maximum Marginal Likelihood	For the task of document-level QA, these constraints ensure that a paragraph is labeled as positive iff there exists a positive answer text span inside the paragraph. The idea of the maximum marginal likelihood method is to define a distribution over coarse labels using the fine-grained model's distribution over fine labels. By expanding the above equations expressing the task dependencies,  $$P(z|x, \theta ) = \sum _{y \in \mathcal {Y}(x)} P(y,z|x, \theta ) = \hspace{-6.0pt}\sum _{y \in \mathcal {Y}(z, x)}\hspace{-6.0pt}P(y|x, \theta )$$   (Eq. 17)  This equation simply says that the probability that a given paragraph $z$ is relevant is the sum of the probabilities of all possible short answer spans within the paragraph.

In EM, we update the parameters $\theta $ to minimize the negative expected log-likelihood of the fine labels with respect to the posterior distribution: $
Q(\theta , \theta ^{old}) &= -\mathop {\mathbb {E}}_{P(y | x, z, \theta ^{old})} \log P(y | x, \theta )\\
&= -\sum _{y \in {\cal Y}(x)} P(y | x, z, \theta ^{old}) \log P(y | x, \theta )
$   By taking a gradient step towards minimizing $Q(\theta , \theta ^{old})$ with respect to $\theta $ , we arrive at a form of generalized EM  . If the loss $Q$ is computed over a mini-batch, this is a form of online EM. We propose a variant of this EM algorithm that is inspired by knowledge distillation methods  ,  , where a student model learns to minimize the distance between its predictions and a teacher model's predictions.

$
Q(\theta , \theta ^{old}) = \sum _{y \in {\cal Y}(x)} \left\Vert P(y | x, z, \theta ^{old}) - P(y | x, \theta ) \right\Vert _2^2
$  In our experiments, we found that squared error outperforms cross entropy consistently. This algorithm also has a close connection to Posterior Regularization  . The coarse supervision labels $z$ can be integrated using linear expectation constraints on the model posteriors $P(y|x,\theta )$ , and a KL-projection onto the constrained space can be done exactly in closed form using equation 20 . Thus the PR approach in this case is equivalent to posterior distillation with cross-entropy and to EM. Note that the posterior distillation method is more general because it allows additional distance functions.

Each document is a sequence of $M$ paragraphs, and each paragraph with index $p$ (where $1 \le p \le M$ ) is a sequence of $n_p$ tokens. The set of possible outputs for the fine-grained task $T_y$ is the set of all phrases (contiguous substring spans) in all document paragraphs. The possible outputs for the coarse task $T_z$ are the paragraph indices $p$ . It is clear that each paragraph output $T_z$0 determines a subset of possible outputs $T_z$1 (the phrases in the paragraph). Fine-grained annotation is provided as $y=(a_{\mathit {p}}, a_{\mathit {start}}, a_{\mathit {end}})$ , where $a_{\mathit {p}}$ indicates the index of the paragraph containing the answer, and $a_{\mathit {start}}, a_{\mathit {end}}$ respectively indicate the start and end position of the short answer.

Formal Definition	We define the fine-grained task of interest $T_y$ as predicting outputs $y$ from a set of possible outputs ${\cal {Y}}(x)$ given inputs $x$ . We say that a task $T_z$ to predict outputs $z$ given inputs $x$ is a coarse-grained counterpart of $T_y$ , iff each coarse label $z$ determines a sub-set of possible labels ${\cal {Y}}(z,x) \subset {\cal {Y}}(x)$ , and each fine label $y$0 has a deterministically corresponding single coarse label $y$1 . We refer to the fine-grained and coarse-grained training data as $y$2 and $y$3 respectively. For our application of document-level QA, $T_y$ is the task of selecting a short answer span from the document, and $T_z$ is the task of selecting a paragraph from the document. The input $x$ to both tasks is a question-document pair.

The shared component with parameters $\theta _s$ maps the sequence of tokens in the document $d$ to continuous representations contextualized with respect to the question $q$ and the tokens in the paragraph $p$ . We denote these representations as $\mathbf {h}(x,\theta _s) = (\mathbf {h}^1(\theta _s),\mathbf {h}^2(\theta _s),\ldots ,\mathbf {h}^{M}(\theta _s)),$  where we omit the dependence on $x$ for simplicity. Each contextualized paragraph token representation is a sequence of contextualized token representations, where $\mathbf {h}^p(\theta _s) = {h_1}^p(\theta _s),\ldots ,{{h}_{n_p}}^p(\theta _s).$

Paragraph-level supervision is provided as $z=(a_{\mathit {p}}, \_,\_)$ , only indicating the paragraph index of the answer, without the start and end token indices of the answer span. The coarse labels $z$ in this case limit the set of possible labels $y$ for $x$ to:  $${\cal {Y}}(z,x) = \lbrace (a_{\mathit {p}}, a^{\prime }_{\mathit {start}}, a^{\prime }_{\mathit {end}})~|~1 \le a^{\prime }_{\mathit {start}} \le a^{\prime }_{\mathit {end}} \le n_p\rbrace .$$   (Eq. 8)  In the presence of the coarsely annotated $D_z$ when the task of interest is $T_y$ , the research question becomes: how can we train a model to use both $D_z$ and $D_y$ in the most effective way?

Multi-task learning for MixedQA	The multi-task learning approach defines models for $T_y$ and $T_z$ that share some of their parameters. The data for task $T_z$ helps improve the model for $T_y$ via these shared parameters (representations). Multi-task learning with representation sharing is widely used with auxiliary tasks from reconstruction of unlabeled data  to machine translation and syntactic parsing  , and can be used with any task $T_z$ which is potentially related to the main task of interest $T_y$ . Let $\theta = \begin{bmatrix}
\theta _y & \theta _z & \theta _{s}
\end{bmatrix}$ be the set of parameters in the two models. $\theta _y$ denotes parameters exclusive to the fine-grained task $T_y$ , $\theta _z$ denotes parameters exclusive to the coarse-grained task $T_z$ , and $\theta _s$ denotes the shared parameters across the two tasks.

Then the multi-task learning objective is to minimize $L(\theta , D_y,D_z)$ :  $$\begin{split}
& -\sum _{(x,y) \in D_{y}} \log P(y|x, \theta _s, \theta _y) \\ -~~~\alpha _z &\sum _{(x,z) \in D_{z}} \log P(z|x , \theta _s, \theta _z)
\end{split}$$   (Eq. 10)  Here $\alpha _z$ is a trade-off hyper-parameter to balance the objectives of the fine and coarse models. We apply multi-task learning to question answering by reusing the architecture from min-seo-hajishirzi:2017:Short to define models for both fine-grained short answer selection $T_y$ and coarse-grained paragraph selection $T_z$ . After the two models are trained, only the model for the fine-grained task $T_y$ is used at test time to make predictions for the task of interest.

Posterior Distillation	In addition to direct maximization of the marginal likelihood for latent variable models  , prior work has explored EM-based optimization  including generalized EM  , which is applicable to neural models  . We present a class of optimization algorithms which we term Posterior Distillation, which includes generalized EM for our problem as a special case, and has close connections to knowledge distillation  ,  . We begin by describing an online generalized EM optimization algorithm for the latent variable model from equation ( 17 ) and show how it can be generalized to multiple variants inspired by knowledge distillation with priviledged information  . We refer to the more general approach as Posterior Distillation. [t] Posterior Distillation Algorithm.

Fine-grained answer selection model	The fine-grained answer selection model $P(y|x,\theta _s,\theta _y)$ uses the same hidden representations $\mathbf {h}(x,\theta _s)$ and makes predictions assuming that the start and end positions of the answer are independent, as in BiDAF  .

The output parameters $\theta _y$ contain separate weights for predicting starts and ends of spans: $\theta _y =
\begin{bmatrix}
\theta _y^{\mathit {start}} & \theta _y^{\mathit {end}}
\end{bmatrix}$  The probability of answer start $a_{\mathit {start}}$ in paragraph $a_{\mathit {p}}$ is proportional to $\exp (h(a_{\mathit {start}}, a_{\mathit {p}}, \theta _s)\cdot \theta _y^{\mathit {start}})$ , where $h(a_{\mathit {start}}, a_{\mathit {p}}, \theta _s)$ is the hidden representation of the token $a_{\mathit {start}}$ in paragraph $a_{\mathit {p}}$ , given shared parameters $\theta _s$ . The probability for end of answer positions is defined analogously.

Paragraph answer selection model	The paragraph selection model for task $T_{z}$ uses the same hidden representations $\mathbf {h}(x,\theta _s)$ for the tokens in the document. Because this model assigns scores at the paragraph granularity (as opposed to token granularity), we apply a pooling operation to the token representations to derive single vector paragraph representations. As in  , we use max-pooling over token representations and arrive at $h^p(\theta _s)=\mbox{max}({h_1}^p(\theta _s),\ldots ,{{h}_{n_p}}^p(\theta _s))$  Using the coarse-grained task-specific parameters $\theta _z$ , we define the probability distribution over paragraphs as: $P(a_p = p | x, \theta _s, \theta _z) = \frac{\exp (h^p(\theta _s) \cdot \theta _z)}{\sum _{p^{\prime }}{\exp (h^{p^{\prime }}(\theta _s) \cdot \theta _z)}} $

The objective function for the coarsely labeled data $D_{z}$ can be expressed as a function of the parameters of the fine-grained task model as:  $$\begin{split}
-\sum _{(x,z) \in D_{z}} \log \sum _{y \in \mathcal {Y}(z, x)} P(y|x, \theta _s, \theta _y)
\end{split}$$   (Eq. 18)  The fine-grained task loss and the coarse-grained task loss are interpolated with a parameter $\alpha _z$ , as for the multi-task approach.

[1] not converge Sample a mini-batch $(x_1,y) \sim D_y$ and $(x_2,z) \sim D_z$ Calculate predicted distribution for current $\theta ^{old}$ $P(\hat{y}|x_2, \theta ^{old})$ Correct and renormalize the predicted distribution using the coarse supervision signal by setting $q(\hat{y}|x_2) \propto {\left\lbrace \begin{array}{ll}
P(\hat{y}|x_2, \theta ^{old}), \hat{y} \in \mathcal {Y}(z)\\
0, \hat{y} \notin \mathcal {Y}(z)
\end{array}\right.}$  Update $\theta $ by taking a step to minimize - $\log P(y|x_1, \theta )$ + $\alpha _z$ distance( $P(y|x,\theta ), q$ ). In EM-like algorithms one uses current model parameters $\theta ^{old}$ to make predictions and complete the latent variables in input examples, and then updates the model parameters to maximize the log-likelihood of the completed data.

We formalize this procedure for our case below. Given a coarse example with input $x$ and coarse label $z$ , we first compute the posterior distribution over the fine labels $y$ given $z$ and the current set of parameters $\theta ^{old}$ :  $$P(y | x, z, \theta ^{old}) &= \frac{[[\hbox{$y \in {\cal Y}(x)$}]] \times P(y | x, \theta ^{old})}{\displaystyle \sum _{y \in {\mathcal {Y}}(z, x)} P(y | x, \theta ^{old})}$$   (Eq. 20)   where $[[\cdot ]]$ is the indicator function.

In our case, we can consider the posterior distribution $P(y | x, z, \theta ^{old})$ to be the teacher, and the model distribution $P(y | x, \theta )$ to be the student. Here the teacher distribution is directly derived from the model (student) distribution $P(y | x, \theta ^{old})$ by integrating the information from the coarse label $z$ . The coarse labels can be seen as privileged information  which the student does not condition on directly. Let us define $Q(\theta , \theta ^{old})$ in a more general form, where it is a general distance function rather than cross-entropy: $
Q(\theta , \theta ^{old}) = \textsc {distance}(P(y | x, z, \theta ^{old}), P(y | x, \theta ))
$  We refer to the class of learning objectives in this form as posterior distillation. When the distance function is cross entropy, posterior distillation is equivalent to EM. As is common in distillation techniques  , we can apply other distance functions, such as the squared error.

The combined loss function using both finely and coarsely labeled data to be minimized is:  $$\begin{split}
& \sum _{(x,y) \in D_{y}} -\log P(y|x, \theta _s) \\ +~~~\alpha _z &\sum _{(x,z) \in D_{z}} Q(\theta ,\theta ^{old},x,z)
\end{split}$$   (Eq. 21)  Figure 2 presents an illustration of the multi-task and posterior distillation approaches for learning from both finely and coarsely labeled data. Algorithm 1 lists the steps of optimization. Each iteration of the loop samples mini-batches from the union of finely and coarsely labeled data and takes a step to minimize the combined loss.

We report final performance metrics on $\mbox{Test}_{y}$ , which is the unseen SQuAD development set.

The contributions for our papers are:

Question: What datasets have this method been evaluated on?
Answer: document-level variants of the SQuAD dataset