Venue: NAACL 2024

# Pregnant Questions: The Importance of Pragmatic Awareness

in Maternal Health Question Answering

Neha Srikanth\({}^{\diamondsuit}\)1 Rupak Sarkar\({}^{\diamondsuit}\)2 Heran Mane\({}^{\spadesuit}\) Elizabeth M. Aparicio\({}^{\clubsuit}\) Quynh C. Nguyen\({}^{\spadesuit}\) Rachel Rudinger\({}^{\diamondsuit}\) Jordan Lee Boyd-Graber\({}^{\diamondsuit}\)3 \({}^{\diamondsuit}\)Department of Computer Science, University of Maryland \({}^{\spadesuit}\)Department of Epidemiology and Biostatistics, University of Maryland \({}^{\clubsuit}\)Department of Behavioral and Community Health, University of Maryland \({}^{\clubsuit}\)Can I breasfreed my baby when I am 1117

Footnote 1: “Pregnant” in our title also refers to its secondary definition, “full of meaning” (as in “a pregnant pause”) alluding to the idea that questions are laden with implicit beliefs.

###### Abstract

Questions posed by information-seeking users often contain implicit false or potentially harmful assumptions. In a high-risk domain such as maternal and infant health, a question-answering system must recognize these pragmatic constraints and go beyond simply answering user questions, examining them in context to respond helpfully. To achieve this, we study assumptions and implications, or _pragmatic inferences_, made when mothers ask questions about pregnancy and infant care by collecting a dataset of 2,727 inferences from 500 questions across three diverse sources. We study how health experts naturally address these inferences when writing answers, and illustrate that informing existing QA pipelines with pragmatic inferences produces responses that are more complete, mitigating the propagation of harmful beliefs.

+
Footnote †: FOOTNOTE:1]Footnote 1: “Pregnant” in our title also refers to its secondary definition, “full of meaning” (as in “a pregnant pause”) alluding to the idea that questions are laden with implicit beliefs.[ENDFOOTNOTE]

## 1 Introduction

Humans have varying information needs when they ask questions Taylor (1962). Sometimes these needs are easily inferred from the surface form, such as in factoid questions (e.g. _"Who is the 44th president of the United States"_). However, in a question such as "_Is there a good non-dairy Baby milk I can supplement for my newborn?_", addressing the underlying false assumption "_Newborns can safely drink non-dairy milk_" becomes part of satisfying the unexpressed information need_. Complete answers to these types of questions must not only address the surface question itself, but also "question the question", critically examining its _pragmatic needs_.

These needs become magnified in sensitive domains, such as consumer health or the legal domain. In these settings, addressing pragmatic needs of questions involves proactively addressing false assumptions or implications in questions to ensure that the asker does not continue holding inaccurate beliefs that they may act on. For example, a complete answer to the question about non-dairy milk for newborns should address that while non-dairy milk is viable for older babies, newborns and infants need human breast milk or dairy-based formula because they offer complete nutrition.

Language models have been shown to exhibit sycophancy Sharma et al. (2023), sometimes adjusting responses to align with a human user's view. However, helpful QA systems should not only _challenge_ false or subjective assumptions in questions Kim et al. (2022) by verifying them against a vetted corpus, but also infer the asker's intent to make sure that its answer satisfactorily addresses their deeper information needs Taylor (1962), just as humans do.1

Footnote 1: “Pregnant” in our title also refers to its secondary definition, “full of meaning” (as in “a pregnant pause”) alluding to the idea that questions are laden with implicit beliefs.

We construct a dataset2 of 2,727 assumptions

Figure 1: We ask public health experts to identify assumptions and implications from questions and find that incorporating them in a qa pipeline produces a more complete answer.



and implications in 500 questions (SS2) collected from three diverse sources to study (1) how humans embed such assumptions and implications in questions, and (2) the extent to which they are _naturally_ addressed in answers written by public health experts. We then ground assumptions and implications, two primary ways humans embed beliefs in questions, in existing linguistic theory of presuppositions and implicatures respectively (SS3). We refer to presupposition and implicative collectively as _pragmatic inference_. While recent work has focused on the task of detecting and addressing false _presuppositions_ in open-domain QA[47], we find that false beliefs of question askers are more likely to present as implicatures than presuppositions (SS4). We experiment with _inducing_ pragmatic behavior in existing qa pipelines with state-of-the-art retrieval and machine reading models (SS5). On questions with at least one highly plausible false pragmatic inference, our expert annotators rated responses from our pragmatic qa system as more helpful and informative.

Thus, qa systems of the future must proactively address assumptions and implications in questions as they are increasingly deployed in sensitive domains.

## 2 Collecting Assumptions and Implications in the Wild

In contrast with factoid qa, systems deployed in _sensitive_ domains such as consumer health must proactively mitigate harm. In these settings, correcting false assumptions is _not optional_: systems must provide contextual answers that balance _information completeness_ with brevity.

Access to high quality healthcare in the United States vastly differs across socioeconomic backgrounds [69]. Such users are often likely to turn to accessible internet resources and--as of late--general purpose chatbots [85]. This motivates us to focus on maternal and infant care, a challenging area of consumer health where patients are concerned with _both_ their own physical health as well as the health of their child.

To effectively study and induce pragmatic behavior in qa systems, the evaluation questions we choose must reflect real-world experiences and situations for which there may not be a straightforward answer explicitly addressed in a single web document. For example, answers to Natural Questions [46, nq]--a popular open-domain question-answering dataset--can be

\begin{table}
\begin{tabular}{l l c} \hline  & **Question** & **Expert-Annotated Pragmatic Inference \& Vercality** \\ \hline rosie & Is it okay for my to color my hair after giving birth? & Hair dye chemicals can pass through breast milk from mother to child. (False/tissue) \\
**QA System** & What is the advantage for not having an epidural during the labor? & Avoiding an epidural contributes to a more “natural” and undecideted birthing experience. (False/ nuisance) \\  & What cough medicine is appropriate for breastfeeding mothers? & Some cough medicines can be secreted in breast milk. (True) \\ \hline
**Reddit** & Is it safe to lay on my stomach at 28 weeks of pregnancy? & Sleeping on the stomach while pregnant may have potential risks (False/unsure) \\  & Is it bad to use different bottles/inpiples during feedings? & Using different bottles or nitpiles for feeding may compromise the baby’s latch. (False/unsure) \\  & How can I increase the time between feedings for my 3-month-old baby? & It may be possible to keep through the night while still ensuring the baby is fed. (True) \\ \hline
**Natural Questions** & When does the fetus begin to develop memory? & Fettuses have the ability to form memories. (True) \\  & What causes a rupture in the amniotic sac? & There may be ways to prevent early amniotic sac rupture. (False/Unsure) \\  & When do the clinical manifestations of an ectopic pregnancy appear? & There may be clinical manifestations of an ectopic pregnancy that do not appear early on. (True) \\ \hline \end{tabular}
\end{table}
Table 1: Health experts identify pragmatic inferences from questions from three sources: Reddit, Natural Questions [46], and questions asked to our domain-specific qa system, rosie[85]. They also determine the veracity of each inference and provide supporting evidence from a trusted web document.

\begin{table}
\begin{tabular}{l r r r} \hline  & \multicolumn{2}{c}{**Matarial Health**} & \multicolumn{1}{r}{**Reddit**} & \multicolumn{1}{r}{**Natural**} \\  & **QA** & & **Questions** \\ \hline
**\# questions** & 200 & 200 & 100 \\
**ans. length (\# sent)** & 3.9 & 6.6 & 5.6 \\
**\# inferences** & 1161 & 1114 & 452 \\ \% false/subjective & 22.5 & 30.8 & 20.1 \\ \% true & 77.5 & 69.2 & 79.9 \\ \hline \end{tabular}
\end{table}
Table 2: Dataset statistics stratified by question source.

 

found directly in short extracted text snippets from Wikipedia (Table 1). In contrast, effectively answering the subjective questions sourced from Reddit requires commonsense reasoning and domain knowledge while identifying the asker's intent.

We carefully construct a dataset of questions from three distributionally distinct sources: a domain-specific qa system we design and deploy to pregnant and postpartum participants we recruit [35], Reddit, and nq. Then, we introduce an annotation scheme to elicit assumptions and implications from these questions, validate their plausibility, and finally collect supporting evidence to determine their veracity. Our final dataset contains 2,727 assumptions from 500 evaluation questions (Table 2). We also include 150 development questions used to train annotators and develop our qa systems.

### Gathering a Diverse Set of Maternal and Infant Health Questions

Maternal Health qa System.We source questions come from a maternal and infant health-specific question answering system that we build [34], henceforth referred to as rosise. Users ask questions pertaining to pregnancy or infant health and are instructed that the qa system does not have any personalized knowledge of their individual medical history or pregnancy.

This system operates over a corpus of web documents we construct3 from trusted sources including United States governmental and hospital organizations on maternal and infant health, and spans salient topics such as pregnancy and postpartum symptoms, developmental milestones, and infant safety. Our end-to-end qa system, rosiE, uses a passage retriever and reranker to provide web passages as answers to study participants via a mobile application. We randomly sample 200 anonymized questions asked to rosiE for our evaluation set and 50 questions for our development set.

Footnote 3: Corpus available upon request. We use Barbaresi2021 to scrape 408,000 web documents which we split into passages of 100 tokens following Karpukhin2020.

Reddit.While the questions asked to rosiE do reflect real-world experiences, they are asked _to an automatic system_ and thus tend to include less situational detail or implicit content. We turn to Reddit4 to capture long-tail questions that are about the diverse set of unique situations a new or expecting parent goes through. Table 1 highlights some distributional differences between questions from Reddit and other data sources. Our questions come from four popular subreddits about maternal and infant health: r/BabyBumps, r/breastfeeding, r/NewParents, r/Mommit, and r/beyondthebump from the pushshift5 dump.

Footnote 4: [https://www.reddit.com/](https://www.reddit.com/)

We develop a series of heuristics as a recall-oriented first step to identify questions with false or subjective assumptions. We begin by selecting questions where an upvoted comment shows assumption-correcting behavior or where a user invokes their medical expertise, identified by a select list of discourse markers (Appendix A). Of these, we only retain posts beginning with a "wh" word, filtering a few hundred thousand posts down to 2,858 questions.

As Reddit encourages community participation, many questions are "community seeking" as opposed to _information_-seeking. To identify information-seeking questions, we use gpt-3.5[69] to filter medical questions from non-medical questions (Prompt A.1) then manually vet the final set of 297 questions. We randomly sample 200 questions for our evaluation dataset and 50 questions for our development set, discarding the rest.

Titles of Reddit posts are often a hook or a summary of the entire post. Using the 50 development questions, we use gpt-3.5 to minimally edit the titular question to include crucial details from the post description, providing a series of exemplars (Prompt A.2). These rewrites mainly include the age of a newborn or the stage of pregnancy from the description, but sometimes include small situational details that contextualize the question. Two authors validate all rewrites, keeping the original title wherever both authors agree that the rewrite changed the communicative goal of the asker.

Natural Questions.Lastly, we include maternal and infant health questions from nq to study pragmatic aspects of factoid-style questions. We embed all questions in the train set of nq using the sentence-transformers [19] implementation of all-mpnet-base-v2 [20], including unanswerable questions [10]. We identify 2500 answerable questions and 2500 unanswerable questions as maternal health-related by identifying the top 100 nearest 

neighbors of 50 randomly sampled questions from the development sets of Reddit and rose.6 From this set, we randomly sample 1007 questions for our evaluation set and 50 for our development set. Though obtained with a nearest neighbors approach, these questions greatly differ from those obtained from our previous sources, as they reflect the factoid qa-oriented tasks and goals of the original dataset creators (Table 2).

Footnote 6: We tried several different filtering heuristics, including keyword-based detection, but the nearest neighbors approach yielded the most topical questions.

Collecting Human Answers from Health Experts.We recruit a team of twenty health experts using Upwork8 to annotate our data including obstetricians and gynecologists (OB/GYNs), nurses, lactation consultants, and public health experts, many of whom have experience with patients. In addition, many of these expert annotators are either currently pregnant or postpartum or have been in the past. We ask a subset of six experts to write helpful and informative long-form answers to all 500 questions in our dataset (Figure 6, bottom panel). While annotators write answers from scratch, they must provide supporting web documents from the same list of verified sources we use to build the corpus for rosie.

Footnote 7: No questions make up a smaller proportion of our evaluation dataset as we avoid diverting large amounts of annotation resources to factoid-style questions.

### Identifying Assumptions and Implications

Inferring _possible_ assumptions, implications, and asker beliefs from patient questions in our domain are challenging. In the past, others have extracted assumptions using shallow signals from the surface form of a question Kim et al. (2021); Parrish et al. (2021). While some assumptions or implications in our dataset can be inferred directly from the question expression, others require deeper domain or experiential knowledge (Table 1).

Eliciting these assumptions and implications from non-linguists is challenging as existing linguistic frameworks (SS3.1) are inaccessible or cumbersome for those unfamiliar with the theoretical concepts behind them. As such, we operationalize large-scale data collection by asking five annotators from a _different_ subset of our expert annotator pool to first write a list of subquestions that an answer to the original question would address (Figure 6, top panel). Doing so primes annotators to reason about the intent behind a question as well as the information needs of an asker. Then, we ask them to write a set of sentences reflecting possible beliefs or assumptions that the patient may hold (or, alternatively, beliefs that any complete answer to the question must address). We emphasize that the assumptions they write can be either medically or scientifically true or false.

Then, we _consolidate_ the set of subquestions and human-written assumptions and beliefs into a single set of assumptions and implications using GPT-3.5 (Prompt B.1).

### Annotating Inference Veracity

Lastly, we ask a new subset of eight expert annotators to annotate whether each assumption and implication in our dataset is medically or scientifically true, false, or subjective and provide a supporting web document from our list of verified sources along with a passage from the document (Figure 6, middle panel).

Validation.To verify that the assumptions and implications we extract are plausibly inferrable from the question, we recruit an additional pair of health experts, which we refer to as expert _validators_, to rate inferences.9 We sample 100 assumptions and implications judged as false or subjective, and 100 true inferences and ask our expert validators to rate the plausibility of an inference on a 1-5 Likert scale based on how likely the question asker is to believe the assumption or implication. Henceforth, we refer to this sample of 200 inferences coming from 152 unique questions as inferenceSample. Both annotators judge the majority of our inferences as plausible, with 80% and 95% rated with a score of at least 3 (see Figure 4 for the rating scale). Spearman's correlation between the two annotators is 0.69. See Appendix C for more detail.

Footnote 9: These expert validators were not a part of our dataset construction.

## 3 Grounding Assumptions and Implications in Linguistic Theory

Assumptions and implications in our dataset map to two well-studied phenomena in linguistic pragmatics: presupposition and implicature Grice (1975); Stalnaker et al. (1977). We begin with a short primer of both types of _pragmatic inference_ (SS 3.1) and then discuss the implications of both types in a QA setting (SS 3.2).

 



### Two Types of Pragmatic Inference: Presupposition and Implicature

A sentence \(S\) is a _pragmatic inference_ of a question \(Q\) if, depending on the _context_ and _conversational goals_ of discourse participants Jeretic et al. (2020), a human would believe that the asker of \(Q\) believes or assumes \(S\) to be true. Henceforth, we refer to the assumptions and implications that we collect in our dataset as _pragmatic inferences_. We review the two most relevant types of pragmatic inferences: _presupposition_ and _implicature_.

Presupposition.Presuppositions are implicit assumptions in utterances taken for granted by discourse participants Beaver (1997). The question _"What vitamins should I stop taking after becoming pregnant?"_ presupposes _"I was taking vitamins before becoming pregnant."_ Presuppositions can often be detected solely by the presence of a lexical or syntactic trigger Levinson et al. (1983). In the example above, the word _stop_ presupposes that an activity was already in motion. We refer to these presuppositions as "trigger-based".

As we observe during the collection of our dataset, domain or world knowledge is often needed to capture presuppositions in real-world data that are not apparent from lexical or syntactic cues Abusch (2002). For example, the question _"Are multiple ultrasounds dangerous for my baby?"_ does not directly result in non-trivial trigger-based presuppositions. However, the asker of the question presupposes that the effects of an ultrasound are additive and hence asks about whether that additive effective is _dangerous_.

Implicature.Implicative is a type of pragmatic inference that is _suggested_ by an utterance as opposed to part of its literal meaning Grice (1975). Consider the question _"Do most babies fit in newborn clothes?"_ While the speaker understands that newborn clothes fit _some_ babies, their question implies that not all babies fit in newborn clothes. As we discover, a significant portion of inferences in our dataset are _implied_ from questions rather than presupposed, but detecting and generating implicatures remain understudied in nlp.

Some implicatures are related to lexical items or syntactic structure of utterances. For example, the statement _"These prenatal vitamins are in gumdrop form, but are healthy"_ implies that gumdrops are usually _not_ healthy. Others are a function of a speaker's intent, beliefs, and other contextual elements Zheng et al. (2021). While they are a part of the content of an utterance, these implicates are not _at-issue_ (e.g. the main point under discussion Potts (2004); Koev (2018)) and are not encoded by the linguistic properties of a sentence Allott (2018). Consider the question _"How can you tell the difference between postpartum depression and exhaustion?"_. Reasoning about asker belief, we may conclude that they are implying that the two conditions should be treated differently, as one is more serious than the other.

### Presupposition and Implicature in qa

In a natural setting, as we discover, humans embed both presuppositions and implicatures nearly equally in questions (SS4). However, from a linguistic perspective, they represent different levels of an asker's commitment to the propositional content of the inference Peters (2016). Presuppositions are already a part of an asker's world model. In contrast, implicatures are _likely_ beliefs that may be negated in an asker's subsequent utterances. Consider the question _"Is it normal for my baby to move more than usual when closer to due date?"_ with both the presupposition _"There are factors that contribute to changes in fetal movement as the due date approaches"_ and the implicature _"It may not be necessary to be concerned if there is a significant increase in fetal movement close to the due date."_ While both are false, the presupposition is stronger, and is clearly in need of addressing in a potential answer. As illustrated, these distinct phenomena must be dealt with differently when answering a question.

Related Work.Existing work in pragmatics in qa focuses on open-domain question answering. Kim et al. (2021) present the first study of presuppositions in Google search queries using the Natural Questions (nq) dataset Kwiatkowski et al. (2019) that are unanswerable due to false presuppositions. However, their system only addresses trigger-based presuppositions, overlooking the type of deeper presuppositions present in our dataset derived from world or domain knowledge. Other work has looked at Google queries with _questionable_ assumptions Kim et al. (2022) and false presuppositions in open-domain Reddit questions Yu et al. (2022). Computational studies of implicature have only focused on specific types, such as scalar implicature (e.g., some \(X\to\) not all \(X\)) Schuster et al. (2020); Zheng et al. (2021); Kabbara and Che

ung, 2022; Jeretic et al., 2020). As a result of the context induced by our domain, implicatures in our dataset extend beyond scalar implicature.

## 4 How do people ask and answer questions?

Before we investigate the behavior of qa systems, we first study how humans embed pragmatic inferences in their questions (SS4.1) as well as to what extent they are _naturally_ addressed by human public health experts (SS4.2).

### Pragmatic Inference Type: Understanding Speaker Commitment

When users ask questions, how strongly are they committed to the inferences that experts identify in their questions? Presupposition is a phenomenon based on _mutual_ acknowledgment of facts: when a human makes a presupposition, not only are they presuming the content of the inference, they are also signaling the belief that their _interlocutor_ (here, a qa system) should believe it too.

On the other hand, implicatures are a softer way for humans to express uncertainty. For example, _"Which immunity injections can I skip for my baby?"_ and _"Is it sufficient if my baby takes most immunity injections"_ have the same underlying inference (_"It is okay to pick and choose vaccines"_), but is taken for granted in the first (presupposition), whereas loosely suggested in the second (implicative). We want to distinguish inferences--separating implications from presuppositions--in our questions to better characterize _so that we can prioritize addressing stronger false inferences._

Annotation Framework.A pair of authors independently annotate all inferences in inferencesample as a presupposition or implicature by first determining whether it is a proposition about the world that the asker believes to be true, without which the question would not be felicitous (pre-supposition) or whether it involves deriving asker belief through communicative principles (implicating). Between authors, Cohen's kappa is \(\kappa=0.85\), indicating strong agreement. Author annotators adjudicated the final inference type (see Figure 2 for overall distribution), but individual annotator labels and adjudication rationales are preserved as a part of our dataset.

Findings.Presuppositions and implicatures are balanced in inference-sample (Figure 2), with a slight majority of inferences as implicatures, indicating that many inferences that health expert annotators identify are more subtle. When an inference is true, it is almost equally likely to be a presupposition or an implicature. However, when users make false or subjective (veracity marked "Unsure") inferences, **they are more likely to do so via implicitity** (Figure 2). Past work has looked into generating and verifying _presuppositions_ in open-domain QA, but identifying and addressing _implicitures_ in an effort to make answers information-complete remains heavily underexplored. This finding highlights a key strength of our work: the ensuing context from our specific domain tests the usefulness of pragmatic inference in QA by allowing us to

Figure 2: Tree-map Shneiderman (1992) visualizing the distribution of expert-annotated pragmatic inferences in inference-sample with their veracity, inference type, and whether or not they were addressed or not in the expert-written answer to the question from which they came from. When users make false or subjective inferences, they are more likely to do so as an implicature. Moreover, when an inference is false, it is more likely to be naturally addressed in an answer by public health experts.

 

extract a greater range of inferences.

In settings that lack such context (e.g. single-turn open-domain QA), we are restricted to leveraging lexical or syntactic signals from the surface form of the question Kim et al. (2021) since reasoning about asker belief is not possible without other contextual signals. For example, in the absence of additional context, the question _"Should I push grandparents for flu shot and tdq?"_ may give rise to inferences involving the safety or effectiveness of these vaccines for the elderly. However, upon learning that this was asked in a web forum by a postpartum mother, we may reason that she believes her infant may be at risk for contracting the flu or other diseases if their grandparents handle them unvaccinated.

### Addressing Inferences in Expert Answers

When health experts are tasked with _answering_ questions, how likely are they to naturally address inferences that users implicitly make? Studying whether or not answers _naturally_ address pragmatic inferences (SS 4.2) gives us better insight into the types of inference health experts, and in turn models, should prioritize when answering questions.

Annotation.We ask two annotators from our expert annotator pool to determine whether each inference in inference-sample is addressed, either implicitly or explicitly, by the human-written answer to its source question.

Findings.The majority of inferences in inference-sample are addressed by the human-written answer _naturally_ (Figure 2). Importantly, **when an inference is false, it is more likely to be naturally addressed.** Moreover, a significant number of _true_ inferences are also addressed by an answer, indicating that health experts not only aim to correct false or subject inferences but also prioritize completeness. This key finding supports one of the main arguments of this work: QA systems _must_ address pragmatic inferences in their answers, just as humans do.

## 5 Inducing Pragmatic Behavior in QA

Inducing pragmatic behavior in QA systems is not straightforward. Existing systems are not trained to proactively reason about asker beliefs, since many popular QA datasets do not necessitate this type of behavior (e.g. factoid QA).

We experiment with eliciting model answers that address the pragmatic needs of questions, such as refuting false inferences, using the pragmatic inferences in our dataset. We inject inferences at each stage of the classic QA pipeline: passage retrieval, reranking, and machine reading (SS5.1) and evaluate outputs against expert-written answers with both automatic and human evaluation (SS5.2).

Experimental Setup Corpus.We use the corpus from Mane et al. (2023) of 408,000 documents from verified web

Figure 3: Our baseline and pragmatic inference-augmented QA pipelines. We experiment with retrieval, reranking, and reading stages and a variety of instruction-tuned and prompt-based models.

 

sources on maternal health and infant care and augment the corpus with the sources that our expert annotators found while both writing answers and determining the veracity of inferences.

Baseline Models.As a baseline system, we use a retrieval, reranking, and reading-based qa pipeline. Contriever (Tlzacard et al., 2022), an unsupervised dense passage retriever, identifies top relevant documents (\(n=100\)) in our corpus given a question. Those documents are reranked using tartfull Asai et al. (2022), a multi-task retrieval system with a cross-encoder architecture (Instruction E.1). tart is instruction-tuned, equipping it with the flexibility to redefine passage relevance for different tasks. We feed the top five reranked documents to three different reader models: flan-ts-xxl Chung et al. (2022), 11 billion parameters), an instruction-tuned, _prompt_-based encoder-decoder model jointly trained on a multiple tasks with a standard answer extraction prompt from Mishra et al. (2022) (Instruction E.3), mistral-7B Jiang et al. (2023) (an open source large language model, Prompt E.4), and gpt-3.5 (Prompt E.4).

Augmenting Systems with Pragmatic Inferences.In addition to retrieving the top 100 passages using the question as input, we retrieve the top 100 passages for each pragmatic inference of the question (\(i_{1}...i_{k}\)) as well. Then, for each pragmatic inference \(i\), we rerank the top 100 passages using a new inference-informed instruction (Instruction E.2) and select the top passage post-reranking. We augment the top five reranked passages from the question with these \(k\) top passages from each pragmatic inference to feed to each reader (Prompt E.5). During reading, we prompt mistral-7B and gpt-3.5 to address all \(k\) assumptions when generating an answer.10 To keep the same _number_ of passages fed to readers in the baseline pipelines as in the inference-augmented pipeline, we add \(k\) extra passages to the top five existing ones. This ensures that while the volume of information presented to machine readers is the same in both pipelines, the nature of the content differs, allowing us to measure the utility of inference augmentation during retrieval and reranking. Figure 3 visualizes our baseline and inference-augmented qa pipelines.

Footnote 10: We do not use fan-t5-xxL here because it struggled with reading in the baseline setting.

### Evaluation

We evaluate answers from seven pipeline variations against expert answers (Table 3): (1) The **No Reader** baselines consist of the top retrieved passage based on the input question (retrievenolly) and the top remarked passage from the input question (rerank-only), (2) three **Baseline Readers** (baseline-flan-t5-xxxL, baseline-mistral-7b, and baseline-gpt-3.5), and (3) two **Inference-Augmented** pipelines inference-mistral-7b, and inference-gpt-3.5).

Automatic Evaluation Metrics.Three automatic evaluation metrics measure the quality of generated answers: rouge Lin (2004) (both F1 and recall),bleurt Sellam et al. (2020), and QAFactEvalFabbri et al. (2022), a more recent QA evaluation metric originally designed to measure the faithfulness of summaries. gpt-3.5 scores the strongest according to QAFactEval, our main evaluation metric because it--of the three metrics--most closely captures information content. However, automatic evaluation of generated answers does _not_ capture several higher-level semantic and pragmatic aspects of the question. Thus, we still need experts to validate the answers.

Human Judgments.We ask our expert _validators_ to score answers from the top-performing baseline and inference-augmented pipeline (baseline-gpt-3.5 and inference-gpt-3.5, on QAFactEval respectively). For each of the 152 questions in inference-sample, expert validators score both model outputs simultaneously from 1-5 based on completeness (instructions in Figure 5). Answers typically received a score of 1 when they were off-topic and missing crucial information, a score of 2 when they were topical but still missing crucial information, 3 when containing all essential information to the question, 4 when most information was present for completeness, and a score of 5 when the answer was information complete. Judging the information completeness of an answer is a subjective task, as reflected by the Spearman rank correlation between their annotations (\(\rho=0.34\)). While the mean score of inference-augmented examples is comparable to baseline answers (\(4.43\) vs. \(4.45\)), annotators rated the inference-augmented answer as equivalent or better than its baseline counterpart in 75% of questions in inference-sample (see Table 5 for examples).

We further focus on annotator preferences on our 

original motivating population of questions--those with highly plausible, false assumptions. Both annotators rate inference-augmented answers higher than the default answers in the subset of questions with at least one _highly plausible_, (plausibility=5) _false_ pragmatic inference (Table 4). We hypothesize that the similar ratings received by the two systems across _all_ questions is due to shortcomings in the instruction-following capabilities of LLMs. Forcing the reader model to address pragmatic inferences distracts it from answering the question more completely, and does not always result in more helpful answers when the pragmatic inferences are true. These results illustrate the promise of inducing pragmatic behavior in qa models and represent a lower bound of their performance, as none of the models we experiment with were trained to optimize for addressing assumptions in questions.

## 6 Can inference extraction be automated?

While pragmatic inferences elicited from health experts are informed by their expertise, they are slow and costly to collect. Our qa experiments use _human-written inferences_ to establish an upper bound of answer quality with existing models. However, a fully automatic pragmatic qa pipeline must first generate pragmatic inferences relevant to a question and _then_ generate an answer that addresses the subset of false inferences. As such, we experiment with generating pragmatic inferences with GPT-3.5Ouyang et al. (2022) to understand to what extent automating the process is feasible with existing prompting and in-context learning.

Experimental Setup.We generate inferences with GPT-3.5 for all questions in inference-sample using seven in-context examples corresponding to 37 different pragmatic inferences, as more in-context examples yields diminishing returns Liu et al. (2022). We select pragmatic inferences written by multiple expert annotators from diverse user questions and randomly shuffle them to prevent unwanted effects emerging from example order Si et al. (2022), including exemplars from all three sources rosie, Reddit, and nq) to capture distributional differences in their pragmatic inferences. As humans naturally did, we let gpt-3.5 generate _varied_ numbers of inferences per question.

Evaluation: Can GPT-3.5 generate human-like pragmatic inferences?For each inference in inference-sample, a pair of authors annotate whether or not each human-written assumption is semantically equivalent to at least one inference generated by GPT-3.5Prompt F.1) with a Cohen's kappa of \(\kappa=0.88\). Post-adjudication, 63% of inferences were not present among model generations. When stratifying by inference type, 53% of presuppositions and 71% of implicatures were not present. This illustrates that just as they are more difficult to detect, implicatures grounded in domain knowledge are more difficult for language models to generate.

## 7 Conclusion

We show that it is possible to induce pragmatic behavior in qa systems to correct latent false assumptions in the sensitive domain of maternal and infant health. Next-generation qa systems deployed in real-world settings _must_ learn to address the pragmatics of user questions. Though we have shown the viability of _explicitly_ inducing pragmatic behavior in models in this work, directions for future work include training retrievers to inherently search for evidence to address pragmatic inferences and readers to reason on top of such evidence to tactfully and effectively challenge user beliefs.

\begin{table}
\begin{tabular}{l c|c c c c|c c} \hline \hline  & \multicolumn{2}{c|}{**No Reader**} & \multicolumn{3}{c|}{**Baseline Reader**} & \multicolumn{2}{c}{**Inference-Augmented**} \\ \hline  & Retrieve & Rerank & Flan-T5-XXL & Mistral-7B & gpt-3.5 & Mistral-7B & gpt-3.5 \\ \hline
**ROUGE-L (F1)** & \(15.6_{(0.10)}\) & \(17.5_{(10.9)}\) & \(15.89_{(13.7)}\) & \(17.4_{(5.1)}\) & \(18.7_{(6.0)}\) & \(16.2_{(4.8)}\) & \(18.6_{(5.9)}\) \\
**ROUGE-L (Recall)** & \(23.0_{(15.7)}\) & \(19.5_{(10.9)}\) & \(36.9_{(2.10)}\) & \(15.2_{(6.8)}\) & \(17.4_{(8.1)}\) & \(13.2_{(5.7)}\) & \(16.6_{(7.6)}\) \\
**BLEUT** & -\(0.72_{(0.34)}\) & -\(0.61_{(0.29)}\) & -\(0.79_{(0.45)}\) & -\(0.38_{(0.22)}\) & -\(0.37_{(0.22)}\) & -\(0.36_{(0.21)}\) & -\(0.38_{(0.22)}\) \\
**QAFactEval** & \(0.69_{(1.05)}\) & \(0.76_{(0.80)}\) & \(1.16_{(1.5)}\) & \(1.02_{(0.65)}\) & \(1.15_{(0.73)}\) & \(0.96_{(0.62)}\) & \(1.17_{(0.75)}\) \\

- & \(4.43\) &
- & \(4.45\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Mean and standard deviations of automatic (rouge, bleurt, QafActEval) and human evaluation metrics per question. We report results for the top retrieved passage and the top reranked passage, and two modes with and without access to human-written assumptions. Inference-augmented models perform competitively with baselines, indicating the promise of inducing pragmatic behavior in qa models to mitigate harm.
 



## Acknowledgments

We thank Philip Resnik, Sweta Agarwal, Sathvik Nair, and other members of the University of Maryland CLIP lab for their helpful feedback. We also thank Ximena Marin Gutierrez, Marina Yue, Michelle Jasczynski, and Amara Channell Doig for their help in the development of rosie. We are grateful for our pool of expert annotators and for users of rosie participating in our clinical trial.

The study reported in this paper was supported by research grants from the National Institute on Minority Health and Health Disparities (grant number R01MD016037) and by the National Library of Medicine (grant number R01LM012849). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the paper.

### Limitations

Data are not multilingual.Although our participants who provide questions come from diverse socioeconomic and racial backgrounds, all of the data we collect are in English. In addition, since we require participants to be located in the United States, the questions provided by participants are only reflective of the healthcare needs of English-speaking residents in the United States.

Choice of a single domain.While our approach can be generalized to any other domain, all of our data and experiments are confined to a single domain (maternal and infant health). We have not validated that our conclusions generalize beyond this particular important domain.

Pragmatics can be annotator-dependent.Finally, some degree of pragmatic inference is always dependent on the annotator, and we have not validated that this is consistent across different annotator backgrounds.

Ethical Considerationsnlp systems are never a replacement for doctors or clinical expertise, especially in high-stakes settings. This work has grown out of collaboration with public health experts to help disseminate medically accurate but _contextual_ information to new or expectant mothers with limited access to healthcare. Upon detection of false or potentially problematic assumptions, patients can then be referred to healthcare providers better able to provide information than current qa systems. All of our data was collected with irb approval in consultation with public health professionals.

## References

* D. Abusch (2002)Lexical alternatives as a source of pragmatic presuppositions. In Semantics and linguistic theory, Vol. 12, pp. 1-19. Cited by: SS5.
* N. Allott (2018)Conversational implicature. External Links: Link Cited by: SS5.
* A. Asai and E. Choi (2021)Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 1492-1504. External Links: Link Cited by: SS5.
* A. Barbaresi (2021)Traflatura: A web scraping library and command-line tool for text discovery and extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 122-131. External Links: Link Cited by: SS5.
* D. I. Beaver (1997)Presupposition. In Handbook of logic and language, H. Waller et al. (Eds.), Elsevier, New York, NY, USA, pp. 939-1008. External Links: ISBN 978-1-4503-3213-X, Document Cited by: SS5.
* G. Becker and E. Newsom (2003)Socioeconomic status and dissatisfaction with health care among chronically ill african americans. American journal of public health93 (5), pp. 742-748. External Links: ISSN 0002-9002, Link, Document Cited by: SS5.
* H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. (2022)Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. Cited by: SS5.
* A. Fabbri, C. Wu, W. Liu, and C. Xiong (2022)QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 2587-2601. External Links: Link Cited by: SS5.
* H. P. Grice (1975)Logic and conversation. In Speech acts, pp. 41-58. External Links: Link Cited by: SS5.



Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESSive? Learning IMPLicature and PRESupposition. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8690-8705, Online. Association for Computational Linguistics.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lilio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b.
* Kabbara and Cheung (2022) Jad Kabbara and Jackie Chi Kit Cheung. 2022. Investigating the performance of transformer-based NLI models on presuppositional inferences. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 779-785, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6769-6781, Online. Association for Computational Linguistics.
* Kim et al. (2022) Najoung Kim, Phu Mon Htut, Sam Bowman, and Jackson Owen Petty. 2022. (qa)2: Question answering with questionable assumptions. _ArXiv_, abs/2212.10003.
* Kim et al. (2021) Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguust invented the lightbulp? presupposition verification for question-answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3932-3945, Online. Association for Computational Linguistics.
* Koev (2018) Todor Koev. 2018. Notions of at-issueness. _Language and Linguistics Compass_, 12(12):e12306.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Levinson et al. (1983) Stephen C Levinson, Stephen C Levinson, and S Levinson. 1983. _Pragmatics_. Cambridge university press.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DecaLO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114, Dublin, Ireland and Online. Association for Computational Linguistics.
* Mane et al. (2023) Herran Y Mane, Amara Channell Doig, Francia Ximena Marin Gutierrez, Michelle Jasczynski, Xiaobe Yue, Neha Pundlik Srikanth, Sourbah Mane, Abby Sun, Rachel Ann Moats, Prague Patel, et al. 2023. Practical guidance for the development of rosie, a health education question-and-answer chatbot for new mothers. _Journal of Public Health Management and Practice_, 29(5):663-670.
* Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In _ACL_.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
* Palancia et al. (2019) Adam Palancia, Peter Flaschner, Anirudh Thommandram, Michael Li, and Yan Fossat. 2019. Physicians' perceptions of chatbots in health care: cross-sectional web-based survey. _Journal of medical Internet research_, 21(4):e12887.
* Parrish et al. (2021) Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, and Tal Linzen. 2021. NOPE: A corpus of naturally-occurring presuppositions in English. In _Proceedings of the 25th Conference on Computational Natural Language Learning_, pages 349-366, Online. Association for Computational Linguistics.
* Peters (2016) Stanley Peters. 2016. Speaker commitments: presupposition. In _Semantics and Linguistic Theory_, pages 1083-1098.
* Potts (2004) Christopher Potts. 2004. _The logic of conventional implicatures_, volume 7. OUP Oxford.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Schuster et al. (2020) Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5387-5403, Online. Association for Computational Linguistics 

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sharma et al. (2023) Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. _arXiv preprint arXiv:2310.13548_.
* Shneiderman (1992) Ben Shneiderman. 1992. Tree visualization with treemaps: 2-d space-filling approach. _ACM Transactions on graphics (TOG)_, 11(1):92-99.
* Si et al. (2022) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2022. Prompting gpt-3 to be reliable.
* Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. _CoRR_, abs/2004.09297.
* Stalnaker et al. (1977) Robert Stalnaker, Milton K Munitz, and Peter Unger. 1977. Pragmatic presuppositions. In _Proceedings of the Texas conference on per\({}^{*}\)formatives, presuppositions, and implicatures. Arlington, VA: Center for Applied Linguistics_, pages 135-148. ERIC.
* Taylor (1962) Robert S Taylor. 1962. The process of asking questions. _American documentation_, 13(4):391-396.
* Tzlaccard et al. (2022) Gautier Tlzaccard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. In _Transactions on Machine Learning Research_.
* Xu et al. (2023) Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3225-3245, Toronto, Canada. Association for Computational Linguistics.
* Yu et al. (2022) Xinyan Velocity Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. Crepe: Open-domain question answering with false presuppositions. _ArXiv_, abs/2211.17257.
* Zheng et al. (2021) Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. 2021. GRICE: A grammar-based dataset for recovering implicative and conversational rEasoning. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2074-2085, Online. Association for Computational Linguistics.

 



## Appendix A Reddit Question Filtering

### Discourse Markers

**Assumption Correcting Markers: "however," "actually."," "as a matter of fact"," "in fact", "not true", "despite what you", "on the contrary", "common misconception", "not exactly", "just to clarify", "you're confusing", "correct me if i?m wrong", "correct me if i am wrong", "you're wrong", "we have to remember that", "while that's true", "could be dangerous", "might not be the best thing" Expertise Invoking Markers: "as a doctor", "as a medical professional", "i'm a doctor", "being a doctor", "as a nurse", "i'm a nurse"

```
Prompt: You will be shown questions about maternal and infant health who specializes in finding out whether a question posed by a new or expecting mother is seeking opinion/community participation, or whether it is a medical question. Given a question, you must answer whether it is question seeking medical advice or if it is seeking personal anecdotes and sharing experience. If it'seeking medical advice,answer with "medical". Otherwise, answer "non-medical". If a question is under-specified,answer with "non-medical".
```

Frompt: You will be shown questions about maternal and infant health asked by users. Each question contains a TITLE and DESCRIPT that elaborates on it, containing details that are both relevant and irrelevant to answering the question. Given a question TITLE and a DESCRIPTION, your task is to incorporate only the relevant details from the DESCRIPTION and rewrite the TITLE into a REWRITE. If there are no relevant details, return the TITLE. As a general rule, keep the rewrite as similar to the original question as possible. The rewrite should be a question in a single sentence. Title: How to Stop Co-Sleeping Description: ... Rewrite: How to wean my 11-month-old out of Co-Sleeping?

## Appendix B Consolidating Subquestions and Assumptions and Implications into Pragmatic Inferences

```
Prompt: Given questions asked by new or expecting mothers, your task is to identify the assumptions in them. For this task, you will be given a QUESTION asked by a new or expecting mother, some ASSUMPTIONS (as a list of beliefs or assumptions) in those questions identified by health experts, and some possible SUBQUESTIONS (as a list) that public health experts have identified to have the same information goals as the original question. Given all three of these, your task is to consolidate the SUBQUESTIONS and ASSUMPTIONS into a single, exhaustive list, called INFERENCES. Turning a SUBQUESTION into an inference may involve just turning it into a declarative sentence, or identifying the assumptions made in the SUBQUESTION. Finally, add the INFERENCS to the list of ASSUMPTIONS and remove any duplicates.
```

Listing 3: Ratings of expert validators of the plausibility of inferences written by health experts in our dataset. The majority of inferences are plausible.

## Appendix C Further Details on Inference Validation

Plausibility scores are the outcome of a three-stage process: (1) a pregnant or new mother holding a belief that is latent while asking a question, (2) a maternal health expert reasoning about these latent beliefs of the mother from the question text, and finally (3) a different expert estimating the likelihood of the beliefs extracted in Step 2 of this process.

The plausibility distribution in Figure 4 represents the results of Step 3. It is important to note that the humans involved in each step of the process are completely disjoint, and have little to no information about each other. In 90% of inferences, the validators agree that the extracted inferences in Step 2 are indeed plausible. The 10% of inferences

Figure 4: Ratings of expert validators of the plausibility of inferences written by health experts in our dataset. The majority of inferences are plausible.



that validators found less plausible is more a reflection of the subjectivity of pragmatic inference in QA in general (see lines 686-690) than our usage of gpt-3.5 a textual transformation tool.

In fact, experts themselves can disagree about answers to questions or background inferences (a finding echoed in Xu et al. (2023)), so we include the plausibility study to reassure audiences that experts do find the inferences in our dataset plausible.

## Appendix D Human Evaluation Details

Human Evaluation Annotation InstructionsSee Figure 5 for annotator instructions.

Annotator preferences on questions with a highly plausible, false assumption.Human annotators prefer inference-augmented answers for questions with a highly plausible, false assumption. See Table 4.

## Appendix E QA Model Prompts

### Reranker Instructions

[title=Example]

```
Proamp BLR/RUSHResultingInstruction Retrieveapassagefrommedicalarticlesonthewebthatanswerthesfollowingquestion.
```

[title=Example]
1:From QLR/RUSHPrementsInterfaceInstuation IwanttocheckifthefollowingassumptionisttrueorfalseRetrieveanevidencepassageformefrommedicalarticlesontheweb. ```
### Reader Prompts

[title=Example]
``` Proamp BLR/RUSHReactiveGroup ```

[title=Example]

YouareanexpertinaternalandinfanthealthYouaregivenafewpassesbelow:Context(context) Usinginformationonlyfromthepassagesabove,respondtothefollowingwithahelpfulandcompleteanswer.Useinformationfrommultiplepassagesifnecessarytensowerthquestion.Again,youareallowedtonlyuseinformationfromthepassagesabove.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.Question.QuestionQuestion.Question.Question.Question.Question.QuestionQuestion.Question.Question.Question.QuestionQuestion.Question.Question.QuestionQuestion.Question.Question.QuestionQuestion.Question.QuestionQuestion.Question.QuestionQuestion.Question.QuestionQuestion.Question.QuestionQuestion.Question.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestion.QuestionQuestionQuestion.QuestionQuestionQuestion.Question 



## Appendix F Inference Generation Prompt

```
//Input B- Inference Generation Prompt when humans ask questions, they often have certain assumptions or implications that are embedded in the questions. These assumptions and implications may be true or false, and they may or may not be present in the surface form of the question. Given a question asked by a new or expecting mother, your task is to identify all relevant assumptions and implications in these questions and write them in a list titled INFERENCES. Each inference under INFERENCES should be an independent and declarative assertion that represents an assumption or an implication that the speaker makes while asking the question.
```

**Algorithm 1** Inference Generation Propt for a bag of words with a given word. The size of the word is determined from the context of the vocabulary. We use the same notation as the one used in our analysis. For example, if we assume that the word has been removed from the vocabulary, then it is recommended to fill the domain with water that feels warm, but not hot, to the inside of your wrist or below. The water temperature should be comfortable for your baby and should not pose a risk of burns or seals. It is important to check the water temperature before placing your baby in the bath to prevent any potential harm. This is to prevent the baby from getting burned or scaled by hot water. **It is important to note that **not as adaptable as adults to temperature change, and they can lose heat rapidly. Therefore, it is crucial to maintain a safe and comfortable water temperature for the baby during bath time.**

According to the information provided by pregnancy kit manufactors, most at-home manufacturers, most at-home pregnancy tests are 98% to 99% accurate when used exactly as instructed. Positive generally be trusted. However, it is important to note that false negative results can occur if the test is taken too soon. Therefore, it is recommended to wait until a sufficient amount of time has passed since the possible conception before taking a pregnancy test to ensure more accurate results.

\begin{table}
\begin{tabular}{l p{256.1pt}} \hline  & Defeatful Answer (AID) \\ \hline When should I give my baby fever reducing mesh after shots? & You should give your baby fever reducing medication after shots if they are experiencing discomfort or extreme discomfort due to the fewer. However, it is important to note that in most cases, a child does not need to be mediated for a fever unless they are extremely uncomfortable. The most important thing to do what a child has a fever is to make sure they stay hydrated by drinking plenty of fluids. If you are unsure whether your child should receive medication to address the fever or other symptoms, it is recommended to consult with your child’s healthcare provider. \\ How warm can my bath be white staying safe for my 4-month-old baby? & To ensure the safety of your 4-month-old baby during bath time, it is recommended to fill the basin with water that feels warm, but not hot, to the inside of your wrist or below. The water temperature should be comfortable for your baby and should not pose a risk of burns or seals. It is important to check the water temperature before placing your baby in the bath to prevent any potential harm. \\ How warm can my bath be white staying safe for my 4-month-old baby? & To ensure the safety of your 4-month-old baby during bath time, it is recommended to fill the basin with water that feels warm, but not hot, to the inside of your wrist or below. The water temperature should be comfortable for your baby and should not pose a risk of burns or seals. It is important to check the water temperature before placing your baby in the bath to prevent any potential harm. \\ Can pregnancy tests be trusted? & According to pregnancy kit manufacturers, most at-home pregnancy tests are 98% to acty as instructed. This means that positive results can generally be trusted. However, it is important to note that false negative results can occur if the test is taken too soon. Therefore, it is recommended to wait until a sufficient amount of time has passed since the possible conception before taking a pregnancy test to ensure more accurate results. \\ Can pregnancy tests be trusted? & According to the information provided by pregnancy kit manufacturers, most at-home manufacturers, most at-home pregnancy tests are 98% to 99% accurate when used exactly as instructed. Positive results from three tests can be trusted. However, it is important to note that false negative results can occur if the test is taken too soon. The test is taken too soon. **Blood tests, on the other hand, can detect pregnancy sooner and are the only tests that can show specific amounts of HCG in the body,..** \\ \end{tabular}
\end{table}
Table 5: Outputs generated by a standard qa system and a qa systems informed by pragmatic inferences. Answers from the pragmatic system better explicate their underlying inferences.



Consider the question asked by an expectant mother to their doctor: _"What kind of music should I play my body in the womb?"_ This question implicitly makes some assumptions that may or may not be true:

Figure 6: Instructions given to each annotator for each phase of annotation. First, we show questions to annotators and ask them to write sub questions and the assumptions present (top panel). Then, after passing these outputs to a prompt-based model to extract consolidated inferences, we ask a different set of annotators to _verify_ the veracity of the inferences along with supporting evidence (middle panel). Simultaneously, we ask a third set of annotators to write answers to questions without any inference supervision (bottom panel).