How to Engage Your Readers? Generating Guiding Questions to Promote Active Reading (2024)

Abstract: Using questions in written text is an effective strategy to enhance readability. However, what makes an active reading question good, what the linguistic role of these questions is, and what is their impact on human reading remains understudied. We introduce GUIDINGQ, a dataset of 10K in-text questions from textbooks and scientific articles. By analyzing the dataset, we present a comprehensive understanding of the use, distribution, and linguistic characteristics of these questions. Then, we explore various approaches to generate such questions using language models. Our results highlight the importance of capturing inter-question relationships and the challenge of question position identification in generating these questions. Finally, we conduct a human study to understand the implication of such questions on reading comprehension. We find that the generated questions are of high quality and are almost as effective as human-written questions in terms of improving readers’ memorization and comprehension. github.com/eth-lre/engage-your-readers

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11749–11765 August 11-16, 2024 ©2024 Association for Computational Linguistics
github.com/eth-lre/engage-your-readers

heading: 1 Introduction
text: Questions play an important role in reading comprehension. Through actively raising questions and seeking answers from the content during reading, readers can deeply engage with the text and achieve better comprehension (Bharuthram, 2017; Syamsiah et al., 2018). However, asking good questions is challenging and requires complex skills.
How can we facilitate readers’ active thinking and questioning during reading?♥1 An effective approach could be presenting valuable questions explicitly in the text, which is a recognized strategy to enhance readability and engage readers (Haggan, 2004). An example is shown in Figure 1. The
1Superscript symbols indicate the roles of questions in this paper, detailed in Section 3.
diverse rhetorical functions during reading to engage readers and improve comprehension.
writer first uses a title question (Q1) to arouse the interest of potential readers. Then, a group of questions (Q2-3) surfaces in the beginning to introduce the central topics to be explored. In what follows, more questions (Q4-5) are raised and elaborated with the moving of discussion, holding the reader’s attention throughout the reading. From a linguistic perspective, these questions not only build up a coherent discourse structure (Curry and Chambers, 2017), but also serve as a communicative device that constructs a virtual dialogue between the writer and potential readers, thereby making the text more engaging and interactive (Hyland, 2002).
In this paper, we refer to these in-text questions as guiding questions. Despite being widely used, there is little understanding of the effect of such questions on human reading. To fill this gap, we analyze how expert writers use guiding questions and explore how to model these questions with advanced language models. Further, we hypothesize that these author-posed questions can complement and encourage readers’ spontaneous selfquestioning, thereby fulfilling the goal of active reading. Based on these motivations, we address the following research questions:
• RQ1: What is the use, distribution, and role of guiding questions in formal writing?♣ • RQ2: How well can language models understand
11749
and generate guiding questions?♣
• RQ3: What is the effect of these questions on human reading comprehension?♣
To answer these questions, we start by curating GUIDINGQ, a dataset of 10,577 guiding questions from research articles and textbooks. Since the two source texts are written by expert writers, we assume the questions are carefully designed to enhance readability and thus ideal for our research. Through qualitative and quantitative analysis on GUIDINGQ, we summarize the question roles based on their discourse and interactional effects, and present their usage and distribution across the domains (RQ1, Sections 3 and 4). Then, we explore various approaches to model these questions from three interrelated aspects (RQ2, Section 5): identifying question positions (where to ask), predicting question focus (what to ask), and finally generating the questions (how to ask). Our results highlight the importance of inter-question relationships and the challenge of question position identification in generating guiding questions.
Finally, to validate whether and how the generated guiding questions can facilitate active reading (RQ3, Section 6), we conduct a carefully designed human study where participants read articles with or without questions, complete a post-reading summarization test, and later evaluate the questions. The results demonstrate that the generated questions are not only of high quality but can help readers produce better summaries, indicating an improved memory retention and understanding of the high-level information of the article.

heading: 2 Related Work
text: Here we focus on discussing the linguistic role of questions. Questions can be used as a tool for discourse planning. The Question Under Discussion (QUD) framework uses questions to interpret the discourse relationship between textual units within a document (Van Kuppevelt, 1995; Roberts, 2012; Benz and Jasinskaja, 2017). For example, the relationship between "Sa: A night of largely peaceful protests ended early Monday in a bloody" and "Sb: Hours earlier, Egypt’s new interim leadership had narrowed in on a compromise candidate to serve as the next prime minister." can be described by question "What happened before the clash?", where Sa, which elicits the question, is called anchor sentence and Sb is the answer sentence. Studying QUD with modern NLP techniques is a relatively new
field where most efforts focus on data construction (De Kuthy et al., 2018; Westera et al., 2020; Ko et al., 2022; Wu et al., 2023).
While QUD provides a more flexible way to represent discourse connections compared to predefined fixed relationships (e.g., elaboration, condition) than other frameworks like Rhetorical Structured Theory (RST; (Mann and Thompson, 1988)), how to use these implicit questions to assist readers is not straightforward. In this paper, we instead focus on questions that are explicitly presented in a text by the author, which we call guiding questions. We argue such questions are more powerful. Besides their discourse effect, these questions directly interact with readers. Therefore, they have the potential to influence readers’ behaviors (Curry and Chambers, 2017). For example, articles titled with questions are found to obtain more downloads (Jamali and Nikzad, 2011).
In terms of the goal, QUD strives to explore the dense space of questions to interpret exhaustive intra-document relationships, while we aim to distill sparse but crucial questions to engage readers without overwhelming them. We highlight the question position identification task as a preliminary step before generating questions. This dimension is neglected by previous QUD or generic Question Generation (QG) studies. However, when questions are used as a communicative tool to interact with readers during reading, when and where to raise them is arguably important. We hope this study sheds light on a greater understanding of the implication of questions in reading comprehension.

heading: 3 Taxonomy of Guiding Questions
text: This section describes our taxonomy of guiding questions. The taxonomy is built upon the discussion in Hyland (2002), which we adapt based on our dataset (Section 4). In particular, we classify questions into five different roles based on their different discourse or interactional effects. Their definitions and examples are provided below. • Arouse Interest.♦ The first category refers to questions that appear in titles. Since the title is generally the reader’s first encounter with a text, formulating it as an intuitive question can grab the reader’s attention.
Questions in titles: How do Philosophers arrive at truth? Is there no quantum form of Einstein Gravity? Why do house-hunting ants recruit in both directions?
Note that although the title question usually reveals the main topic of an article, the breadth of the content is not always encapsulated by it
• Frame Purpose.♣ This type of question often surfaces and clusters in the beginning section to foreground the central topics to be explored.
Several aspects of this theory need further investigation. Is it possible to achieve predictable refractive changes? Can this be achieved through an intact epithelium? ... This paper describes the use of a novel device ...
Writers pose these questions to provide an agenda for the article and then pick them up again in later sections to direct readers through the reading.
• Organize Discourse.♠ Questions can also serve as subheadings to structure the text, guiding readers by explicitly introducing shifts in information and identifying what will be discussed in the ensuing section.
What are the advances of telecommuting? The term telecommuting emerged in the 1970s to ... What are the drawbacks of telecommuting? In 2013, Yahoo’s thenCEO, Marissa Mayer, ended ... What are the ethical challenges of telecommuting? ...
Noticeably, such questions usually appear multiple times throughout an article, collectively creating a sense of progression toward a greater understanding of the topic.
• Establish Claim.♥ Another use of questions is to introduce and emphasize the writer’s arguments rather than to seek the reader’s interaction or viewpoint.
What contributes to a corporation’s positive image over the long term? Many factors contribute, including a reputation for treating customers and employees fairly and for engaging in business honestly.
A distinct feature of such questions is that the writer often provides a clear answer (i.e., the argument), usually close to the question, thereby limiting the reader’s alternative interpretations to the preferred one.
• Provoke Thought.★ Finally, there are some “genuine” questions that do not anticipate specific responses within the text. Therefore, they can facilitate the reader’s active thinking to the greatest extent.
If the technological resources of today’s governments had been available to the East Germany Stasi and the Romanian Securitate, would those repressive regimes have fallen? How much privacy and freedom should citizens sacrifice to feel safe? [END]
It’s worth noting that different roles are not mutually exclusive. For instance, an AROUSE INTEREST question may also provoke thoughts and vice versa. Nevertheless, we focus on understanding the main role of a question.

heading: 4 GUIDINGQ DATASET
text: In this section, we first discuss the choice and rationale of source texts (§ 4.1), followed by the construction pipeline (Figure 2) of the GUIDINGQ dataset (§ 4.2). Then, we present a series of distributional features of guiding questions (§ 4.3).

heading: 4.1 Source Texts
text: We select scientific articles and textbooks as the source texts to build the dataset. Our choice is based on two considerations. First, their writerreader discourses have a clear communicative intent, either peer-to-peer or teacher-to-student, which can motivate the use of questions. Second, they are formal texts written by experts, ensuring

heading: GuidingQ Textbook Research Articles
text: that questions presented in them are strategically used to enhance readability. Specifically, we use textbook chapters collected from an online free publisher OpenStax2 (Singh et al., 2023) and research papers from the arXiv and PubMed datasets (Cohan et al., 2018).

heading: 4.2 Construction Pipeline
text: We describe the main steps of collecting and annotating GUIDINGQ below. S1: Question Extraction. We start by extracting questions from source texts by detecting interrogative marks. We only keep documents with at least three questions, indicating the writer actively used questions in the writing. S2: Question Completion. Since the extracted questions are a part of the source texts, they are not always semantically complete due to omissions or unclear pronouns, e.g., What central point might constitute such a code? Therefore, we first identify and complete such questions based on the context. We do this because it is the first step to understanding the meaning of such questions. S3: Question Answering. Next, we generate the answer to each question. In particular, the answer should be detailed enough and solely based on the article. Therefore, we use the article’s words as the answer whenever possible. If a question is not discussed in the article (e.g., PROVOKE THOUGHTS question), we label it as "NO ANSWER." S4: Evidence Extraction. Given a produced answer, we automatically extract supporting sentences from the article as evidence. We do this by greedily searching a set of sentences that has the maximum Rouge score with the answer, which is the standard way to find Oracle sentences for extractive summarization systems (Nallapati et al., 2017). For questions without answers as per S3, we directly label them as "NO EVIDENCE."
S5: Question Role Identification. Finally, we identify the role of each question. We make this the last step as the information collected in previous
2openstax.org
steps (complete question, answer, evidence) can help in understanding the role of a question.
Recent studies have shown that LLMs with carefully designed prompts are comparable to or even better than human annotators (He et al., 2023; Zhang et al., 2023; Törnberg, 2023). To reduce human effort and enable data scaling in the future, we use ChatGPT to complete the annotation steps S2, S3, and S5. See prompts in Tables 8, 9, and 10.
S6: Quality Control. We adopt a Human-LLM coannotation paradigm (Li et al., 2023). Concretely, we ask the model to provide a confidence level for each of its outputs. Examples below a certain level are reviewed by human annotators, followed by an overall quality review, detailed in Appendix A.

heading: 4.3 GUIDINGQ Analysis
text: The statistics of the GUIDINGQ dataset is summarized in Table 1. In general, textbook chapters use questions slightly more frequently than research articles. This is possibly because teacher-to-student interactions are more inclined to involve questions than peer-to-peer ones.
What is the distribution of question roles?♠ In Figure 3, we compare the distributions of different questions on the two subsets. As can be seen, research articles contain more FRAME PURPOSE and ORGANIZE DISCOURSE questions, while textbooks favor PROVOKE THOUGHT questions possibly due to their educational purpose. Besides, AROUSE INTEREST (title) questions are more common in articles than textbooks, although both proportions are small. A possible reason is that research articles exist in a competitive environment where potential readers are confronted with a large number of papers, under which circumstances interrogative titles could help attract readers (Haggan, 2004; Jamali and Nikzad, 2011).
We omit AROUSE INTEREST questions as they are in titles by definition.
How diversely do human writers use guiding questions?♠ To understand this, we measure the number of question roles used in an article. The results are shown in Table 2. Since research articles contain fewer questions per article, their question roles are slightly less diverse than those in textbooks. Overall, the majority of articles (70%+) use less than three types of questions, suggesting room for improvement in human questioning strategies.
Where are guiding questions asked?♠ We divide articles into five equally sized segments and calculate the percentage of questions that appear in each part. As in Figure 4, there is a strong positional bias among different questions. FRAME PURPOSE questions mostly appear in the first and second segments. ORGANIZE DISCOURSE questions are largely located in the middle segments, while PROVOKE THOUGHT questions tend to emerge in the last segment, consistent with their expected functions. ESTABLISH CLAIM questions are skewed towards the end of the text. A possible explanation is that writers make more and more conclusive arguments with the progression of discussion.
Where are guiding questions answered?♠ We measure the distance between a question and its farthest evidence sentence (if any) in Figure 5. This can be considered the scope a question acts on the discourse of the document, i.e., from when the question is raised to when it is closed. In

heading: Max Evidence Distance
text: this sense, PROVOKE THOUGHT questions go beyond the content as they do not have specific answers. We find the two subsets show a similar distribution. FRAME PURPOSE questions serve as the outline of a document, therefore having the largest range. ORGANIZE DISCOURSE questions are usually answered within one or two paragraphs, consistent with their subheading role. In contrast, ESTABLISH CLAIM questions have prompt answers as the claims.

heading: RST Relationship Distribution
text: How are questions related to other text units?♠ As guiding questions are integral to the article, we also analyze the discourse relationships between questions and other text units3 within the same document using an RST (Mann and Thompson, 1988) parser4. In Figure 6, we compare the distribution of discourse relationships concerning questions with those of all text units. The analysis reveals that questions exhibit a higher proportion of concrete relationships, such as "elaboration" and "explanation," and a lower proportion of general relationships, such as "span" and "same-unit" compared to other units. This suggests incorporating ques-
3Following RST, we consider Elementary Discourse Unit (EDU) as the minimal unit.
4github.com/EducationalTestingService/rstfinder
tions and their answers in writing might add to the coherence of the discourse structure.

heading: 5 Guiding Question Generation
text: In this section, we first describe our methods to model guiding questions and then report experimental results.

heading: 5.1 Task Formulation
text: Given a document D = {s1, ..., sn} of n sentences, our goal is to learn a sequence of questions Q = {q1, ..., qm}, their positions in the article P = {p1, ..., pm}, and (optionally) their answer information A = {a1, ..., am}. In particular, 1 ≤ pi ≤ n is the index of the sentence after which qi should be asked. We call these sentences anchor sentences {sp1 , ..., spm}.

heading: 5.2 Data Preparation
text: To construct training examples, we remove questions from articles and reconstruct them based on the corrupted articles such that the model can learn how to use guiding questions as human writers. In concrete, given an article D, we extract its questions and locate their positions to obtain Q and P . For each question qi, its answer information is a set of keywords ai = {w1, ..., w∣ai∣} extracted from its evidence sentences obtained during the annotation (Section 4.2). We take this form to exclude redundant information in the full answer and reduce input (output) length for efficiency consideration.
Since deleting sentences would create incoherence and reduce learning P to identify where sentences are removed, we use gpt-3.5-turbo-1106 to assess the coherence of missing positions and, if necessary, eliminate the incoherence by making small edits around the missing positions. Since there could still be nuanced differences in edited positions, we perform the same “delete and smooth” operation on 1% randomly selected non-question sentences as noise, detailed in Appendix B.1.

heading: 5.3 Modeling
text: In what follows, we describe three popular QG paradigms considered for our task: Pipeline, Multitask, and Joint Generation (Ushio et al., 2023). All approaches are unified as a text generation task and use Flan-T5 (Chung et al., 2022) as the backbone model, which we finetune on our dataset.
Pipeline. We decompose the task into three subtasks to learn {P,A,Q} with independent models.
First, a Position Predictor (PP) identifies the positions of questions. A naive way is to directly generate the position indices {p1, ..., pm} conditioned on D. However, this requires learning a mapping between sentences and numerical symbols. Instead, we opt to identify the anchor sentences by training PP to copy them from D:
C̃ = argmax C PθPP(C∣D). (1) where the output C̃ = [sp1∣...∣spm] is a concatenation of anchor sentences separated by "∣". P can be obtained by relocating copied sentences in D. When there is no exact match, we use BM25 to get the most similar sentence.
Then, we highlight the target position in D by inserting a special mark [Question] after the anchor sentence and use an Answer Extractor (AP) to generate answer keywords ai.:
D(i) = [s1, ..., spi , [Question], ..., sn], (2) ãi = argmax
a PθAE(a∣D(i)), (3)
where D(i) is the document marked at position pi. Finally, a Question Generator (QG) generates questions based on predicted positions and extracted answers:
q̃i = argmax q PθQG(q∣D(i), ai). (4) Note that the PP model predicts all positions in one pass while the other two generate output one by one.
MultiTask. The multitask model still consists of the three components described above. However, instead of independently training three models, we train a unified model for all tasks in a multitask learning manner. In practice, we mix the training examples of the three tasks and distinguish them by adding different task prefixes before the inputs.
Joint Model. As observed in Figure 1, guiding questions tend to be related to each other. Therefore, we consider jointly generating all questions at the same time. To be specific, we use a template function to convert{P,A,Q} into a flattened sequence T (P,A,Q) ={t(p1, a1, q1)∣t(p2, a2, q2)...} where t(p, a, q) = "Position:sp#Answer:a#Question:q". The
model is trained to directly generate the whole target sequence T based on the article D:
T̃ = argmax T PθJT(T ∣D). (5) In doing this, each question is also conditioned on previous ones, enabling the model to learn the inter-question connections.
For the joint model, we also introduce a variant that additionally generates question roles. This is done by inserting "Role: question role" between the position and answer field of each entry. We denote this model as JointR.
Since the above methods are all formalized as a text generation task, their training objectives take a similar form:
L = − ∣Y ∣ ∑ i
logPθ(yi∣y1, ..., yi−1, X), (6) where X and Y are the input and output sequence of the corresponding task. See Appendix B.2 for training details.

heading: 5.4 Automatic Evaluation
text: The main results are presented in Table 3, where we also include zero-shot prompted GPT-4 (gpt-4-1104-preview) with the prompt in Table 13. We report the average number of generated questions # Q, Rouge (L) (Lin, 2004), Meteor (Banerjee and Lavie, 2005), and BertScore (Zhang et al., 2020). Besides, we use Dist-N (1/2) (Li et al., 2016), the percentage of distinct n-grams in generated questions, to measure the balance between diversity and relevance of guiding questions. Since there is no one-to-one map between generated and ground-truth questions, we concatenate
all the questions as a whole sequence to compute reference-based metrics.
Overall, fine-tuned models generate fewer questions than references, while zero-shot GPT-4 generates more. We found that Flan-T5 tends to output short sequences; therefore, we attribute this to their different pre-training paradigms. Jointly generating question roles can boost performance, which is expected as they are indicative of a series of distributional features. The best fine-tuned models achieve remarkable BertScores. This suggests that the generated questions successfully replicate the main information of human questions.
For the textbook dataset, the joint model generally performs the best among fine-tuned approaches. In particular, it best resembles the Dist1/2 results of human questions, while others tend to generate more independent questions (higher Dist1/2). This suggests that the success of this model is possibly because it better learns the inter-question relationships. As for the scientific set, the multitask model achieves competitive results with the joint model. We conjecture the more complex content and sparser questions increase the difficulty of learning the question relationship, which could diminish the advantage of joint generation.
Finally, we scale up the parameter size of the best-performing model JointR up to 11B. We can see that scaling the model size results in significant performance gain on the textbook set but little on the scientific set. This demonstrates the challenge of understanding scientific language with LLMs.
Question Position. Since the generated questions are different from references, naively matching their positions (e.g., recall, precision) is not a suitable way. Instead, we evaluate a question’s
position by measuring how well it fits the context using perplexity. Specifically, let p be the index of question q. We insert q in p and calculate the average word perplexity of sentences[sp−3, sp−2, sp−1, q, sp+1, sp+2, sp+3], each conditioned on its previous three sentences. The results are presented in Figure 7. Interestingly, human questions in textbooks create a local peak of PPL in the context. We explain this phenomenon from the connection between surprisal and salience: text units with higher salience are usually less expected, making them stand out from the context and attract the reader’s attention (Rácz, 2013; Zarcone et al., 2016; Blumenthal-Dramé et al., 2017). This is consistent with the interactional purpose of using questions. However, none of the models replicate this salience effect, either lowering the PPL (joint model) or increasing the PPL without a quick decline (pipeline/multitask).
The scientific articles show a different pattern where human questions are with lower PPL. We conjecture this is because scientific articles usually make sufficient discussions before proposing a question. As a consequence, their questions are highly predictable by prior sentences. Finetuned models manage to replicate the effect, possibly because research articles and their question usage are usually structured (e.g., proposing questions in the introduction) and thus easier to learn.

heading: 6 Human Study
text: Finally, we conduct a between-group human study to investigate the impact of guiding questions on reading comprehension. Participants are asked to read articles with (or without) questions, after which we gather their feedback and analyze their information retention and understanding to gain a holistic view of the effect of guiding questions.

heading: 6.1 Experiment Design
text: Procedure. The human study runs on a crowdsourcing platform Prolific and consists of 4 stages.
1. Demographic Questionnaire: We collect participants’ demographic information (Appendix C.1) and consent before the experiment. 2. Article Reading: Participants read an assigned article in 20 minutes with or without questions beside the article. Details of the reading interface are in Appendix C.2. 3. Comprephension Test: After reading, participants are asked to write a summary of at least 100 words without access to the article. 4. Evaluation: Finally, we ask participants to rate the usefulness and quality of questions (If any).
Participants. We select 45 participants with the criteria being at least C1 command of English. Each participant received 10 GBP/hour on average. We randomly assign them to one of three groups:
• Control: reading w/o questions. • Reference: reading w/ expert-written questions. • Generated: reading w/ generated questions.
Test Articles. We use the JointR model to generate questions for five textbook chapters5 selected from different domains spanning Business, Philosophy, Sociology, Political Science, and Psychology. A few small adaptations are made to make them better fit the study (e.g., length reduction). To focus on question quality, we generate the same number of questions as reference6. Each article is assigned to 3 participants to average-out the effect of individual articles.
5We only consider textbook passages because scientific articles are challenging for general readers.
6We do this by truncating over-generated questions or disallowing the <eos> token until we get enough questions.

heading: 6.2 Question Evaluation
text: Participants evaluate the intrinsic quality of each question from three aspects: relevance, position, and importance (detailed in Table 14). From a later survey, we found that participants may have different interpretations of these dimensions despite our instructions. Nevertheless, the results in the first section of Table 4 prove that the average quality of generated questions is on par with (if not better than) ground truth ones.

heading: 6.3 Effect on Reading Comprehension
text: User Perceived Usefulness. Results in the second part of Table 4 prove that the generated questions are as good as human questions in perceived usefulness. See the questionnaire in Table 14.
Improved Memorization and Comprehension. We use summary quality as a proxy to measure participants’ memorization and comprehension. The evaluation consists of three dimensions: Coherence, Consistency, and Informativeness, and is based on GPT-4, which has shown a superior correlation with human annotations on summary evaluation (Liu et al., 2023), detailed in Appendix C.3. As shown in Table 5, the reference and generated groups achieve comparable scores, and both outperform the control group by a large margin. An additional between-group summary analysis, including summary time, length, and n-gram overlap, is shown in Table 15.
An intuitive explanation for the improved quality is that users memorize question-related information and incorporate them into the summary. To verify this, we measured the entailment score between summaries and answers to guiding questions using BertScore recall (BSr):
ENTSCORE = 1∣S∣ ∑ s∈S 1∣Qs∣ ∑q∈Qs BSr(s, aq), (7)
where s is a summary, Qs is the set of guiding questions of the source article of s, and aq is the answer to q.
The results in Table 6 show that the reference and generated groups incorporated more answer information compared to the control group, indicating improved memorization. However, the difference in recall scores is not as pronounced as the difference in summary quality (Table 5). This suggests that readers do not simply compile questionanswer pairs into summaries. Therefore, we believe the improvement in summary quality also reflects a deeper understanding facilitated by the guiding questions.
Reading Time. The reading speeds of different groups are shown in Table 7. When questions are displayed next to the articles, the users spend a longer time reading. On the one hand, this is a signal of enhanced engagement. On the other hand, this potentially implies an extra cognitive load caused by guiding questions.
Participant Feedback. Finally, we gathered user feedback through a preliminary interview-based study, detailed in Appendix C.5. In summary, participants confirmed the benefits of guiding questions and discussed various aspects of these questions, highlighting both consistent preferences and nuanced complexities. We hope these insights will inspire future research in this area.

heading: 7 Conclusion
text: This paper studies the discourse and interactional role of guiding questions in textbooks and scientific articles. We explore various approaches for modeling these questions, providing insights into how to model this task and highlighting challenges to be solved. We validate our results with human studies, which demonstrate reading with guiding questions can improve the high-level memorization and understanding of human readers.
Broader Impacts
In this study, we analyzed the use of questions in academic and educational articles, demonstrating their benefits for reading comprehension. While questions can enhance engagement, they can also increase readers’ cognitive load, as evidenced by longer reading times (Table 7). Additionally, questions may introduce unintended nuances for communication, such as creating unequal social relationships (Hyland, 2002). Therefore, it is important to be aware of these mixed effects when using guiding questions in writing.

heading: Limitations
text: We summarize the limitations of this study into the following open questions.
Is our question role taxonomy generalizable to other domains?★ Our investigation of the role of guiding questions is initially focused on textbooks and scientific articles. However, different domains might use questions differently. Nevertheless, our analysis (Section 4.3) uncovers distributional features that are indicative of question functions, such as their positions and question-answer relationships. These findings offer insights that could be generalized to understand the roles of questions in broader contexts.
How to align guiding questions with individual preferences?★ Our model aims to replicate guiding questions crafted by human writers. However, these questions may not always resonate with individual readers, given their different reading goals and prior knowledge. We expect that personalized generation (Cui and Sachan, 2023), which takes into account user profiles, would yield more helpful questions.

heading: How has the role of questions evolved?★ It is
text: important to note that the use of questions could change over time. For instance, Ball (2009); Jiang and Hyland (2022) have analyzed the distribution shift of questions in titles over the past decades. In this study, we did not take the temporal dimension into account, and the conclusions are based on contemporary texts. Therefore, the findings of this paper may not remain consistent in the future.

heading: A Annotation Details
text: We use gpt-3.5-turbo-1106 with a temperature of 0.2 for data annotation. The prompts used for question completion (QC), question answering (QA), and question role identification (QRI) are listed in Tables 8, 9, and 10.
We ask the model to provide confidence levels for outputs of QA and QRI as these two tasks are arguably more challenging. 437 questions have a confidence level of 1 on both tasks. An expert anno-
[Task Description] You will be given an article and some questions. In particular, these questions are presented in the article and play different roles. Your task is to identify their role. The definition and example of each question role are described below. {Definitions and examples of question roles} Make sure you understand the above instructions clearly. Start by reading the article carefully and locating the positions of the given questions. Check each question-answer pair and write down your analysis about its role based on the above definition. Finally, output its role and provide a confidence level from 1 to 5 for your judgment, where 1 is the lowest and 5 is the highest.
[Input] Article: {article} Question 1: {question 1} Answer 1: {answer to question 1} . . .
tator (author of this work) manually validated these questions. Note that our goal is not to annotate as many examples as possible, but to check whether there are unknown question roles beyond our taxonomy and analyze common bad cases to improve prompts. We found that most cases with low confidence are because of their mixed roles. For example, when a question introduces an argument and meanwhile starts related discussions, it could be both ESTABLISH CLAIM and ORGANIZE DISCOURSE question. For such ambiguous cases, we recommend determining their main roles based on their question-answer relationships, as shown in Figure 5. For example, if the abovementioned question has an immediate answer, its main role should be ESTABLISH CLAIM . Note that our taxonomy is to provide a holistic understanding of question functions rather than a hard classification system.

heading: Task Criteria Yes (%)
text: Finally, we sample 10 research articles and 10 textbook chapters with 116 questions in total and
review the accuracy of annotation. As shown in Table 11, decent results are observed on all tasks. We plan to scale the dataset and update the annotation with more powerful LLMs available in the future.

heading: B Experiment Details

heading: B.1 Article Processing
text: To process an article into the training format, we delete questions from the article and eliminate incoherence by prompting gpt-3.5-turbo-1104 with the instruction in Table 12. To reduce the cost, we build the paragraph using the 5 sentences before and after the deleted question, which is enough to assess or restore the coherence of the local context according to our qualitative inspection. When performing this operation on random sentences, we disallow sentences around (distance<10) any already deleted or question sentences to be selected in order to avoid severe incoherence.

heading: B.2 Training setup
text: We split the dataset into 90% training and 10% test sets. Our implementations are based on the Transformers Library (Wolf et al., 2020). In concrete, for all approaches, we fine-tune the Flan-T5 for up to 10 epochs with a learning rate of 5e − 5 and batch size of 32. Following Raffel et al. (2020), we employ the AdaFactor (Shazeer and Stern, 2018) optimizer and do not use warm-up. An early stop strategy is applied when the loss on the validation set does not decrease in three continuous epochs. We use 4 Nvidia Tesla A100 cards with 40 GB GPU memory for training. One epoch takes around half an hour. At inference, we use beam search
[Task Description] Given an article, your task is to incorporate several questions into the text to enhance its readability and make it more engaging. For each question, first determine its position in the article by copying the sentence after which the question should be raised, then provide a set of keywords of the answer to the question. Finally, generate the target question.
[Input] {article}
decoding with a beam size of 4. All evaluations are conducted with the default parameters in their public implementations.
Since the articles in our dataset are relatively long, we truncate them and keep sentences whose indices fall within [max(0, p0 − 5), min(∣D∣, pm + 5)], where ∣D∣ is the number of sentences in article D, and p0, pm are the index of the first and last question respectively. In other words, we keep at least 10 sentences as the context for predicting each question. If the truncated document exceeds the model’s context length, we split it into segments of roughly the same length and run each segment separately.

heading: C Human Study Details

heading: C.1 Demographic Information of Participants
text: The average age was 29 years, with 25 female and 20 male participants. The simplified ethnicity distribution is: 23 white, 15 black, and 7 Asian. All information is on a self-identification basis.

heading: C.2 Reading Interface
text: Figure 8 shows a screenshot of the reading interface. To highlight guiding questions, we display them on the right side of the article. In order to measure paragraph-level reading time, participants need to click the "Finished" button at the end of each paragraph to reveal the next one.

heading: C.3 Summary Evaluation
text: We prompt GPT-4 (gpt4-1104-preview) with a temperature of 0.7 to evaluate the quality of collected summaries. The prompt is shown in Table C.3, where we adopt the chain-of-thought (Wei et al., 2022) and batch evaluation (Yuan et al., 2023) prompting strategy.

heading: C.4 Additional Summary Analysis
text: We provide additional analysis about collected summaries in Table 15, including summary time, length, and n-gram overlap between summaries and their sources. We can observe the reference and generated group produced longer summaries than the control group and spent longer time accordingly. The authentic group’s summaries and the generated group’s summaries are similar in quality, but the former is more concise. This can be attributed to the greater significance of reference questions, consistent with user ratings in Table 4. In addition, summaries from both the reference and specifically generated groups show a larger n-gram
overlap, providing further evidence of improved memorization.

heading: C.5 Observations from Preliminary Study
text: Before the reported human study, we conducted preliminary interviews and follow-up surveys with 15 participants, five from each group, as detailed in Sec. 6.1. The aim is to validate the experiment design and gather user preferences and feedback on presenting questions during their article reading experience. We refer to participants as group_id (R/C/G) + user_id (1-5), where R, G, and C stand for Reference, Generated, and Controlled.
Many participants expressed positive feedback on the generated questions, noting that “the questions are easy to understand” (G5). Notably, compared to the author-curated questions with relatively complex terms and sentences, our generated questions had simpler vocabulary and shorter sentences and facilitated quicker context comprehension. Some participants even found it helpful to use the questions to “remember the content of the whole article” and “better understand it again” (G1). We summarize the observations from the interview into three points.
Prompt Template
[Task Description] Your will be given three summaries written for an article titled “{title}”. Please act as an impartial judge and evaluate the quality of these summaries in terms of {metric}. Please make sure you read and understand the following instructions carefully. Please keep this document open while reviewing and refer to it as needed.
[Evaluation Criteria] {metric description}
[Evaluation Steps] Read the source article carefully and identify the main topic and key points. Read each summary carefully. Check if the summary meets the above criteria and provide an explanation for your judgment. Assign a {metric} score for each summary on a scale of 1 (lowest) to 5 (highest). Decimal scores are particularly encouraged.
[Input] Article: {article} Summary 1: {summary_1} Summary 2: {summary_2} Summary 3: {summary_3}
Consistent Question Preference. While each participant has their unique criteria for defining a good question, we have observed a consistent preference for questions that are both intriguing and informative. Among the 10 participants exposed to supported questions, whether authentic or generated, seven conveyed a preference for questions that stimulate thought and reflection. These questions may offer insights (R5) or highlight specific details (G1), but they must be inherently challenging and motivate reading (G5). Conversely, participants tend to dislike questions that are too easy or have immediate answers within the article (R4, R5, G5). This observation provides valuable insights for refining our future question generation process by allowing us to better control the level of difficulty and align with participants’ preferences.

heading: Relevance, Distractibility, and Helpfulness. In
text: the survey, we asked participants to assess the relevance, distractibility, and helpfulness of each question on a 5-point Likert scale. Half believe a question’s relevance depends on whether the following sentences answer it. Meanwhile, shorter questions tend to be less distracting, with their distractibility rating inversely proportional to perceived helpfulness. Interestingly, conflicting viewpoints arose, with some participants considering certain questions neither distracting but useless (R1) or helpful yet irrelevant (G1, G4). This nuanced understanding of human complexity calls for more research in human-AI collaboration research to find adaptive question generation solutions.
Question Position and Scenario Matters. The two observations mentioned above may vary slightly depending on the question’s position and the reading scenario. Notably, five participants experienced a “cold start” during the reading task, expressing difficulty in “getting into the context of an article at first” (R1). Therefore, their initial preference leans towards easier and more relevant questions at the article’s beginning. As they familiarize themselves with the context after reading a few paragraphs, their inclination shifts towards more intriguing and divergent questions. Furthermore, participants’ standards for a good question may fluctuate in different scenarios. Distinctions were made between first-time reading versus content reviewing (R2), learning scenarios versus examination scenarios (R4), and serious learning versus casual reading (G4). These considerations could serve as additional factors in our future iterations.

heading: Future Work
text: These considerations could serve as additional factors in our future iterations