MedNgage: A Dataset for Understanding Engagement in Patient-Nurse Conversations
Year: 2023

Abstract:
Patients who effectively manage their symptoms often demonstrate higher levels of engagement in conversations and interventions with healthcare practitioners. This engagement is multifaceted, encompassing cognitive and socio-affective dimensions. Consequently, it is crucial for AI systems to understand the engagement in natural conversations between patients and practitioners to better contribute toward patient care. In this paper, we present a novel dataset (MedNgage), which consists of patient-nurse conversations about cancer symptom management. We manually annotate the dataset with a novel framework of categories of patient engagement from two different angles, namely: i) socio-affective engagement (3.1K spans), and ii) cognitive engagement (1.8K spans). Through statistical analysis of the data that is annotated using our framework, we show a positive correlation between patient symptom management outcomes and their engagement in conversations. Additionally, we demonstrate that pre-trained transformer models fine-tuned on our dataset can reliably predict engagement categories in patient-nurse conversations. Lastly, we use LIME (Ribeiro et al., 2016) to analyze the underlying challenges of the tasks that state-of-the-art transformer models encounter. The de-identified data is available for research purposes upon request 1.

heading: 1 Introduction
text: Due to the ease of use and efficiency of digital health interventions (DHIs) (Greaves et al., 2018), we are witnessing a surge in online conversations between patients and healthcare providers. Literature suggests that actively engaged patients are more likely to obtain the full benefits of an intervention and exhibit better outcomes (Yardley et al., 2016). Therefore, it is critical to understand patients’ engagement in online conversations and
1https://github.com/YanRayray/MedNgage
extract insights to aid healthcare professionals in providing in-time support to the patients.
While previous research has made progress in modeling engagement in human-human and human-agent conversations (Reddy et al., 2021; Sano et al., 2016; Xu et al., 2020), these works do not translate well to modeling engagement in patient-provider conversations. To bridge this gap, we introduce a novel resource called MedNgage, which consists of patient-private asynchronous message boards from an online intervention led by study nurses for symptom management. The dataset encompasses 2.1K turns between 68 patients and the nurses. We hypothesize that patient socio-affective engagement (e.g., sharing cancerrelated experiences) and cognitive engagement (e.g., information-seeking question) can predict their symptom management outcomes. It is important to note that datasets containing patientprovider conversations are rarely found in the ex-
4613
isting literature. To effectively model engagement in our proposed dataset, we create a novel framework called Socio-Affective Cognitive Engagement (SACe), drawing inspiration from linguistic theories of discourse (Asher et al., 2003), cognitive science of grounding in communication (Clark and Brennan, 1991), and the model of social presence (Swan et al., 2009). With SACe, we manually annotate the dataset, classifying patient engagement into eight categories of socio-affective engagement (e.g., personal information (Swan et al., 2009)) across 3.1K spans and seven categories of cognitive engagement (e.g., information-seeking question, clarification question (Asher et al., 2003)) across 1.8K spans within conversations. Additionally, we investigate the extent to which this framework can predict patient symptom management outcomes. Figure 1 showcases excerpts from our dataset, which have been annotated using the SACe framework.
We conduct Spearman’s rank correlation and Kruskal-Wallis tests to analyze the relationship between engagement categories and changes in patients’ perceived symptom control. The tests provide empirical evidence that patients who have a higher level of engagement in certain categories gain more control over their symptoms. Thus, automated prediction of engagement from conversations can help practitioners to detect low patient engagement levels and identify effective intervention strategies so that they could tailor content for improved outcomes. To facilitate this, we conduct a range of experiments evaluating the efficacy of both traditional machine learning (e.g., SVMs) and state-of-the-art transformer models in predicting engagement, followed by an analysis using LIME (Ribeiro et al., 2016) to identify the challenges of the tasks. Thus, our contributions to this paper are:
• We present a novel framework for measuring engagement in patient-nurse conversations, as well as a novel dataset annotated with (i) socioaffective and (ii) cognitive engagement. The framework can be generalized to interactions in other healthcare scenarios that involve adopting healthy behaviors and improving clients’ mental and physical well-being.
• With our statistical tests, we show a positive correlation between patient engagement level in conversations and perceived symptom control.
• We show transformer models can reliably predict
engagement (F1 score > 78) when trained on our data to help practitioners identify patients with low engagement and intervene accordingly. We analyze classifier errors using LIME to provide insights into the challenges of the tasks.

heading: 2 Framework
text: Inspired by theories of discourse coherence, we explore linguistic and conceptual models of engagement across various disciplines. Through iterative collaboration with expert nurses and linguists during the initial content analysis phase, we have generated a set of categories that effectively capture the nuances of our data. Therefore, our categorization of engagement combines existing literature with data-driven adaptation to match the distinctive features of our dataset, MedNgage. We will discuss the theoretical underpinnings behind our proposed SACe framework and the categories of engagement for annotating our dataset.
Socio-Affective Engagement Interventions are social: in our online nurse-led symptom management intervention, the nurses worked to create a relationship with the patient based on trust, respect, and closeness and encouraged patients to talk about their feelings and concerns (Phillips, 2016). Thus, it is important for patients to feel affectively connected to the nurses and intervention. Adapted from social presence in the Community of Inquiry (CoI) framework (Swan et al., 2009), for socio-affective engagement, we look for positive evidence of developing an affective connection (a trusting, close, and respectful therapeutic relationship) with the nurses. Appreciation, emotional expression, self-disclosure, greetings and salutations, and vocatives are examples of CoI categories that signal patient engagement in conversations. The mapping of CoI framework (Swan et al., 2009) to categories of socio-affective engagement is in Appendix A. However, CoI focused on online learning in higher education. Engagement indicators such as patients sharing their cancer experiences and interest in further communication are not included in this theory. Moreover, we excluded classes such as "humor" that do not appear commonly in our context. The eight final socio-affective categories are listed in Table 1.
Cognitive Engagement The primary goal of conversations between a patient and nurse in our dataset is to complete the intervention following
the intervention protocols with some degree of individualization. To do so, it takes the nurse and the patient together to coordinate the content and plan of action on the message boards. They do it by building common ground (mutual understanding, assumption, and knowledge) and updating their common ground moment by moment (Clark and Brennan, 1991). Therefore, for cognitive engagement, we look for the positive evidence of patients coordinating the content and process of the intervention with the nurse within collaborative goals—acknowledgment and initiation of the relevant next turn (e.g., answering protocolized questions, clarification questions), based on Clark and Brennan (1991). The mapping of grounding in communication framework (Clark and Brennan, 1991) to categories of cognitive engagement is in Appendix A. Moreover, we found taking initiative to be an important patient engagement marker in our corpus that is not defined by Clark and Brennan (1991). This category demonstrates patients taking the initiative on managing their symptoms without being prompted by the nurses.
We have also studied discourse frameworks such as Segmented Discourse Representation Theory (SDRT) (Asher et al., 2003). Discourse relation classes from SDRT such as information-seeking
question, and clarification question overlap with (Clark and Brennan, 1991) and belong to our framework. The background class is closely related to Cancer-related experience in our data. Although there is an overlap between frameworks of engagement and discourse, engagement is a broader construct that captures a speaker’s involvement in a conversation. While the discourse frameworks focus on logical and semantic relationships at Elementary Discourse Unit (EDU) level, engagement is observed across multiple EDUs in a conversation. As such, the remaining categories of SDRT and other discourse frameworks are not applicable directly. The final seven categories of cognitive engagement are listed in Table 1.

heading: 3 Dataset
text: In this section, we describe our data source, followed by our annotation protocol and a brief discussion about the patient characteristics in our dataset.

heading: 3.1 Data Source
text: The data source for our work is asynchronous message boards between patients2 with recurrent ovarian cancer and study nurses. All interactions be-
2Patient informed consent and approval of our institutional review board were obtained.
tween the nurses and patients are captured verbatim. Patients interacted 1:1 with a nurse to go through the theory-guided intervention elements to develop individualized Symptom Care Plans for 3 target symptoms that they hoped to gain better control over (e.g., fatigue, pain, and nausea). A description of the intervention process is listed in Appendix B.
One of the primary outcomes of the intervention is patient self-reported symptom controllability (i.e., an individual’s confidence in one’s ability to control symptoms with medications and behaviors), which was assessed by a validated and reliable measure —Symptom Representation Questionnaire (Donovan et al., 2008). First, the patients completed a 28-item symptom inventory (e.g., pain, fatigue, depression, nausea) and reported symptom severity at its worst in the past week from 0 (did not experience the symptom) to 10 (as bad as I can imagine). The patient then identified three target symptoms they would like to control better. The Symptom Controllability Scale (e.g., “I can do a lot to control this symptom”; five items) was used for each targeted symptom on a 0 (strongly disagree) to 4 (strongly agree) scale at the beginning and the end of the intervention.

heading: 3.2 Manual Annotation
text: Based on the theoretical framework discussed earlier, two trained raters (one graduate and one undergraduate nursing student) independently begin by abstracting the sentences in the patients’ posts that reflect cognitive and socio-affective engagement as meaning units. To determine the minimal meaningful units and their respective categories, the raters initiate the analysis by examining the first word of a patient’s post. We then gradually expand the analysis, observing for points where the category of engagement changed. When a category shift is identified, the section analyzed up to that point is marked as a "span." If the current span does not fall under either the cognitive or socioaffective engagement category, it is skipped, and the analysis continues until the next category shift is detected. This iterative process is repeated until
the entire post is classified into smaller sections or spans, each corresponding to a distinct engagement category. Examples from our annotation are listed in Table 1. An overview of the annotated dataset is provided in Table 2.
Inter-rater Agreement Code differences between the two raters were discussed and decided by the principal investigator of the intervention. A coding scheme with examples (Appendix B) is developed in an iterative manner to ensure interrater reliability. Inter-rater reliability is evaluated by Cohen κ statistic across 131 turns by two annotators. Cohen κ for cognitive and socio-affective engagement are 0.87 and 0.86, respectively.
Patient Characteristics In the dataset we annotated, the mean age of the patients was 59.7 (SD = 9.5), ranging from 24 to 83. The majority (75%) were married or living with a partner, and 51.5% had a bachelor’s degree and above. Based on the Charlson Comorbidity Index (CCI), 45.6% of the patients did not have any comorbidity, and 54.4% had at least one comorbidity.

heading: 4 Analysis
text: In this section, we first report the distribution of different engagement categories in our data. Then we illustrate the relationship between engagement and symptom controllability. The significance level for statistical analysis is p < 0.05.

heading: 4.1 Distribution of Engagement
text: The most common socio-affective engagement categories are Greetings (mean = 8.81, SD = 7.48) and Vocatives (mean = 8.6, SD = 9.11). The least common one is expressing Negative sentiment (mean = 0.35, SD = 0.99).
The most commonly used cognitive category is Intervention information (mean = 12.41, SD = 9.36). The least common ones are Disagreement (mean = 0.19, SD = 0.4), Clarification question (mean = 0.21, SD = 0.59 ), and Initiative taking (mean = 0.22, SD = 0.59). Table 3 lists the mean and median of the frequency of each engagement category.

heading: 4.2 Symptom Controllability and Engagement
text: Of 68 patients, 58 reported symptom controllability scores at baseline and at the end of the intervention (8 weeks). The average controllability score change is 0.21 (SD = 0.61), ranging from -1.85 to 1.73. Based on symptom controllability score change, we divided patients into three groups: (1) improved (n = 23) with changes greater than +0.3 SD (> 0.389), (2) stable (n = 14) with changes within +/-0.3 SD (0.029-0.389), and (3) worsened (n = 21) with changes greater than -0.3 SD (0.029).
Spearman’s rank correlation was computed to assess the correlations between controllability and engagement. Most categories’ frequencies have a significant positive relationship with changes in patient controllability scores from baseline to 8 weeks (0.2<ρ< 0.4). Table 4 shows the correlations between patient controllability scores and the frequency of each category. Kruskal-Wallis tests
(Kruskal and Wallis, 1952) were used to examine the differences in the frequency of engagement categories among patients with an improved vs. stable vs. worsened sense of control over their symptoms. When a significant difference was detected, a post-hoc Dunn’s test (Dunn, 1961) with Bonferroni adjustment (Napierala, 2014) for multiple pairwise comparisons was applied to distinguish the significant and non-significant pairs. We used ϵ2 to calculate the effect sizes for the Kruskal-Wallis tests and biserial correlation r for post-hoc Dunn’s tests. Although the sample size is smaller, we were able to detect a moderate to large effect size.
We discovered variations in the frequency of specific engagement categories between patients who experienced improved control and those with stable or worsening control. Regarding cognitive engagement, patients who reported improved perceived control over symptoms provided information toward intervention (i.e., content, process, and technical issues) more often (Mdn = 17) than those who reported worsened control (Mdn = 8), with an effect size of 0.4 (Figure 2a). Individuals who reported improved control acknowledged the nurse’s contributions more frequently (Mdn = 1) than those who had a stable sense of control (Mdn = 0), with an effect size of 0.41 (Figure 2b). Although marginally significant (P = 0.053), patients who reported improved control agreed with the nurse or agreed to do the intervention activities more frequently (Mdn = 4) than those who reported worsened control (Mdn = 1), with an effect size of 0.36.
In terms of socio-affective engagement, individuals who reported improved controllability shared cancer-related experience (e.g., cancer story, vulnerability) more often (Mdn = 4) than those who reported worsened symptom control (Mdn = 1), with an effect size of 0.38 (Figure 2c). Patients who reported an improved sense of control addressed nurses by their names more frequently (Mdn = 11) than those who reported worsened symptom control (Mdn = 3), with an effect size of 0.39 (Figure 2d). Moreover, patients who reported improved controllability appreciated and recognized the nurses’ contributions significantly more often (Mdn = 6) than those who reported worsened symptom control (Mdn = 2) (Figure 2e). Although marginally significant (P = 0.053), patients who appeared to report an improved sense of control expressed positive sentiment toward intervention more frequently (Mdn = 4) than those
who reported worsened symptom control (Mdn = 1), with an effect size of 0.37.

heading: 5 Experiments
text: In order to assess the feasibility of predicting cognitive and socio-affective engagement, we train traditional SVMs and fine-tuned pre-trained transformer models on our dataset.

heading: 5.1 Dataset for experiments
text: For our experiments, we isolate the cognitive and socio-affective spans and use 80% of the data for training, 10% for validation, and 10% for testing. All numbers are reported on the test set. Due to the relatively small number of instances, we leave out the low-frequency classes ("Disagreement" and "Clarification questions"), yielding eight socio-affective and five cognitive engagement categories.

heading: 5.2 Classification
text: SVMs As baselines, we train SVMs on word bigrams and character n-grams (2-5), weighted by term frequency-inverse document frequency (tf-idf) using scikit-learn’s3 default parameters.
3https://scikit-learn.org/stable/ modules/generated/sklearn.svm.LinearSVC
Pre-trained transformers We fine-tune four pretrained transformers: i) bert-base-cased (Devlin et al., 2019), ii) BioClinicalBERT (Lee et al., 2019), iii) RoBERTa-base (Liu et al., 2019) and iv) XLNetbase-cased (Yang et al., 2019). All the transformer
models are trained for 3 epochs under the same settings: learning rate of 8e-8, batch size of 16, and a maximum length of 200 tokens.
Experiment results The experiment results are presented in Table 5. We report macro-averaged precision, recall, and F1 scores. Each experiment is repeated three times and the mean results, along with the standard deviation are reported. Since SVMs are deterministic, the standard deviation is 0. We observe that fine-tuned transformer models can reliably predict engagement in our dataset with mean F1 scores of 78.5 and 78.8 respectively for socio-affective and cognitive categories.

heading: 5.3 Error Analysis
text: Despite the promising results achieved by transformer models, it is important to assess the limitations of these models due to the sensitivity of patient conversations. Thus, we manually annotate all instances where the best model (BioClinicalBERT for cognitive, and RoBerta for socio-affective engagement) make errors. To aid our analysis, LIME (Ribeiro et al., 2016) is used to identify the words
and phrases that support the models’ selection of a particular class. LIME determines the contribution score of specific words on the prediction of a classifier by generating variations of an input sentence by randomly removing a word and observing changes in the prediction. The "contribution score" represents how much weight the word had in the original prediction by the model.
Observing the output of LIME, we identify four main categories of errors: i) missing context, ii) inaccurate weighting of words; iii) pragmatic failures; and iv) human error in the annotation. Figure 3 shows the examples of errors and Figure 4 shows the weighted percentages of errors based on the count of categories for the two tasks.
Cognitive Errors: Context mistakes and inaccurate weighting account for around 63% of the errors in the prediction of cognitive engagement after being adjusted for the count of the cognitive engagement categories. Due to these two types of errors, the model commits a high rate of errors in the categories of Intervention information and Agreement (15% of the errors examined after the
adjustment for each type of error). Of these two categories, human mistake rates also account for 15% of the errors. Due to inaccurate weighting, the model made errors in differentiating the categories of Information-seeking question and Agreement (16% of the errors after adjustment).
Socio-affective Errors: The most frequent error types for socio-affective engagement, apart from human error, are pragmatic failures and inaccurate weighting, which account for 53% of the errors after being adjusted for the count of the socioaffective engagement categories. Due to pragmatic failure (10% of the errors evaluated after the adjustment), the model tends to make mistakes between the categories of appreciation and positive sentiment. Similarly, the human error rate is also the highest between these two categories (10% of the errors analyzed after the adjustment).
We find a large difference in pragmatic errors and context errors between cognitive and socioaffective tasks. We think this is because when predicting socio-affective engagement, the model needs to consider more pragmatic factors (e.g., situational context, the individuals’ mental states) than cognitive engagement. For example, in Figure 3b which shows LIME output, we observe that the model focuses on the word “welcome”, which is an indicator of “Appreciation”, but in practical usage, it was used as interest in communication. On the other hand, detecting cognitive engagement relies more on tone and structure, leading to context errors in Figure 3a.

heading: 6 Implications on DHIs
text: By examining the relationship between patients’ engagement and their symptom controllability, our study provides empirical evidence of the potential pathways of patient learning and behavior change
to gain better control over symptoms through meaningful engagement. The classification models presented in this paper can be used to track patient engagement based on patient-provider interactions on asynchronous message boards. The models could be used as complementary tools to augment clinicians’ capabilities to identify patients with a low engagement level so that they can focus their energy and time (i.e., tailoring content and communication accordingly) to enhance engagement and help those who struggle the most to obtain the maximum benefits of a given DHI. Based on predicted engagement markers, or lack thereof, the provider can tailor the intervention content (e.g., suggest more strategies based on professional experience, segment a series of questions into a few posts, provide timely emotional support) to achieve common ground and cultivate an affective connection with the patient. For example, nurses can encourage the patients to i) provide relevant information to complete intervention activities and tasks (e.g., describing their symptom beliefs), and ii) share their feelings, cancer treatment, and symptom experiences. For patients participating in DHIs to achieve desired outcomes, it is also important to feel close to the provider, for example, addressing the provider by name, acknowledging the provider’s message, and recognizing their contributions to the care and support.
Our dataset, MedNgage contains rich narratives of various symptom experiences and management processes from women who were fighting advanced cancer in an online cognitive behavior intervention. The Representational Approach (Donovan et al., 2007) underlying the intervention is diseaseagnostic and designed to help clients understand how their disease representations relate to behaviors. Therefore, the annotation system and model developed in this study can be applied to various healthcare scenarios involving the promotion of healthy behaviors and the enhancement of clients’ mental and physical well-being. The corpus includes comprehensive representations of patient symptoms, concerns, and the obstacles they faced while seeking the best symptom management. This presents an excellent opportunity for various clinical NLP tasks, such as developing a dialog system, optimizing patient self-report/expression review, generating text summaries of patient conversations, and customizing automated relevant responses to improve patient engagement, as suggested in recent
works on summarization (Xu et al., 2020) and style transfer (Atwell et al., 2022). For example, the system can use patient-specific language or generate text that encourages patients to ask questions or express their concerns.

heading: 7 Related Work
text: Engagement in human-human conversation has been studied from both socio-affective and cognitive aspects, resulting in numerous frameworks and theories for modeling engagement. One of the inspirations behind our work, the Community of Inquiry framework (Swan et al., 2009), which builds on the educational philosophy and practice of Dewey, models conversations in the online learning environment in terms of social presence, cognitive presence, and teaching presence. Rourke et al. (1999), proposes a community of inquiry framework that synthesizes pedagogical principles with the dynamics of computer conferencing, focusing on social presence. Clark and Brennan (1991), the inspiration behind the cognitive aspect of engagement in our work, studies how grounding in conversation is shaped by purpose and medium of the conversation. The authors highlight different grounding references that we adapt to our work.
Previous research has investigated engagement in medical interactions, including conversations. Studies have specifically examined the cognitive engagement of individuals with schizophrenia in conversations, with a focus on medication adherence (McCabe et al., 2013; Howes et al., 2012). In contrast, our study employs various methods to identify and predict patient engagement in a complex intervention aimed at modifying multiple health behaviors, including diet, physical activity, relaxation, and medication adherence. Further, we investigate the impact of engagement on patient symptom outcomes, providing practical strategies for nurses and other practitioners to effectively engage patients and deliver optimal care.
Other studies have explored engagement in different contexts such as political argument settings (Shugars and Beauchamp, 2019), conversations around terrorist attacks (Chiluwa and Odebunmi, 2016), socio-affective aspects of conversations such as emotion (Yu et al., 2004), student engagement in online discussion forums (Liu et al., 2018), cognitive engagement in MOOC forums (Wen et al., 2014), real-time engagement in reducing binge drinking through intervention text
messages (Irvine et al., 2017), user engagement in online health communities (Wang et al., 2020). However, none of these studies specifically consider the dynamics of socio-affective and cognitive engagement in online conversations between patients and healthcare providers. Our work is the first to develop a dataset with aggregated annotations and models for computationally modeling engagement in patient-provider scenarios.

heading: 8 Conclusions
text: We have developed a framework (SACe) that effectively captures patient engagement in patient-nurse conversations. Through the analysis of a unique dataset (MedNgage) consisting of online patientnurse conversations, we have identified eight categories for socio-affective engagement and seven categories for cognitive engagement. These findings provide valuable insights for behavioral scientists to understand and monitor patients’ engagement in healthcare interactions. The approach could be applied in other fields, such as online education, where engagement plays a crucial role. Our analysis confirms that higher levels of engagement, including the increased coordination and emotional connections between patients and healthcare practitioners during the intervention, result in improved symptom control. Additionally, we have demonstrated that fine-tuned transformer models can reliably predict fine-grained engagement in conversations so that practitioners can adjust their communication style and tailor strategies promptly, to promote patient engagement for improved outcomes. Our analysis of model output using LIME has shown the challenges that state-of-the-art transformer models encounter for the two tasks, which can be used to improve these models for similar tasks in the future. We expect our system could also aid subsequent text generation tasks, such as summarization of patient conversations, and tailoring automated responses in clinical settings.
Limitations
While this dataset is unique and pioneering, its size is limited, and it involves specific patients. To enhance the generalizability of the findings, a larger dataset may be required. Similarly, although our framework is innovative, we anticipate the development of more comprehensive and informative annotation protocols in the future. For instance, we observed a higher frequency of the "Intervention in-
formation" category within cognitive engagement, likely because the intervention predominantly follows a Q (nurse) & A (patient) format. We hope that the coding scheme established in this study can aid future research in refining this category with finer granularity, based on specific intervention theories.
Ethical Considerations
We used the NLM Scrubber offered by NIH to produce HIPAA-compliant deidentified health information for scientific use, including dates, and places. Two independent annotators evaluated the NLM-Scrubber on the dataset to make sure no events or other people in patients’ posts can allow patients to be traceable. The de-identified version of our data will be shared with researchers upon request who have completed an ethical review from their institution and a data request application form from us.
Since the domain of our dataset is specific, the models trained on our dataset may exhibit subtle biases on out-of-domain data. Further, pre-trained models that we use in our work have been shown to exhibit biases (Li et al., 2021). We hope future researchers could use these models with caution regarding the biases that these pre-trained models have.
The long-term goal of our work is to aid healthcare providers to quickly identify poorly engaged patients to allocate their energy and resources to provide in-time support. Models trained on our data should not be deployed in the real world without human supervision because, despite the potential of transformer models, they cannot be relied on completely in sensitive medical scenarios.

heading: Acknowledgements
text: The randomized clinical trial (WRITE Symptoms intervention) was supported by a grant from the National Institutes of Health, National Institute of Nursing Research NIH-NINR R01NR010735-NRG Oncology GOG-259 (H.S.D., PI) as well as NIH-NCI grants to NRG Oncology (U10CA180822), NRG Operations (U10CA180868), and UG1CA189867 (NCORP). This study is supported by the Eta Chapter Sigma Theta Tau International Honor Society of Nursing Research Award (Y.W., PI) and the Margaret E. Wilkes Scholarship Fund Award (Y.W., PI). We extend our gratitude to Hannah Bergquist for her
invaluable contributions to the annotation of the asynchronous message boards. Furthermore, we are immensely grateful to all the women who participated in this trial, as their involvement has been instrumental to the success of this study.

heading: A Appendix: Mapping of Engagement Categories

heading: B Appendix: Description of Intervention
text: Process and Coding Scheme
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section at the end of the paper
3 A2. Did you discuss any potential risks of your work? Section 6 and Ethical Considerations section at the end of the paper
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sections 2-5
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We are presenting a new dataset and we are adhering to the rules of our institutions ethics review board.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Researchers who wish to gain access to our dataset, will have to complete appropriate ethics review protocols
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 6
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 1
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5.2
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We do not do hyper-parameter search.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.3
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We describe key points of annotation protocol. We don’t provide full text because of space limits.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3.1
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 3.1
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 3.1
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 3.2

heading: Future Work
text: Our analysis of model output using LIME has shown the challenges that state-of-the-art transformer models encounter for the two tasks, which can be used to improve these models for similar tasks in the future. We expect our system could also aid subsequent text generation tasks, such as summarization of patient conversations, and tailoring automated responses in clinical settings. Limitations
While this dataset is unique and pioneering, its size is limited, and it involves specific patients. To enhance the generalizability of the findings, a larger dataset may be required. Similarly, although our framework is innovative, we anticipate the development of more comprehensive and informative annotation protocols in the future. For instance, we observed a higher frequency of the "Intervention in-
formation" category within cognitive engagement, likely because the intervention predominantly follows a Q (nurse) & A (patient) format. We hope that the coding scheme established in this study can aid future research in refining this category with finer granularity, based on specific intervention theories. Ethical Considerations
We used the NLM Scrubber offered by NIH to produce HIPAA-compliant deidentified health information for scientific use, including dates, and places. Two independent annotators evaluated the NLM-Scrubber on the dataset to make sure no events or other people in patients’ posts can allow patients to be traceable. The de-identified version of our data will be shared with researchers upon request who have completed an ethical review from their institution and a data request application form from us. Since the domain of our dataset is specific, the models trained on our dataset may exhibit subtle biases on out-of-domain data. Further, pre-trained models that we use in our work have been shown to exhibit biases (Li et al. , 2021). We hope future researchers could use these models with caution regarding the biases that these pre-trained models have. The long-term goal of our work is to aid healthcare providers to quickly identify poorly engaged patients to allocate their energy and resources to provide in-time support. Models trained on our data should not be deployed in the real world without human supervision because, despite the potential of transformer models, they cannot be relied on completely in sensitive medical scenarios