Do Clinicians Know How to Prompt? The Need for Automatic Prompt Optimization Help in Clinical Note Generation (2024)

Abstract: This study examines the effect of prompt engineering on the performance of Large Language Models (LLMs) in clinical note generation. We introduce an Automatic Prompt Optimization (APO) framework to refine initial prompts and compare the outputs of medical experts, non-medical experts, and APO-enhanced GPT3.5 and GPT4. Results highlight GPT4APO’s superior performance in standardizing prompt quality across clinical note sections. A human-in-the-loop approach shows that experts maintain content quality post-APO, with a preference for their own modifications, suggesting the value of expert customization. We recommend a two-phase optimization process, leveraging APO-GPT4 for consistency and expert input for personalization 1.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 182–201 August 16, 2024. ©2024 Association for Computational Linguistics
182

heading: 1 Introduction
text: Large Language Models (LLMs), including iterations of the Generative Pre-trained Transformer (GPT) series, have dramatically expanded the scope of natural language processing (NLP). Their applications now range from simple Q&A to the intricate demands of clinical documentation, necessitating the craft of prompt engineering (Brown et al., 2020; Sanh et al., 2021; Chowdhery et al., 2022; Longpre et al., 2023; OpenAI, 2023; Wang et al., 2023a; Yang et al., 2023b). The quality of a prompt is paramount, as it is typically created by a human mentor to guide an LLM mentee to generate the document. Yet, this prompt creation process is encumbered by the complexities of human expression—rich in subtleties and cultural nuance—that often surpass the computational confines of LLMs, resulting in a cognitive gap (Zamfirescu-Pereira et al., 2023). Variances in prompt quality lead to differences in prompt efficacy, which can fluctuate considerably (1) when switching between LLM
∗* Indicates equal contribution 1https://github.com/seasonyao/Automatic_
Prompt_Optimization_Physician_Prompting
mentees (As shown in Figure 1, ‘mentor’ modifies the prompt to allow ‘mentee’ to perform the targeted task better) and (2) across various sections of the documentation or (3) among different human mentors, as illustrated in Figure 1. This inherent variability underscores the need for a consistent tool that standardizes prompt quality to achieve reliable uniformity in LLM performance.
In the clinical domain, where the stakes are particularly high, optimizing prompt engineering is critical to help busy clinicians most efficiently use LLMs for clinical practice. Our study adopts Automatic Prompt Optimization (APO) (Prasad et al., 2022) as a novel solution to address these challenges. APO refines the initial prompts provided by clinicians, adapting them to the nuanced requirements of different clinical note sections for AI-assisted clinical documentation. Thus, the resulting clinical notes are significantly enhanced in
quality and efficiency. Through a comprehensive comparative analysis,
our research elucidates how APO, when used with human experts, substantially elevates the refinement process of prompts. Our first experimental set pits generic prompts, modified by medical experts, non-medical experts, and APO-enhanced GPT3.5 and GPT4, against each other. The results highlight APO-GPT4’s remarkable ability to elevate content generation, revealing an inherent capacity for selfimprovement that aligns with recent academic discourse. Our second experimental set delves into the potential of human-in-the-loop systems. Here, we further refine APO-generated prompts with human experts. Contrary to non-expert interventions, which often detracted from the quality of the content, expert modifications maintained the high standards set by APO. Moreover, our human preference feedback suggests that, while experts may not significantly alter the content quality, they prefer the results of their own modifications, pointing to a personalized touch without sacrificing the quality of the content.
In light of our findings, we advocate a twopronged approach to prompt optimization: initially employing APO-GPT4 to standardize prompt quality, followed by expert-led customization based on preference. This strategy offers a pragmatic balance, effectively harnessing the power of AI while respecting the nuances of human expertise.

heading: 2 Related Work
text: Soft prompts and parameter adjustments offer promising results for open-source LLMs (Li and Liang, 2021; Lester et al., 2021; Hu et al., 2021), while discrete prompt searches (Shin et al., 2020; Wen et al., 2023) and reinforcement learning (Deng et al., 2022; Zhang et al., 2022) push the boundaries further. Closed-source LLMs, conversely, necessitate gradient-free optimization, relying on iterative prompt refinement and natural language feedback for efficacy (Prasad et al., 2022; Xu et al., 2022; Guo et al., 2023; Fernando et al., 2023; Zhou et al., 2022; Xu et al., 2023; Pryzant et al., 2023; Yang et al., 2023a; Wang et al., 2023d; Dong et al., 2023; Li et al., 2023; Sun et al., 2023).
In the clinical context, synthesizing such optimization techniques has been pivotal. Foundational work in automated note generation (Krishna et al., 2020; Song et al., 2020; Yim and Yetisgen-Yildiz, 2021; Su et al., 2022; Giorgi et al., 2023; Wang
et al., 2023b,c; Yao et al., 2023) informs our approach, integrating APO to streamline medical documentation. This research leverages both iterative enhancement and expert feedback, embodying the iterative, gradient-free optimization approach to improve the precision of clinical LLM applications.

heading: 3 Method
text: We are given a dataset D of n i.i.d training clinical data, comprised of f features (D ∈ Rn×f ) including the doctor-patient dialogue, the name of a SOAP (Podder et al., 2021, 2023) note section 2, the ground truth section clinical note summary, the model-generated section clinical note summary, etc. Our method broadly consists of a “forward pass” (3.1) and a “backward pass” (3.2). First, an LLM generates summaries for a batch h from a section s ∈ S using a generic prompt p0 provided by the user. An LLM is then asked via a fixed prompt p∇ to provide suggestions to make p0 more suitable for s given the ground truth and generated summaries, producing an answer g. Afterward, another fixed prompt, pδ, is used to command the LLM to use g to fix p0, outputting a new prompt p′. p′ should now be slightly more tailored to generate better summaries for s, closer to the theoretical optimal prompt p∗. This is executed for all S utilizing a random sample of data h (batch) from each section, where h ⊆ n. This process is illustrated in Figure 2 and detailed in Algorithm 1 3.

heading: 3.1 Forward Pass
text: The forward pass utilizes an LLM to generate summaries (ŷ) for h from section s by passing in a generic user-provided prompt (p0), doctorpatient dialogue (x), and s. We use black box LLMs via API, denoted as LLMp(i) 4. This API yields a probable text continuation, symbolized as ŷ, given a prompt. This prompt is a fusion of p and i. Mathematically, LLMp(i) is approximated by argmaxŷ∈LPLLM(ŷ|p, i), where it selects the most likely continuation ŷ from the set of natural language tokens L. The ones used for our method are OpenAI’s GPT3.5 and GPT4 5.
2SOAP structure details can be found in the Appendix A.1. 3Algorithm 1 is simplified to use one data point’s dialogue (x). In reality, a batch (h) of data is used. Note that iterations for batch h involve a single type but not multiple types of sections.
4i is defined as all the inputs to the prompt (dialogue, section, etc.).
5We use OpenAI’s gpt-3.5-turbo-0613 and gpt-4-0613 in our experiments.
p0 is a generic prompt such as the one shown in Figure 2 or Appendix A.4 that, in our use case, would be provided by a medical professional such as a clinician. It is a prompt that only instructs the model, in this step LLM a. p0 and x are passed into a to output a generated summary ŷ. This first ŷ is likely to be very suboptimal for s.
Algorithm 1 SOAP Note Prompt Optimization 1: p0 = “Generate a SOAP summary.” 2: p∇ = “What’s wrong with p0?” 3: pδ = “Use g to fix p0.” 4: procedure FORWARD(s, x) 5: p0 = p0 + s+ x 6: return a(p0) ▷ LLM a 7: end procedure 8: procedure BACKWARD(s, x, y, ŷ) 9: p∇ = p∇ + p0 + s+ x+ y + ŷ
10: g = b(p∇) ▷ LLM b 11: pδ = pδ + p0 + g 12: return b(pδ) ▷ LLM b 13: end procedure 14: procedure MAIN 15: for i = 1 to k do 16: for c = 1 to j do 17: ŷ = FORWARD(x, s) 18: p′ = BACKWARD(s, x, y, ŷ) 19: p0 = p ′ 20: end for 21: end for 22: end procedure

heading: 3.2 Backward Pass
text: This segment of the algorithm represents the key transformational stage. The backward pass consists of (1) utilizing the same or a different LLM as before to provide suggestions on what is wrong with ŷ, (2) utilizing the LLM in step 1 to fix p0 using the suggestions provided in step 1. Step 1 generates “gradients” and step 2 performs “backpropagation”.
The backward pass starts by passing in a fixed prompt (p∇), p0, x, s, the ground truth summaries (y), and ŷ into an LLM b to generate suggestions (g) on how to fix p0 to make it more suitable for generating summaries for s. An example is shown in Appendix A.4. These suggestions are named “gradients”, the reason p is labeled with ∇. Note that a ?= b, i.e. a may or may not be equal to b.
Next, a fixed prompt (pδ), like the one shown in Appendix A.4, commands b to use g to fix p0. g, p0, and pδ are passed into b. “gradient descent” happens here. pδ resembles differentiation in tradi-
tional neural network training by using g (the “gradient”) to guide the model toward a lower “loss”. Hence the p is labeled with δ. A new prompt p′
is outputted by b, which should be closer to the optimal prompt p∗. p∗ = argmaxp∈L{m(p, T )}, where m(·) represents a metric function and T is all the training data for s. p′ should be an edited version of p0 that is in the opposite semantic direction.

heading: 3.3 Iterations & Validation
text: At this point in the algorithm, the same h is summarized again using a, but this time with p′. The new summaries are evaluated against y. p′ is set to p0 and the “iteration” restarts, repeating j times. After j iterations, the “epoch” is finished, and the final prompt, p′final, is used to generate summaries for a validation dataset E. These summaries are evaluated against y to check the performance of p′final. The epochs are repeated k times.

heading: 3.4 Human-in-the-Loop Prompt Refinement
text: Enhancing the APO framework, we incorporate a human-in-the-loop component for prompt refinement. Post-APO, medical experts and laypersons review and adjust p′final for each s, adding clinical acumen to the AI’s output. These revised prompts, p′final−human, are then evaluated by generating new summaries and scoring them against ground truths. The goal is to determine if there is a potential for human-AI collaboration on this task, and whether it should be with experts or not.

heading: 4 Experiments

heading: 4.1 Dataset
text: With 1.7k total doctor-patient dialogues and summaries, MTS-Dialog supports advances in automatic clinical note generation (Abacha et al., 2023b,a). For our initial exploration of which GPT variants are the best across most sections (more details in Section 4.4), we use the original evaluation split of 100 data points. For APO, since the evaluation split is small, we merge the training and evaluation data into a single pool. The data is comprised of 20 SOAP sections. We discard sections with less than 10 data points, resulting in 14 sections that meet the criteria for further experimentation. Then, we randomly sample 5 data points from each section as training data. Detailed
data distribution for these sections is outlined in the Appendix Table 3.

heading: 4.2 Metrics
text: Models are evaluated with full-length F1-scores of ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005). We use QuickUMLS6 to extract medical concepts from both model-generated and ground truth summaries and then calculate F1scores for these two lists of concepts, which is named UMLS-F1 (Adams et al., 2023). We also add human preferences in Experiment Set-2.

heading: 4.3 Experimental Setup
text: We put the details of our dataset in Appendix 4.1. First, we designed the experiment to use the generic prompt, outlined in Appendix A.4, on six different GPT models 7. This objective was to evaluate which variants are the best across most sections, thereby guiding our selection for use in APO. We then divided our experiments into two sets 8:
Set-1: Comparative Analysis of APO and Human Contributions in Clinical Note Generation. This experiment aims to assess how APO, compared with humans, can assist in improving content generation for different sections of clinical notes. Specifically, we introduce a generic prompt and training data for distinct sections. The goal is
6https://github.com/Georgetown-IR-Lab/QuickUMLS 7text-ada-001, text-babbage-001, text-curie-001, text-davinci-003, gpt-3.5-turbo-0613, and gpt-4-0613
8After we got the different sets’ prompts, we then ran gpt-3.5-turbo-0613 or gpt-4-0613 API with selfconsistency and zero-shot settings (Wang et al., 2022), where temperature=0.3, run numbers=5. We used the default numbers for all other parameters in OpenAI API.
to aid AI systems, such as GPT3.5 and GPT4, in identifying suitable section prompts that enhance content generation in each section. Our experiment involves four groups of prompters: medical experts 9, non-medical experts 10, GPT3.5 (with APO), and GPT4 (with APO). Each group modifies the generic prompt based on the training data for each section. We then compare the effectiveness of these modified prompts in assisting AI to generate summaries for different sections, using the results of the generic prompt as a baseline.
Set-2: Enhancing AI-Generated Clinical Content through Humans Prompt Modification PostAPO. In this set of experiments, we take the results of GPT3.5 (with APO) and GPT4 (with APO) as new baselines and invite medical experts and non-medical experts to further modify the prompts based on their knowledge and preferences. This approach examines how human intervention, postAPO implementation, affects the quality of AIgenerated content in various clinical note sections. We analyze the effectiveness of these modifications by comparing them against the baseline established by APO-modified prompts, focusing on the nuances introduced by the domain-specific knowledge and preferences of the two human groups.

heading: 4.4 Results
text: For our initial experiment, the findings indicate that GPT-4 and GPT3.5 emerged as the most effective variants, in descending order of performance, as detailed in Appendix A.5. As a result, they were used for our proposed algorithm.
Set-1: Comparative Analysis of APO and Human Contributions in Clinical Note Generation. Upon examining the ‘X guides GPT3.5’ results from Table 1 11, we observed that expert
9One licensed physician 10One has a master’s degree, and one has a bachelor’s de-
gree. They do not have any medical background. 11The details can be found in Appendix Table 5
and non-expert modifications resulted in slight improvements compared to the generic (baseline) results. However, according to the ROUGE and METEOR scores, ‘expert guides GPT3.5’ did not yield better outcomes than ‘non-expert guides GPT3.5’; non-experts led regarding factuality (UMLS-f1) scores. The performance of APO-GPT3.5 did not significantly differ from the baseline, whereas APO-GPT4 markedly surpassed all other methods. Compared to human modifications, APOGPT4 enhanced summary quality, a feat APOGPT3.5 did not achieve. For the same Table 1 ‘X guides GPT3.5’ experiment, the results indicated that prompts modified by experts, non-experts, and APO-GPT3.5 all fell short of the generic prompt across various sections, with expert modifications slightly outperforming non-experts, and both human groups surpassing APO-GPT3.5, especially in terms of factuality score. Consistent with the ‘X guides GPT3.5’ findings, APO-GPT4 again significantly elevated the scores across the board. Finally, the results in the Appendix Table 5 show the helpful effect of APO-GPT4 on problem (2) and (3) in Figure 1. These results further demonstrate GPT4’s emergent abilities in self-critique (Madaan et al., 2023), self-feedback (Huang et al., 2022), and selfexplanation (Zhao et al., 2023).
Set-2: Enhancing AI-Generated Clinical Content through Humans Prompt Modification PostAPO. In this experiment, we continued to explore the outcomes of the human-in-the-loop paradigm on top of APO. From the previous experiments in Table 1, it was evident that APO-GPT4 significantly boosted the summary quality, raising the lower bound of AI performance on this task and providing a new baseline for users to engage in further prompt engineering. We refer to the process of experts post-editing APO-refined APO-GPT4 prompts as ‘Exp-APO’ and the analogous postediting by non-experts as ‘NoExp-APO’. We compared Exp-APO and NoExp-APO modifications, with the term ‘APO’ now exclusively referring to the results achieved by APO-GPT4. In Table 2, we found that for both ‘X guides GPT3.5’ and ‘X guides GPT4’, Exp-APO modifications did not significantly differ from APO-GPT4 in terms of ROUGE, METEOR, and UMLS-f1 scores, whereas NoExp-APO modifications notably degraded summary quality, particularly factuality scores, suggesting a loss of key information or the introduction of hallucinations.
In a detailed comparison between Exp-APO
and APO-GPT4, we curated a human evaluation dataset from 100 randomly selected instances within the evaluation set. This allowed experts who contributed to Exp-APO to assess and provide feedback on their preference for summaries generated from their revised prompts compared to those produced by the original APO-GPT4 prompts. The outcome showed a preference distribution where 75% favored Exp-APO, 3% indicated no preference, and 22% preferred APO-GPT4. These results show that while factuality scores remained closely comparable, there was a slight decrease in ROUGE scores for Exp-APO, yet the expert preference was markedly in favor of Exp-APO. This can be attributed to how APO tends to enforce certain structural elements within prompts, such as explicitly stating ‘None’ in the absence of information. Experts tended to remove such repetitive formulations, which, although potentially reducing the strict adherence to format and the ROUGE score, did not impact the factuality score. Moreover, experts’ preferences are less influenced by rigid formatting and more by their own knowledge and experience. These expert insights, incorporated through the human-in-the-loop approach, may have introduced a degree of personalization to the prompts, aligning the AI-generated content more closely with human evaluative criteria and contributing to the overall preference for Exp-APO. This suggests that while expert post-editing prompts may not markedly enhance the quality of APO-GPT4 summaries, they align more closely with user preferences, offering a more personalized result without sacrificing summary quality.

heading: 5 Conclusion
text: Our investigation has demonstrated the profound impact of prompt engineering on the effectiveness of LLMs, specifically in clinical note generation. Implementing our APO framework has notably advanced the standardization of prompt quality, particularly with GPT4, which has shown superior performance in generating clinical notes. Incorporating a human-in-the-loop approach further validated the importance of expert involvement, indicating a clear preference for expert-modified prompts, suggesting that personalized tweaks to APO-generated prompts yield user-preferred outcomes without compromising the content’s integrity.

heading: 6 Limitations
text: Our research, while insightful, acknowledges several limitations. The task-specific nature of our findings implies that even if prompts perform well within our dataset, this does not guarantee similar success in real-world, complex scenarios. The MTS-Dialog dataset’s limitations also pose challenges; many sections had insufficient data, leading to exclusion and a lack of comprehensive coverage. Even after preprocessing and filtering, data imbalance remains a concern. Moreover, our evaluation metrics—ROUGE, METEOR, and UMLS-f1—may not fully encapsulate the qualitative subtleties of clinical note generation, potentially overlooking nuances apparent to human experts. The number of human mentors involved was constrained by time and financial resources, possibly introducing bias into the results.
Recent advancements in APO have seen the development of more sophisticated algorithms aimed at enhancing efficacy and stability (Fernando et al., 2023; Wang et al., 2023d; Dong et al., 2023; Li et al., 2023; Sun et al., 2023; Opsahl-Ong et al., 2024); however, these were not compared in our study. Additionally, our approach to prompting with APO and human experts primarily focused on general quality without targeting specific aspects such as hallucination (Huang et al., 2023). Tailoring the APO algorithm to improve particular model performances (e.g., factuality) could yield more targeted enhancements. The integration of external resources, like databases, information retrieval systems, or writing assistant tools, could also provide additional information to aid AI in making more accurate suggestions during the forward pass and refinements during the backward pass, overcoming some of the AI’s knowledge limitations (Petroni et al., 2019; Sung et al., 2021; Yao et al., 2022a,b; Singhal et al., 2022).
Moving forward, we plan to delve deeper into the nuances of prompt engineering, exploring the boundaries of personalization and the potential for even more sophisticated AI-human collaboration models. We aim to expand the diversity of expert input and examine the impact of such variations on the overall system performance. Furthermore, future work will also investigate the scalability of our approach to other domains within NLP, testing the generalizability and robustness of the APO framework. In addition, we are also interested in the emergent ability of GPT4 that can perform APO for
other AI and itself well, and we plan to distill this ability into trainable LLMs, such as the LLaMA family (Touvron et al., 2023a,b), by creating a batch of synthetic instruction learning data (Wang et al., 2022; Tran et al., 2023).

heading: 7 Ethics Statement
text: In conducting this research, we have adhered to ethical guidelines, ensuring that all patient data used in the dataset was anonymized and used strictly for research purposes. We have also considered the potential implications of our work on clinical practice, emphasizing the enhancement of AI tools as assistive rather than replacement technologies to support medical professionals. As we progress, we remain committed to upholding these ethical standards and continuously assessing the societal impacts of our research.

heading: A Appendix
text: A.1 SOAP Structure The SOAP (Subjective, Objective, Assessment, and Plan) structure is commonly used by providers (Podder et al., 2021).
∗ The Chief Complaint section is a brief description of a patient’s conditions and the reasons for the visit.
∗ The Subjective section is a detailed report of the patient’s current conditions, such as source, onset, and duration of symptoms, mainly based on the patient’s self-report. This section usually includes a history of present illness and symptoms, current medications, and allergies.
∗ The Objective section documents the results of physical exam findings, laboratory data, vital signs, and descriptions of imaging results.
∗ The Assessment section typically contains medical diagnoses and reasons that lead to medical diagnoses. The assessment is typically based on the content of the chief complaint and the subjective and objective sections.
∗ The Plan section addresses treatment plans based on the assessment.
A.2 Human Annotation Guideline
199
A.4 APO Iterations Examples Scores Suggestions & Prompt Initial: summary_rouge1 0.1041 summary_rouge2 0.0085 summary_rougeL 0.1041 summary_meteor 0.0926
In this task, we ask for your expertise in writing SOAP notes from the doctor-patient conversation. Mainly we provide the target section in the SOAP note and the conversation snippet. We need you to generate a summary for the respective snippet
First epoch (good direction): summary_rouge1 0.1674 summary_rouge2 0.0462 summary_rougeL 0.151 summary_meteor 0.2417
final suggestion in this step: The instruction should ask the model to generate a concise summary of the patient’s chief ↪→ complaint based on the doctor-patient dialogue. The summary should focus on the main reason for the visit, ↪→ excluding unnecessary details unless they are directly relevant to the chief complaint. The use of medical ↪→ terminology should be appropriate and not overused, with the focus being on clear and accurate communication.
after curr epoch, the target_trainable_instruction is: Based on the doctor-patient dialogue, please generate a concise summary of the patient’s chief complaint, which is the
↪→ primary reason for their visit to the doctor. The summary should be straightforward and focus on the main issue, ↪→ excluding any unnecessary details unless they are directly relevant to the chief complaint. Use medical ↪→ terminology appropriately, but ensure the summary remains clear and accurate. Remember, you are a smart doctor ↪→ and a professional medical scribe, your goal is to accurately reflect the conversation.
Second epoch (good direction): summary_rouge1 0.2079 summary_rouge2 0.0704 summary_rougeL 0.1897 summary_meteor 0.2376
final suggestion in this step: The summary should be a succinct reflection of the patient’s chief complaint based on the ↪→ doctor-patient dialogue. It should focus solely on the main issue, excluding any details that do not directly ↪→ contribute to understanding the chief complaint. The summary should be as brief as possible without losing the ↪→ essence of the complaint. Use standard medical terminology appropriately, but ensure the summary remains clear ↪→ and accurate. The goal is to distill the conversation into the most brief, yet comprehensive summary possible, ↪→ using appropriate medical language.
after curr epoch, the target_trainable_instruction is: Please generate a succinct summary of the patient’s chief complaint, which is the primary reason for their visit, based on
↪→ the doctor-patient dialogue. The summary should focus solely on the main issue, excluding any details that do ↪→ not directly contribute to understanding the chief complaint. While using medical terminology appropriately, ↪→ ensure the summary is as brief as possible, without losing the essence of the complaint. The goal is to distill ↪→ the conversation into the most brief, yet comprehensive summary possible, using appropriate medical language. ↪→ Remember, you are a smart doctor and a professional medical scribe, your task is to accurately and succinctly ↪→ reflect the conversation.
Third epoch (good direction): summary_rouge1 0.2224 summary_rouge2 0.0927 summary_rougeL 0.2051 summary_meteor 0.255
final suggestion in this step: The summary should be a concise reflection of the patient’s chief complaint, capturing the ↪→ essence of the doctor’s inquiries and the patient’s responses. It should focus on the main issue, excluding any ↪→ extraneous details that do not directly contribute to understanding the chief complaint. The use of standard ↪→ medical terminology is crucial, but the summary should remain brief and comprehensive. The goal is to distill ↪→ the conversation into a succinct summary that accurately reflects the doctor’s understanding of the chief ↪→ complaint.
after curr epoch, the target_trainable_instruction is: Please generate a concise summary of the patient’s chief complaint, which is the primary reason for their visit, based on
↪→ the doctor-patient dialogue. The summary should capture the essence of the doctor’s inquiries and the patient’s ↪→ responses, focusing solely on the main issue and excluding any details that do not directly contribute to ↪→ understanding the chief complaint. Use standard medical terminology to describe the complaint, ensuring the ↪→ summary is as brief as possible, without losing the essence of the complaint. The goal is to distill the ↪→ conversation into the most brief, yet comprehensive summary possible, using appropriate medical language. ↪→ Remember, you are a smart doctor and a professional medical scribe, your task is to accurately and succinctly ↪→ reflect the conversation using standard medical terms.
Third epoch (bad direction): summary_rouge1 0.1851 summary_rouge2 0.0595 summary_rougeL 0.1715 summary_meteor 0.2537
final suggestion in this step: The instruction should guide the model to generate a concise summary of the patient’s chief ↪→ complaint based on the doctor-patient dialogue, focusing on the main issue and excluding irrelevant details. The ↪→ summary should reflect the level of certainty or uncertainty expressed in the conversation, and accurately ↪→ represent any symptoms or conditions the patient denies experiencing. The use of standard medical terminology is ↪→ important, but it should not lead to verbosity. The summary should be written from the doctor’s perspective, ↪→ reflecting the doctor’s role in the patient’s care.
after curr epoch, the target_trainable_instruction is: Please generate a concise summary of the patient’s chief complaint, which is the primary reason for their visit, based on
↪→ the doctor-patient dialogue. The summary should capture the essence of the doctor’s inquiries and the patient’s ↪→ responses, focusing solely on the main issue. Exclude any details that do not directly contribute to ↪→ understanding the chief complaint. Reflect the level of certainty or uncertainty expressed in the conversation. ↪→ If the patient denies experiencing certain symptoms or conditions, ensure to reflect this accurately in the ↪→ summary. Use standard medical terminology to describe the complaint, ensuring the summary is as brief as ↪→ possible, without losing the essence of the complaint. Avoid verbosity in the use of medical terminology. The ↪→ summary should be written from the doctor’s perspective, reflecting the doctor’s role in the patient’s care. The ↪→ goal is to distill the conversation into the most brief, yet comprehensive summary possible, using appropriate ↪→ medical language. Remember, you are a smart doctor and a professional medical scribe, your task is to accurately ↪→ and succinctly reflect the conversation using standard medical terms.
Table 11: APO iterations of good and bad examples from the ‘CC’ section.
A.5 GPT Variants Per Section Section Variant Average Best Variant MEDICATIONS text-ada-001 0.02255639098 text-davinci-003 MEDICATIONS text-babbage-001 0.1096938776 text-davinci-003 MEDICATIONS text-curie-001 0.09467405383 text-davinci-003 MEDICATIONS text-davinci-003 0.2071920384 text-davinci-003 MEDICATIONS gpt-3.5-turbo-0613 0.2035366419 text-davinci-003 MEDICATIONS gpt-4 0.1999162675 text-davinci-003 PASTSURGICAL text-ada-001 0.03455261137 gpt-3.5-turbo-0613 PASTSURGICAL text-babbage-001 0.02777777778 gpt-3.5-turbo-0613 PASTSURGICAL text-curie-001 0.08775603992 gpt-3.5-turbo-0613 PASTSURGICAL text-davinci-003 0.1024338849 gpt-3.5-turbo-0613 PASTSURGICAL gpt-3.5-turbo-0613 0.1309354758 gpt-3.5-turbo-0613 PASTSURGICAL gpt-4 0.1283720208 gpt-3.5-turbo-0613 ALLERGY text-ada-001 0.04682662539 gpt-4 ALLERGY text-babbage-001 0 gpt-4 ALLERGY text-curie-001 0.1891025641 gpt-4 ALLERGY text-davinci-003 0.1002458291 gpt-4 ALLERGY gpt-3.5-turbo-0613 0.2307379782 gpt-4 ALLERGY gpt-4 0.2795421063 gpt-4 FAM/SOCHX text-ada-001 0.02921216026 gpt-4 FAM/SOCHX text-babbage-001 0.03212721942 gpt-4 FAM/SOCHX text-curie-001 0.1216424461 gpt-4 FAM/SOCHX text-davinci-003 0.1441214133 gpt-4 FAM/SOCHX gpt-3.5-turbo-0613 0.2415016373 gpt-4 FAM/SOCHX gpt-4 0.26145789 gpt-4 ASSESSMENT text-ada-001 0.0388869863 text-curie-001 ASSESSMENT text-babbage-001 0.005281690141 text-curie-001 ASSESSMENT text-curie-001 0.1543199765 text-curie-001 ASSESSMENT text-davinci-003 0.1242746478 text-curie-001 ASSESSMENT gpt-3.5-turbo-0613 0.106788819 text-curie-001 ASSESSMENT gpt-4 0.1281340914 text-curie-001 CC text-ada-001 0.03660714286 gpt-4 CC text-babbage-001 0 gpt-4 CC text-curie-001 0.1886569845 gpt-4 CC text-davinci-003 0.2283677945 gpt-4 CC gpt-3.5-turbo-0613 0.2139382547 gpt-4 CC gpt-4 0.2475876016 gpt-4 EXAM text-ada-001 0.08333333333 text-curie-001 EXAM text-babbage-001 0 text-curie-001 EXAM text-curie-001 0.2142857143 text-curie-001 EXAM text-davinci-003 0.08333333333 text-curie-001 EXAM gpt-3.5-turbo-0613 0.15 text-curie-001 EXAM gpt-4 0.18 text-curie-001 EDCOURSE text-ada-001 0.1304407442 text-davinci-003 EDCOURSE text-babbage-001 0.02094356261 text-davinci-003 EDCOURSE text-curie-001 0.1772495791 text-davinci-003 EDCOURSE text-davinci-003 0.2750014022 text-davinci-003 EDCOURSE gpt-3.5-turbo-0613 0.2590712521 text-davinci-003 EDCOURSE gpt-4 0.2440284049 text-davinci-003 ROS text-ada-001 0.03748626835 gpt-4 ROS text-babbage-001 0.0340848458 gpt-4 ROS text-curie-001 0.08547537401 gpt-4 ROS text-davinci-003 0.0952141002 gpt-4 ROS gpt-3.5-turbo-0613 0.1714490651 gpt-4 ROS gpt-4 0.1762812153 gpt-4 DISPOSITION text-ada-001 0 gpt-3.5-turbo-0613/gpt-4 DISPOSITION text-babbage-001 0.1584821429 gpt-3.5-turbo-0613/gpt-4 DISPOSITION text-curie-001 0.2519607843 gpt-3.5-turbo-0613/gpt-4 DISPOSITION text-davinci-003 0.2091346154 gpt-3.5-turbo-0613/gpt-4 DISPOSITION gpt-3.5-turbo-0613 0.2608359133 gpt-3.5-turbo-0613/gpt-4 DISPOSITION gpt-4 0.2608359133 gpt-3.5-turbo-0613/gpt-4 DIAGNOSIS text-ada-001 0.05555555556 gpt-3.5-turbo-0613 DIAGNOSIS text-babbage-001 0 gpt-3.5-turbo-0613 DIAGNOSIS text-curie-001 0.05555555556 gpt-3.5-turbo-0613 DIAGNOSIS text-davinci-003 0.2532051282 gpt-3.5-turbo-0613 DIAGNOSIS gpt-3.5-turbo-0613 0.3211143695 gpt-3.5-turbo-0613 DIAGNOSIS gpt-4 0.245994832 gpt-3.5-turbo-0613 PASTMEDICALHX text-ada-001 0 gpt-3.5-turbo-0613 PASTMEDICALHX text-babbage-001 0 gpt-3.5-turbo-0613 PASTMEDICALHX text-curie-001 0.07830882353 gpt-3.5-turbo-0613 PASTMEDICALHX text-davinci-003 0.14375 gpt-3.5-turbo-0613 PASTMEDICALHX gpt-3.5-turbo-0613 0.2317706867 gpt-3.5-turbo-0613 PASTMEDICALHX gpt-4 0.2045185666 gpt-3.5-turbo-0613 PLAN text-ada-001 0.05696640316 gpt-4 PLAN text-babbage-001 0 gpt-4 PLAN text-curie-001 0.07544836116 gpt-4 PLAN text-davinci-003 0.1067404817 gpt-4 PLAN gpt-3.5-turbo-0613 0.2096407229 gpt-4 PLAN gpt-4 0.2272458144 gpt-4 GENHX text-ada-001 0.05855827354 gpt-4 GENHX text-babbage-001 0.0200537811 gpt-4 GENHX text-curie-001 0.09488431364 gpt-4 GENHX text-davinci-003 0.1421504194 gpt-4 GENHX gpt-3.5-turbo-0613 0.3101982791 gpt-4 GENHX gpt-4 0.3141274328 gpt-4
Table 11a: The best GPT variant for each section when using the generic prompt. Note: The Average column is the mean of the Rouge1, Rouge2, RougeL, and RougeLsum scores.
Variant Count text-curie-001 2 text-davinci-003 2 gpt-3.5-turbo-0613 3 gpt-4 6 gpt-3.5-turbo-0613/gpt-4 1
Table11b: The number of sections where
each variant is the best. Note: The last row
is where two variants are tied for the
“Disposition” section.

heading: Future Work
text: Furthermore, future work will also investigate the scalability of our approach to other domains within NLP, testing the generalizability and robustness of the APO framework. In addition, we are also interested in the emergent ability of GPT4 that can perform APO for
other AI and itself well, and we plan to distill this ability into trainable LLMs, such as the LLaMA family (Touvron et al. , 2023a,b), by creating a batch of synthetic instruction learning data (Wang et al. , 2022; Tran et al. , 2023)