Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue
Year: 2023

Abstract:
Factual correctness is often the limiting factor in practical applications of natural language generation in high-stakes domains such as healthcare. An essential requirement for maintaining factuality is the ability to deal with rare tokens. This paper focuses on rare tokens that appear in both the source and the reference sequences, and which, when missed during generation, decrease the factual correctness of the output text. For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are important and (b) uplift their conditional probability. We introduce the “utilization rate” that encodes knowledge and serves as a regularizer by maximizing the marginal probability of selected tokens. We present a study in a knowledgerich domain of healthcare, where we tackle the problem of generating after-visit care instructions based on patient-doctor dialogues. We verify that, in our dataset, specific medical concepts with high utilization rates are underestimated by conventionally trained sequence-tosequence models. We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency. 1

heading: 1 Introduction
text: Recent advances in language modeling (c.f. Dong et al. (2021); Erdem et al. (2022) for survey) have enabled applications across multiple domains including education (Shen et al., 2021), jurisprudence (Bell et al., 2021), e-commerce (Zhang et al., 2020; Xiao et al., 2021), and healthcare (Valmianski et al., 2021; Compton et al., 2021; Alambo et al., 2022; Krishna et al., 2020).
One of the central challenges in deploying these models in-the-wild is that rare words tend to have underestimated conditional probability during generation (Luong
∗Work done while at Curai 1Code is available at https://github.
com/curai/curai-research/tree/main/ careplan-charting.
et al., 2014; Chintagunta et al., 2021; Holtzman et al., 2020). However, in high-stakes applications, many of these rare words are semantically important and need to be preserved. For example, some symptoms, diseases, and medications can be both rare and important (Mottaghi et al., 2020) (e.g. knowing that the patient is taking warfarin is extremely important, even if the word “warfarin” occurs infrequently).
Prior approaches for handling rare word generation utilize a copy mechanism (See et al., 2017; Joshi et al., 2020; Xu et al., 2020; Choi et al., 2021). This facilitates copying from the source text using a probabilistic switch to decide if the next output token is generated or copied from the input (See et al., 2017). However, it doesn’t properly resolve the main challenge: not all rare tokens are important. Only specific rare tokens (e.g. warfarin) have a high probability of appearing in the reference sequence when found in the source sequence. In cases where the training data does not have enough structure to disambiguate which rare words are essential, the copy mechanism becomes overly extractive (Gehrmann et al., 2018; See et al., 2017).
Also relevant to this paper are previous works that integrate knowledge into language models (Duan et al., 2020; Liu et al., 2022). In entity-centric summarization, Keskar et al. (2019); Liu and Chen (2021) add key phrases to the prompt, which through the self-attention mechanism influence the output distribution. However, for prompts containing rare tokens, self-attention struggles to capture the prompt-reference dependency, and the marginal probability of rare tokens remains underestimated. Joshi et al. (2020) extends this approach by not only explicitly including the medical concepts in the input sequence, but also adding a related term to the loss function. However, they still find that for rare tokens the model underestimates the conditional probability during generation.
Finally, dictionary look-up of rare and out-ofvocabulary words has been studied in Yu et al. (2022); Ruzzetti et al. (2022). However, these papers focus on finding good representations of specific tokens. In this paper, we tackle the problem of uplifting important rare tokens even when a good representation is not available.
We base our work on the premise that specific rare tokens (e.g. warfarin) have a high probability of appearing in the reference sequence if they also appear in the
2373
source sequence. The main questions we tackle in this paper are the following: How do we know which rare tokens have a propensity to appear in both the source and the reference? How do we encode this information into the model?
We study our approach in the healthcare setting, for the concrete problem of after-visit care instruction generation from a medical dialog between patient and medical professional. We define the medical concept utilization rate and utilization-rate-aware training objective in section 2, discuss the care plan generation problem and data collection in section 3, describe the sequence-tosequence model setup in Figure 4, and report experimental results in section 5.
Our contributions are the following:
1. We are the first to explicitly focus on identifying and modeling specific rare tokens that appear in both the source and the reference. We call them “high utilization concepts.”
2. We propose a measure of “utilization rate” to identify tokens that comprise “high utilization concepts.” We use external knowledge to help with this computation as these tokens can be extremely rare.
3. We introduce a regularization term during training that leverages token utilization rate to uplift the conditional probability of important rare tokens.
4. We demonstrate the application of our approach to the concrete task of generating after-visit care instructions from medical professional-patient dialogue.
We observe performance improvement with both automatic metrics and human evaluation with medical experts.

heading: 2 Approach
text: In many sequence-to-sequence tasks, certain rare concepts have a high probability to appear in the reference sequence (y) if they also appear in the source sequence (x). We call these concepts “high utilization concepts” (c ∈ CHU) and formally define them in Equation 1. These concepts are comprised of one or more tokens c = [ν0, ν1, ...]. We hypothesize that a source of factuality errors in many sequence-to-sequence tasks is that learned model underestimate the conditional probability of high utilization concepts p̂(yi = ν, |y<i,x, ν ∈ c, c ∈ x, c ∈ CHU) < p(...), where p̂ denotes the model estimated probability and p is the true probability.
Definition 2.1 (High utilization concepts) Given a universe of concepts C, the set of high utilization concepts CHU is defined as
CHU =
{ c ∈ C : p(c ∈ y|c ∈ x) p(c ∈ y) ≫ 1 }
(1)
Equation 1 answers the question “How do we know which rare tokens have a propensity to appear in both source and target?” while at the same time it works for rare tokens.
This key insight leads us to define two goals for this work: learn to identify high utilization concepts, and build a utilization-rate-aware training objective.

heading: 2.1 Identifying high utilization concepts using externally provided knowledge
text: The major challenge in identifying high utilization concepts in real datasets is that the concepts we are interested in are present in very few examples. This means that it is hard to directly estimate p(c ∈ y|c ∈ x) and p(c ∈ y) from Equation 1 due to the high variance. In particular, a frequency-based estimate of probability has an uncertainty proportional to 1/sqrt(N) where N is the number of samples for a given concept. However, these rare concepts can still be very impactful to the overall performance of the model. This is because, for a given reference, y, it is unlikely that a particular high utilization concept will be present (∀c ∈ CHU, p(c ∈ y) ≪ 1), but it is also unlikely that no high utilization concept will be present ( ∏
c∈CHU p(c ̸∈ y) ≪ 1). This is well documented in the medical domain, where medical concepts have a very long-tailed distribution (Prabhu et al., 2019; Mottaghi et al., 2020), yet may appear in almost every relevant sequence. As an illustration, imagine a list of medication instructions. Every instruction may have a different medication so no medication token appears more than once; however, each instruction is rendered useless if it doesn’t include the relevant medication (e.g. see “Medication Plan” instructions in Figure 1).
To overcome this challenge, we propose computing what we call “utilization rate”, rϕ, which we define in Equation 2. This function relies on the concept equivalence class map ϕ : Csel → E where Csel ⊆ C and E is a set of equivalence classes. (ϕ, Csel, E) cannot be derived from the data or the model, but instead are provided from an external source of knowledge. If ϕ is an identity (id) then rid(cn) = p̂(cn ∈ y|cn ∈ x), (x,y) ∈ D.
1. Develop a method for identifying high utilization concepts, CHU for a dataset D = {(xi, yi)}Ni=1.
2. Develop a method for augmenting the training procedure of sequence-to-sequence models to correctly estimate the conditional probability of tokens forming high utilization concepts.
Definition 2.2 (Utilization rate) The utilization rate of concept cn is defined as
rϕ(cn) = ∑ c∈Csel ∑N
j=1 1[c∈xj ,c∈yj ,ϕ(c)=ϕ(cn)]∑ c∈Csel ∑N j=1 1[c∈xj ,ϕ(c)=ϕ(cn)] (2)
Here, Equation 2 tries to make the intuition from Equation 1 applicable to a real dataset. We gener-
ally cannot compute the lift because for rare words the dataset frequency derived probability estimates are poor.
Note that Equation 2 combines both externally provided knowledge (ϕ, Csel, E) and dataset derived values. This allows us to inject domain-specific information. Because concepts are mapped to equivalence classes, every concept in a particular equivalence class has the same utilization rate. If a concept cn ∈ Csel has marginal probability to appear in the reference sequence that is much lower than rϕ(cn) then it is a high utilization concept.

heading: 2.2 Utilization-rate-aware seq2seq training
text: Our analysis in section 5 (see Figure 3) shows that conventionally trained seq2seq models underestimate the utilization rate (rϕ) for many rare concepts. While we cannot optimize the utilization rate directly, we can optimize the approximate marginal probability p(ν|x) of a token ν given a source sequence x, as seen in Equation 3.
p(ν|x) = ∑
y<t
p(ν|y<t)p(y<t) ≈
≈ ∥y∥∑
t=1
p(ν|y<t)p(y<t) p(y<t) is uniform≈
≈ 1∥y∥
∥y∥∑
t=1
p(ν|y<t)
(3)
Given the source sequence x, the tokens for which we aim to optimize the marginal probability are {ν ∈ c, c ∈ x ∩ CHU}. We define the unweighted utilization loss.
Definition 2.3 (Unweighted utilization loss)
lu(x) =− 1
∥{ν ∈ c, c ∈ x ∩ CHU}∥ × (4)
× ∑
ν∈c,c∈(x∩CHU) log p(ν|x) (5)
However, not all concepts in CHU are equally likely to appear in the reference given their appearance in the source. To better reflect we also propose a weighted utilization loss where the weight for each token is determined by its utilization rate.
Definition 2.4 (Weighted utilization loss)
lw(x) = − ∑
ν∈c,c∈(x∩CHU) rϕ(c) log p(ν|x)∑ ν∈c,c∈(x∩CHU) rϕ(c)
(6)
Note that Equation 6 directly injects externally provided knowledge through its dependence on ϕ.
We use utilization loss as a regularization term and augment the objective function. We use α > 0 to balance the strength of the regularization:
l(x,y) = lnll(y) + α · lu or w(x) (7)
where lnll = − ∑|y|
t=1 log p(yt|y<t,x) and lu or w is either lu from Equation 5 or lw from Equation 6.

heading: 3 After-visit care instruction generation: task and data description
text: After-visit care instructions (care plan) are a set of actions (instructions) that a medical professional writes in the patient’s electronic health record (EHR) as a followup to the patient’s visit. A care plan often includes a list of medications with appropriate directions, further medical evaluations, or educational information for preventive care. Before writing the care plan, the medical professional discusses it with the patient, and together, they jointly agree on the next course of action. This joint decision-making implies that most of the necessary information for writing the care plan is already available in the conversation.
In Figure 1, we show two examples. In each example, we present the (a) segment of the conversational dialog corresponding to provider messages discussing the care plan with the patient and (b) corresponding care plan charted in the EHR. We can see that the instructions
FI ND
IN G
M ED
IC AT
IO N
DI SO
RD ER CL AS S CH AR AC TE R TH IN G HI ST OR Y AC TI VI TY PR OC ED UR E LA TE RA LI TY CO UR SE QU AL IF IE R DE M OG RA PH IC TO PI C M EA SU RE M EN T LA BO RA TO RY ON SE T TI M IN G SU RG ER Y
Semantic Types
0.00
0.01
0.02
0.03
0.04
0.05
M ar
gi na
l P ro
ba bi
lit y
(a) Average marginal probability of every semantic type.
FI ND
IN G
M ED
IC AT
IO N
DI SO
RD ER CL AS S CH AR AC TE R TH IN G HI ST OR Y AC TI VI TY PR OC ED UR E LA TE RA LI TY CO UR SE QU AL IF IE R DE M OG RA PH IC TO PI C M EA SU RE M EN T LA BO RA TO RY ON SE T TI M IN G SU RG ER Y
Semantic Types
0.0
0.2
0.4
0.6
0.8
1.0
Ut iliz
at io
n Ra
te
(b) Semantic type utilization rates.
Figure 2: Empirical concept marginal probabilities and utilization rates estimated from the dataset.
are written in a directive format, using action verbs and often paraphrasings of the corresponding text in the dialogue. The care plan does not always have all the medical concepts mentioned in the conversation. In the first example, “serotonin syndrome" and “Celexa" are rare, but the care plan includes only the latter. We need a model that is robust to rare medical concepts and can discern which knowledge needs to be carried forward.
We tackle the problem of taking the relevant section in the conversations corresponding to the care plan as input and automatically derive care plan instructions that the medical professionals can approve. We do not assume access to 1-1 mappings between the sentences in the conversation to the care plan instructions. However, we develop a method to derive a dataset of 1-1 mappings, albeit noisy, which we use for model training.
Dataset construction. We use a dataset with 14K medical professional-patient encounters collected on a virtual primary care platform. Each encounter has a text-based conversation between the medical professional and the patient. We applied an in-house conversation discourse parser to extract only those dialogue turns from the medical professional’s corresponding to the care plan discussion. We also have the associated care plans written from the patient’s electronic health record for that encounter. On average, each encounter has 9 dialogue turns corresponding to care plans and 4 care plan instructions.
We need a parallel corpus with pairs of dialogue turns and care plan instructions for our model. Getting manual annotations for each encounter would be expensive as it requires expert knowledge. Therefore, we automatically construct a paired dataset, albeit noisily, from the paired encounter level care plan and provider dialog turns. We get sentence-level embeddings for every sentence in each turn and instructions in the care plan and pair those with the highest cosine similarity (We provide additional details in the Supplementary Material).
At the end of this, we have 48,000 source-reference pairs, where the source is a sentence in the conversational dialog and reference is the mapped instruction. We randomly sample 3000 pairs for testing, 1000 for validation, and the remaining 44,000 pairs for training.
We use medical concepts from UMLS (Bodenreider, 2004) and in particular SNOMED-CT and RXNorm ontologies. The synonyms are pooled from all ontologies in UMLS that map to the corresponding concept in SNOMED-CT and RXNorm.
To identify the concepts, we use an in-house lookupbased concept recognizer. It uses a sliding window strategy to find maximal matches of text corresponding to medical concepts and their synonyms. It ignores stop words while doing the match. Finally, it has an agglomeration step that leverages a concept hierarchy. If we have overlapping spans corresponding to two concepts where one is a child of another (eg “lower abdominal pain” and “abdominal pain”) then only the more specific concept is extracted. If two different concepts have a span overlap and are not hierarchically related, then the concept linking is greedily selected with the concept on the left being given priority.
Identifying high utilization concepts. We limit Csel to only medical concepts and choose ϕ such that it maps them to their SNOMED CT semantic types (which informs our choice of E). In our case study this narrows down 758 unique medical concepts to their 19 semantic types. The marginal probability p(c ∈ y) for each semantic type c is shown in Figure 2a while the utilization rates are shown in Figure 2b. Comparing them we can see that utilization rates are 10-100x larger than the marginal probabilities. This suggests that all medical concepts are part of high utilization tokens set (CHU = Csel). It also means that many kinds of medical concepts that are present in the source sequence do not get generated in the output sequence, which drastically hurts medical correctness.

heading: 4 Experimental setup
text: We follow the standard practice (Ott et al., 2018) of training our sequence-to-sequence models using FairSeq framework (Ott et al., 2019). We use byte-pair encoding implemented in the fastBPE package (Sennrich et al., 2016). We use a transformer architecture for our model and train models on our data from scratch2.
Model architecture We use the transformer_iwslt_de_en architecture in FairSeq for experiments. It consists of 6 encoder and decoder layers with 4 self-attention heads followed by feed-forward transformations. Both encoder and decoder use embeddings of size 512 while the input and output embeddings are not shared. Both the encoder and decoder use learned positional embedding. We early-stop training based on the validation performance. Evaluation is done on the test set.
Training We use Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9 and β2 = 0.98. We use the inverse square root learning scheduler with 4,000 warmup steps. We use the initial learning rate of 5 × 10−4, dropout rate of 0.3 (Srivastava et al., 2014) , and weight decay with its rate set to 10−4. We use label smoothing with 0.1 of probability smoothed uniformly during training. We modify the training objective Equation 7 by adding oversmoothing loss (Kulikov et al., 2021) with a coefficient of 0.9 and unlikelihood loss (Welleck et al., 2019) with a coefficient of 0.5. All training was performed on VMs with single V100 GPUs, we estimate 200 GPU hours as the total amount required for the completion of this work.
Early stopping We use early stopping for model selection based on the value of the objective function computed on the validation set. We evaluate the model on the development set every 2K updates (∼4K tokens per update). We stop training when the objective has not improved over more than 5 consecutive validation runs. It takes approximately 75K updates to an early stop.
Decoding We use beam search implementation from FairSeq. We decode using the beam size of 5. We set the lower- and upper-bound of a generated output to be, respectively, 0 and 1.2 · ||x||+ 10. We do not use either length normalization or length penalty since we apply oversmoothing loss.
Lexically constrained decoding baseline Apart from using the unregularized version of the model as a baseline, we compare the proposed approach with the lexically constrained decoding approach (Post and Vilar, 2018). We stick to the LexicallyConstrainedBeamSearch implementation of the Dynamic Beam Allocation (DBA) algorithm that ensures the presence of provided tokens in the generated output. DBA implements an optimized
2Informally, we also tried a pre-trained BART (Lewis et al., 2019) but the results were worse.
version of the Grid Beam Search (Hokamp and Liu, 2017). DBA is training-agnostic and is used only during generation. We apply DBA for the baseline model. Given the non-uniform distribution of utilization rates, for each source we leave only medical concepts c with rid(c) > τ for some threshold τ . We report results for τ = 0.6, which we select by running an extensive grid search.

heading: 5 Results

heading: 5.1 Effect of knowledge injection during training on model’s utilization rate
text: We evaluate whether the knowledge injection through regularization (subsection 2.2) has the desired effect of improving model estimate of the utilization rate, rϕ. Because the test set is too small to effectively estimate per-concept utilization rate, we instead compute it for semantic types. In Figure 3 we use semantic relative error (Equation 8) to compare models trained with α ∈ {0, 0.25, 0.5, 0.75, 1} that either use unweighted loss lu (which uplifts all medical concepts equally, “Unweighted") or a weighted loss lw with the ϕ being identity (“Concept weighted”) or mapping concepts to semantic types (“Semantic weighted”). In addition, as a baseline we also compare an unregularized model that uses DBA for generation (“DBA”). For a detailed breakdown of relative errors for each combination see the Supplementary Material.
Definition 5.1 (Semantic relative error) Relative error for semantic type s computed from r̂ϕ estimated from model derived output sequences and rϕ estimated from reference sequences. cs is any concept for which ϕ(c) = s holds and the value of ϵs in not dependent on the choice of cs.
ϵs = ∥r̂ϕ(cs)− rϕ(cs)∥
rϕ(cs) (8)
In Figure 3a we present the relative error for different α as a function of semantic type frequency in the test set. For each point (a given semantic type and α) we take the lowest relative error among {“Unweighted”, “Concept weighted”, and “Semantic weighted”}. The highest relative errors are seen for α = 0, which corresponds to no regularization. For other values of α the difference is not statistically significant, although, for very rare semantic types, α = 0.25 appears to perform worse than models with higher regularization strength. This shows that our external knowledge informed regularization has a significant impact on a relative error, but the utilization rate estimate is not sensitive to the exact weight of the regularization term.
In Figure 3b we present relative error for different training procedures, {“Unweighted”, “Concept weighted”, and “Semantic weighted”}, as well as a baseline of “DBA.” For each point (a given semantic type and training procedure) we choose an α that gives the lowest relative error. We find that “DBA" baseline,
(a) Relative error in the utilization rate for each regularizer strength α. Note that α = 0 means there is
no regularization.
(b) Relative error in the utilization rate for each model (and best α for that model).
Figure 4: Entropy of the conditional distribution p(y|y<t,x) with respect to different α values. Filled regions denote the standard deviation across training runs according to Figure 4.
α BERTScore Concept-F1 GPT-2 Perplexity
Baseline 0.0 22.48 ±0.66 57.43±3.73 5.53±0.04 DBA - 23.59±0.28 79.83±0.43 11.96±0.05
Unweighted (ours)
0.25 25.09±0.69 58.19±2.11 5.91±0.07 0.5 25.42±0.56 58.91±6.83 5.65±0.03 0.75 26.22±0.35 60.83±5.96 6.28±0.02 1.0 26.74±0.43 61.05±7.48 6.18±0.05
Concept weighted (ours)
0.25 28.29±0.19 60.87±3.86 6.93±0.05 0.5 28.19±0.20 60.36±2.03 8.49±0.05 0.75 28.08±0.15 64.09±1.85 7.95±0.080 1.0 27.82±0.25 63.05±2.49 9.37±0.10
Semantic weighted (ours) 0.25 28.97±0.56 69.10±2.12 7.01±0.29 0.5 30.54±0.78 74.98±3.91 6.84±0.03 0.75 31.48±0.86 75.77±3.30 6.96±0.11 1.0 30.59±0.63 75.02±2.18 6.94±0.12
Table 1: Automated metrics scores for different model setups. We report average score and standard deviation over five random seeds. We highlight in bold the best average and all scores having overlapped standard deviation intervals with the best score.
which is a constrained generation procedure applied to an unregularized model, performs worse than any of the regularized models, although it does outperform the unregularized model (α = 0 in Figure 3a). While not significant, we also see that for rare semantic types “Semantic weighted” seems to perform the best, which aligns with our expectation that the utilization rate is hard to estimate for very rare concepts.

heading: 5.2 Effect of knowledge injection during training on model’s uncertainty
text: We analyze the effect of utilization regularization on the model’s uncertainty at every timestep. Uncertainty at timestep t is defined as an entropy of model’s distribution on each timestep t (here y<t is the decoded sequence up to t-th timestep, y is an arbitrary token from the target vocabulary):
Ht(y<t,x) = − ∑
y
p(y|y<t,x) log p(y|y<t,x) (9)
We consider the defined uncertainty on earlier timesteps, where the model’s distribution is closer to marginal. As the proposed method pushes up the marginal probability of the medical concepts, we claim that models’ uncertainty decreases with the regularization. Moreover, care plan instructions typically introduce crucial concepts at the beginning of an instruction. Thus, we claim that early timesteps uncertainty matters for the precise decoding of instructions.
This is confirmed by Figure 4. We observe that uncertainty drops monotonically as the α weight increases. In particular, uncertainty on early timesteps heavily drops as a result of utilization minimization. Hence, the model becomes more confident in selecting principal concepts at the beginning of an instruction. In contrast to the baseline, all regularized models’ uncertainty start to increase for t > 10. As fewer concepts appear in the instruction end, the marginal probability maximization flattens the conditional distribution. However, the uncertainty does not degrade in comparison to the baseline. Thus, the proposed regularization effectively improves the confidence of the model on early timesteps.

heading: 5.3 Results on Care plan instructions task
text: Automated evaluation: The precise and complete concepts utilization directly affects the quality of instruction. We first quantify the quality by calculating automatic metrics to judge the relevance, fluency, and concept utilization rate in comparison to the reference instructions. We use BERTScore (Zhang et al., 2019) to estimate the similarity between reference and candidate, GPT-2 perplexity for (Nguyen, 2021) to assess the coherence (fluency) of the candidate, and concept overlap (Joshi et al., 2020) to measure the percentage of medical concepts used in both candidate in reference.
Table 1 presents the automatic evaluation results. The scores indicate that incorporating knowledge correlates with relevance and concept overlap. We highlight three
observations. First, the regularization is effective in terms of quality and concept overlap. We observe significant quality improvement compared to both the baseline and DBA. Moreover, weighted versions of the model outperform the unweighted setup. Thus, injecting more knowledge into the model, such as empirical utilization weights, results in better quality. Second, the impact of the regularization hardly depends on the α weight. Third, the GPT-2 perplexity degrades. This demonstrates that the regularization impacts the model distribution, so the fluency of the model may deteriorate. This trade-off, however, has no negative impact on the quality given the improved BERTScore. For qualitative results, please see the Supplementary Material. Medical experts evaluation: To get a more precise medical assessment, we conduct human evaluation with medical experts. We randomly sample 100 dialogues from the test set and generate candidates with each model setup setting α = 1.0. We ask five doctors to evaluate the relevance to the dialogue, medical usability (if the generated instruction can be used in any care plan), and grammatical correctness (fluency) on a scale from 1 to 5. Additionally, we ask assessors to indicate degenerate generations, i.e., premature or repetitive sequences. Exact questions and interface screenshots can be found in the Supplementary Material.
As shown in Table 2, we claim that both weighted versions achieve significant improvement in relevance and usability, which are target medical metrics. In contrast to the GPT-2 perplexity, medical experts report equal fluency for all models but DBA. We explain this discrepancy with vocabulary shift as GPT-2 is not trained on a healthcare corpus. Finally, utilization rate regularization does not affect the number of degenerate outputs. Hence, the proposed solution effectively induces knowledge in the model distribution without corrupting generated text correctness. This is not true for DBA, which struggles from a lack of coherence and degenerate outputs while producing more relevant and usable instructions.

heading: 6 Conclusion
text: In this work, we tackle the problem of under-generation of rare but important tokens in sequence-to-sequence models. We show that external knowledge can be effectively injected into the sequence-to-sequence models and mitigate the problem of lexical precision. We characterize the problem by identifying a set of lowfrequency but important concepts and defining their utilization rate, which estimates the probability of a concept that is present in the source to be also present in the reference. We confirm that modern welltrained sequence-to-sequence models suffer from underestimating utilization rates, and propose a way to directly maximize it during training. We design a differentiable proxy based on the marginal entropy and propose a regularized training objective. Since some concepts may be omitted from the reference, we extend the approach by applying weights, which restrict the
regularization impact of low-utilized concepts or their semantic types.
We perform a case study in automatic care plan generation from medical dialogues. We experiment with a custom internal dataset and observe the effectiveness of the approach. We also compare a previous approach for external knowledge injection – dynamic beam allocation (DBA). First, we find that regularization improves the model’s utilization rate by pushing it closer to the empirical values observed in reference sequences. Second, regularization reduces the model’s uncertainty at early timesteps: exactly where concepts are typically introduced. Third, we observed a significant (in terms of standard deviations) quality improvement. More specifically, we did a human evaluation of relevance, concept overlap, medical usability, and fluency using five medical experts. The results revealed the enhanced relevance and usability of generated instructions while, unlike DBA, maintaining high fluency and low degeneracy.
Ethics Statement: This work was done as part of a quality improvement activity as defined in 45CFR §46.104(d)(4)(iii) – “health care operations” secondary research.
Reproducibility statement: Code used for training regularized sequence-to-sequence models in this paper is available at https: //github.com/curai/curai-research/ tree/main/careplan-charting. However, data will not be shared due to patient privacy and HIPAA compliance. as it contains significant amount of Patient Health Information (PHI) and cannot be shared.
Privacy concerns: Our research aims to utilize knowledge to enhance NLG systems. However, we also acknowledge the privacy concerns associated with leveraging sensitive medical information. All training data was anonymized during preprocessing step, and all personally identifiable information (PII) was removed to protect patient identities in generated outputs. Another privacy consideration is inference leakage, where NLG systems unintentionally reveal sensitive information during generation. We suggest incorporating differential privacy mechanisms to prevent the association of rare tokens or medical concepts with specific individuals.

heading: 7 Limitations
text: There are several important limitations to this work that can be split into two categories: (1) method applicability to other domains and (2) method scalability to much larger models.
Method applicability to other domains. Utilization rate computation and regularization are possible when there is some external knowledge that can be used to infer which tokens are “important.” In particular, our highest-performing model uses token semantic type to compute utilization rates. This limits our approach to sub-domains where there is an external knowledge source that can inform us about important tokens and give us higher-order semantic information about how to group the important tokens. For example, our approach will likely not be very helpful for open-domain conversations.
Method scalability to much larger models. We have evaluated our approach for models on the scale of O(108) parameters. However, modern state-of-the-art models often involve O(1011) parameters, three orders of magnitude larger than models in our experiments. Large language models (LLMs) often still suffer from the under-generation of rare tokens, but our study is insufficient to determine if our approach would still work. We suppose that utilization-rate-based regularization is most likely to be beneficial in the fine-tuning step of LLMs, but verification of this is left for future work.

heading: A Semantic relative errors
text: Section 5.1 in the main text discusses the relative error (Equation 7 in the main text) in model computed utilization rate for different semantic types as a function of α ∈ {0, 0.25, 0.5, 0.75, 1} and regularization type. The regularizations are lu (“Unweighted") or a weighted loss lw with the ϕ being identity (“Concept weighted”) or mapping concepts to semantic types (“Semantic weighted”). For α = 0 all mentioned models are equivalent to the baseline, that does not use any knowledge injection. Figure 5 shows the exact values of relative errors for every combination of models.

heading: B Human evaluation
text: B.1 Human evaluation UI The screen shot of the UI provided to medical experts for evaluation is shown in Figure 6.
B.2 Questions We used the following set of questions for medical experts to evaluate every sample:
1. Usability: How clinically usable is the candidate instruction in any context? Please rate on a scale from 1 to 5.
2. Relevance: How relevant is the candidate instruction to the highlighted portion of the dialgoue? Please rate on a scale from 1 to 5.
3. Fluency: How fluent/grammatically correct is the candidate instruction? Please rate on a scale from 1 to 5.
4. Degeneracies: Is the candidate instruction degenerate (either instruction ends mid sentences of words are repeated in a row)? Yes or No.
B.3 Evaluation task description Table 3 presents the description of the task that was provided to the medical experts. We also presented it personally to clarify the goals and answer questions.

heading: C Qualitative examples
text: A complete example of synthezing training samples is given in Table 4 and qualitative comparison between different models for the final task is in Table 5.
D Identifying source dialogue turns The training data includes only parts of the dialogue relevant to the care plan discussion, which is achieved by the internal segmentation model [work will be published and cited here prior to camera ready]. We then train a FastText model (Joulin et al., 2016) on all provided segments. We use spacy framework (Honnibal and Montani, 2017) to split dialogue turns into sentences x and generate an embedding E(x) for every
sentence by averaging the FastText embeddings e(xt) of the words in a sentence Equation 10.
E(x) = 1
∥x∥
∥x∥∑
t=1
e(xt) (10)
We repeat the procedure for the true care plan instructions y. Next, we use a cosine similarity c (Equation 11) between FastText embeddings of x and y with a threshold of 0.85 to map a sentence to the relevant care plan instruction. We omit the unmapped sentences and care plan instructions from the dataset.
c(x,y) = E(x) · E(y)
∥E(x)∥∥E(y)∥ (11)
To improve computational efficiency, we utilize the FAISS framework for mapping (Johnson et al., 2019).

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Yes, Section 7.
3 A2. Did you discuss any potential risks of your work? Our method is generally applicable to a wide range of sequence models including those which may generate harmful content. However, our method does not aim to mitigate these risks explicitly. Nevertheless, we discuss privacy concerns after Section 6.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and section 1 discuss main contributions.

heading: 7 A4. Have you used AI writing assistants when working on this paper?
text: Left blank.
B 3 Did you use or create scientific artifacts? Section 4.
3 B1. Did you cite the creators of artifacts you used? Section 4 cites code base we have used in our work.

heading: 7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: We used open source tools. The code of our method will be open sourced and free to use.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Collected data contains sensitive patient information. We discuss this in the Ethics Statement after Section 6.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Data is described in Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, "Dataset construction”.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Sections 4-5.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Sections 4 and 5.1 discuss hyperparameters of the model, give overview of the model performance w.r.t. different hyperparameter values, and highlight the best-performing ones.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We show descriptive statistics by running experiments with multiple random initializations.

heading: 7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used the main fairseq branch as the code base.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3, Appendix Section B.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? See appendix section B.

heading: 7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Medical experts are full-time workers and the requested information cannot be disclosed due to the company NDA.

heading: 7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Medical experts are full-time employees of the company and signed the agreement which contains the consent. Details of the agreement cannot be disclosed due to the NDA.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? See Ethics statement after Section 6.

heading: 7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Cannot be disclosed since workers are full-time employees.

heading: Future Work
text: We suppose that utilization-rate-based regularization is most likely to be beneficial in the fine-tuning step of LLMs, but verification of this is left for future work