SYMPTOMIFY: Transforming Symptom Annotations with Language Model Knowledge Harvesting
Authors: Bosung Kim
Ndapa Nakashole
Venue: EMNLP 2023 Findings (2023)
Keywords: Symptom Recognition
TL;DR: Introducing SYMPTOMIFY: a comprehensive resource of patient reports for symptom recognition, enriched with annotation explanations and background knowledge about symptoms.

Abstract: Given the high-stakes nature of healthcare decision-making, we aim to improve the efficiency of human annotators  rather than replacing them with fully automated solutions. We introduce a new comprehensive resource, SYMPTOMIFY, a dataset of annotated vaccine adverse reaction reports detailing individual vaccine reactions. The dataset, consisting of over 800k reports, surpasses previous datasets in size. Notably, it  features  reasoning-based explanations alongside background knowledge obtained via  language model knowledge harvesting.  We evaluate performance across various methods and learning paradigms, paving the way for future comparisons and benchmarking.

SYMPTOMIFY: Transforming Symptom Annotations with Language Model Knowledge Harvesting
Bosung Kim and Ndapa Nakashole
Computer Science and Engineering University of California, San Diego
La Jolla, CA 92093
bosungkim@ucsd.cm edu, nnakashole@eng, ucsd. edu
Abstract
Drug Safety Monitoring System (e.g., VAERS, Vigibase,
Given the high-stakes nature of healthcare decision-making, we aim to improve the ef- ficiency human annotators rather than re- placing them with fully automated solutions. We introduce new comprehensive resource, SYMPTOMIFY, a dataset annotated vaccine adverse reaction reports detailing individual vaccine reactions. The dataset, consisting of over 800% reports, surpasses previous datasets in size. Notably, features reasoning- based explanations alongside background knowledge obtained via language model knowledge har- vesting. We assess data quality, and evaluate performance across various methods and learn- ing paradigms, paving the way for future parisons and benchmarking."
G (Anyone)
(Trained
-Prientc
Report suhmission
personnel) ModERA ctaling,...
Public release
B
Symptom Recognition Model
This report mentions presence Anosmia. statee "90% loss smell/taste next indicatinn significant
reductiio ability to pereeiiesmelll, smells, whichis wharacarratacctcccyynneeee of Anosmia"
This report mentions presence of Hypoxia. It states that the paticat found to have an oxygen saturation of 86%. which indicutes a lower-thun norunnn-oooallll level oxygen blood
1 Introduction
C illustrate the stages of drug safety monitoring systems like VAERS. We aim to improve the efficiency of skilled human coders in step B by facilitat- ing the development of systems that identify symptoms and offer explanations to aid coders, as demonstrated by model-generated output in step D.
Figure 1:
Drug safety monitoring systems like Vigibase (Lindquist, 2008) and VAERS² are essential for evaluating the safety of medications and vaccines. These systems enable anyone to report drug reac- tions, facilitating early detection of vaccine safety issues. For example, if a mother observes vomiting and fever in her two-year-old after the chickenpox vaccine, she may be curious about the frequency of similar reactions reported by others. VAERS online browser can provide the answer. Being able to provide such answers involves normalizing the raw patient reports in order to address variations caused by polysemy, abbreviations, and other fac- tors present in these informal and potentially noisy reports.
Dictionary for Regulatory Activities (MedDRA) (Brown et al., 1999; Shimabukuro et al., 2015). To illustrate the coding process, consider the report: R1: "I had 90% loss of smell/taste the next day. Also felt irritable and nervous but manageable." This report annotated with these MedDRA terms: Ageusia, Anosmia, Irritability, Nervousness.
Our aim is to support the decision-mann making of hu- man coders during annotation, by supporting the development of symptom recognition models that can predict and explain identified symptoms (Fig- ure 1). Given the high-stakes nature of this field, where data interpretation consequences, fully automated solutions not yet sufficiently accurate. Specifically, we present resource of annotated reports detailing people's reactions to vaccines, and enriched in several ways. In summary, our contributions are as follows:
VAERS employs trained staff review and "code" each submitted report, assigning formal medical terms to the symptoms experienced by the patient. VAERS terms come from the standard- ized medical terminology known as the Medical
carry life-or-death
'The dataset code are available at https://github. com/ 'Vaccine Adverse Event Reporting System https:// vaers. hhs. hww.shhhh..oo.aaaaa gov/data.htmll

---

1. SYMPTOMIFY Dataset: We present a symp- tom recognition dataset derived from VAERS informal medical reports with over 800k en- tries. Beyond its scale, we enhance the dataset by adding explanations and back- ground knowledge substantially boosting its utility for advancing systems that aid human coders in annotation.
Dataset
Source
Symptoms
Size
SYMPTOMIFY (ours)
VAERS
11,472
871,373
COMETA (Basaldellaa et al., 2020)
Reddit
7.645
20,000
MedRed (Sespnnviic
2020)
Roddit
2,000
RedMed (Lavertu and Altman. 2019)
Roddit
2.978
waa
Twitter TwiMed (Alvaro et al., 2017)
Twitter
3,344
2,000
CADEC (Karmmi et al., 2015)
AstaPatient
6.754
1,253
ADR (Niklarjam
Twitter
1,784
Table I: SYMPTOMIFY surpasses the scale of previ- ous related datasets for entity recognition in informal health reports. Additionally, SYMPTOMIFY contains notation explanations and background knowledge about symptoms.
2. Annotation Explanations: We enrich SYMP- TOMIFY with annotation explanations, care- fully harvested from LLMs. We showcase their usefulness on the newly released Falcon language model.
differ from the focus of our work, which centers around user-generated text.
3. Background Knowledge Augmentation: We augment SYMPTOMIFY with background knowledge about symptoms, improving per- formance, particularly for rare symptoms.
Table 1 summarizes datasets closely related ours, with many, like ours, targeting pharmacovig- ilance, and unlike ours, originating from social media platforms such as Twitter and Reddit. Like our work, these efforts aimed to identify symp- toms mentioned in unstructured and informal tex- tual sources. SYMPTOMIFY surpasses previous datasets in size. and notably, it incorporates annota- tion explanations and provides background knowl- edge about symptoms.
4. Performance Evaluation: We implement and evaluate various baselines across different methods and learning paradigms, facilitating future comparisons and benchmarking.
2 Related Work and Existing Datasets
Our work related to the old task of Named Entity Recognition (NER), which is the task of mapping mentions of entities in text to standardized entity names (Sutton and McCallum, 2004; Ratinov and Roth, 2009; Ritter et al., 2011; Hoffart et al., 2011; Cao al., 2021). Our work goes beyond recogniz- ing entities as we wish support human decision making. Additionally, our work is complemen- tary to work on Explainable Artificial Intelligence (XAI) (Doshi-Vekee and Kim, 2017; 7; Ribeiro et al., 2016), and can be used to build more explainable models. Furthermore, there are efforts to enhance open-source LLMs by extracting instruction-oun output pairs from massive, paid-APl LLMs. The goal is to gather valuable data that can be utilized for instruc- tion tuning of open-source models. This method is particularly beneficial for projects that lack the financial resources to hire people for human feed- back, thereby democratizing access to advanced LLM capabilities (Peng et al., 2023).
LM Knowledge Harvesting
Access to the most powerful LMs is often limited to paid APIs with query restrictions. In dynamic settings, such continuous MedDRA report anno- tation, rapid query response is crucial. Therefore, propose to extract a portion of LM knowledge to enrich an existing dataset. This approach allows greater control, transparency, and balance between performance, cost-efficiency, and autonomy. Enriching existing datasets meets a widespread need as under-specified datasets are common. A prominent example comes from Question Answer- ing tasks where only few datasets fully labeled with question, answer, passage, and span.
In our work, key steps to upgrade a dataset via LM knowledge harvesting include: (i) Identify- ing Shortcomings: Assess the dataset to determine missing elements and areas needing improvement. (ii) Creating Good Prompts: Design prompts using existing dataset examples to minimize potential am- biguitics. (ii) Checking Data Quality: Implement strategies to assess LLM output quality, ensuring it is accurate, useful, and safe. (iv) Maintaining De- tailed Records: Document the harvesting process, including prompts, LLM configurations, and dates, for improved transparency and reproducibility.
Closer to our work are prior efforts on creating datasets for medical entity recognition from diverse sources, including death certificates (Goeuriot et al., 2017), scientific publications (Verspoor et al., 2012: Mohan and Li, 2019), and electronic health records (EHRs) (Suominen et al., 2013). While these cor- pora contribute to medical entity recognition, they

---

4 The SYMPTOMIFY Dataset
Avg: 104.3
With VAERS² as the starting dataset, we used LM knowledge harvesting to create SYMPTOMIFY. The dataset spans three years (2019-2021) and includes 839, 215 reports, incorporating symptom texts, rel- evant MedDRA terms, and associated metadata like age, sex, and vaccine type.
50
100
150
200
250
Explanation
Figure 2: Distribution of explanation lengths in number of words.
4.1 Quantitative Analysis MedDRA coders
VAERS uses certified MedDRA coders to assign formal medical terms to symptoms. We evaluated the quality of these annotations via crowdsoure- ing on Amazon Mechanical Turk (AMT). We first randomly selected 1, 000 symptoms and then se- lected corresponding patient report (symptom text) mentioning that specific symptom. A symp- considered correctly annotated if two three evaluators agreed.* Based on this accuracy metric, VAERS staff correctly labeled 93.4% of the 1, 000 tested symptoms, highlighting the high quality of their annotations. Still, this figure might underestimate the true accuracy, as most challeng- ing examples were medical test results not stated in patient reports (such as C-reactive protein lev- els metabolic function tests). The agreement rate among evaluators was moderately strong (0.55 Fleiss' Kappa*), ), similar related annotation tasks reported in the literature which showed agreements in the range: 0.55 0.62 (Wadhwa et al., 2022; Nye et al., 2018; Deleger et al., 2012).
b-scatiooed
petiras is pertieen
the gateer rignn
the patiect orgariznunt
ahs tedicaas the
Eoganess from ceeent
supore stoee thar
the paunts
in pooder pan an
Figure 3: Visualizing the top 30 most frequent 4 g-grams explanations. These phrases often indicate that the explanations quote spans from the patient reports.
4.2 Harvesting Annotation Explanations
We augment VAERS data by incorporating expla- nations to compensate for the absence of span-level annotations. Using GPT-3.5, we generated expla- nations to add to the dataset. The prompt used was: "Explain why the patient report mentions '(SYMP. TOMJ'. quoting report fragments where possible: '(REPORT)' ChatGPTTs responses were then used as explanations. These explanations offer extended context beyond the report's explicit content, like identifying Hypoxia from a report referencing an "oxygen saturation of 86%", correctly interpreting as low blood oxygen. By directly quoting from the patient report, the explanations also provide span-level information.
ChatGPT as the 4-th annotator. ChatGPT (based on GPT-3.5) served as fourth annotator alongside three human coders, working on the same 1, 000 tasks. The provided prompt was: "Does the following patient report mention '[SYMPTOM)': (REPORT)??. ChatGPT agreed with MedDRA 76.7% the cases. However, this figure might overestimate ChatGPT's symptom recognition ca- pacity, as this binary task significantly simpler than unguided symptom identification. We further probed ChatGPT's abilities through tests more au- thentically reflecting symptom recognition.
Qualitative Analysis of Explanations. Figures 2 and 3 illustrate the distribution of explanation lengths and the 30 most recurrent 4-grams within the explanations, respectively. Often, these fre- quently appearing phrases refer the symptom mentions in patient reports, including phrases like "is mentioned the," " "the report states that," " "frag-
*VAERS is jointly managed by the U.S. Centers for Dis- Control and Prevention (CDC) the U.S. Food and Drug Administration (FDA)
Detaiil the annotation task are provided in Appendix A.I.
'Overall, the annotation process cost 8723 (US dollars) and involved 201 workers.

---

Vaccine
Symptons
Explanation Fragment
Dizziness postural
This crident from following frogment repoct: becume esnemmel dizzy stand.
COVIDIY
lynghuulnepuihy
This is evident from the statement "My lymph nodes were swallen.
FLUA3
Throat tighmess
The following fragments frem
indicare
of symptom:: "throat feeling like is about clese" "swelling tump
right
HEPA
Thirs
The following Fragments from the report support this explanation: "He then hegun ing lange amounes of water thruggouu the day and even waking at night to ask for drink multiple times."
Arthragiia Dysproo
report states "Shoulder pain" which common sy mympiom arthralgia. The symptom "Dysjeooa" is mentional in the pasient report bacause the putiens is experiencing difficalty breuthing, which common symptoom of dysproes.
FLUIR4
Table 2: Example explanation fragments featuring some of the most common 4-grams, demonstrating direct quotation spans from the reports and the use of background knowledge about symptoms when necessary.
ments from the report," etc. Table 2 shows frag-
Sympton
LLM Detinition
UMLS Detinition
ments from example explanations that feature these common 4- gramm,, illustrating the direct quotation of spans from the reports and the use of background symptom knowledge.
Melsar
stukl, Eypically caused by pasovinnntinal boodiig..
The back lary, foul-mucling Bhat contiis dags blood.
Echaenia
Lack Nood
bood coused blockage blood vessel.
This obsnuctionn
problau blood voucl..
Table 3: Examples of symptom definitions derived from both UML. and a Language Model (I.M). The com- parison with UMLS definitions underscores the LM's capability to generate high-quality, medical term defini- tions.
Quantitative Analysis of Explanations: Honesty and Helpfulness. We evaluated the explanations two primary dimensions: 1) Honesty: Is the ex- planation factually accurate, without fabrications distortions? and Helpfulness: Is the explana- tion correct and beneficial in clarifying why a symp- tom mentioned?. We randomly picked 1, 000 explanations and employed a three-choice task for assessment: A) The explanation is both honest and helpful; B) The explanation is honest but not de- tailed enough to be helpful; and C) The explanation is neither honest nor helpful (i.e., incorrect explana- tion). Each task was evaluated by three annotators, with label deemed accurate if at least two anno- tators agreed on it. Our findings show that 91.3% the explanations were honest and helpful, 7.5% were honest but not helpful, and 1.2% were nei- ther honest nor helpful. For this relatively complex task, we hired three graduate students for annota- tion. The agreement rate was moderately strong, with Fleiss's 's Kappa of 0.60.
its definition. We harvested symptom definitions using the LLM Here one could also use knowl- edge graphs such as the Unified Medical Language System (UMLS) (Bodenreider, 2004), but we found coverage to be incomplete. In our exper- iments, these definitions were used in zero-shot learning baselines. But, there are many other poten- tial applications that future research could explore. Currently, SYMPTOMIFY'S symptom background knowledge primarily consists of definitions. How- ever, there are avenues to expand this knowledge further. For example, more complex relational structured data, such as known relationships be- tween symptoms, could be extracted from large LMs in future endeavors.
4.3 Harvesting Background Knowledge
Qualitative Analysis of LLM Definitions. Table 3, we display two example definitions gen- erated by the LMM comparing them with UMLS definitions to highlight the LM's proficiency in pro- ducing high-quality explanations. Certain terms used in drug safety monitoring systems may have dual meanings, creating ambiguity both within and outside the medical context. To bolster a medically- focused comprehension, particularly regarding ad- verse effects, we recommend integrating the phrase "ad "adverse effect" into the definition prompt, and sup-
Sometimes, recognizing symptoms in report re- quires understanding beyond what's written. For example, report might state "BP 173/109" without mentioning hypertension directly. But, given that hypertension is defined as blood pres- sure above 140/90, the report implics this condi- tion. This kind of understanding demands external knowledge. Thus, we aimed to augment SYMP- TOMIFY with background knowledge about symp- toms.
As a basic form of background knowledge we link each unique symptom in SYMPTOMIFY with
"We used, from the GPT series, the text-davinci-eon version.

---

plementing the prompt with example reports from VAERS that mention the symplom in a few-shot style. This method mitigates potential misunder- standings from multiple interpretations of a term.
Full tess
UMLS muppedd
Model
Macro
Micro F1
Macro Fl
Micm Fl
String Matching
30.38
10.41
32.98
MetaMap Symptom Embedding
3,74
10.65
0.87
Quantitative Analysis of LLM Definitions. To quantitatively evaluate the definitions, we ran- domly selected 1, 0.0 definitions and assigned three crowd-sourced annotators to rate each def- inition's s accuracy. A definition was deemed ac- curate and useful if it gained agreement from at least two annotators. The evaluation results indi- cated that 96.1% of the definitions were accurate. The inter-rater agreement, Fleiss' Kappa, was 0.34, which indicates fair degree of consensus among the raters.?
Table 4: Performance of string matching and symptom embedding methods. These methods do not utilize the training data.
0.03
8,01
Pyrexta
Fattique
Throat clasing, dafficulty breathing
0.79
Drapharyngeal disconfort
Higu
5 Experimental Study
Throat clesing,
Byspnoea
difficulty breathing
Trenor
Data. Our experiments the following data splits: Training set with 671,373 instances, vali- dation set with 83, 921 instances, and test set with 83, 921 instances. We further segment the test set into three subsets for more granular analysis: i) Full: The entire test set. ii) UMLS-mapppd: Ex- clusively contains symptoms that can be mapped to UMLS, identified via exact string matching, for fair comparison with previous UMLS- S-ocused stud- ies, specifically Metamap. iii) Rare: Only includes 'long-tail' symptoms, those occurring reports, representing over 80% of all symptoms. More details of the experimental configuration can be found in the appendix.
Figure 4: An illustration of symptom embedding base- line. is similar to apan-based string matching, but per- forms matches using continuous representations rather than discrete symbols.
Uses pre-trained lan- guage models fine-tuned on our dataset.
Pretrain-Finetune
4. Prompting: Leverages prompts to generate responses from language models in zero-shot and few-shot settings.
or less
5.1 String Matching Baselines
Our initial baselines attempt to align symptom men- tions with MedDRA terms without considering context or utilizing training data. We employ two methods: (i) MetaMap: A National Library of Medicine-developed tool that maps text spans symptom names in the UMLS dictionary. For fair- ness, we only evaluated MetaMap on the test set subset that could be UMLS-mapped. (ii) Exact String Matching: A stringent method that requires precise string matches, offering a stricter compari- son than MetaMap's span matching.
Task. To the initial benchmarks on our dataset, we started by focusing on the accuracy of symp- tom recognition, without considering the added information from explanations or symptom back- ground knowledge. We represented each data point pair Si), where x is patient report and S = (s,,.., sn|s; E S} is the set of symptoms mentioned in The goal is predict the Med- DRA names of symptoms mentioned in the input report x. We explore a variety of baseline methods across different learning paradigms:
The results in Table 4 demonstrate the task's complexity, these string matching methods per- form poorly. The findings underline the shortcom- ings of simplistic approaches in effectively han- dling this task.
1. String Matching: Involves matching symp- tom names directly with the report text.
2. Symptom Embedding: Represents symptom names and report texts as dense vectors for matching.
5.2 Symptom Embedding Baseline
As shown Figure 4, our symptom embedding method uses dense retrieval from a frozen lan- guage model. Symptom vector representations
'The annotation process engaged 1, 437 workers and amounted to a cost of $522.

---

are created from symptom definitions in SYMP- TOMIFY. For each symptom 8, we construct an embedding ha; using its name and definition. Similarly, we generate n-gram span embeddings for each n-gram in the patient report using Clin- icalBERT (Alsentzer et al., 2019) and the span embedding method by Ujiie et al. (2021). We then compare each symptom embedding with all n-gram embeddings using the scoring function Fi,
Oytoneee
Beghuryegral discesfart
Model
cdrrsouity breatning
Figure 5:
The joint learning method combines discrimi- native and generative approaches in the pretrain-fime finetunitee paradigm. The classifier head (yellow) and the genera- tion head (blue) are illustrated.
hs: h[z,, (n- We re- call spans with scores exceeding a certain thresh- old, determined by the best validation score. This approach similar to zero-shot entity linking by Wu et al. (2019). In our experiments, with n = 2. the embedding method did not surpass the string matching method, suggesting that further training of embeddings may be needed (see Table 4).
Model
test
Macro Fi
Micro FI
Macro FI
Miaro F1
BERT base
1.34
15.81
1.45
18,80
BART-base
ClinicaBERT }
1,53
16.32
1.76
19.46
ClinicaBERRT CNN
7.29
62.68
8.12
65.79
GEN w/ BART
31.15
78.90
31.22
81.27
GEN w/ TS-base
32.01
79.88
32.19
82:25
5.3 Fretrain-Finetuns Baselines
Joinr C GRN
32.89
79.93
32.70
82,97
Given the substantial data available in SYMP- TOMIFY, we investigated fine-tuning baselines.
Table 5: Pretrain-finetune performance. І denotes clas- sification methods. Models without symbols are genera- tive. Joint C GEN refers both classification and generative method (T5 + ClinicalBERT).
Classification Approach. In multi-label clas- sification context, we experimented with various pre-trained language models including BERT (De- vlin et al., 2019), BART (Lewis et al., 2020), and ClinicalBERT (Alsentzer et al., 2019). We used BERT's classification paradigm, using the [CLS] vector for the sentence representation and a classi- fier layer. Additionally, we implemented Convo- lutional Neural Network (CNN) top of BERT's last hidden states (BERT + CNN) to capture more granular features from the contextualized repre- sentations, following the approach by Safaya et al. (2020).
Joint Classification & Generation. To bring together the strengths of both discriminative and generative approaches within the pretrain-finetune framework, we implement a joint learning strat- egy. We use transformer- models for both approaches, with shared transformer layers but dis- tinct output layers for each task. This joint model structure is depicted in Figure 5.
Results. The generative techniques substantially outperformed the most effective string-matching method, demonstrating the power of such mod- els. Leveraging T5 as the base model, the gener- ative model achieved impressive miero FI scores of 79.88% and 82.25% the Full and UMLSS- mapped test sets respectively. Interestingly, the multi-label classification didn't surpass the string matching approach, potentially due to the large class count (over I 11 and the bottleneck caused by the [CLS] token. However, incorporating a CNN on top of ClinicalBERT enhanced classification performance, reaching 62.68% the full set and 65. 79% on the UMLS Mapp-dd set, but still lagging behind the generative model.
Generative Approach. In the context of symp- tom generation, we adopt the generative entity re- trieval approach of GENRE, utilizing transformer- based encoder and decoder (Cao et al., 2021; Lewis al., 2020). In this framework, the model gener- ates a sequence of symptom names as the target output, rather than individual entity classification. To retrieve multiple symptoms, GENRE needs an- notated spans referring to each symptom. However, mentioned, VAERS annotations lack symptom mention spans; we only know if a symptom was mentioned or not. Thus, generate target se- quence as a comma-separated list, e.g., given the input report: "I have muscle pain and the tar- get sequence is: "Myalgia, Pyrexia". We consider basclines using BART and T5 as the pretrained models.
The joint learning model showed modest per- formance improvements, confirming the benefit of combined strategies.
The variance between micro and macro FI

---

Model
Fall test
UMLS-mapped
Target: y
Macro Fl
Micro Fl
Macro Fl
Micro F1
Report: Hodid feces, indica: forfor blood the feces...
GPT-2 (Supervised)
13.93
59.41
14.27
61.90
Input
CharGP (Zero-shot)
39.75
22.23
Symptom: Melaena
CharGPT (One-shog shot)
29.17
42,74
27.97
41.51
Symptom Back-
ground Knowledge:
LLM (GPT-2)
Loss (3.
Melaena refers
Table 6: Performance of prompting methods. While these methods fall short of the pnetrain-fineture tech- niques, it is notable that the vnetrainffinttue approach leverages the extensive SYMPTOMIFY dataset, underlin- ing the considerable benefits of fine-tuning for improved results in settings with massive amounts of data such as SYMPTOMIFY.
black, tarry, foul-
smelling
contain dograaded
Generated: 3
Report: I Lorem ip- sumn dolor amee, consectes....
Figure 6: Synthetic Report Generation: the LM takes symptom names input, and to encourage semantically faithful reports, also takes background knowledge as input (A), During training, target (B) and generated reports (C) are used to compute the loss.
Rare symptoms test set
Model
Macro F1
Micro Fl
String match
6.91
7.79
GEN (TS-base)
21.78
24.98
prompting methods, and shows that they are unable to surpass the performance of the pretrain-finetune methods. This is expected since the pretrain- finetune models are trained
GEN w/ definitions
21.72
25.16
GEN synthetic reports
23.31
25.73
GPT-2 Prompting
6.63
7.87
ChatGPT Prompting (zero-shot)
11.75
8.56
large training data of over 600K reports, whereas the zero-shot and one-shot prompting methods have access to zero only one report, respectively. It is worth noting that the large-scale nature of SYMPTOMIFY highlights the significant benefits of fine-tuning for improved results in settings with massive amounts of data such as in S SYMPTOMIFY.
ChatGPT Prompting (onc-shot)
16.06
10.49
Table 7: Macro Fl and Micro Fl of the symptom detec- tion task the rare symptoms test set.
scores in all configurations due to the dataset's imbalance: 80% of symptoms are rare with 50 or fewer reports mentioning them, thus making it harder for the model to learn to accurately detect these infrequent symptoms.
5.5
Semantically Faithful Synthetic Data for Rare Symptoms
5.4 Prompting Baselines
In our experiments, all methods struggled to iden- tify rare symptoms. To address the rare symptoms problem, we used data augmentation techniques to generate more data points for these symptoms. We tried using symptom definitions as synthetic reports but found a stark discrepancy between the language in real reports and definitions. We instead used a simple synthetic data generation method to create more realistic data points for rare symptoms, illustrated in Figure 6. We trained GPT-2 model on prompts comprising symptom name and defi- nition. The definition is there to ensure that the report is semantically faithful to the meaning of the symptom, also reducing ambiguity in compar- ison conditioning on the symptom name alone. Using Top-k multinomial sampling, we generated 100 synthetic reports per symptom, with optimal performance using 30 reports each. We then inte- grated this augmented data into our training set to improve the model's handling of rare symptoms. Results. Table illustrates the low performance achieved on the rare symptoms dataset, as indicated by both the macro and micro Fl scores across all
Given the widespread success of LM promoting, we also experimented with prompting baselines.
Language Model Prompting. We prompted ChatGPT using the following prompt: æ = Extract a comma-separated list of symptoms from the fol- lowing report: "(REPORT)" In addition to zero- shot prompting, we explored the few-shot setting, specifically the one-shot setting with single test set example added to the prompt.
Discriminative models assign mentions to spe- cific class labels, while generative language models have a vocabulary that may differ from the formal symptom names in MedDRA. For instance, when generating the symptom "Pyrexia." " ChatGPT often produces "Fever" instead. This necessitates answer engineering. We employ embedding- based meth- ods using SBERT (Reimers and Gurevych, 2019). The prediction with the highest cosine similarity score between the predefined symptom set and the generated output is selected.
Results. Table contains the results of the

---

Sympliom
(CI) Report
Zem--shol Falcon
(C44 Falemn with SYMPTOMIPY Explanation
Explanation
(RI) Arthralgia
Putient having shoulder puin. Made appoint- with doctur, MRT completed.. Passible SIRVA.
Arthoalgio mentioned in the putient report sympptom resusiatel SIRVA (Shoulder Impingenent Syndrome Radial Neuropathy)
The symptom "Arthralgia" is mentioned in pulies rejett bazanse putient repertdd shoulder pain. This is evident from mext Tatient hoving shoulder puin'. symptom "Pyresia" mentioned the tient report because patient experiencing fever.
it
(82) Pyresia
all over body rash and fever
Pyrexia is mentioned the patient coport
common symptom flver
(R3) Muscle spusems
Polient stated he gulling musele the back after iuu shot.
Muscle speeems common side effect of the received vaccine and experienaed musale spusuns as result.
The symptom "Musclo squeams" is mentioned in putient report becuse the putient periencing muscle quuerr in the nock and after receiving a flu shot.
The punient
(R0) Mood altered
Pever, cold chills, restless sleep, rodness site of injectios, right leg screness unable to stand on that lez. loss of appetite, moodinesskrping. After 12 hours of injection
The patient reported moed altered result injection. mentioned patient report as it potential side effect of the med- ication.
The sympeom "Mood altored" mentioned patient repect because patient repocted experiencing change in their moed. This is evidens from the statement 'I was moodylery. ing"
Table 8: Falcon LM struggles to generate quality explanations (column C3) compared to ChatGPT. Yet, supple- menting it with SYMPTOMIFY explanations improves its performance (column C4). Blue indicates accurate report fragments; red signifies paraphrased, non-exact matches, hallucinations.
methods. Furthermore, we observe a modest im- provement when incorporating synthetic reports into the training data. While the improvement is small, it demonstrates the (perhaps small) potential benefits of leveraging synthetic data to enhance the model's s ability to handle rare symptoms.
expose Falcon to a larger volume of SYMPTOMIFY explanations, thus improving its ability to gener- ate high-quality explanations. However, due to the context size limitations in the prompting, the ideal solution leverage these explanations effectively would be to implement the pretrain-finetur ap- proach to use the massive SYMPTOMIFY dataset more productively as training data.
5.6
Leveraging SYMPTOMIFY'S Explanations in Falcon
We carried out an experiment to study the effec- tiveness of the explanations in SYMPTOMIFY using a prompting approach. For this, we used Falcon- 7B-Instruct", a new, instruction-tuned iteration of the open source Falcon renowned for its superior performance on key NLP benchmarks. The experi- ment had two configurations:
6 Conclusion
We introduced SYMPTOMIFY, large-scale dataset of annotated reports, reflecting reactions to medica- tions and vaccines. It includes MedDRA symp- toms, annotation explanations, and background knowledge about symptoms, designed to facilitate development of systems that can aid human anno- tators operating in the critically important public health domain. Despite current performance on rare symptoms being low across all baselines, the continual evolution in language models and strides in few-shot learning offer promise for improve- ment.
1) Zero-shot Falcon Explanations: We executed standard zero-shot in-context learning with Falcon. 2) Falcon with SYMPTOMIFY Explanations: We introduced few-shot learning in this setup, using selected SYMPTOMIFY explanations as our few- shots.
Table 8 provides comparative analysis of Fal- con's explanation generation under the two scenar- ios. Initially, Falcon struggled produce benefi- cial explanations in the zero-shot learning context. However, its performance significantly improved terms of specificity and context quoting when it was supplemented with few-shot learning, us- ing a handful of SYMPTOMIFY explanations as a guide (contrast columns C3 and C4, specifically (R),C4)). Nonetheless, false quoting of halluci- nated segments, illustrated in (R4, is prob- lem. To tackle this, one possible solution is to
There are several avenues for improving SYMP- TOMIFY and expanding its utility. Integrating the explanations into a wider range of models may lead to the better generation of explained predictions with open source models such as Falcon. Expand- ing background knowledge to include relational knowledge, such as inter-sympton relationships, could yield performance gains. Further, focusing on laypeople's 's perspectives and acquiring more layman medical terms could facilitate broader, more inclusive understanding of symptoms. Look- ing further ahead, as patient reports can sometimes contain not only textual descriptions but also ref-
hhttps: htt.://ggggnggaceeeeeeme wwt.cc//iiiuie falcon- 7b-instruct

---

erences medical images, pictures submitted by patients, and lab lest results, the incorporation of these non-textual elements using multi-modal lan- guage models, could be useful.
sessing the safety of medicines and vaccines. They can aid in the timely identification of potential risks and contribute to individual and public health. However, the availability of data to train such mod- els, which is the main contribution of this work, raises concerns about individual privacy. Releas- ing the data, which includes free text reports and metadata from VAERS, may risk the exposure of personal information or identities. We rely the privacy monitoring efforts of the CDC and FDA, the co-managers of VAERS. Furthermore, Follow- ing prior work on automated health systems, our goal to be clear and transparent about the con- tent of SYMPTOMIFY and the capabilities of the methods developed in this work.
Limitations
Our work, while it made useful contributions, has a number of significant limitations. First, the VAERS database, our primary data source, may not be rep- resentative of the larger population due to potential sampling biases. It also limits language diversity it contains only English-language reports.
The symptom distribution in the reports presents a distinct challenge, with strong skew towards certain symptoms that hinders model performance, including that of our most successful methods, like pretrain pretrain-fintune. While our attempts to rectify this through data augmentation techniques have yielded some improvement, the performance still falls short. This indicates the need for innovative methods that go beyond synthetic data expansion for addressing this challenge effectively.
References
Emily Alsentzer, John R. Murphy, Willie Boag. Wei- Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. 2019. Publicly available clinical bert embeddings.
Nestor Alvaro, Yusuke Miyao, Nigel Collier, 2017. Twimed: Twitter and pubmed comparable corpus of drugs, diseases, symptoms, and their relations. JMIR public health and surveillance, 3(2):e6396.
The explanations present in SYMPTOMIFY are a rich resource that, as of now, we" ve only begun to explore. The explanations coupled with the pub- lic availability of the dataset offer valuable base for other researchers integrate into their work. Nevertheless, more exhaustive studies are needed fully harness the potential of these explanations. The potential biases of language models like ChatGPT, known to mirror the biases in the data they're trained on, is another limitation. Although we have not observed any explicit biases in our current study, no thorough testing for bias was con- ducted, and it remains a concern.
Marco Basaldella, Fangyu Liu, Ehsan Shareghi, and Nigel Collier. 2020. Cometa: A corpus for medical entity linking in the social media. arXiv preprint arXiv:2010.03295.
Olivier Bodenreider. 2004. The unified medical lan- guage system (umls): integrating biomedical termi- nology. Nucleic acids research. (issupp_1):D2265 D270.
Elliot Brown, Louise Wood, and Sue Wood. 1999.
The medical dictionary for regulatory activities (med- dra). Drug safery, 20(2):109-11)
Finally, the use of non-medical experts from Amazon Mechanical Turk, as well as Computer Science graduate students for the annotation pro- cess introduces the possibility of inaccuracies into our assessment. Although we attempted to mitigate this by implementing rigorous review process and utilizing multiple annotators, this factor may still have influenced the accuracy of our human evaluations. Future work may benefit from the involvement of medical experts in the annotation process, increase the validity of our findings, and by extension the datasct.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference Learning Representa- tions.
Louise Deleger, Qi Li, Todd Lingren, Megan Kaiser, Katalin Molnar, Laura Stoutenborough, Michal Kouril, Keith Marsolo, Imre Solti, et al. 2012. Build- ing gold standard corpora for medical natural lan- guage processing tasks. In AMIA Annual Symposiun Proceedings, volume 2012, page 144, American Med- ical Informatics Association.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for
Ethics Statement
Models for symptom recognition have the potential to provide valuable insights for monitoring and as-

---

Computational Linguisties: Human Language Tèch- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang. Iain Marshall, Ani Nenkova, and Byron C Wal- lace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes support language processing for medical literature. In Pro- ceedings of the conference. Association for Computa- rional Linguistics. Meeting, volume 2018, page 197. NIH Public Access.
Finale Doshi Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08600
Lorraine Goeuriot, Liadh Kelly, Hanna Suominen, Au- rélie Névéol, Aude Robert, Evangelos Kanoulas, Rene Spijker, Joao Palotti, and Guido Zuccon. 2017. Clef 2017 ehealth evaluation lab overview. In Ex- perimental IR Meets Mulriliigguaitt, Multimodal- ity, and Interaction: 8th International Conference of the CLEF Association, CLEF 2017, Dublin, Ire- land, September 1 11-14, 2017, Proceedings 8, pages 291 303- Springer.
Baolin Peng, Chunyuan Li. Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXtv:2304.03227.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. CoRR, abs/1910.10683.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference Compu- tational Natural Language Learning (CoNLL-2009). pages 147-155.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau. Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 201 Conference on Em- pirical Methods in Natural Language Processing, pages 782-792.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNU) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015. Cadec: A corpus ad- verse drug event annotations. Journal of biomedical informatics, 55:73-81. 81.
Adam Lavertu and Russ B Altman. 2019. Redmed: Ex- tending drug lexicons for social media applications. Journal of biomedical informatics, 103107
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should trust you? explain- ing the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1144-
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online, Association for Computa- tional Linguistics.
Alan Ritter, Sam Clark, Oren Etzioni, al. 2011. Named entity recognition in tweets: an experimen- tal study. In Proceedings of the 2011 conference empirical methods in natural language processing. pages 1524-1534.
Marie Lindquist. 2008 Vigibase, the who global iesr database system: basic facts. Drug Information Jour- nal, 4195449-4449
Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT- CNN for offensive speech identification in social me- dia. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054-2059, Barcelona (online), International Committee for Computational Linguistics.
Tyya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR,
abs/1711.05101.
Sunil Mohan and Donghui Li. 2019, Medmentions: A large biomedical corpus annotated with umls con- cepts. arXiv preprint 0et.:090220000
Sanja Scepanovic, Enrique Martin-Lopez, Daniele Quer- cia, and Khan Baykaner. 2020. Extracting medical entities from social media. In Proceedings of the ACM Conference Health. Inference, and Learning. pages 170-181.
Azadeh Nikfarjam, Abeed Sarker, Karen O'connor, Rachel Ginn, and Graciela Gonzalez. 2015. Pharma- covigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association, 22(3):671-66 1-681.
Tom T. Shimabukuro, Michael Nguyen, David Martin, and Frank DeStefano. 2015. Safety monitoring in the vaccine adverse event reporting system (vaers). Vaccine, 33(36):4398444)) 4405.

---

Hanna Suominen, Sanna Salanterä, Sumithra Velupil- lai, Wendy W Chapman, Guergana e Elhadad, Sameer Pradhan, Brett R South, Danielle I. Mowery, Gareth JF Jones, et al. 2013. Overview of the share/clef ehealth evaluation lab 2013. In Information Access Evaluation. Multilingualit, Mul- timodality, and Visualization: 4th International Con- ference of the CLEF Initiative, CLEF 2013, Valencia, Spain, September 23-26, 2013. Proceedings 4, pages 212-231. Springer.
Charles Sutton and Andrew McCallum. 2004. Col- lective segmentation and labeling of distant entities in information extraction. Technical report, MAS- SACHUSETTS UNIV AMHERST DEPT OF COM- PUTER SCIENCE.
Shogo Ujii,, Hayate Iso, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2021. End-to-end biomedical entity linking with span- dictionary matching. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 162-167, Online. Association for Computational Linguistics.
Karin Verspoor, Kevin Bretonnel Cohen, Arrick Lan- franchi, Colin Warner, Helen I. Johnson, Christophe Roeder, Jinho D Choi, Christopher Funk, Yuriy Malenkiy, Miriam Eckert, al. 2012. corpus of full-text journal articles is robust evaluation tool for revealing differences in performance of biomedical natural language processing tools. BMC bioinformat- ics, 13(1):1-26.
Somin Wadhwa, Vivek Khetan, Silvio Amir, and By- Wallace. 2022. Redhot: A corpus of annotated medical questions, experiences, and claims on social media. arXiv preprint arXiv:2210006331.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2019. Scalable zero- shot entity linking with dense entity retrieval, arXiv preprint arxiv: 1911:99 0.0384..

---

A Appendix
the roject iiterond cou
A.1 Annotation Instructions for Mechanical Turk
th diker heere ticiton neminng aclive on de ah lins che cilin lu catira surecck
penton bur untorrsii mnsrr
fuber
CiTe aentin coneeer un undliin
Cop waater palctt rip persus Tappre coeeral fertu becoe Light den de mas
We conducted three assessments via the Amazon Mechanical Turk (AMT) system and our annota- tion process. In this section, we describe detailed instructions and web interface screenshots used in the assessments.
cau sent
plortiin captraor Horeel
Figure A.2: The web interface used evaluate the quality of explanations in SYMPOMIPYY
symptom is mentioned in the report. Choose A if the explanation correct, and helpful in describ- ing why the symptom mentioned. Choose if the explanation correct, but lacks details, thus not helpful to explain why the symptom mentioned. Otherwise choose C." Below are the examples we have provided for detailed instructions.
Select an aption
Symptom (or Diagnoste)
Incaaded Not incudee
Chronic aportsreous articarla (Detattio of tomplon a skón dboweer chanaterbeed the recurence of tomtosti
allegie
Report
Ongoinn cironi spontanoous unkaat peose ring 108920011 (WO wooks oficr sociro paree spends sah seseerrulane 2y relkono
rosppeero
24 loue,, bul symttore relurn when dey dorige teihed Fir mnvw uooling aleraaa ressionn the fnst mediim ansti, only bocater
Flu-like symptoms, extreme chills (shivering ), sweats for 2 days
Report:
Figure A.1: The AMT web interface used to evaluate the quality of MedDRA annotation in VAERS.
Symptom : Chill
1) VAERS MedDRA Annotation Quality Assess- ment As described in section 4.1, we evaluate the annotation quality of MedDRA annotations in VAERS. The instruction given to workers is "Given a patient's 's report describes their medical condi- tions. The goal of this task is check whether the given symptom is included in the report not." For the detailed instruction, we provided the examples as below:
The patient report mentions "extreme chills (shivering)" as symptom, referring to the presence of extreme chills shivering experienced by the patient for 2 days.
Explanation
The answer should be A, since the ex- planation correctly describing as "The report mentions extreme chills (shivering) as a symptom' with a reason based on the report.
Symptom/Diagnosis: Anosmia
Patient report 90% loss of smell/taste next day. Also feel irritable and nervous but manageable.
Explanation This report mentions pres- ence of Chill.
~Thh answer should be even the explana- tion is correct, since the explanation is NOT helpful and too short to conjecture why the symptom is mentioned.
+ The answer should be Included since the report mentions "90% loss of smell" in the report.
Symptom/Diagnosis: Headache Patient report 90% loss of smell/taste next day. Also feel irritable and nervous but manageable.
Explanation This report mentions pres- ence of Chill. It states the patient experi- enced dizziness and shivering for 2 days.
The answer should be C since the patient did NOT experience 'dizziness', i.e,, the ex- planation is not correct.
+ The answer should be Not included
We also added the definition of the symptom in case that the workers are not familiar with the symptom (see Figure A.1).
3) GPT-3 Definition Quality Assessment The instruction given workers is "Given a term and definition, answer Yes or No whether the definition is correct for the term or not." We have provided the examples below for detailed instructions.
2) Explanation Quality Assessment The instruc- tion given to workers is "Given a patient report and the symptom, the explanation is describing why the

---

option
ischaemic encephalopathy (W wag you google
No
Detinition: daraaee seat cauaedilty by oaggen org
Figure A.3: The AMT web interface used to evaluate the quality of GPT-3 Definitions.
Figure A.4: Symptoms by patient age. Age group 0-9 is more likely to die after taking a vaccine.
Term Anosmia
Definition Anosmia a condition that results in a loss of the sense of smell.
+ The answer should be Yes
Injetton Pyesis
Twmmle
Injecton sic seed Ing
Term:
Headache
ivectiion cite prutiss
Definition A reaction that occurs immedi-
Inection die r3.h
ately following an injection.
Skn
Migrae
+ The answer should be No
harredbbeec 2ra
Injectior 012th
We also added guidance encourage annotators to Google search they are unaware of the symptoms.
0. 0.2 0.6
10
Figure A.5: Symptoms by patient sex. The dashed line denotes average ratio of male to female, and shows that females report more. Symptoms such Paraesthesia oral, and Migraine, are more likely to observed in females than males.
A.2 Metadata Analysis
VAERS contains over pieces of metadata, for example patient's basic information (c.g., age and sex), medical background (e.g., allergies, and medi- cations), and vaccination information (e.g., vaccine type, and vaccine provider). We analyze this meta- data. First, we observed that some symptoms occur more frequently in certain age groups, shown in Figure A.4. The age group 0-9 more likely die after taking vaccine than other groups. On the one hand, teenagers (age group 10-19) are more likely to get Syncope, Loss of Consciousness, and Pallor. Second, symptoms such as Paraesthesia Oral, and Migraine, are more likely to be observed in females than males. Syncope, Death, and Pallor are more common in males shown in Figure A.5, the vaccine type. Headache and Pyrexia are the most common side-effects of COVID 19 vaccines, however, Diar- rhoca and Hacmatochezia mostly occur after RV5 vaccination (see Table A.I).
standard deviations are presented in Table A.4. For generative models, adopted GENRE's (Cao al., 2021) experimental settings with 256 of maximum input length, 128 of maximum output length, 64 batch size, and of beam search size. We used the pre-trained BART-base (Lewis et al., 2020) with 3c-5 of learning rate and T5- base (Raffel et al., 2019) with le-4 of learning rate following the original papers. We fine-tuned each model epochs
training set.
Third, symptoms vary depending
Evaluation of Generative Models While dis- criminative models can identify exact classes, gen- erative models cannot predict discrete classes. our work, we form the target text as sequence symptoms separated by "!". We split the generated text by " and match each chunk with symptom names by exact string match. For those symptoms that not exactly matched within symptom set, post-processed answer engineering as de- scribed in Section 5.4.
A.3 Experimental Setup Details
Table A.2 describes hyperparameters and search spaces we considered in experiments. For training tranformer-based models, used AdamW opti- mizer (Loshchilov and Hutter, 2017) for training all transformer-based models, and hyperparameters are set on the best validation performance. We ran experiments times with different seeds, the re- ported scores are average of them. The results with
Evaluation of MetaMap We use Metamap for one of zero-shot baseline models. Given the input text, MetaMap outputs UMLS entities with con- fidence scores. We experimented with thresholds

---

COVID-19
FLU4
HPV9
TDAP
RV5
Headache
Pain
Syncope
Injection site pain
Diarhhoa
Pyrexia
Injection site pain
Dizziness
Pain
Hacmatochezia
Fatigue
Pain extremity
Hcadache
Pain in extremity
Vomiting
4
Chills
Injection site erythema
Injection site pain
Injection site swelling
Intussusception
5
Pain
Pyrexia
Nausea
Injection site erythema
Pyrexia
Table A.1: Top 5 most frequent symptoms by vaccine type.
Metamap
Symptom
threshold
0.1 [0.05, 0.1, 0.15, 0.2. 0.25)
1
Headache
2
Pyrexia
3
Fatigue
Multi-label Classification
4
Chills
Learning rate
5e-5 (2e-5, 5e-5)
5
Puin
6
Nausea
max input length batch size threshold
256 [128, 256
7
Pain in extremity
32
Dizziness
0.1
9
Injection site pain
10.05, 0.1, 0,15, 0.2, 0.25)
10
Myalgia
Generative Models
Table A.3: Top 10 symptoms SYMPTOMIFY
Learning rate (BART-buse) Learning rate (TS-base)
3e-5 [2o-5, 3e-5, 5c-5)
1e.4 (3e-5, Se-5, 1e-4, 2e-4)
max input length max output length (decoder) beam search batch size
256 (128, 256]
Input
renagashhe
barget senbemee: sgaanss, anoars imittally samouannsr
128 [64, 128, 256)
Genertted
64 (32, 64)
Joint Learning (C GEN)
<b spaua
Learning rate
3e-5 (e--5,3e-5, 5e-5] 256 (128, 256]
input length
cm aye
max output length (decoder) batch size
128
64 [32, 64)
0.1 |0.05, 0.1, 0.2, 0.5)
Attention map accomting to decoding step genentive model
I.M Prompting (GPT-22)
Learning rate max input length batch size
5e-5 (2e-5, 3e-5, Se-5]
Ex1
E21
512
(1) Amertion map of the claseiicationn
64 (32, 64)
Figure A.6: Generation vs Classification: Attention patterns in each approach.
Table A.2:
Best- Best-perforning hyperparameters and search space. Values in parentheses denote search space.
proach. In the generative model, the decoder at- tends input tokens that are related to symptoms at each decoding step. On the other hand, the classifi- cation model uses one aggregated vector to predict all symptom classes. We can conclude that con- sidering most examples have multiple symptoms, predicting once sufficient to recall multiple entities.
(0.05, 0.1, 0.15, 0.2, 0.25, 0.3) and regarded enti- ties as positives over the threshold. The threshold of 0.1 was determined on the best validation score.
Computing Infrastructure We ran experi- ments on a single NVIDIA GTX 108 (12GB) with
CUDA 10.1 version.
Negation Performance Due to the prevalence of complex negation in our dataset, we analyzed how the model performs in the presence of negation. We sampled 103 examples of those patterns from the development set and tested them with different training sizes. Figure A.7 presents performance
A.4 More Analysis
Generation vs Classification We can observe the significant gap performance between classi- fication and generation models. Figure A.6 shows the difference of attention patterns in each ap-

---

Full test
UMLS mapped
Model
Macro F1
Micro F1
Macro Fl
Micro FI
Exact String Matching
8.87
30.38
10.41
32.98
MetaMap
3.74
10.65
Symptom Embeddings
0.72
3.32
0.87
5.87
BERT-base
1.34 (#0.13)
15.81 (+0.14)
1.45 (#0.09)
18.80 (#0.08)
Pretrain Finetune
BART-hase $
1.37 (#0.18)
15.96 (#0.11
1.55 (+0.12)
18.98 (#0.13)
ClinicalERRT $
1.53 (+0.12)
16.32 (#0.12)
1.76 (#0.11)
19,46 (=0.11)
ClinicalBERT + CNN
7.29 (+0.35)
62.68 (#0.25)
8.12 (#0.47)
65.79 (#0.25)
GEN w/ BART
31.15 (#0.37)
78.90 (+00.4)
31,22 (#0.19)
81.27 (+0.25)
GEN TS-base 1
32.01 (#0.31)
79.8 (+0.27)
32.19 (+0.22)
82.25 (+0.25)
Joint C++ GEN
32.89 (#0.21)
79.93 (#0.19)
32.70 (+0.43)
82.97 (#0.31)
Prompt
GPT-2 (Supervised)
13.93
59.41
14.27
61.90
CharGPT (Zero-shot)
24.69
39.75
22.23
38.10
ChatGPT (One-shot)
29.17
42.74
27.97
41.51
Table A.4: Performance summary. I denotes classification methods and denotes generative methods. Joint C GEN refers to both classification and generative method (T5 ClinicalBERT),
Type
Example
no Isymptom name]
I feel cold, but do not have fever, am not having chills, nausea, etc. Fever of 101, 102 for 2 Days; Chills; Nausea with Vomiting.
no visible signs of rash or irritation.
Medical history
Medical history included Ankylosing. spondyltis/psorii
cardiac ablation from Jan2017 Family medical history included her dad had rheumatic fever and high hlood pressure. Past medical history included no adverse event. concurrent medical conditions included Ulcer and Hy plyooyyyooidim.
current symptom)
Other negation expressions
Patient didn't I experience fever and rash, but feel musele pain
All but the pain and soreness the injection site subsided by the next day Denies any further fevers or chills. Denies any nausea vomiting. Denies loss stasis smell.
Table A.5: Examples of negation patterns in SYMPTOMIFY
on 103 negation samples as we varied the num- ber of train examples. As the size the training set increases, the model performs better on nega- tion expressions and the performance saturates at 200k. Through this result, we can conclude that our dataset contains sufficient negation data points and effectively learns to deal with a large number of them.
0.765
0-798
0.664
0.703
Micro
We observed that the simple negation expression "no [symptom name]" indeed an easy form of negation, the model performs well on it, even with a small amount of train set around 50k examples. A more thorough analysis of which negation ex- pressions the model still fails after saturation, is left as future work. Table A.5 shows examples of negation expressions in SYMPTOMIFY.
10k
50k
200k
(Full)
# of train examples
Figure A.7: Micro Fl of the generative model with T5- base on data points involving negation the size of the training data is varied. As the size of the training set increases, the model performs better on negation expressions and the performance saturates around 200k.


Reviews:
[{"review_id": "4v0sn1tSkc", "reviewer": "Reviewer_Y8AL", "summary": "", "questions": "A. Line 158: It is not clear where this specific number of symptoms come from? Does the dataset have 1000 symptom texts or is this randomly chosen?\nB. What is the agreement between  Chat-GPT as the 4th annotator and the other 3 human annotators?\nC. Line 172: What are these 1000 tasks?\nD. Line 214: Are these 1000 samples statistically significant? Same with Line 275.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and thoughtful review. We are fully committed to incorporating these suggestions to improve the clarity and quality of our paper. \n\n**In relation to points A and C (choice of evaluated patient reports):** \nSYMPTOMIFY has 839,215 patient reports  (symptom texts). To carry out our analysis, we selected 1000 symptom **names** at random. For each of these symptom names, we also randomly selected a corresponding patient report (symptom text) that mentions that specific symptom. Our objective was to incorporate a broad spectrum of symptoms within our evaluations, thereby avoiding any inclination towards analyzing only a skewed subset of patient reports that might disproportionately focus on common symptoms like headache or pyrexia. We have included  instructions and screenshots in the Appendix to offer further details of our methodology, and we will clarify these statements in the main paper body.\n\n**In relation to point B (agreement between ChatGPT and human annotators).** The agreement between ChatGPT and the 3 human annotators is 0.54 Fleiss' Kappa representing moderate agreement rate that is similar to related annotation tasks reported in the literature where  agreements fell within the range: 0.55-0.62 (Wadhwa et al., 2022; Nye et al., 2018; Deleger et al., 2012). Note that these prior settings differ from our setting because they do not include ChatGPT as a fourth annotator as we did here. We will clarify these details in the paper.\n\n**In relation to point D (Evaluation sample size).** \nIn terms of statistical significance, we evaluated a set of 1000 randomly selected symptoms from the total pool of 11,472 symptoms; this represents over 8% of the entire symptom set  within the SYMPTOMIFY dataset. This subset serves as a representative sample for our analysis, allowing us to draw meaningful conclusions.\n\n**Others (typos & presentation):** Thank you for this presentation improvement comment, regarding the typo on line  427.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.Motivation and background is well written. The work is also nicely situated in the relevant areas of research. \n2.The LM knowledge harvesting is interesting in this context\n3.Overall, the authors put forward an approach which utilizes the power of LMs and make AI accessible. The thought of using AI as a tool, rather than a human replacement is commendable.\n", "reasons_to_reject": "The annotation process of the Symptomify dataset is not clear. Since the main contribution of the paper is the dataset, the correctness of the annotation can be an issue. Also, there are some unanswered questions (Please see Questions for Authors). ", "questions_to_authors": "A. Line 158: It is not clear where this specific number of symptoms come from? Does the dataset have 1000 symptom texts or is this randomly chosen?\nB. What is the agreement between  Chat-GPT as the 4th annotator and the other 3 human annotators?\nC. Line 172: What are these 1000 tasks?\nD. Line 214: Are these 1000 samples statistically significant? Same with Line 275.\n", "paper_topic_and_main_contributions": "This paper is about aiding symptom annotation by taking advantage of the LMs. The main contribution is the symptomify dataset having medical reports, annotation explanations using LM knowledge harvesting and background information regarding symptoms. The authors also explore performance of various baseline models on this data.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 427: “usng”"}, "extra_text": {}}, {"review_id": "c6Dv2YNnWl", "reviewer": "Reviewer_7Q9D", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and questions. We are committed to addressing the concerns and suggestions you've raised to improve the clarity and quality of our work. \n\n**Scope of Contributions.** Our annotations of explanations and background knowledge complement MedDRA codes by providing additional context and supporting human coders in decision-making during the annotation process. Furthermore, our annotations facilitate the development of symptom recognition models capable of both prediction and explanation, a capability unavailable in the original VAERS dataset. Given the critical role of explainability in healthcare, we believe our contributions add value. We will state this with more clarity in the paper.\n\t\n**Annotations Verification Process.** The VAERS dataset has been initially annotated by trained professional MedDRA coders. Our supplementary validation relied on perspectives of three distinct groups who inspected the  VAERS annotations across varied tasks: i) crowdworkers, ii) ChatGPT, and iii) graduate students that have experience in biomedical entity linking. Across these annotation tasks, we observed agreement rates similar to related annotation tasks reported in the literature which showed agreements in the range: 0.55-0.62 (Wadhwa et al., 2022; Nye et al., 2018; Deleger et al., 2012). When we manually inspected the annotation from crowdworkers, the majority of errors come  from medical test results not explicitly mentioned in patient reports (such as C-reactive protein levels or metabolic function tests). Thus, although we lacked access to trained MedDRA coders, we adhered to (and went further than) the established standards in NLP annotation protocols.  We will clarify this in the paper.\n\n**Poor Experimental Results for Certain Baselines.** Our analysis showed that the main factor contributing to poor results is the long-tail nature in the dataset. Since most symptoms (> 80%) are rare.  Another factor influencing the low F1 scores  in classification baselines (in contrast to generative methods) is the bottleneck created by the classifier layer commonly employed in the classification setup of Language Model models (LLMs). Typically, the classification model utilizes a single aggregated vector (such as the [CLS] vector in BERT) along with an additional classifier layer positioned atop the LLM architecture. Following this practice, in our task setup, we rely on only one aggregated vector to predict outcomes across all symptom classes (a total of 11,472 classes). It becomes apparent that this approach is not sufficient to predict that many entities at once, considering that most examples within SYMPTOMIFY involve multiple symptoms. We present an analysis of this issue, outlining the notable disparity of attention patterns between classification and generation models, in Appendix A.4. We will clarify these statements in the main paper body.\n\n**Title Refinement.** We appreciate your suggestion for refining the title to accurately represent our paper's content. We will update the title to reflect this characterization of \"Leveraging Knowledge in a Large Language Model,\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The new dataset is large (800k entries) and would be an additional resource for the community.\n\n- The experiments include interesting and very up-to-date attempts, e.g. addressing rare symptoms, addressing the zero-shot scenario, and incorporating the Falcon-7B-Instruct model.", "reasons_to_reject": "- The VAERS dataset already contains MedDRA codes of symptoms. The new dataset provides textual spans and additional information of the symptoms. In my opinion, the added value is limited.\n\n- The annotations of the VAERS dataset are trained professionals. It is a good idea to verify the quality of the annotations, but this needs to be done by professionals as well, not crowd sourcing workers. Therefore, I consider the results of this verification untrustworthy.\n\n- Some of the experimental results are very poor, e.g. F1 scores as low as 1.34! Something seems wrong (model setting? bug in code?) and urgently needs further explanation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors extend the VAERS dataset with annotations of symptom information with the help of ChatGPT. In additional, the authors assess the quality and utility of the new dataset by performining the task of symptom recognition using a variety of approaches.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The title is misleading. \"Knowledge harvesting\" refers to the established task of mining large amounts of text to create a knowledge base. The way ChatGPT is used in this paper would be better characterized as \"leveraging knowledge in a large language model.\""}}, {"review_id": "mjnqTuVBlr", "reviewer": "Reviewer_jY5d", "summary": "", "questions": "1. The effectiveness and accuracy of harvesting background knowledge from LLM are uncertain. Are there problems such as hallucinations that lead to the wrong knowledge? I think these tasks require professionals to annotate.\n2. The results of different baselines are placed in different tables, so it is not easy to compare with each other.\n3. Some of the experimental results appear to be very poor, with some F1 scores less than 2.0 in Table 4 & 5.\n  ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough, and constructive review of our work. We are committed to addressing the concerns and suggestions you've raised to improve the clarity and quality of our work.\n\n**Scope of evaluation and ablations.** We agree that comprehensive experiments are a stronger contribution.  We implemented and evaluated various baselines across different methods and learning paradigms, \n*i) String Matching*: Involves matching symptom names directly with the report text based on discrete symbols.\n*ii) Symptom Embeddings*: Represents symptom names and report text as dense vectors for matching in vector space.\n*iii) Pretrain-Finetune*: Uses pre-trained language models fine-tuned on our dataset.\n*iv) Prompting*: Leverages prompts to generate responses from language models in zero-shot and few-shot settings.\n\nWe believe this variety of  baselines facilitates future comparisons and benchmarking.\n\n**Poor Experimental Results for Certain Baselines.** Our analysis showed that the main factor contributing to poor results is the long-tail nature in the dataset. Since most symptoms (> 80%) are rare.  Another factor influencing the low F1 scores  in classification baselines (in contrast to generative methods) is the bottleneck created by the classifier layer commonly employed in the classification setup of Language Model models (LLMs). Typically, the classification model utilizes a single aggregated vector (such as the [CLS] vector in BERT) along with an additional classifier layer positioned atop the LLM architecture. Following this practice, in our task setup, we rely on only one aggregated vector to predict outcomes across all symptom classes (a total of 11,472 classes). It becomes apparent that this approach is not sufficient to predict that many entities at once, considering that most examples within SYMPTOMIFY involve multiple symptoms. We present an analysis of this issue, outlining the notable disparity of attention patterns between classification and generation models, in Appendix A.4. We will clarify these statements in the main paper body.\n\n**One table for different baselines.** Our intention was to present the results through various approaches: the 'zero-shot approach' (Table 4), the 'Pretrain-finetune approach' (Table 5), and the 'Prompting approach' (Table 6). Additionally, we provide separate results for the Rare symptom test set (Table 7), which are results on a different test set unlike the others. To enhance readability, we will include a comprehensive table that allows for a consolidated view of all comparisons in a single place.\n\n**Types of annotators used.** The VAERS dataset has been initially annotated by trained professional MedDRA coders. Our supplementary validation relied on perspectives of three distinct groups who inspected the  VAERS annotations across varied tasks: i) crowdworkers,  ii) ChatGPT, and iii) graduate students that have experience with biomedical entity linking.   Across these annotation tasks, we observed agreement rates similar to related annotation tasks reported in the literature which showed agreements in the range: 0.55-0.62 (Wadhwa et al., 2022; Nye et al., 2018; Deleger et al., 2012). When we manually inspected the annotation from crowdworkers, the majority of errors come  from medical test results not explicitly mentioned in patient reports (such as C-reactive protein levels or metabolic function tests). Thus, although we lacked access to trained MedDRA coders, we adhered to (and went further than) the established standards in NLP annotation protocols. We will clarify this in the paper.\n\n**The quality of harvested knowledge from LLM.** The explanations we harvested from the LLMs are encouraged to cite segments of mentions from the patient reports, thus this grounding of mentions of symptoms discourages hallucination. We found that ChatGPT avoided hallucination  in explanations especially with prompts that included this grounding requirement, whereas the open source, smaller Falcon LLM  struggled with hallucinations even with prompts that required grounding; in the paper, we  provided examples of Falcon hallucinations of spans in Table 8 on page 8.\n\n**Sizes of Figures 2 and 3:** Thank you for this presentation improvement comment, we will change their sizes to be consistent. Figure 2 has less content, thus it is currently smaller, but we agree that the current presentation and contrast with the much larger Figure 3 is not ideal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The article introduces a new method using language model knowledge to improve the SYMPTOMIFY dataset. This method expands existing research on symptom recognition and has the potential to advance medical decision-making.\n2. By incorporating LLM, the article improves the SYMPTOMIFY dataset by adding explanations and background information. This enhancement provides valuable context and detailed information for symptom recognition, improving the overall quality and usefulness of the dataset.", "reasons_to_reject": "1. The evaluation of the proposed method is limited, and it would be beneficial to have more extensive evaluation and ablation studies.\n2. The idea of validating the quality of annotations is good, but also needs to be done by professionals, not crowdsourcing. Therefore, the verification results cannot be trusted.", "questions_to_authors": "1. The effectiveness and accuracy of harvesting background knowledge from LLM are uncertain. Are there problems such as hallucinations that lead to the wrong knowledge? I think these tasks require professionals to annotate.\n2. The results of different baselines are placed in different tables, so it is not easy to compare with each other.\n3. Some of the experimental results appear to be very poor, with some F1 scores less than 2.0 in Table 4 & 5.\n  ", "paper_topic_and_main_contributions": "The paper introduces SYMPTOMIFY, a dataset of annotated vaccine adverse reaction reports that aims to improve the efficiency of human annotators in healthcare decision-making. The dataset contains over 800k reports, reasoning-based explanations, and background knowledge obtained from language model knowledge harvesting. The paper evaluates various baselines across different learning paradigms and provides insights for future comparisons and benchmarking.\n**Contribution:**\n- The paper addresses a significant problem in healthcare decision-making and proposes a dataset that can aid human annotators.\n- The dataset is comprehensive, featuring a large number of reports and including explanations and background knowledge.\n- The paper provides insights and evaluation results for various baselines and learning paradigms, which can guide future research and comparisons.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The size of Figure 2 and Figure 3 can be adjusted to be consistent.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
In this paper, the authors introduce SYMPOMIFY, an extension of the VAERS dataset that includes annotations of symptom information. These annotations were generated using Large Language Model (LLM) knowledge harvesting, particularly with the assistance of ChatGPT. The dataset comprises over 800,000 reports, along with reasoning-based explanations and background knowledge. Additionally, the authors evaluate the quality and utility of the new dataset by conducting symptom recognition tasks across various learning paradigms, providing insights for future comparisons and benchmarking.

**Strengths:**
The reviewers unanimously agree on the following strengths:
1. The article introduces the SYMPTOMIFY dataset which incorporates explanations and background information, offering valuable context and detailed data for symptom recognition, thereby improving the overall quality and utility of the dataset.
2. The newly introduced dataset in this work is notably substantial, with over 800,000 entries, providing an additional and valuable resource for the research community.
3. The experiments conducted in the paper are both interesting and up-to-date, including efforts to address rare symptoms, address zero-shot scenarios, and incorporate the Falcon-7B-Instruct model.
4. The method outlined by the authors expands the existing field of the research on symptom recognition and holds the potential to advance medical decision-making.
5. The paper offers valuable insights and evaluation results for various baselines and learning paradigms, serving as a guide for future research and comparisons.
6. The authors' approach harnesses the capabilities of Large Language Models (LLMs) to make AI an accessible tool, rather than a human replacement.

**Weaknesses:**
All the reviewers share the same concerns regarding this work. 
First and foremost, the annotations in the VAERS dataset are carried out by trained professionals. While it's a commendable idea to assess the quality of these annotations, it should ideally be conducted by experts rather than crowdsourced workers. Consequently, the results of the verification in this work may lack reliability. Given that the primary contribution of the paper is the dataset itself, the accuracy of the annotations becomes a critical concern.
Additionally, the evaluation of the proposed method is somewhat limited, and it would be advantageous to have a more extensive evaluation and conduct ablation studies.

**Author-Reviewer discussion and acknowledgment:**
The authors have provided clarifications in response to the concerns raised by the reviewers and have outlined the planned improvements to be made during the rebuttal response and discussion phase. All reviewers have responded and acknowledged the authors' arguments.

**Conclusion:**
The paper is well-motivated, and the background is well-written. The work is also appropriately contextualized within the relevant areas of research. However, reviewers suggest that the authors correct the identified typos. Furthermore, reviewers recommend that the authors improve the paper by addressing the questions and points raised during the discussion phase.
Decision: Accept-Findings