SlowGuess/NAACL_2022_OCR huggingface dataset:
- Include fragmented annotations by excluding the text that is not relevant to the feature (e.g., if the feature is No relief with Imodium or Cipro, only the underlined text of the following excerpt should be annotated: Has tried Imodium (aggravated condition), and Cipro  $250\mathrm{mg}$  BID (has taken 9 tablets) from prior episode of diarrhea in Kenya of lesser severity (no effect))
Each feature should be marked up as a separate annotation, and the annotation should include all, but not more than, the text that captures the meaning of the corresponding entry in the feature (e.g., if the key essential is No blood in stool, only the underlined text of the following excerpt should be annotated: No blood or mucus in stool).
- Annotations should include quantifiers (e.g., twice, four times, some), intensifiers (e.g., mild, severe), and temporal modifiers (e.g., two weeks, several years) that are specified in the corresponding entry in the feature, as well as the object that is being described (e.g., pain, cough).
- Annotations should not include articles (e.g.,  $a$ ,  $the$ ) or references to the patient (e.g.,  $her$ ,  $he$ ) that occur at the beginning of note entries, or end punctuation (e.g., periods); however, it is not necessary to fragment annotations if words or characters, such as these, occur within relevant text and do not modify the meaning of the feature entry.
- Annotations may overlap; that is, they may share text with other annotations. For example, negations (e.g., negative for, no, denies) frequently will be shared among several annotations. In the phrase Negative for fever, chills, nausea, vomiting, hematochezia, the negated nouns refer to different features and should be annotated as Negative for fever, Negative for chills, Negative for nausea, etc.
- Mark up every instance of the feature whether it is identical to an existing annotation or not.
History: Describe the history you just obtained from this patient. Include only information (pertinent positives and negatives) relevant to this patient's problem(s).
Karin Moore is a 45 yo F here for nervousness. A few weeks ago she noticed that she was feeling more nervous than usual and that it has been worsening. It is exacerbated by family and work. She feels especially nervous on Sunday night and Monday morning when she is preparing for the week. She is unable to fall asleep and doesn't want to eat anything, though she does make herself eat. Nothing helps her nervousness. She otherwise denies significant changes in appetite, weight loss, or overall wellbeing. She denies fevers, chills, nausea, constipation, diarrhea, skin changes, racing heart, shortness of breath, dizziness, headaches or rashes.ROS: otherwise negativePMH: None; PSH: NoneMeds: Tylenol for occasional HAFHX: Father had an MI, died at 65yoAllergies: NKDASH: Lives at home with husband, mother, and youngest son. Is an english literature professor at a local college.Has 2 drinks/mo, no tobacco or drug use.
Physical Examination: Describe any positive and negative findings relevant to this patient's problem(s).Be careful to include only those parts of examination you performed in this encounter.VS: Blood Pressure: 130/85 mm HgHeart Rate: 96/minGen: No acute distress, conversational, thinNeck: No thyromegaly, no lymphadeopathyHeart: RRR, no murmurs, rubs or gallops. Radial pulses +2 bilaterallyLungs: Clear to ascultation bilaterally, no wheezesPsych: Well-groomed. Non-pressed speech, linear though process.
Data Interpretation: Based on what you have learned from the history and physical examination, list up to 3 diagnoses that might explain this patient's complaint(s). (...)General anxiety disorderPanic disorderHyperthyroidism
For example, if the feature is NSAID-use and the examinee wrote Uses NSAIDs as well as took ibuprofen, both snippets of text should be annotated. If the exact snippet Uses NSAIDs appeared more than once in a note, it should be annotated every time it appears in the note.
- Gender is a special case of a feature and should only be annotated once for the first mention. Subsequent phrases that may be linked to gender such as she or his should not be annotated.
C Inter-annotator Agreement Per Case
Table 1: Example of a PN. The dataset features only the history portions of the PNs.
Case	f1	Jaccard	κ	yes_no_f1
201	.72	77.37	.82	.96
202	.90	91.48	.93	.98
203	.93	94.93	.96	.99
204	.92	93.05	.95	.99
205	.71	72.61	.78	.94
206	.88	90.05	.93	.99
207	.79	86.09	.89	.98
208	.85	87.02	.89	.97
209	.88	86.40	.89	.97
210	.87	86.54	.89	.97
Mean	.84	86.55	.89	.97
SD	.075	6.89	.057	.014
Table 2: Inter-annotator agreement per case, where the gold standard is the annotation of the senior annotator. The columns represent (in order): Micro F1 character-position based agreement, Jaccard distance, Cohen's  $\kappa$  coefficient, and a binary F1 score for whether the annotators agree that a given feature expression was found in a PN (1 if found, 0 if not found). As can be seen, the annotators agree very well on whether a feature was found in a PN or not, with some differences in agreement about the exact span of characters that represent that feature.