SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 5: Selected spans among top 100 negative scores for Amazon
Span Text	Score
was terrible it tasted like we were licking an ashtray. it has a burnt grounds flavor. i highly recommend not wasting your money on this product.	-7.30
the one can i tasted and threw out to the food pantry.	-7.30
really stale items from amazon.com and this was one. unedible. beware of the quality of food items on this website that are on special as they can be very close to due dates or in this case, not expired but stale and unedible just the same.	-7.30
this product claims and hours of entertainment. my dog had it completely destroyed on 20 minutes. i'm completely disappointed.	-7.30
life threatening . undigested pieces of these chews were in his waste. i do not reccomend	-7.30
this is disgusting, it doesn’t taste like watermelon at all. it’s actually a blend of several different juices, plus the ascorbic acid, and the blend does not meld at all. it’s just bitter, overly sweet, and has a nasty aftertaste.	-7.29
Table 6: Selected spans among top 100 positive scores for ASD
Span Text	Score
subjective intake chief complaint problems with sleep, inattention, and beha-ioral concerns both in the home and school setting. DATE, recently more anger and recent tic like behavior	+6.95
psychologist presenting problem NAME is a 3 year, 4 month old female who was referred for a neurodevelopmental assessment due to concerns regarding her overall development, behavior, and social emotional functioning and to assess for autism spectrum disorder	+6.82
problem list diagnosis • disruptive behavior disorder • impaired speech articula-tion • daytime enuresis • other subjective visual disturbances • hypermetropia of both eyes • ADHD attention deficit	+6.81
problem list diagnosis • anemia of prematurity • history of colitis • meconium tox for the • extreme immaturity of newborn, 27 completed weeks • nasal congestion of newborn • presumed	+6.78
motor delay DATE • hypotonia DATE • clasped thumb DATE • polydactyly DATE • developmental	+6.74
therapy NAME was seen for developmental support during rop eye exam today. the	+6.65
Table 7: Selected spans among top 100 negative scores for ASD
Span Text	Score
subjective NAME is a 5 y.o. female who presents for her 5 year well child visit. history was obtained today by father. concerns had om a few weeks ago. check her throat.	-5.91
CLINIC sick visit patient active problem list diagnosis • routine child health maintenance chief complaint patient presents with • fussy x several days. dad ? possible ear infection hpi has never	-5.90
NAME is a male child here for his 15 month well child visit. concerns none diet varied voiding and stooling well. past medical history active ambulatory	-5.87
evaluation was performed today unless otherwise noted. assessment encounter diagnosis name primary? • regular astigmatism of both eyes yes plan 1. astigmatism	-5.83
breast bottle vitamins formula no 0 oz. per feeding of feedings in 24 hours 0 solids yes juice no elimination patterns loose sleep sleeps all night development	-5.81
subjective is a 17 m.o. male and is here for a well child visit. history was obtained today by is 17 months old and is here for a 15 month exam. he is doing well.	-5.76
Table 8: Selected spans among top 100 positive scores for ADHD
Span Text	Score
behavioral parent training patient and family response to interventions we discussed parenting stress and consistent plan to move to pdi 4 next week. objective mental status exam behavioral	+5.33
sensory disorder subjective pain assessment no pain. patient caregiver comments NAME is reportedly behind in reading is likely going to need summer school reports of difficulty keeping place when reading. family considering testing to rule out ADHD. objective goals demonstrate improved	+5.23
outpatient prescriptions on file prior to visit medication sig dispense refill • clonidine hcl catapres 0.1 mg tablet take 0.1 mg by mouth nightly. 2 • melatonin 3 mg tablet take 3 mg by mouth nightly. • methylphenidate concerta 54 mg	+5.18
list diagnosis • dyslexia, developmental • intermittent asthma • right elbow pain • food allergy • fire ant sting past medical history active ambulatory problems diagnosis date noted • dyslexia, developmental DATE • intermittent asthma DATE	+5.18
diagnosis • gestational age, NUMBER weeks • apnea of prematurity • breech presentation • unconjugated hyperbilirubinemia	+5.15
5 y.o. male who presents with h o developmental delay, speech disorder, sensory and fine motor disorders, challenging behavior and who would likely continue to benefit from continued evaluation to address identified concerns. will obtain information to assist r o ADHD	+4.92
Table 9: Selected spans among top 100 negative scores for ADHD
Span Text	Score
5 y.o. female presenting with 3 days of runny nose, congestion, sore throat, cough. today she woke up with a little bit of drainage from her right eye. as the day has gone	-3.08
presents for an established patient office visit here for wcc. is doing well hopes to go to early hs past medical history past medical	-3.06
y.o. female is here today for the influenza vaccine. vaccine administered today influenza quad patient guardian reviewed or provided the hard copy of the YEAR influenza vaccine information	-3.01
diagnosis • healthy infant or child • abdominal pain, periumbilical current outpatient	-2.99
3 y.o. female here for evaluation of woke up with abdominal pain temp to 100 at daycare mother says she vomited x 1 this morning her sister was seen last week for strep patient active	-2.95
subjective pain assessment no pain. patient caregiver comments NAME is doing very well in swim and basketball. he coped very well when parents were recently out of the in the care of extended family. objective goals NAME will... 1. open snack	-2.91
Table 10: Selected spans among top 100 positive scores for Asthma
Span Text	Score
history diagnosis date • rad reactive airway disease , unspecified history re- viewed. no pertinent surgical history. family history problem relation age of onset • asthma mother outpatient prescriptions marked as taking for the DATE encounter office visit with NAME medication sig • albuterol	+7.75
pt goal home airway clearance dates start DATE, description patient will in- crease airway clearance at home to three times a day when	+7.74
pediatric icu progress note DATE hospital day 1 icu admission indication 3 m.o. female with principal problem respiratory distress active problems hypotonia laryngomalacia chromosomal abnormality NAME is a 3	+7.68
patient active problem list diagnosis • gestation period, 28 weeks • respiratory insufficiency • breech birth overnight none medications caffeine citrated 5 mg kg	+7.63
CLINIC sick visit patient active problem list diagnosis • tof tetralogy of fallot • sacral dimple • chromosome abnormalities chief complaint patient presents with • nasal congestion • cough • sneezing • breathing problems hpi NAME is	+7.58
inhalation started as ordered and held near nose and mouth for ventilation administration.	+7.53
Table 11: Selected spans among top 100 negative scores for Asthma
Span Text	Score
vaccine less than 7yo im • hib prp omp conjugate vaccine 3 dose im pedvaxhib • pneumococcal conjugate vaccine 13 valent im prevnar 13 • hepatitis a vaccine pediatric	-5.67
19 m.o. here today with a red swollen slightly tender distal right 4th finger. patient injured that finger in a cabinet roughly 6 days ago. last 2 to 3 days it	-5.66
20 m.o. male who presents today for the evaluation of chief complaint patient presents with • cough • nasal congestion history was obtained today by father. uri symptoms for gt 1 week. no worse but	-5.65
motor runs and climbs well throws a ball stacks 3 or more blocks fine motor uses spoon and cup scribbles, tries to use	-5.61
21 m.o. with problems with gait and balance, some crying at night. grandmother thinks he is having difficulty with constant falling and running with some bruising of his head or face. no limp, no deformity or swelling. he has	-5.60
20 m.o. female brought in by mother. hpi NAME presents with a 2 days history of the fever, with maximum temperature of 104. she was seen in er 2 days ago and diagnosed with viral illness. she is still running fever, has runny nose and is fussy.	-5.60
Appendix B. AUC, span size, and span overlap with  $J = 10$
Figure 5 illustrates the performance of our model for a fixed value of  $J = 10$ , larger than that chosen for each dataset in the main paper (4, 3, and 7 for IMDb, Amazon, and the health datasets, respectively). While AUC is generally higher for each dataset compared to that obtained with a smaller value of  $J$ , we find that span length and overlap are now more sensitive to  $\theta$  and drop more rapidly as  $\theta$  is increased. In practice, we employ smaller values of  $J$  and adjust  $\theta$  to achieve a desired level of span size and overlap to (1) allow for finer control of the tradeoff in performance, span size, and span overlap, and to (2) avoid overparameterizing our model.
(b) Average span size
(c) Average intersection-over-union
Figure 5: Trends in performance. Baseline AUCs - IMDb: 0.938, Amazon: 0.931, ASD: 0.702, ADHD: 0.804, Asthma: 0.630. Performance tends to drop slightly as  $\theta$  is decreased, but spans become more concise and distinct.
Appendix C. Training loss vs. epoch
In Figures 6 through 20, we show the traces of training loss as a function of epoch for each experiment. For all models except the baseline, we separate our loss into two components: one for the negative log likelihood (denoted "loglik") and another for the negative JSD (denoted "uniqueness"). In each experiment we find that the lower bound (LB) for negative JSD is not violated, providing experimental support for our proof of an upper bound on the modified JSD. Note that the bottom right subplot in each non-baseline model – titled "span_size" – can be ignored as this is related to a feature that was ultimately not incorporated into the model. There is no contribution to the total loss from this component; hence, the value is zero across all epochs.
Figure 6:IMDb:Baseline
(a) No penalty
(b)  $\theta = 0.5$ $\mathrm{LB} = -1.39$
(c)  $\theta = 0.475$ ,  $\mathrm{LB} = -1.32$
(d)  $\theta = 0.45$ $\mathrm{LB} = -1.25$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.11$
(f)  $\theta = 0.25$ $\mathrm{LB} = -0.69$
Figure 7:IMDb:  $J = 4$
(c)  $\theta = 0.475$ $\mathrm{LB} = -2.19$
(d)  $\theta = 0.45$ $\mathrm{LB} = -2.07$
(b)  $\theta = 0.5$ ,  $\mathrm{LB} = -2.30$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.84$
Figure 8:IMDb:  $J = 10$
(f)  $\theta = 0.25$ ,  $\mathrm{LB} = -1.15$
Figure 9: Amazon: Baseline
(b)  $\theta = 0.5$ $\mathrm{LB} = -1.10$
(c)  $\theta = 0.475$ ,  $\mathrm{LB} = -1.04$
(d)  $\theta = 0.45$ $\mathrm{LB} = -0.99$
(e)  $\theta = 0.4$ $\mathrm{LB} = -0.88$
(f)  $\theta = 0.25$ $\mathrm{LB} = -0.55$
Figure 10: Amazon:  $J = 3$
(b)  $\theta = 0.5$ $\mathrm{LB} = -2.30$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.84$
Figure 11:IMDb:  $J = 10$
Figure 12: ASD: Baseline
(b)  $\theta = 0.5$ ,  $\mathrm{LB} = -1.95$
(c)  $\theta = 0.475$ ,  $\mathrm{LB} = -1.85$
(d)  $\theta = 0.45$ $\mathrm{LB} = -1.75$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.56$
(f)  $\theta = 0.25$ $\mathrm{LB} = -0.97$
Figure 13: ASD:  $J = 7$
(c)  $\theta = 0.475$ ,  $\mathrm{LB} = -2.19$
(b)  $\theta = 0.5$ $\mathrm{LB} = -2.30$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.84$
Figure 14: ASD:  $J = 10$
Figure 15: ADHD: Baseline
(b)  $\theta = 0.5$ $\mathrm{LB} = -1.95$
(c)  $\theta = 0.475$ $\mathrm{LB} = -1.85$
Figure 16:ADHD:  $J = 7$
(b)  $\theta = 0.5$ ,  $\mathrm{LB} = -2.30$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.84$
Figure 17:ADHD:  $J = 10$
Figure 18: Asthma: Baseline
Figure 19: Asthma:  $J = 7$
(d)  $\theta = 0.45$ ,  $\mathrm{LB} = -2.07$
(b)  $\theta = 0.5$ $\mathrm{LB} = -2.30$
(e)  $\theta = 0.4$ $\mathrm{LB} = -1.84$
Figure 20: Asthma:  $J = 10$
Appendix D. Code
The IMDb and Amazon datasets were processed using the following scripts:
- imdb_baseline.ipynb - to train baseline model on IMDb and obtain AUC
- imdb_main.ipynb - to train SpanPredict model on IMDb and obtain (1) AUC and (2) average span length
- imdb_spans.ipynb - to obtain spans for human evaluation and supplementary material from IMDb
- amazon基礎.ipynb - to train baseline model on Amazon Food Reviews and obtain AUC
- amazon_main.ipynb - to train SpanPredict model on Amazon Food Reviews and obtain (1) AUC and (2) average span length
- amazon_spans.ipynb - to obtain spans for human evaluation and supplementary material from Amazon Food Reviews
- imdb_amazon_IoU.ipynb - to obtain average intersection over union from both datasets on non-baseline models
The ASD, ADHD, and Asthma datasets were processed using the following scripts:
- asd_adhd_asthma_baseline.ipynb - to train baseline model on ASD / ADHD / Asthma datasets and obtain AUC
- asd_adhd_asthma_main.ipynb - to train SpanPredict model on ASD / ADHD / Asthma datasets and obtain (1) AUC and (2) average span length
- asd_adhd_asthma_spans.ipynb-to obtain spans for human evaluation and supplementary material from ASD /ADHD/Asthma datasets
- asd_adhd_asthma_IoU.ipynb-to obtain average intersection over union from ASD /ADHD/Asthma datasets on non-baseline models
To train a SpanPredict model from scratch, use the {imdb, amazon}__main script.  $J$ ,  $\alpha$ , and  $\theta$  can be set using the num_spans, uniqueness_weight_, and JSD_weight fields of the model_options variable, respectively. Note that JSD_weight is defined as a tuple:  $(\theta, 1 - \theta)$ . For instance, to set  $J = 10$ ,  $\alpha = 0.1$ , and  $\theta = 0.25$ , use: model_options = {'num_spans': 10, 'uniqueness_weight': 0.1, 'JSD_weight': (0.25, 0.75), ...}.
All other parameters of model_options and all parameters of training_options can be left as is. This will create a database which stores every span extracted from every document in the testing corpus. The {imdb, amazon}_spans and {imdb, amazon}_IoU scripts can then be used to obtain (1) example spans (e.g., those tabulated in Appendix A and those selected for human evaluation) and (2) intersection-over-union scores, respectively.
Note that due to privacy restrictions, we are unable to share models trained on the ASD, ADHD, and Asthma datasets. However, we provide baseline, no penalty, and  $\theta = 0.5$  model checkpoints for the IMDb and Amazon datasets (with  $J = 4$  and  $J = 3$ , respectively, for the non-baseline models). These were produced using the {imdb, amazon} _main script and can be loaded by making the appropriate modifications.
For all our models, we employ 100 dimensional GloVe embeddings, which can be found here: http://nlp.stanford.edu/data/glove.6B.zip.