SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Top 20 section titles extracted by the original DF-IAPF algorithm (Raw) and titles selected by medical experts based on Raw (Selected).
Rank	Raw	Rank	Selected
1	history of present illness	1	history of present illness
2	date of birth	2	date of birth
3	sex f	3	sex
-	sex m	-
4	discharge date	4	discharge date
5	admission date	5	admission date
6	social history	6	social history
7	past medical history	7	past medical history
8	discharge medications	8	discharge medications
9	medications on admission	9	medications on admission
10	discharge diagnosis	10	discharge diagnosis
11	discharge condition	11	discharge condition
12	discharge instructions	12	discharge instructions
13	major surgical or invasive procedure	13	major surgical or invasive procedure
14	brief hospital course	14	brief hospital course
15	pertinent results	15	pertinent results
16	followup instructions	16	followup instructions
17	family history	17	family history
18	chief complaint	18	chief complaint
19	attending	19	attending
20	physical exam	20	physical exam
extracted by DF-IAPF in the top 20 section titles appear in the top 30 titles. However, the titles that are missing in the rule-based method have very low ranks. It shows that even though the rules are carefully designed by humans, they may not be applicable to all clinical notes or titles. Therefore, we can conclude that our DF-IAPF algorithm is more universal than the rule-based method since it can effectively locate section titles and require less human effort.
Quantitative analysis To numerically demonstrate the effectiveness of our proposed DF-IAPF algorithm, we randomly select 50 clinical notes and manually extract the section title set  $\Omega_{i}$  for each clinical note by medical experts. To evaluate the coverage of the top-20 extracted section titles  $\hat{\Omega}$  by DF-IAPF and the rule-based method, we use an average intersection rate between  $\Omega_{i}$  and  $\hat{\Omega}$ :  $\frac{1}{50} \sum_{i=1}^{50} \frac{|\Omega_{i}| \cap |\hat{\Omega}|}{|\Omega|}$ . The rate of DF-IAPF is 0.87, while the rate of the rule-based method is 0.83. The rates are less than 1 due to the absence of the bottom 3 titles in Table 4. Additionally, some clinical notes contain less frequent titles including "facilities", "addendum", etc. However, the rate of DF-IAPF is still higher than the rule-based method because "chief complaint", "discharge date", and "sex" are all top frequent section titles, while "discharge disposition" is a relatively less frequent title. Moreover, we report the frequency of section titles after segmentation using the 23 section titles in Table 4. We can see that all section titles have high frequencies. Together with the intersection rate, it further proves the coverage and accuracy of the extraction algorithm. Note that the rank of the section titles extracted by the rule-based method is different from the order of frequencies. This is because the rank is determined by the number of extracted section titles based on colons before segmentation. However, not all section titles are followed with a colon. Therefore, after segmentation, the frequencies may be different from title extraction.
It is worth noting that the top 20-30 titles mainly contain some special tokens, such as “[**first name3**]”, which are masked tokens in the original dataset for privacy concerns. In the contrastive learning part, we do not use sections that have little relation to ICD codes, including “date of birth”, “sex”, “admission date”, “discharge date”, “attending” and “service”, and use the remaining titles to pre-train the clinical note encoder. In the training of ICD coding models, we use all 23 section titles (top 20, 23, 28, and 29) so that we make the least change to the completeness of clinical notes. For some less frequent section titles such as “addendum” mentioned before, we do not segment sections by applying them as separators, but merge them with adjacent sections. In this way, the content of these sections is reserved for training.
Table 6: Results (\%) of MIMIC-full when trained with/without the proposed contrastive pre-training and masked training (CM) strategies. Cells with the green color denote an improvement of w/ CM compared to w/o CM. Here, we do not provide a  $p$ -value since we run backbone models one time.
Model	w/o CM	w/ CM
Macro F1	Micro F1	P@8	P@15	Macro F1	Micro F1	P@8	P@15
MultiResCNN	8.5	55.2	73.4	58.4	9.3	55.9	74.0	58.8
HyperCore	9.0	55.1	72.2	57.9	9.6	55.6	73.0	58.5
JointLAAT	10.7	57.5	73.5	59.0	11.5	58.3	73.9	59.4
EffectiveCAN	10.6	58.9	75.8	60.6	11.3	59.4	76.2	61.1
PLM-ICD	10.4	59.8	77.1	61.3	10.6	60.0	77.2	61.5
MSMN	10.3	58.4	75.2	59.9	11.4	58.8	75.6	60.2
Role of medical experts discussion In Section 4.1, we mentioned that the selection of the extracted section title is performed by medical experts. To eliminate the selection bias, we list the originally extracted titles by our algorithm and the selected titles by medical experts in Table 5. We can see medical experts only need to correct "sex m" and "sex f". Since the extracted titles are mostly correct, there is actually little effort required by medical experts. Therefore, the role of medical experts in this process is to validate the extracted titles by the proposed DF-IAPF method, which further evaluates the effectiveness and accuracy of the DF-IAPF method.
C.3 Results of MIMIC-full prediction
We report the results of MIMIC-full in Table 6. Here, w/o CM and w/ CM mean the results without and with the proposed CM strategies, respectively. In this task, we directly use the w/o CM results from the MSMN paper [30]. For the w/ CM results, we report the result of one run since this experiment requires a lot of time. For the results of w/o CM, all the backbone models have a relatively low Macro  $F_{1}$  score due to the large size of the label set and long tail distribution of ICD codes, while PLM-ICD is the best in terms of Micro  $F_{1}$ , P@8, and P@15. As for the result w/ CM, the cells with green color indicate an improvement. From the comparison, we notice the proposed contrastive pre-training and masked training can improve the performance of the backbone models, among which the Macro  $F_{1}$  score is increased by  $7.1\%$  on average. However, the PLM-ICD model does not improve as much as other backbone models. We infer it is because the PLM-ICD model already split clinical notes into chunks with a fixed length. Even with our training strategies, it somewhat breaks the information between sections so that the variability cannot be largely reduced by our proposed training strategies.