## 3 Results  
Table 1 lists the characteristics of the clinical notes selected for review. The number of progress notes within 9 days following admission ranged from 7 to 17. The average number of lines per progress note varied from 15 to 35 and increased with the number of progress notes (Figure 2A). The inter-rater agreement (kappa value) for manual annotation between the two attending physicians was 0.767, indicating substantial agreement. Based on the results of manual annotation, 34% to 78% of the lines of the progress notes were determined to contain new information. The proportion of lines with new information was negatively correlated with the average number of lines per progress note (Figure 2B).  
\begin{table}
\begin{tabular}{c c c c c c} \hline  & **Total Lines in** & **Number of** & **Total Lines in** & **Lines per Progress** & **Lines with New** \\  & **Admission Note** & **Progress Notes** & **Progress Notes** & **Note, Mean** & **Information** \\ \hline Case 1 1  & 74 & 11 & 248 & 23 & 60\% \\ Case 2 & 47 & 8 & 183 & 23 & 58\% \\ Case 3 & 36 & 8 & 120 & 15 & 78\% \\ Case 4 & 117 & 9 & 142 & 16 & 66\% \\ Case 5 1  & 57 & 9 & 191 & 21 & 61\% \\ Case 6 1  & 31 & 13 & 286 & 22 & 40\% \\ Case 7 & 83 & 17 & 593 & 35 & 48\% \\ Case 8 & 69 & 12 & 379 & 32 & 34\% \\ Case 9 & 55 & 7 & 178 & 25 & 49\% \\ Case 10 1  & 58 & 9 & 187 & 21 & 39\% \\ \hline \end{tabular}
\end{table}
Table 1: Characteristics of the clinical notes selected for review.  
Table 2 gives the performance of the automated identification of new information across different numbers of preceding notes used in the bigram language model. The highest F1 score (0.833) and accuracy rate (0.814) were achieved when at most four preceding notes were employed to build the bigram language model. Therefore, the optimal number (\(N\)) of preceding notes was set to 4.  
Clinical notes from case 1, 5, 6, and 10 (Table 1) were selected for the user experiment. The new information in each note was highlighted using the bigram language model mentioned above. Four physicians and eight nurse practitioners participated in the experiment. Table S1 lists the characteristics of the participants. Table 3 gives descriptive statistics of task demands, performance, and perceived workload for each testing scenario. No significant differences in task demands were observed between scenarios as quantified by the time to completion as well as the total number of mouse clicks, mouse wheels, or mouse movements.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Previous Notes, \(N\)** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline
1 & 0.764 & 0.690 & 0.953 & 0.801 \\
2 & 0.792 & 0.723 & 0.945 & 0.819 \\
3 & 0.811 & 0.747 & 0.938 & 0.832 \\
4 & 0.814 & 0.751 & 0.936 & 0.833 \\
5 & 0.814 & 0.754 & 0.929 & 0.832 \\
6 & 0.810 & 0.752 & 0.921 & 0.828 \\
7 & 0.809 & 0.753 & 0.918 & 0.827 \\
8 & 0.808 & 0.752 & 0.916 & 0.826 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of automated identification of new information using manual annotation as the reference standard by increasing the number of previous notes.  
Figure 2: The relationship between lines per progress note and number of progress notes (**a**) and the relationship between lines with new information and lines per progress note (**b**).  
As for task performance, there was no difference between scenarios in the sub-scores for questions regarding the admission note. In contrast, the overall scores and the sub-scores for questions regarding the progress notes were significantly higher in the testing scenario with notes in the highlighted condition. The overall perceived workload of reviewing highlighted notes was significantly lower than that of reviewing original notes. The workload in different dimensions of the NASA-TLX decreased significantly except for the dimension of perceived overall performance.