Language Models for Fall Risk Assessment in Children with Cerebral Palsy using Electronic Medical Records

Authors: thasina  tabashum
karen m kruger
jon r davids
susan e sienko
shou-jen  wang
adam  graf
anita  bagley
jeremy  bauer
joseph j krzak
ross s chafetz
jason  woloff
mark v albert
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-13 01:25:54

Abstract: Children with Cerebral Palsy (CP) face a heightened risk of falls, complicating treatment outcomes. Traditional manual scoring methods like the Cummings Fall Assessment Score are subjective and labor-intensive due to the diverse characterization of CP. Leveraging electronic medical records (EMRs) and advanced language models (LMs) offers a data-driven alternative for fall risk assessment. Because CP is a varied heterogeneous cohort LMs have not been thoroughly applied to assess fall risk. To address this, we utilized unstructured EMR data from 1,604 patients with CP from the Shriners Children’s Hospital Network, employing Clinical BioBERT, BioBERT, and BERTbase to predict fall risk. We explored two approaches: continued pre-training followed by fine-tuning with labeled data, and direct fine-tuning with supervised labeled data. Our findings indicate that continued pre-training does not guarantee performance improvements on downstream tasks, relative to only fine-tuning. This reduces the need for an extensive pre-training process. Only fine-tuned models were able to achieve a 0.71 F1 in prediction fall assessment risk, and 0.74 F1 scores when we excluded the borderline fall assessment score during training. The best performance achieved by Clinical BioBERT is a recall of 0.72 and a specificity of 0.80. Furthermore, CP is a complex, multifaceted condition often involving lengthy clinical notes that exceed 30,000 characters, which many LMs cannot process in a single context window. We propose a process where notes are assessed as a group of samples that fit the context window. Then a collective decision is made by probability weighted majority voting (PWMV). This approach improved model performance by 1\% to 3\% and demonstrated its effectiveness in enhancing fall risk prediction for children with CP. Our work lays the groundwork for evidence-based, data-driven treatment planning in pediatric CP clinical practice and research, significantly improving the efficiency and accuracy of CP patient care.

Main paper: 
 i. introduction: 
 Cerebral palsy (CP) is a lifelong movement disorder that affects individual muscle control. The severity and specific symptoms of CP vary widely among individuals [1] . One major concern among individuals with CP is falls, previous studies showed that children with CP are more prone to falls [2] . Falls represent a significant cause of death and injury in pediatric hospitals, accounting for 25% to 52% of all treated child injuries [3] . Furthermore, falls can result in injuries that deteriorate the subject's condition. According to Jamerson et al., over half of the children who fell were not recognized as high-risk before the incident occurred [4] . This points to a gap in the fall risk assessment process, and standardized fall assessment tools can assist clinicians in evaluating the individual's conditions.Assessment is important in CP management, including diagnosis, progress monitoring, quality of life, social skills, motor abilities, and more [5] . Continuous re-intervention and re-evaluation are needed for the dynamic nature of CP, as a result maintaining consistent assessment by different clinicians is challenging [6] , [7] . To assess individuals with CP, clinicians maintain a large amount of documentation that can be found in electronic medical records (EMR). EMR contains rich information about subjects growing at a fast rate in clinical practice [8] . Traditional practices can be significantly improved with the integration of artificial intelligence (AI) and EMR [9] , [10] . As recent advancements in language modeling in numerous domains showed promising results, these language models (LMs) can be valuable in incorporating significant information on the subject's overall condition to develop fall risk assessment tools.One major transformer-based [11] LM is Bidirectional Encoder Representations from Transformers (BERT) [12] that showed promising results in the different avenues of natural language tasks such as question answering, natural language inference, and translation. BERT has also gained popularity in the medical domain, for instance, BioBERT is trained on medical literature [13] , and performed well in medical entity recognition, question answering, and relation extraction. BERT also presented an ideal performance on longitudinal structured EMR data, such as Med-BERT [14] , BEHRT [15] , CEHR-BERT [16] , and HiBERT [17] . A specialized clinical model on unstructured EMR data has also been proposed Clinical BioBERT [18] on intensive care unit (ICU) datasets, MIMIC [19] . Previously, BERT-based inpatient fall detection is proposed using Alberta hospital notes [20] . Although there are significant advancements in LMs, no previous LM addresses CP concerns. To address this limited population need, we utilized unstructured EMR data of 1,604 CP subjects from Shriners Children's Hospital Network and trained three BERT models, Clinical BioBERT, BioBERT, and BERT BASE on fall risk assessment. To our knowledge, no study has utilized LM on unstructured EMR data of CP subjects in rehabilitation settings to assess fall risk.LMs require a significant amount of computation power and money, generally, accounting for two steps, pre-training and fine-tuning [21] . For instance, it took 23 days using 8 NVIDIA V100 (32 GB) GPUs to pre-train BioBERT [13] . Pre-training is self-supervised training without a downstream task label. As the domain narrows, continued pre-training is general practice [14] , [20] . The two main objective functions for pre-training are masked language modeling (MLM) and next sentence prediction (NSP). Fine-tuning is a training model with labeled data. Pre-training is not always essential when substantial labeled data is available. Zoph et al. demonstrated that high-quality labeled data pre-training decreases accuracy [22] . With a robust and diverse dataset, models can often learn effectively without pre-training. If one can bypass the pretraining phase, it is possible to eliminate a computationally intensive step. However, it is crucial to investigate whether this omission impacts model performance. To analyze the necessity of pre-training in our settings, we demonstrated that extensive pre-training is not always required when high-quality labeled data is accessible.In addition one of the biggest challenges for LMs is the context window. The context window is the amount of text a model can process at a time. LMs generally have predefined context window sizes. Even large LMs with billions of parameters, for example, LLaMA can process a maximum of 2048 tokens [23] . Clinical notes are much longer than the context window. Training models with longer context windows requires a significant amount of computation power and time and increases operational costs. For example, Chen et al. demonstrated that further pre-training Transformers for 1000 batches the model only increased the context window from 2048 to 2560 [24] . Therefore, increasing the context window is not always computationally efficient. To overcome the context window challenge and maximize the information, we utilized confidence weighted majority voting (CWMV) [25] with the extension of the equality affect parameter proposed by Meyen et al [26] . We integrated CWMV into our settings using the model's probability classification layer output to calculate the weights. This approach is straightforward, easily adaptable, and computationally efficient.In summary, we conducted a comprehensive comparison between pre-trained and non-pre-trained Clinical BioBERT, BioBERT, and BERT BASE in predicting fall risk assessment risk in CP. To our knowledge, this work is the first to explore the application of LMs for CP and, more specifically, for fall risk assessment. Non-pre-trained models can predict fall risks effectively, highlighting that general LMs, when fine-tuned appropriately, can be robust tools for healthcare applications. Moreover, to mitigate the limitation of the context window we utilized a probability weighted majority voting (PWMV) strategy to make collective decisions across multiple segments of text. PWMV allows us to combine predictions from different text segments in a way that gives more weight to certain positions, ensuring that important information is not lost due to context window constraints. By using PWMV, we improved the overall model performance by 1-3%. In this study, we propose a robust and effective strategy with valuable insights and well-validated generalized models with real-life hospital data on the CP cohort to predict fall risk. For the fall risk assessment predictions task, we used the Cummings Fall Assessment Score. Cummings Fall Assessment Score has 6 components: history of fall, physical alterations/impairments, equipment, functional status, cognitive/psychological, and medications that alter equilibrium. Each component is scored by clinicians. Table III describes each component of the total Cummings Fall Assessment Score. If the total score is 8 or more that is considered high-risk, 0 is considered no-risk, otherwise low-risk. For this study, we considered 0-7 as low-risk and 8-16 as high-risk. The threshold between 7 and 8 is practiced by this hospital network. We had 11, 213 samples from 1, 144 subjects that include the Cummings Fall Assessment Score, encompassing 14 different note types. Fig. 1b describes the lengths of these 11, 213 samples vary, ranging from 525 to 24, 589 characters.We listed the note types with the number of subjects and samples in Table IV . Subjects can be assessed multiple times throughout treatment. One subject might not have the same fall score throughout their treatment. We showed score distribution by the number of subjects in For pre-training, we removed fall scores. We split all notes following Clinical BioBERT [18] pre-processing steps, and Scispacy [27] is used to perform sentence extraction. After that, we have 12,067,470 samples for pre-training. Similarly, we performed section extraction for fall risk assessment prediction. Each note is addressed as one sample for fall risk assessment prediction, and portions that fit the context window from one note are addressed as sections. We have 423, 188 We used the MLM objective function for pre-training and pre-trained Clinical BioBERT, BioBERT, and BERT BASE in the same setup. We pre-trained all the models with a 15% mask percentage. Following the original BERT [12] masking strategy, we replaced the i-th token with the mask token 80% of the time, a random token 10% of the time, and unchanged 10% of the time. Following the Clinical BioBERT [18] , we fixed other hyper-parameters such as the duplication factor set to 5. We used 128 max length for the tokenizer with padding of maximum length, and setting truncation true. The batch size is 32. The learning rate is 5 × 10 -5 with 10,000 warmup steps. The weight decay is 0.01. The models are pre-trained to predict the original token with cross-entropy loss. We achieved a satisfactory loss on the test set within 31,562 steps. We used Adam Optimizer.For fine-tuning BERT models we used the same hyperparameters for all models. For classification, we added a dropout layer with 0.1, and fully connected layers with 128 dimensions with Relu activation. The learning rate was set 1 × 10 -5 , batch size 32. For classification, softmax is used. The models were fine-tuned using 32 batch sizes for 2 epochs.During training individual sections were treated independently.To aggregate the predictions from sections from one sample, we used probability weighted majority voting (PWMV) following Meyen et al. strategy [26] . The individual weight is calculated using equation 2 which is logarithmic odds. Here, individual output probabilities are p i . max(p i ) is the maximum probability output of the classification layer between two classes. Aggregating these weights w i and y i , group output y P W M V q that is collective output for all the sections from one sample is calculated using equation 3. β is the equality effect that weighs individual votes. Here β can be between zero to infinity, β ∈ [0, ∞]. if β = 1 the weights become unchanged which will be naive PWMV. On the contrary, if β = 0, every weight becomes 1 which results in majority voting (MV). The final group prediction is decided based on equation 4.EQUATIONEQUATIONEQUATIONEQUATIONEvery subject in the CP cohort exhibits unique characteristics and responses to treatment. To ensure the model's generalizability and robustness, subject-wise testing is crucial [28] , [29] . To properly validate the model, we tested by subject. For pre-training, we used a test set of 20% subjects. Next for the fall risk assessment prediction, the training set is split 80% (n=916) subjects, and 20% (n=288). The test subjects were excluded from pre-training and fine-tuning. The total sections for the fall risk assessments are 342,616 in the training set and 80,572 in the test set. In the training set the number of low-risk is 220, 963 and high-risk is 121,653. For evaluation, each model was fine-tuned using the training set, and all models were tested in the same separate test set of 288 subjects.We pre-trained ClinicalBioBERT, BioBERT, and BERT BASE for 31,562 steps and 18,750 steps. We evaluated the pretrained models by MLM loss on the test set. There is not a significant difference between 18k to 31k steps. The losses of 31,562 steps on the test set were 0.235, 0.241, and 0.245; the losses on the training set were 0.107, 0.119, and 0.110 for ClinicalBioBERT, BioBERT, and BERT BASE respectively. The loss until 18K steps is depicted in Fig 5 . The results indicate that all models demonstrate comparable performance on the MLM objective. Initially, BioBERT underperformed compared to the other models. However, by the 3,000-step mark, all models achieved similar performance levels.To aggregate the decision from individual sections, we used equation 3 to make one decision per sample. We grid search β for each model in the training set from 0 to 4 with the step size of 0.01. The β scores are listed in table V. For comparison, we also applied β = 0 which is majority 2) Prediction Results: The evaluation metrics F-1, precision, and recall on the test set are shown in Table VI . Comparing the results Clinical BioBERT and BERT BASE decrease performance when pre-trained, whereas BioBERT improves after pre-training. PWMV (β > 1) gave the best results. The best F1 score is 0.71 achieved by non-pre-trained Clinical BioBERT and pre-trained BioBERT using PWMV (β > 1) . Pre-trained BioBERT with PWMV (β = 2.90) achieved the best recall of 0.73. The specificity of all the models is more than 0.77. The specificity Clinical BioBERT achieved is 0.80.To analyze model performance on different Cummings Fall Assessment Scores, we showed the confusion matrix of each score 0 to 16 in Fig 6 with recall values on the top of the confusion matrix. The recall values decrease for the 7 and 8 scores which are the thresholds between high and low-risk. The recall value for 7, and 8 is only 0.5. As illustrated in Fig 6 , the model can predict better if subjects are further away from the threshold. The difference between 7 and 8 is considerably hard to learn. To investigate the effect of borderline samples (7 and 8), we fine-tuned the model without 7 and 8 scores and compared the results on all testing samples (including 7 and 8). The model's performance improved significantly after we excluded borderline samples. The F1 score increased from 0.71 to 0.74, and the recall improved from 0.72 to 0.77. Further testing without these borderline samples showed an increase in the F1 score from 0.74 to 0.78, and recall rose from 0.74 to 0.80. This suggests that the borderline samples might lack sufficient certainty, impacting the model's performance.Clinicians might also find it challenging to distinguish between cases labeled as 7 and 8. This leads to uncertainty in true labels due to the inherent ambiguity in these cases. Moreover, we evaluated the non-pre-trained Clinical BioBERT with PWMV (β = 3.90) in a smaller subset of training subjects. We decreased 50% subjects in the training set and listed the evaluation metrics in Table VII . As the number of subject data increases the model performance can be increased. To assess the predictive capability of each note type in determining fall risk, we reported success rates in the test set, presenting the result in Table VIII alongside 95% binomial confidence intervals (CI). Our findings indicate that certain note types, such as Patient Summary, Outpatient, and PACU Assessment PowerForms, achieved success rates exceeding 90%, with 95% CI ranging from 71% to 98%. Each note type contains different kinds of information. For instance, patient summary outpatient notes often contain rehab-related information that may be more directly associated with fall risk. In contrast, admission notes include general details, such as skin color and temperature, which might be less predictive. The predictive power of these note types could vary across different hospital networks, depending on documentation practices and the specific patient populations. However, it's important to note that some note types had small sample sizes, so we can not conclude their informativeness. Nonetheless, these findings provide valuable insights into the potential value of each note type for fall risk assessment. We leveraged unstructured EMR data to assess fall risk in children with CP, intending to reduce falls and improve patient outcomes. Despite the variability associated with CP patients, our findings demonstrated that BERT-based models can effectively perform fall risk assessment. By utilizing the available data and harnessing the power of generalized LMs, we are the first to show that CP fall risk assessment management can be significantly enhanced, providing valuable support to clinicians in decision-making.There are a few limitations of this work. Our study is confined to BERT-based models. For comparisons, we plan to explore other LM that showed promising results in recent years such as T5 [30] , ClinicalT5 [31] , BART [32] . Additionally, quality data can significantly enhance the performance of models. With more data availability we can increase model performance as shown in Table VII , by increasing 50% of subject number model performance increases 10%. This enhancement suggests that a larger dataset allows the model to learn more nuanced patterns and relationships within the data. The models can become better at generalizing from the training data to new unseen data, leading to improved accuracy and robustness in predictions. Moreover, our study is limited to fall risk assessment. Our goal is to extend predictive modeling using unstructured EMR in other prediction tasks that are important for children with CP, such as pain score prediction, and functional activity level evaluation.We provide foundational baselines and experimented with both pre-trained and non-pre-trained for Clinical BioBERT, BioBERT, and BERT BASE in CP fall risk assessment. Our findings demonstrated that it is possible to leverage the capabilities of generalized LMs without the need for extensive domain-specific pre-training. This can significantly reduce the computational resources and time required for model pretraining. For instance, pre-training Clinical BioBERT took 17 -18 days of runtime using GeForce GTX TITAN X 12 GB [18] . Our results indicate Clinical BioBERT and BERT BASE did not improve performance by pre-training, and BioBERT improved slightly. Further experimentation with more diverse datasets from multiple hospital networks is necessary to explore the benefits of pre-training, particularly with larger, more varied data sources. However, we showed utilization of existing pretrained models for further tailored predictive modeling is both beneficial and cost-effective.Moreover, to overcome LM's inability to process long text, we can make collected decisions from smaller pieces of longer notes. We utilized PWMV which can yield accurate decisions by giving more weight to individual samples with higher confidence [25] . Moreover, Meyen et al. introduced an equality effect parameter β; as β gets larger that enhances the impact of high confidence segments relative to low confidence ones [26] . Without the equality parameter (β = 1), the PWMV treats each individual's confidence equally, whereas β = 0 can represent simple majority voting without weighing based on confidence. However, Meyen et al. demonstrated that using an equality parameter gives more preference to decisions supported by fewer individuals with very high confidence rather than a larger number of individuals with lower confidence. With PWMV we can make collective decisions, and model performances improved. This method is readily adaptable and can be applied, unlike approaches where increasing the context window model is expensive.Utilizing cross-validated approaches can effectively reduce the uncertainty associated with evaluating fall risk. We achieved 0.71 F1 scores using BERT models in subject-wise testing. Subject-wise testing in a CP cohort is critical for providing the model's generalizability. It ensures that each person's unique challenges are addressed, facilitating better management of the condition and enhancing the quality of life for those affected. Fall risk changes over time, with an interplay between disease progression, age, and treatments. We have 622 out of 1144 subjects who changed over time between high and low risk according to the Cummings Fall Assessment Score. In this model, we are making risk assessment predictions when the notes were taken. For future work, we would like to make future predictions of fall risk, that is, anticipate changes in fall risk, but that is beyond the scope of this effort. Furthermore, we demonstrated that removing borderline samples improves model performance by 3%-4%. This indicates that models can effectively learn without the inclusion of hard samples, which may only have marginally differentiable features. These samples can be challenging not only to the model but also to clinicians, who might find it difficult to assess them accurately due to the lack of clear distinguishing characteristics. The presence of such samples can introduce noise and reduce the overall effectiveness of the model, highlighting the benefit of focusing on more distinct and well-labeled examples for training.The models in our experiments showed signs of overfitting during training. For instance, the F1-scores for the fully finetuned Non-Pretrained Clinical BioBERT, BERT base, and BioBERT on the training set were 0.99, 0.98, and 0.98, respectively. To mitigate overfitting, we conducted further experiments by partially fine-tuning the Non-Pretrained Clinical BioBERT model. Instead of fine-tuning the entire model, we fine-tuned only the last encoder layer along with the added classification layers. This approach reduced overfitting, with the following results: on the training set, F1: 0.78, Precision: 0.87, Recall: 0.70, and on the test set, F1: 0.74, Precision: 0.80, Recall: 0.69. All the results mentioned above were obtained within the PWMV setup. Implementing more robust techniques to address overfitting could further enhance model performance.Moreover, we show that certain types of samples possess greater predictive power than others. It provides clinicians with a clearer understanding of which types of data are the most critical for making decisions. By identifying these key data types, clinicians can focus their efforts on gathering and analyzing the most impactful information, thereby enhancing the accuracy of their assessments. In summary, our study leads to more reliable predictions, better-informed clinical decisions, and ultimately, improved outcomes for children with CP.There is significant assistance needed in the current fall risk assessment in CP. This study suggests implementing BERTbased models for this population can significantly impact the evaluation process. By collecting more data, these models improve their accuracy and can offer a powerful tool for integrating and analyzing diverse data sources, providing personalized and accurate risk predictions, and supporting more effective clinical decision-making. By addressing the current limitations in fall risk assessment, BERT-based models can improve outcomes and a higher quality of care for individuals with CP. Implementing these advanced models aligns with the growing emphasis on data-driven, personalized healthcare, and can potentially transform fall risk management in the CP population. 
 
 Appendix:  



Reviews: review: title: title: Important problem and valuable analysis
paper_summary: paper_summary: The paper proposes a solution that can assess the risk of falls in children with cerebral palsy. It uses BioBERT, Clinical BioBERT, and BERT_base to predict fall assessment risk. The method uses clinical notes to categorize patients into high-risk and low-risk groups.
strength_weakness: strength_weakness: None
questions: questions: None
title: title: New application but with limited techinical novelty
paper_summary: paper_summary: This paper uses the Language Model to evaluate fall risk using electronic medical records. The authors explored the impact of pre-training and fine-tuning on performance. They also used a voting-based method to handle the extreme-long context info.
strength_weakness: strength_weakness: Technical novelty is limited, pre-training and fine-tuning are both mature techniques. 

Also as other reviewers have mentioned, given the single source of the dataset (only from one hospital), the conclusion (pre-training is not necessary) is less convincing.
questions: questions: You mention 'certain types of samples possess greater predictive power than others', is it because of certain exams? Can you please provide a few examples of such samples? Do you think the same type of samples will still hold if at other hospitals?
title: title: Language Models for Fall Risk Assessment in Children with Cerebral Palsy using Electronic Medical Records
paper_summary: paper_summary: The paper applies BERT-based language models, specifically Clinical BioBERT, BioBERT, and BERTBASE to predict fall risk in children with CP using unstructured EMR as input. The study introduces a Probability Weighted Majority Voting (PWMV) strategy to improve model performance by aggregating predictions from segmented clinical notes.
strength_weakness: strength_weakness: (1) It is unclear on what sample of data the 0.71 and 0.74 F1 scores are achieved on. The methods section states 80/20 train/test split, where the training data is further split into 80/20 for validation. Presumably, the test set of 228 subjects is not used in any part of the pre-training or fine-tuning of the models; however, this is not clear to the reader. 
(2) Dataset appears to be from a single institution, while EHR documentation practices can vary by institution and EHR software used.
questions: questions: (1) Would like to know the F1 score on the training and validation dataset to assess how much overfitting between training data and unseen EHR.
(2) Is fall risk constant or does it change over time? How might this impact your model and its downstream clinical implementation?