Title: Noise reduction in BERT NER SLM models for clinical entity extraction in clinical trials
Primary area: foundation or frontier models, including LLMs
Keywords: SLM, NER, Noise removal, Density Map, BERT, Clinical, Clinical Trials, Biomedical entity

Abstract: Precision is of utmost importance in the realm of clinical entity extraction from clinical notes and reports. Small Language Models (SLM) fine-tuned for Named Entity Recognition (NER) are an efficient choice for this purpose. We pre-trained an in-house BERT SLM over clinical data and then fine-tuned it for NER. These models performed well on recall but suffered on precision. To address this challenge, we developed a Noise Removal model that refines the output of NER. The NER model assigns token-level entity tags along with probability scores for each token. Our Noise Removal (NR) model then analyzes these probability sequences and classifies predictions as either weak or strong. One might assume that simply filtering upon lower probabilities would be sufficient to identify weak predictions. While  intuitive, this approach is not entirely reliable. Due to the limitations of the SoftMax function, transformer-based models often assign high confidence scores even to weak predictions, making them harder to distinguish. To overcome this issue we used a supervised modeling approach, where our NR models capture the effects of positional encodings in transformers and integrates them with the probability density distributions generated for each token sequence. This approach enables the model to classify predictions as weak or strong with significantly improved accuracy. With the help of these Noise Removal models we were able to reduce False Positives across various clinical NER models by 50\% to 90\%.

TL;DR: A light weight post processing technique applied after BERT based NER models to remove Noisy prediction to achieve high precision