Preprocessing Pathology Reports for Vision-Language Model Development

Authors: ruben t lucassen
tijn  van de luijtgaarden
sander p j moonemans
willeke a m blokx
mitko  veta
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-24 13:25:10

Abstract: Pathology reports are increasingly being used for development of vision-language models. Because the reports often include information that cannot directly be derived from paired images, careful selection of information is required to prevent hallucinations in tasks like report generation. In this paper, we present a language model for subsentence segmentation based on the information content, as part of a preprocessing workflow for 27,500 pathology reports of cutaneous melanocytic lesions. After initial clean up, the reports were first translated from Dutch to English and then segmented by separate language models. Both models were developed using an iterative approach, in which the development dataset was expanded with manually corrected model predictions for previously unannotated reports before finetuning the next version of the models. Over the course of eight iterations, the development dataset was in the end scaled up to 1,500 translated and annotated reports. On the independent test set of 3,597 sentences from 150 reports, 219 translation errors (6,1%) of different severities were counted. The subsentence segmentation model achieved a strong predictive performance on the test set with a macro average F1-score of 0.921 (95% CI, 0.890-0.940) and a weighted average F1-score of 0.952 (95% CI, 0.944-0.960) over 13 different classes. The remaining 25,850 unannotated reports were translated and segmented using the final models to complete the dataset preprocessing. Differences in word count and class distribution between section types of the reports were explored in preparation for future vision-language modelling. The presented methodology is generic and can, therefore, easily be extended to multiple or different pathology domains beyond melanocytic skin lesions. Code and trained model parameters are made publicly available.

Main paper: 
 introduction: 
 Vision-language models in computational pathology have recently gained attention for tasks like multimodal pre-training, image retrieval, and image captioning. (Huang et al., 2023; Lu et al., 2024b; Vorontsov et al., 2024; Lu et al., 2024a ) Some of the first models that combined vision and language for histopathology, such as PLIP (Huang et al., 2023) and CONCH (Lu et al., 2024b) , were trained using histopathology images paired with captions, primarily scraped from social media or scientific literature. However, for case-level tasks like pathology report generation, which can potentially reduce the workload of pathologists, digitally archived whole slide images and corresponding pathology reports likely form a more suitable resource for vision-language model development. This is in line with more recent models, such as PRISM (Vorontsov et al., 2024) and PathChat (Lu et al., 2024a) .Pathology reports are often diverse in content, including sentences about cell and tissue appearance on hematoxylin and eosin (H&E)-stained slides, expression patterns in immunohistochemistry (IHC), results of molecular testing, clinical information, patient history, (differential) diagnoses, and treatment recommendations. (Scolyer et al., 2013) Using sentences with information that cannot be derived from the paired images during training can lead to hallucinations (i.e., generated statements that contradict the source content) (Ji et al., 2023) , which can be dangerous for high-stakes applications such as in healthcare, motivating the need for careful selection of information from the reports. Whereas preprocessing steps for whole slide images (e.g., tessellation, pen marking removal, and stain normalization) are well-documented and extensively studied, pathology report preprocessing for vision-language modeling is comparatively new and best-practices are still developing.In this work, we present a language model for segmentation of subsentences based on the information content as part of a preprocessing workflow for pathology reports. After de-identification, removal of irrelevant sentences, and sectioning, the reports were first translated from Dutch to English and then segmented by separate language models, both developed using an iterative model development approach (Figure 1 ). While the results shown in this paper are specific to reports of cutaneous melanocytic lesions, the presented methodology is more general and can straightforwardly be extended to multiple or different pathology domains. The code and trained model parameters are made publicly available 1 . 2 Materials and Methods 
dataset: 
 The dataset used in this work consists of 27,500 pathology reports describing skin biopsy and excision specimens with melanocytic lesions, accessioned between January 1, 2013 and December 31, 2020, and obtained from the digital archive of the Department of Pathology at the University Medical Center Utrecht, the Netherlands. The reports were made as part of routine clinical practice by different general pathologists and more specialized dermatopathologists. The study was conducted in compliance with the hospital's research ethics committee guidelines. Reports for patients who opted out of the use of their data for research purposes were excluded. Several manual preprocessing steps were performed at the start to clean up the pathology reports. All reports were de-identified by removing names or abbreviations of pathologists and hospitals, as well as dates and case numbers. Identifying patient information was never included in the reports, with the exception of sex and age at accession. Irrelevant sentences were also removed. If multiple unique specimens were originally combined into a single report, the original report was divided into a separate report for every specimen. Each report can include four sections: (1) a structured report; (2) a free-text description of all observations and results of diagnostic testing; (3) a free-text discussion of the findings and the interpretation thereof; (4) a structured or free-text conclusion with key findings, the (differential) diagnosis, and treatment recommendations. Not all sections are present in each report.Separate language models were used for the translation and segmentation task. Both models were iteratively improved over the course of eight iterations by expanding the development dataset with manually corrected model predictions for previously unannotated 3 in the Supplementary Material. The reports were translated and corrected by two translators (R.L., T.L.), annotated and corrected by three annotators (R.L., T.L., S.M.), all under supervision of a pathologist (W.B.). Additionally, a test set of 150 reports, selected similarly from different patients, was annotated independently (i.e., not starting from model predictions) by a single annotator (R.L.) and verified by a pathologist (W.B.) for evaluation of the models. An example of an annotated report with the 13 defined classes is shown in Figure 2 . Punctuation marks were included in the annotations in such a way that a sentence remains coherent as much as possible when classes would be filtered out. Annotation and correction of model predictions was performed using the open-source Label Studio software2 . The remaining 25,850 reports were translated using the final translation model and identified translation errors that remained were corrected before subsentence segmentation by the final segmentation model.The vast majority of the pathology reports in the dataset were originally written in Dutch (99.7% in Dutch, 0.3% in English). We decided to translate the reports from Dutch to English, as most of the publicly available, pre-trained language models have English as primary language, which can potentially be useful in the downstream development of visionlanguage models for computational pathology.We built upon a pre-trained Dutch-to-English translation model, opus-mt-nl-en3 , from the OPUS-MT project (Tiedemann and Thottingal, 2020) , which is a Transformer model (Vaswani et al., 2017) with an encoder-decoder architecture of 182 million parameters, using SentencePiece (Kudo and Richardson, 2018) for subword tokenization. The model was finetuned on Dutch sentences as input with the English translation as output for 100 epochs with a batch size of 32 and a starting learning rate of 1 • 10 -4 . Data augmentation consisted of the introduction of spelling errors in the input sentence, changing the input and output sentences to upper, lower, or title case, adding parentheses around all words or the full sentence, and using the output sentence also as input. To reduce translation errors with respect to numbers or abbreviations (e.g., age, measured distances, percentages, and names of genes), we additionally included a set of template sentence pairs with a range of possible values inserted during training. The HuggingFace framework was used for model training and inference. (Wolf et al., 2020) For segmenting subsentences in the translated pathology reports based on the information content, we built upon FLAN-t5-large 4 (Chung et al., 2024), which is an instructionfinetuned Transformer model (Vaswani et al., 2017) with an encoder-decoder architecture of 783 million parameters, pre-trained on numerous language tasks. Custom tokens were added to the default SentencePiece tokenizer (Kudo and Richardson, 2018) , primarily for the class labels. The model was finetuned on input-output pairs to learn the pathology report-specific subsentence segmentation task for 10 epochs with a batch size of 12 and a learning rate of 1 • 10 -4 at the start. Input prompts consisted of a single sentence to be segmented, in combination with a context window of sentences that precede and follow it. The output format includes the single sentence again, where now each subsentence segment is concluded by the corresponding class token and delimited by a newline token except at the end. An example input-output pair with a context window size of 1 can be seen below: Input: "segment pathology sentence: 'A cutaneous metastasis cannot be excluded for certain, but there is no history of prior melanoma.' Context: 'The Breslow thickness is at least 1.6 mm. <SENTENCE> No ulceration, no regression, no satellites.'" Output: "A cutaneous metastasis cannot be excluded for certain,<CON><Nline>but there is no history of prior melanoma.<HIS>" Based on a hyperparameter search, evaluated on the validation set, we selected a context window of 7 (from a range of 0-10) and oversampled sentences with annotations of medical history (HIS) and/or calculations and counts (CAL) (i.e., the two smallest classes) by a factor of 5 during training. The FLAN-t5-large variant of the model showed slightly better performance than smaller variants. In addition, we investigated including the section name in the input prompt, which improved the performance when using a small context window size, but did not show benefit for larger context windows. The HuggingFace framework was used for model training and inference. (Wolf et al., 2020) 3 Results 
translation performance: 
 The predictive performance of the translation model was evaluated on the independent test set of 3,597 sentences from 150 reports by counting and categorizing the type of errors. Translation errors that changed the meaning of a sentence and were undetectable without knowing the input sentence were counted 21 times (0.6%), which mostly concerned (decimal) numbers, anatomical locations, and missed or introduced negation. Spelling and grammar errors were counted 69 times (1.9%). Incorrect translations that were detectable without knowing the input sentence or missing words that did not considerably change the meaning of the sentence were counted 129 times (3.6%).The predictive performance of the segmentation model was evaluated on the independent test set at word level using the precision, recall, and F 1 -score for each class. The overall performance was calculated by macro averaging (i.e., using equal class weights) and weighted averaging (i.e., using proportional class weights) the class scores. Bootstrapping (R = 10,000 samples) was used to calculate 95% confidence intervals (CIs). Out of the 4,660 subsentences in the test set, the model incorrectly reproduced the original subsentence 30 times (0.6%) in the output. These reproduction errors, which did not affect the predicted class tokens, were manually corrected before the evaluation.The results of the segmentation model evaluation are shown in Table 2 . The F 1 -scores per class ranged between 0.773-0.974, with a macro average F 1 -score of 0.921 (95% CI, 0.890-0.940) and a weighted average F 1 -score of 0.952 (95% CI, 0.944-0.960). The model performance was substantially lower for the HIS and IHC+ classes in comparison to the other classes. For words annotated but not predicted as part of the HIS-class, CON and IHC were the most frequent alternatively predicted classes. For words annotated but not predicted as part of the IHC+-class, the most frequently predicted alternative classes were H&E and IHC. N indicates the number of reports for which the corresponding section was not empty. Annotations were used for class-specific word counts if available. Model predictions were used for the remaining reports without available annotations. A logarithmic scale was used for the number of words per class.Statistics for the complete dataset of 27,500 pathology reports for downstream visionlanguage modeling, extracted after preprocessing of the unannotated reports, are shown in Figure 3 . Substantial differences are seen between the sections in terms of the presence, the words per section, and words per class. Almost all reports have a conclusion section (99.9%), which tends to be shorts with a median of 13 words and mostly related to conclusions, clinical information, and the radicality. The majority of the reports also have a description section (71.8%) with a median of 70 words. The descriptions mainly contain observations based on H&E-stained slides, followed by information related to IHC expression patterns and the radicality. The discussion section is much less common (2.5%) and has the largest median length of 128 words. Results of molecular testing and treatment advice are more frequently included in the discussion section compared to the description section. Structured report sections, typically only used for melanomas, are the least common (1.3%) and have a median length of 78 words. In contrast to the other sections, the distribution of the word count for the structured reports does not have a single, clear mode. The class distribution is fairly mixed, with H&E-stained slide observations, clinical information, conclusions, and unrelated information contributing close to equally as largest classes.IHC-stained slides are not mentioned in the structured reports. Subsentences related to patient history, as well as calculations and counts, are uncommon across all sections. 
discussion and conclusion: 
 As part of a preprocessing workflow for pathology reports of melanocytic lesions, we developed separate language models for translation from Dutch to English and subsentence segmentation based on information content. This enables selective use of text data from the reports for vision-language modeling, which can prevent hallucinations in tasks like report generation by training only on information that can be derived from the paired images.After initial preprocessing, the reports were first translated from Dutch to English using the language model for translation. Before finetuning this model on text from the melanocytic lesion reports, many translation errors were seen with respect to pathologyspecific words. The number of errors greatly reduced during the iterative model development as the dataset for finetuning increased in size. In addition, the model was found to be more robust to spelling errors in the input sentences when these errors were also introduced during finetuning as a form of data augmentation. Despite the improvements, words and (decimal) numbers in the input sentences that were not seen during training remained as a source of error and were corrected before segmentation. Hence, we expect the generalizability of the current translation model to other pathology domains to be limited.The language model for subsentence segmentation showed a strong predictive performance on the independent test set with a macro average F 1 -score of 0.921 (95% CI, 0.890-0.940) and a weighted average F 1 -score of 0.952 (95% CI, 0.944-0.960). The model reached a F 1 -score of above 0.90 for 11 out of the 13 classes, with only a lower performance for the HIS and IHC+ classes. Patient history is only present in a relatively small subset of reports and can be fairly diverse, likely affecting both the model performance and representativeness of the evaluation for the HIS-class. The IHC+-class concerns subsentences with observations based on IHC-stained slides, which are also visible in the H&E-stained slides. Ambiguity regarding some subsentences and possibly also the strong context-dependence could explain the lower segmentation performance for the IHC+-class. A limitation of the presented approach for text segmentation is the possibility of incorrect reproduction of the original sentence in the input. Although this occurred rarely, corrections to the model output were required. A benefit of segmentation for structured reports, which are already intended to ease information retrieval, is robustness to changes in the report format over time.The developed language models were used to translate and segment the unannotated reports to complete the preprocessing of the entire dataset of 27,500 pathology reports. In future work, we plan to use the preprocessed dataset in combination with the corresponding whole slide images to investigate vision-language modeling for tasks like multimodal pretraining and report generation in the domain of cutaneous melanocytic lesions. 
 
 Appendix: https://labelstud.iohttps://huggingface.co/Helsinki-NLP/opus-mt-nl-en 



Reviews: review: title: title: Review of paper 9
main_review: main_review: The article addresses the issues of Dutch-to-English translation of pathology reports and the segmentation of translated reports by fine-tuning large language models, and conducts a detailed evaluation of their performance. The project is substantial and holds promising application potential. Importantly, both models are publicly accessible, providing valuable resources and baselines for the integration of pathology and NLP.

I recommend accepting the paper, but some details should be expressed more clearly before publication.

1. The process of manual correction is unclear. During each iteration, are the mistakenly predicted samples identified from the unannotated set and then added to the training and validation sets? If that is the case, I would like to see a figure showing the increasing performance during iterations between model and pathologists (with more corrected developing set samples).

2. The article mentions that the development set ultimately comprises 1500 reports, consistent with Table 1. Starting with how many samples did the iterative correction process begin? How many samples were added to the training and validation sets in each iteration? (whether errors happens frequently? if so, which sample have priority to be corrected?)

3. Section 3.1 mentions translation errors. For translation tasks with non-unique solutions, how are errors defined and evaluated?

Additionally, I would like to inquire whether the datasets involved in the study could potentially be made publicly available (in future publications). This would be a highly valuable resource for the community.
title: title: Iterative preprocessing techniques to improve vision-language models for pathology reports
main_review: main_review: This paper presents a methodology for preprocessing pathology reports to develop vision-language models. The authors focus on subsentence segmentation and translation from Dutch to English for 27,500 pathology reports of cutaneous melanocytic lesions. The preprocessing workflow includes de-identification, translation, and segmentation using iteratively improved language models. The final models achieved high predictive performance, with macro and weighted average F1-scores of 0.921 and 0.952, respectively. The methodology is designed to prevent hallucinations in report generation by ensuring only image-derivable information is used. The approach is generalizable to other pathology domains, and the code and trained models are publicly available.

Strengths
The paper outlines a detailed and systematic approach to preprocessing pathology reports, which is crucial for accurate vision-language model development.
The use of iterative development and manual correction to enhance model performance is a robust approach, leading to significant improvements in translation and segmentation accuracy though it may be exhaustive and resource consuming.
The models achieved good F1-scores, indicating strong predictive capabilities.
Making the code and trained models publicly available promotes transparency and facilitates further research in the field.
The methodology is designed to be applicable to various pathology domains, not just melanocytic lesions, enhancing its utility.

Weaknesses (Constructive Feedback)
The need for manual correction of model outputs indicates a dependency on human intervention. Though it is a trustful way however, developing more robust error detection and correction mechanisms within the models could reduce this dependency.
While the dataset is extensive, it is specific to a single institution and pathology type. Expanding the dataset to include reports from multiple institutions and diverse pathology types could enhance the robustness and applicability of the models.
The paper primarily lies in data collection and preparation rather than the development of new models. In addition to the mentioned future work of combining WSIs for multimodal prediction, authors could focus on innovating the model architecture or introducing novel techniques in the preprocessing steps.
The translation model’s generalizability to other pathology domains is limited. Exploring domain adaptation techniques or multi-domain training for enhanced generalizability can be additional contributions.
As the segmentation model showed lower performance for certain classes (HIS and IHC+), further investigation on the reasons can be beneficial for the readers. Moreover, incorporating additional training data or context-specific adjustments could be ways to think of for improving the performance.

Final thoughts:
This paper seems to be an intermediate report for a larger direction. The big picture of the project seems interesting. Although this paper is limited to the preprocessing and clean data collection techniques, as the details are well described and the code will be released it can benefit the community. The release of the whole dataset will also be beneficial.
title: title: In this paper, the authors focus on the pre-processing of pathology reports to be used for the development of a vision language model.
main_review: main_review: The study provides a workflow for the pre-processing of pathology reports for the development of a visual language model. The pathology reports were first translated from Dutch into English and then the sub-sentences were segmented. Both tasks were performed by two language models. Both tasks were performed by two language models iteratively over a period of eight iterations. The translation model was scored based on the number of errors of varying severity, while the sub-sentence segmentation model was scored based on F1 scores.

During the review, the following points were raised, and it is recommended to improve the manuscript accordingly:

1.	The authors mentioned that the models were developed iteratively to correct predictions. It would be beneficial to include performance data for each iteration to assess whether the model improved continuously or if improvements were random. Additionally, it would be helpful if the authors clarified what they mean by an iteration, how many reports (from the 1350 annotated reports) were considered for each iteration, and whether this number remained constant throughout all iterations.

2.	Were the reports translated to simplify the development process (if the AI researcher does not know Dutch) or for another reason? What was the main reason for translating the reports from Dutch to English? Are there any development limitations? To understand the entire process, it is important to know whether the model will require an additional model to translate the outcomes back into the original language in practical use.

3.	What were the common errors corrected in the translation when the model was applied to the unannotated 25,850 reports before using the segmentation model? Did these errors affect the performance of the segmentation model?

4.	The HIS class segmentation shows worse performance, likely due to an insufficient number of evaluated samples. Did the authors consider using weighted augmentation or a class weight approach in training the model? Similarly, was the same consideration given for the IHC+ class?