# Language Model-Based Text Augmentation System for Cerebrovascular Disease Related Medical Report
## 3 Methodology
### Annotator  
The classification model used in this study is a BioBERT-based model tailored for cerebrovascular disease classification. Bidirectional Encoder Representations from Transformers (BERT) [30] is a language model that uses a masked language model based on a bidirectional transformer. BioBERT is an extension of BERT that has been further pre-trained on biomedical text. For this purpose, it was pre-trained on the corpus used for BERT as well as biomedical domain-specific texts from PubMed abstracts (PubMed) and PubMed Central full-text articles (PMC). This additional pre-training allows BioBERT to understand and process the biomedical domain's language better.  
Furthermore, BioBERT was fine-tuned using approximately 73,000 cerebrovascular disease-related report data to perform binary classification on cerebrovascular disease-related text.  
To annotate the generated synthetic text, thresholds of 0.24 and 0.89, representing the top and bottom 30% of predictions, were set. These two thresholds serve as reference points for the model to classify the text accurately. Threshold selection plays a critical role in the filtering process. For instance, if a narrower threshold, such as the top and bottom 20%, were used, the number of samples classified as ICH or normal would decrease, potentially reducing misclassification by the model. However, this might lead to a shortage of data needed for performance improvement. On the other hand, setting a broader threshold, such as the top and bottom 40%, could increase the number of samples available for training,improving the model's performance. However, this approach also risks including uncertain samples near the prediction boundary, which could degrade the model's performance.  
Subsequently, we passed the generated synthetic text through the classification model. This model outputs a prediction value indicating whether the input text is ICH or normal. The text is annotated as normal if the prediction value is less than 0.24. If the prediction value exceeds 0.89, we annotate it as ICH. Texts with prediction values between these two thresholds are excluded, as they will likely introduce ambiguity into the model's judgment. Such texts, with unclear prediction values, could negatively impact model performance if annotated incorrectly. Equation (2) illustrates annotating and filtering the generated synthetic text using the classification model.  
\[Classifier(S)=\left\{\begin{aligned} ICH,&&\qquad\text{ if }P(y|S;)>0.89\\ Normal,&&\qquad\text{if }P(y|S;)<0.24\\ Excluded,&&\qquad\text{if }0.24\leq P(y|S;)\leq 0.89 \\ \end{aligned}\right. \tag{2}\]  
This process is essential for maintaining the quality and consistency of the generated synthetic text. By annotating the text through the classification model, we enhance the reliability of the dataset and refine the data for model training. This annotation process maximizes the effectiveness of data augmentation using synthetic text, ultimately playing a crucial role in improving the model's performance. Table 1 shows examples of the augmented synthetic text.