# Language Model-Based Text Augmentation System for Cerebrovascular Disease Related Medical Report
## 3 Methodology
### Generative Model  
The generation model used in this study is a DistilGPT2-based model for generating synthetic text related to cerebrovascular diseases [29]. DistilGPT2 is a distilled version of the GPT-2 model, which retains most of GPT-2's capabilities while being smaller and more efficient. GPT-2, developed by OpenAI ([[https://openai.com/](https://openai.com/)]([https://openai.com/](https://openai.com/)), accessed on 2 July 2024), is a state-of-the-art transformer-based language model known for generating coherent and contextually appropriate text based on the input it receives. DistilGPT2 maintains these abilities while being lighter and faster, achieved through knowledge distillation, where the smaller model is trained to replicate the behavior of the larger model.  
Figure 1: Proposed System. The text from cerebrovascular disease-related reports is input into a generation model to create synthetic text. This synthetic text is passed through a classification model, which annotates and filters it to produce annotated and filtered synthetic text.  
The architecture of DistilGPT2 is as follows. It has a vocabulary size of 50,260, which allows it to handle a wide range of medical terms and expressions commonly found in medical reports. The model consists of 6 layers, providing sufficient depth to learn complex language patterns. With a hidden size of 768 dimensions, it can richly represent the contextual meaning of the text. Additionally, the model uses 12 attention heads, enhancing its ability to simultaneously consider various text parts and understand the context more effectively.  
Specifically, DistilGPT2 was fine-tuned using a corpus of medical records to tailor it for medical text generation. Fine-tuning involves further training a pre-trained model on a specific domain's dataset, making the model suitable for that domain. In this study, DistilGPT2 was fine-tuned with medical records related to cerebrovascular diseases to generate relevant text. As a result, the synthetic text generated by this model reflects the style, terminology, and contextual characteristics of medical texts, closely resembling actual data.  
To assess the impact of augmented synthetic text on model performance using small amounts of text data, we randomly extracted 5000, 10,000, 15,000, and 20,000 samples from the entire report dataset. This step was crucial to reflect real-world scenarios where obtaining annotated medical data is challenging, ensuring the model can effectively learn from limited data. To evaluate the generalized performance of the model, we applied a 5-fold cross-validation technique. This involves dividing each original dataset into five distinct training and test pairs, maintaining an 8:2 training--test split ratio for each pair. This approach helps verify the model's generalization ability and reliability across different parts of the data.  
Subsequently, we generated synthetic text using the training data from each pair. Equation (1) illustrates the structure of the synthetic text generation process.  
\[S=G_{distilGPT2}(text_{train}) \tag{1}\]  
We used the generated synthetic text to augment the training datasets and improve the performance of future classification tasks. This approach creates more affluent and more diverse datasets, significantly enhancing the model's learning and generalization capabilities. Consequently, it contributes to developing more accurate and reliable medical AI systems.