# Language Model-Based Text Augmentation System for Cerebrovascular Disease Related Medical Report
## 4 Experimental Result
### Result
#### 4.4.4 Comparison  
Table 3 presents each model's accuracy based on the data's size. We progressively added synthetic text in 20% increments of the original dataset size. Regardless of the model or the size of the original data, adding synthetic text generally improved performance. The baseline models showed higher accuracy as the original data size was increased. Although ClinicalBERT had the highest accuracy overall, its performance improvement was the smallest, with an increase of 2.44% for 5000 data points, 1.97% for 10,000, 1.26% for 15,000, and 0.77% for 20,000.  
On the other hand, BiomedBERT, which had the lowest baseline accuracy, showed the most significant performance improvements, with accuracy increases of 20.28% for  
Figure 4: Results of the BiomedBERT models fine-tuned with the original data and augmented with synthetic text. (**a**) Original dataset size of 5000 samples and (**b**) original dataset size of 20,000 samples.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{
\begin{tabular}{c} **Data** \\ **Size** \\ \end{tabular} } & \multirow{2}{*}{**Baseline**} & \multirow{2}{*}{**+20\%**} & \multirow{2}{*}{**+40\%**} & \multirow{2}{*}{**+60\%**} & \multirow{2}{*}{**+80\%**} & \multirow{2}{*}{**+100\%**} & \multirow{2}{*}{**+120\%**} & \multirow{2}{*}{**+140\%**} & \multirow  
linebreak 5000 data points, 4.96% for 10,000, 1.04% for 15,000, and 2.22% for 20,000. BioBERT showed accuracy increases of 4.18% for 5000 data points, 3.07% for 10,000, 0.97% for 15,000, and 0.89% for 20,000. While the performance improvement for BioBERT was smaller than that of BiomBERT, it was more extensive than that of ClinicalBERT.  
Additionally, all three models demonstrated a trend where more synthetic text resulted in relatively better performance gains. Furthermore, the smaller the original dataset, the more pronounced the performance improvement became.  
Table 4 shows the F1 scores of each model based on the dataset size. Most models experienced performance improvements regardless of the size of the original dataset. BioBERT and ClinicalBERT both achieved the highest F1 scores at 0.85. ClinicalBERT showed an improvement of 0.01 for 5000 data points, 0.04 for 10,000, 0.03 for 15,000, and 0.01 for 20,000. In contrast, BioBERT demonstrated more varied enhancements, with an increase of 0.13 for 5000 data points, 0.02 for 10,000, 0.03 for 15,000, and 0.03 for 20,000, resulting in a significant difference in the magnitude of performance enhancement between the two models.  
BiomedBERT, while achieving a maximum F1 score of 0.84--lower than the other two models--showed the most considerable performance improvements. The F1 score increased by 0.14 for 5000 data points, 0.02 for 10,000, and 0.1 for 15,000, but no improvement was observed for the 20,000 data point case. Additionally, both BioBERT and BiomedBERT showed more pronounced performance improvements with smaller original dataset sizes, a less evident trend in ClinicalBERT.  
These results demonstrate that synthetic text is crucial in enhancing model performance. Specifically, adding an appropriate amount of synthetic text significantly improved model accuracy. Conversely, adding an inappropriate amount of synthetic text could degrade model performance. This indicates that the quantity of synthetic text significantly impacts model performance, highlighting the importance of determining the optimal amount of synthetic text.