# Synthetic Text as Data: On Usefulness and Limitations
## 1 Introduction  
Large language models (LLMs), such as ChatGPT, have demonstrated remarkable capabilities in generating fluent, contextually appropriate, and human-like text. This has opened new possibilities in natural language processing--not only in text generation tasks but also in augmenting datasets for downstream applications. However, a fundamental question remains underexplored: can LLM-generated text be treated as reliable training data?  
To be more specific, if we replace or supplement real-world data with synthetic text generated by GPT, how close will the resulting model performance be? Will it be identical, noticeably worse, or somewhere in between? And, if it is somewhere in between, can we quantify the gap? These are critical questions for understanding whether GPT-generated text is merely stylistically convincing or genuinely informative in the way real data are. Addressing this issue requires moving beyond the generation quality and evaluating synthetic text from a data-centric perspective.  
In this study, we investigate the representational value of GPT-generated text by comparing its utility to that of real data in supervised learning. Our working hypothesis is straightforward: if GPT-generated samples are as informative as real samples, models trained on them should perform similarly. If there is a significant performance gap, then despite the surface-level fluency, synthetic text may lack task-relevant information presentin human-written examples. In this sense, we view classification accuracy not just as a metric of model performance but as an indirect proxy for data quality.  
To construct a measurable evaluation framework, we use MBTI personality classification as a benchmark task. This dataset features both class imbalance and low-resource conditions, which we deliberately leverage--not as problems to solve, but as controlled scenarios in which the effectiveness of synthetic data becomes testable. Experiment 1 evaluates whether GPT-generated text can mitigate class imbalance, while Experiment 2 simulates a few-shot setting to assess how well synthetic samples substitute for scarce real data. In both cases, we compare models trained with real, synthetic, and mixed data to measure the relative utility of each.  
Our findings lead to two main contributions. First, we show that GPT-generated samples can supplement real training data in meaningful ways. In settings involving data scarcity--such as class imbalance or few-shot learning--synthetic augmentation leads to measurable improvements in model performance. This suggests that GPT can serve as a practical augmentation tool when real data are limited. Second, we also clarify the limitations of synthetic text. While useful as a supplement, it does not function as a full substitute: models trained solely on GPT-generated data consistently underperform, particularly in fine-grained classification tasks. This emphasizes the importance of treating GPT-generated content not as a direct replacement for real data but as a conditional support whose value depends heavily on task complexity and granularity. Our study thus contributes both a practical takeaway and a cautionary insight into the role of LLMs in data-centric AI workflows.