# Synthetic Text as Data: On Usefulness and Limitations
## 4 Experiments
### Assessing the Substitutive Value of GPT-Generated Text
The experimental results are summarized in the Experiment 2 tab at [[https://seoyeonc.github.io/MBTI_dashboard](https://seoyeonc.github.io/MBTI_dashboard)]([https://seoyeonc.github.io/MBTI_dashboard](https://seoyeonc.github.io/MBTI_dashboard)) (accessed on 10 May 2025), with only a subset visualized in Figure 5. Overall, we found that GPT-generated text could improve performance relative to the one-shot baseline. However, its effectiveness remained consistently below that of real data, even when the quantity of synthetic samples increased. This suggests that while GPT outputs may appear fluent and coherent on the surface, they often lack the subtle, class-discriminative signals necessary for fine-grained classification. Figure 4: Visualization of ENFJ data distribution in training and testing sets, comparing the baseline Kaggle dataset against incremental GPT-augmented data integration. The \(x\)-axis quantifies the number of samples. Blue bars signify samples derived solely from Kaggle data, visible on the left side of the chart, and red bars represent additional GPT-generated samples included for the training set, shown on the right side. In conclusion, GPT-generated text appears to be a useful supplement to training data, especially under conditions of scarcity. However, it falls short of functioning as a true substitute for real-world samples in terms of informational richness and diversity. This finding highlights a key limitation and provides an important cautionary insight into the role of synthetic text in supervised learning tasks. The previous experiment demonstrated a consistent performance gap between GPT-generated text and real-world data in fine-grained classification. However, while this result was clear, it also raised a new question: does such a difference imply a binary distinction between usefulness and uselessness, or might synthetic data still retain partial informativeness? Rather than concluding that GPT-generated text is entirely lacking in value, we sought to explore whether it might preserve some--but not all--of the characteristics necessary for accurate classification. For instance, consider the ENFJ personality type. To convincingly replicate an ENFJ-written post, a model must reflect all four dimensions--E, N, F, and J. If GPT-generated text reflects only some traits (e.g., E and N), it may not resemble a complete ENF profile, but rather a partial form like EN-- or E--J (in this notation, the \"\(-\)\" symbol indicates an MBTI trait that is unclear or not strongly represented in the text). In such cases, the synthetic text may enable coarse-grained differentiation (e.g., extroverted vs. introverted) but fail in more nuanced distinctions between similar personality types. This motivated a refined hypothesis: while GPT-generated content may underperform in subtle classifications, it could still be effective when distinguishing between types that are more linguistically and conceptually distant. To examine this hypothesis, we designed a binary classification task in which ENFJ is contrasted against five MBTI types presumed to be relatively dissimilar: ISTP, INTP, ISFP, ISTJ, and ESTP. The experimental configuration followed the same few-shot structure as before. The ENFJ class contained only a single real training example, with 30 fixed test samples. GPT-generated ENFJ texts were added to the training set in increments of 20, up to a total of 140 samples. The data distributions for the comparison types were kept consistent with the original Kaggle dataset. The full results of this experiment are available in the Experiment 3 tab at [[https://seoyeonc.github.io/MBTI_dashboard](https://seoyeonc.github.io/MBTI_dashboard)]([https://seoyeonc.github.io/MBTI_dashboard](https://seoyeonc.github.io/MBTI_dashboard)) (accessed on 10 May 2025), with key findings visualized in Figure 6. Interestingly, the results diverged from previous patterns.