EPIC: Effective Prompting for Imbalanced-Class Data Synthesis in Tabular Data Classification via Large Language Models

Authors: jinhee  kim
taesung  kim
jaegul  choo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:14:06

Abstract: Large language models (LLMs) have demonstrated remarkable in-context learning capabilities across diverse applications. In this work, we explore the effectiveness of LLMs for generating realistic synthetic tabular data, identifying key prompt design elements to optimize performance. We introduce EPIC, a novel approach that leverages balanced, grouped data samples and consistent formatting with unique variable mapping to guide LLMs in generating accurate synthetic data across all classes, even for imbalanced datasets. Evaluations on real-world datasets show that EPIC achieves state-of-the-art machine learning classification performance, significantly improving generation efficiency. These findings highlight the effectiveness of EPIC for synthetic tabular data generation, particularly in addressing class imbalance.

Main paper: 
 introduction: 
 Tabular data, consisting of mixed variable types such as numerical and categorical variables, represents a widely applicable and essential data format [30] . It plays a crucial role in enhancing decisionmaking and efficiency in various real-world applications, such as finance, healthcare, manufacturing, and natural sciences [1, 3, 26] . However, challenges such as data scarcity and class imbalance, particularly for rare yet crucial events, often significantly degrade the performance of machine learning (ML) models, often resulting in reduced accuracy for underrepresented classes. To address these challenges, efficient synthetic data generation methods have been developed to augment tabular datasets. Traditional methods, such as synthetic minority oversampling technique (SMOTE) and its variants [5, 11, 21] , focus on generating minority class samples to alleviate class imbalance. Recently, advanced generative models, such as TVAE [35] , CTAB-GAN [39] , and TabDDPM [19] , have shown promising results in producing high-quality synthetic tabular data.Large language models (LLMs) have also demonstrated significant potential for generating realistic tabular data, effectively handling both numerical and categorical variables [4, 37] . However, these models often require extensive, data-specific fine-tuning, which can increase the risk of overfitting to majority classes or dominant feature values, especially in small and highly imbalanced tabular datasets. An alternative approach involves leveraging the in-context learning capabilities of LLMs.Existing work has shown that LLMs can solve complex reasoning tasks, learn and mimic patterns from input data, and augment textual datasets without parameter updates [13, 20, 34] .However, designing effective prompts to optimize this capability for tabular data generation is challenging, especially for imbalanced datasets. This is because tabular data is not naturally expressed in textual form and involves unique challenges, such as identifying feature correlations and accurately representing underrepresented attributes. Therefore, carefully crafted prompt design tailored to synthetic tabular data generation is essential to fully leverage the capabilities of LLMs. While several studies have employed in-context learning with LLMs for synthetic tabular data generation [27, 37] , few have conducted comprehensive investigations into optimal prompt designs that significantly impact data quality and generation efficiency.In response, this study examines key components of prompt design and identifies an effective method for generating high-quality synthetic tabular data, particularly addressing class imbalance. We introduce a novel approach, EPIC, which leverages the in-context learning capabilities of LLMs to produce synthetic tabular data with balanced class representation. EPIC incorporates prompt design strategies such as CSV style formatting, balanced class grouping, and a unique variable mapper, which together contribute to generating synthetic data that accurately represents class-specific distributions and feature correlations, as illustrated in Fig. 1 .Extensive evaluations across six real-world datasets demonstrate that EPIC significantly improves the data quality and generation efficiency, achieving state-of-the-art performance. As shown in Fig. 2 , baselines exhibit low sensitivity, struggling to accurately generate minority class samples due to inherent class imbalances. In contrast, EPIC achieves high sensitivity and balanced performance for all classes, underscoring its robustness and practicality for real-world applications. Figure 3 : Overview of our approach. Our prompt includes repeated data example sets consisting of feature names and class-balanced groups, with the feature name at the end serving as a trigger for the LLM to generate realistic synthetic tabular data. The proposed unique variable remaps categorical values to distinct alphanumeric strings, ensuring clear distinction and variability among variables.In summary, our key contributions are as follows:• Our study investigates the effectiveness of LLMs in generating realistic synthetic tabular data through in-context learning, offering prompt design guidelines for imbalanced datasets.• We propose EPIC, a simple yet effective prompting method that uses balanced, grouped data samples with unique variable mapping to generate tabular data that accurately represents both minority and majority classes, preserving feature correlations and overall data distribution.• The proposed approach is model-agnostic, generally applicable to various LLMs, and easy to implement for any tabular data with minimal preprocessing requirements.• Extensive experiments on six real-world public tabular datasets and one toy dataset demonstrate the effectiveness of our approach, significantly improving ML classification performance and data generation efficiency.By addressing class imbalance and enhancing classification outcomes, our work contributes to the advancement of the crucial field of tabular data research, significantly impacting various domains. 
method: 
 In this study, we investigate various prompt design components to maximize the in-context learning capabilities of LLMs for generating high-quality tabular data. Our objective is to develop an optimized approach that reliably and effectively produces realistic tabular data, accurately representing both minority and majority classes to improve ML classification performance, especially addressing class imbalance. Formally, given a tabular dataset T of dimensions n × m, with n samples and m variables, we aim to generate a synthetic dataset T of dimensions n ′ × m that accurately reflects key characteristics of T , including class-specific attributes, feature correlations and data distributions.To achieve this, we explore the following key prompt design elements: data format (Section 2.1), class presentation methods (Section 2.2), variable mapping (Section 2.3), and task specification (Section 2.4). These design choices are evaluated through extensive analyses on public datasets from diverse domains, focusing on ML classification performance and generation efficiency. Detailed analysis is provided in Section 3, with Fig. 12 in Appendix C.1 illustrating the ablated versions.Building on these analyses, we introduce EPIC, a structured prompting method designed to guide LLMs in synthetic data generation while effectively addressing class imbalance, as depicted in Fig. 3 . EPIC begins with optional variable descriptions, followed by a series of structured data sample sets, each containing balanced samples for each class. Each set consists of feature names and samples organized by target class. To prompt the LLM to generate a corresponding set of synthetic samples, feature names are appended at the end of the prompt to serve as a trigger, leveraging the pattern recognition capabilities of LLMs. The following sections detail each design component of EPIC.We investigate effective data formatting methods to represent tabular data within prompts, ensuring that LLMs can interpret the demonstrations and generate new data accurately. Tabular data can be formatted as plain text or as comma-separated values (CSV style). Let the feature name of the k-th variable be v k , and the i-th observation of this variable be o k,i . According to previous work [4] , this value can be transformed into a sentence-style representation as [v k , "is", o k,i , ", "]. In contrast, the CSV style can be expressed as [o k,i , ", "], presenting only values, with variable names for all variables specified as v = [v 1 , ", ", v 2 , ", ", • • • , v m ] at the beginning of the data sample presentation.Here, the sentence style redundantly uses variable names and "is" tokens for every value, leading to higher token usage and computational cost than the CSV style. The CSV-style format enables a higher volume of in-context learning examples within the same token constraints. Thus, our method utilizes a CSV-style format to maximize the number of examples, as providing a large number of data samples is crucial for ensuring sufficient representation of the original dataset in in-context learning.Class presentation methods aim to ensure adequate representation of all target classes within a prompt, particularly addressing underrepresented classes in imbalanced datasets. The construction of data samples within the prompt is crucial, as it can either worsen or mitigate existing imbalances. To address this, we explore three prompt design options and propose a class balancing and grouping approach that enables accurate and representative generation of data samples across all target classes.Single-class vs. Multi-class generation When generating data under class conditions, a primary consideration is whether to generate data for one class at a time or for all classes simultaneously. Single-class generation allows the model to focus on the unique characteristics of a specific class, while multi-class generation enables direct comparison across classes. Our experiments indicate that generating data for multiple classes simultaneously in a structured manner (discussed in the following sections) yields samples that more accurately capture the distinctive features of each class. This finding suggests that generating data with contextual awareness of other class characteristics enhances the representativeness of synthetic samples.When constructing prompts to generate multiple classes simultaneously, simple random sampling from the original dataset T often results in an imbalanced distribution of samples across target classes. This imbalance may cause the LLM to overfit to the majority class, limiting its ability to learn the characteristics of the minority class. To mitigate this, we employ a balanced sampling approach, equalizing sample numbers for all classes rather than strictly following the original class distribution. Our findings reveal that balancing sample sizes substantially improves the quality of generated data for minority classes, indicating that providing balanced data enables LLMs to learn more effectively and replicate the accurate characteristics of all target classes.Listing vs. Class grouping Within a prompt, data samples can either be presented sequentially, in the order they are sampled, or grouped by class. Grouping emphasizes contrasts between different classes and reinforces similarities within each group, enabling the LLM to generate more distinct and coherent samples for each group. Empirical evidence suggests that grouping yields a better representation of class-specific feature correlations in the generated data. Class grouping also contributes to constraining the LLM to generate data with specific attributes more efficiently. Without grouping, guiding the LLM to produce samples with targeted attributes can be challenging, often requiring numerous iterations to achieve the desired outcome. In contrast, grouping samples by specific features facilitates conditioning the LLM to generate data in groups with particular attributes or conditions. Although this work primarily focuses on class conditions, this approach can also be applied to feature grouping, enabling controlled generation based on selected criteria.Unlike existing methods that often require extensive preprocessing and handling of noisy or missing data [19, 40] , our approach minimizes these steps, preserving the integrity of the raw data, including original feature names and values, similar to other LLM-based approaches [4] . However, a critical challenge arises when a dataset contains numerous categorical variables with identical values (e.g., extensive boolean variables). In such cases, data examples in the prompt may become monotonous and repetitive, making it difficult for the model to distinguish between variables, which can significantly degrade the quality of the generated data. Moreover, when faced with such repetitive input, LLMs tend to struggle to generate valid samples, leading to a notable decline in generation efficiency.To address this, we propose a unique variable mapping method, which remaps the values of each categorical variable to distinct random alphanumeric strings, as illustrated in Fig. 3 . Uniform substitution is applied across the dataset, maintaining the integrity of the data structure while introducing necessary variation. For example, consider a categorical variable v 1 with values of 't' and 'f'. Our approach remaps these values to distinct three-character strings, such as 'JY0' and 'GGN,' respectively, and applies them consistently across all data points in the dataset. Other categorical variables are similarly remapped to unique values. The transformed dataset is then used as in-context learning demonstrations. Although these new values do not hold inherent semantic meaning, they effectively represent categorical distinctions as symbols, allowing the LLM to identify and utilize patterns within them. This approach ensures that each variable has a unique representation, making the variables clearly distinguishable and leading to more accurate and efficient data generation.A crucial aspect of the prompt design is clearly guiding the LLM to generate synthetic data as intended. To achieve this, we consider two options and ultimately find that prompting the LLM to learn the pattern from a structured, consistent format with optional descriptions for variables is more effective for reliable generation than providing explicit instructions.To clarify the task for the LLM, a prompt might include explicit instructions, such as "generate new data samples." However, crafting specific instructions can be challenging, as LLMs are highly sensitive to subtle prompt variations, making it inefficient to determine an optimal phrasing for tabular data generation. A simpler and more effective approach is to provide patterns within the prompt for the LLM to mimic by formatting data samples without explicit instructions, leveraging its advanced pattern recognition capabilities. Formally, let the i-th set of data samples across all c classes be structured asS i = [v, s i,1 , • • • , s i,c ],where each s i,k contains n samples for the k-th class. Here, v represents feature names and serves as a header to indicate the start of a set. This approach enables the LLM to recognize the pattern in which each set of data samples is structured and consistently begins with v. We then input t such sets in a prompt, [S 1 , S 2 , . . . , S t ], to establish a recognizable pattern, expecting the LLM to generate the next set,S t+1 = [v, s t+1,1 , • • • , s t+1,c ].To reinforce this, we place a trigger at the end of the prompt by including the new header [v] for S t+1 , signaling the LLM to complete the sequence with[s t+1,1 , • • • , s t+1,c ].By structuring the input prompt in this way, we guide the LLM to generate a consistent number of samples, n, for each target class.Providing contextual information Optionally, adding contextual information to the prompt can help the LLM understand the characteristics of the original dataset and generate accurate samples. Therefore, when available, we include line-by-line variable descriptions at the beginning of the prompt. Additionally, we treat the target class variable as one of the attributes and position it as the first variable to clarify that each group is organized by the target class.The synthetic data generation process begins by creating a prompt template and determining whether the unique variable mapping is needed. The prompt template specifies parameters such as the target class, sample size per group, the number of groups, and the number of sets to construct demonstrations.We then randomly sample data examples from the dataset to populate the template, ensuring that examples do not overlap within each prompt to maintain diversity. Using the provided examples, the LLM generates new data in a CSV format, which is subsequently converted into a structured tabular format. Instances containing categorical variable values not present in the original dataset are discarded to maintain authenticity. This generation process is repeated until the desired sample size is achieved, with new examples sampled with replacement in each iteration. This approach enables the LLM to encounter diverse combinations of real samples and produce rich synthetic samples (see Appendix B.1 for analysis). Although each individual generation reflects the distribution of the provided subset, successive iterations enable the LLM to cover a broader range of the original data cumulatively. This iterative method ensures that the output generated by the LLM comprehensively reflects the characteristics of the original data. Final prompting examples are available in Appendix C. 5  
experiments: 
 This section describes the experimental setup and provides a comprehensive evaluation of our method, assessing its effectiveness across diverse real-world datasets.Summary We conduct three unique experiments to analyze the utility of our method in enhancing ML classification performance: augmenting the original dataset with generated data (Tables 1, 19 , Figs. 2, 14) , augmenting only the minority class similar to SMOTE (Table 6 ), and using only generated data (Table 7 ). Additionally, we evaluate our approach using three LLMs: Mistral, Llama2, and gpt3.5-turbo (Table 2 , Figs. 4, 5 ). We perform ablations on prompt elements to compare classification performance (Tables 3, 5 ) and conduct unique analyses of token usage and LLM generation efficiency (Tables 4, 8, 9 ). To examine feature correlations, we separately analyze minority and majority classes, comparing results across all prompt variations (Figs. 4, 5 ). We further investigate how varying the number of generated samples affects classification performance (Figs. 6, 7 ). Using a toy dataset, we analyze how different prompts influence the accuracy of generated data distributions (Figs. 1, 9 ). Lastly, we explore the sampling of input examples and their corresponding outputs in LLMs using the toy dataset, providing insights into the variability and reliability of generated data (Figs. 10, 11) .Datasets We evaluate our method using six real-world public tabular classification datasets from diverse domains: Travel (Marketing), Sick (Healthcare), HELOC (Finance), Income (Social science), Diabetes (Healthcare), and Thyroid (Healthcare). The Thyroid dataset, released after the training cutoff date for GPT-3.5-turbo-0613, provides a more rigorous validation of our approach on completely unseen data. For binary classification datasets, the minority class is designated as one. All duplicate data samples are removed from the datasets. Each dataset is randomly split into 80% training and 20% test sets. We retain the original data, including missing or noisy features. The exception is the Sick dataset, where we follow the source's method. Further details are provided in Appendix C.2.Evaluation measure We evaluate our method based on ML classification performance, generation efficiency, feature correlation, and analysis of generated data distribution. For classification, we report F1 score, sensitivity, specificity, and balanced accuracy (BAL ACC). Feature correlation is measured using Pearson correlation for numerical variables and Cramér's V correlation for categorical variables.Baseline models In our study, we compare our method with various generative models for tabular data, including SMOTE [5] , SMOTENC [5] , TVAE [35] , CopulaGAN [22] , CTGAN [35] , CTAB-GAN [39] , CTAB-GAN+ [40] , GReaT [4] , and TabDDPM [19] . To compare prompting methods, we also use the prompts from CuratedLLM [27] and LITO [37] .Experimental details Our method utilizes the GPT-3.5 models (GPT-3.5-turbo-0613 and GPT-3.5-turbo-16k-0613), Mistral-7b-v0.1 [15] , and Llama-2-7b [29] . Unless stated otherwise, we use the GPT-3.5 model for our method. The number of synthetic data samples is based on the size of the original datasets. Across all experiments, unless stated otherwise, we report results from four top-performing ML classifiers: XGBoost [7] , CatBoost [24] , LightGBM [17] , and Gradient boosting classifier [10] , known for their strong performance, often surpassing recent deep learning models on tabular datasets. Each classifier is executed in five independent runs, with results averaged over a total of 20 runs to ensure robustness. Further details are available in Appendix C. We evaluate the quality of the synthetic data samples by assessing the machine learning classification performance when the synthetic data are added to the original dataset. Here, our method utilizes the GPT-3.5-turbo model. As shown in Table 1 , our method achieves state-of-the-art F1 scores and balanced accuracy across all six datasets, surpassing the baselines that require model training. Notably, our method is the only one that consistently outperforms the original data in both F1 score and balanced accuracy across all six datasets. For a machine learning model to perform well in classification, the correlation between input data and labels in the training data must be precise. Thus, the consistent improvement across six datasets demonstrates that the data generated by our method aligns well with the class labels. Other methods often result in worse performance than the original data, indicating that the data generated by baselines disrupts the original data distribution. These findings underscore the robustness and effectiveness of our method, affirming its superiority over baselines, even for challenging datasets.Moreover, baselines such as GReaT and TabDDPM exhibit significantly lower sensitivity compared to specificity, particularly for Travel and Income. Their high balanced accuracy is due to high specificity, but their severely low sensitivity indicates a failure to generate appropriate minority class data. In contrast, our method significantly improves sensitivity and achieves a balance between sensitivity and specificity while also attaining the best-balanced accuracy and F1 score compared to the baselines. For example, in Travel, our method achieves a sensitivity of 78%, which is 19%p higher than the second-best method, GReaT, at 58.8%. In Income, our method shows 66.45% sensitivity, surpassing the second-best score of 60.69%. These results highlight the effectiveness of our method in accurately representing minority classes in tabular datasets.Overall, our findings demonstrate that our approach is generally applicable across various imbalanced tabular datasets and excels in generating high-quality samples that improve ML classification performance. Further analyses exploring the impact of augmenting the original dataset for the minority class (Table 6 ) and replacing the original dataset with synthetic data (Table 7 ) are available in Appendix A.We also apply our prompting method to open-source LLMs, including Llama2 [29] and Mistral [15] . As shown in Table 2 , our method exhibits robust performance when used with open-source LLMs across various datasets. Notably, in the Sick dataset, Mistral demonstrates the highest balanced accuracy, indicating its effectiveness in generating high-quality tabular data. These results validate the broader applicability and general effectiveness of our approach with open-source LLMs. We investigate key prompt design choices and validate our method through extensive ablation studies across multiple datasets from diverse domains, focusing on ML performance and generation efficiency, as detailed in Tables 3 and 4 , respectively. Given a fixed number of input samples, we evaluated (1) the number of input tokens required, (2) the number of valid generated samples, and (3) the generation success rate. Our results indicate that CSV-style prompting generally outperforms sentence-style prompting in ML performance with the same number of input samples. While generating data for one class at a time yields good results for the Sick and Income datasets, it results in a low F1 score of 58.67% on the Travel dataset. However, generating data for multiple classes simultaneously with balanced samples improves this score to 70.37%. Furthermore, applying unique variable mapping consistently enhances F1 scores on the Sick dataset. The combined approach of random mapping and grouping achieves higher F1 score of 91.95% compared to 86.60% with random mapping but without grouping, underscoring the positive impact of this combined approach. These findings indicate that prompt design significantly influences the quality of generated data, emphasizing the need for carefully tailored prompts to generate high-quality synthetic tabular data.For the Travel and Income datasets, performance without grouping is slightly better than with grouping, but the differences are within 0.5%p, making their performance on par. However, these show significant differences in LLM generation efficiency. Without grouping, an average of 0.88 samples are generated per iteration, whereas grouping increases this to an average of 8.6 samples per iteration. Repeating the set twice within the prompt further improves generation efficiency, boosting the average output to over 10 samples and the success rate to 94%, by creating input patterns that LLMs can mimic to generate data. Unique variable mapping also significantly enhances LLM generation efficiency and success rates, as detailed in Table 8 of Appendix A.3.Furthermore, we conduct an ablation study on the task specification elements, as shown in Table 5 .Our approach of triggering completion significantly enhances overall ML performance compared to the instruction-based prompting methods. Interestingly, adding class distinctions within the prompt achieves the best sensitivity, indicating that the distinction facilitates better generation for minority classes. Providing variable descriptions improves balanced accuracy but reduces the F1 score, suggesting that this element may be optional rather than a definitive improvement. A comparison of generation efficiency using these methods is provided in Table 9 of Appendix A.3.We compare the feature correlations of the categorical variables between original and synthetic data, as illustrated in Fig. 4 . Our synthetic data generated with Mistral and Llama2 exhibit the closest feature correlation with the original data for both minority and majority classes, outperforming the baselines and ablated versions. Additionally, our grouping method significantly contributes to preserving feature correlation for the minority class. These findings indicate that our method generates data that more accurately matches each class, particularly for underrepresented classes, compared to other methods. Additional results on the Sick dataset are available in Fig. 5 of Appendix A.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We publicly release the source code, along with instructions to reproduce the experimental results. Guidelines: • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations and future work: 
 When the training dataset is large and cannot be fully included in the LLM prompt due to token size limitations, only a subset of the data can be used as examples for generating samples. If these prompt samples do not fully represent the original data distribution, the generated data may be incomplete and of low quality, as LLMs are limited to producing data based only on the patterns present in the input. To overcome this, our method employs multiple rounds of random sampling with replacement to create a synthetic dataset that more comprehensively represents the original distribution, resulting in improved machine learning classification performance (a comprehensive analysis is provided in Appendix B.1). However, this approach still carries the risk that samples may not fully capture the original data distribution. Future research will focus on developing methods to identify key examples that more accurately represent the entire dataset. 
conclusion: 
 This study demonstrates the effectiveness of using LLMs for synthetic tabular data generation. We introduces EPIC, a simple yet effective solution for generating realistic, high-quality data without additional training, specifically designed to address class imbalance. Our method demonstrate significant improvements over state-of-the-art generation models across six real-world public datasets, generating data with highly accurate feature correlations and significantly improving ML classification accuracy for minority classes. These results underscore the substantial impact of our approach in real-world applications, making a significant contribution to the field of tabular data research. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The paper specifies the training and test details, including splits, hyperparameters, and their selection process, in Section 3 and Appendix C. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: The paper reports standard deviations in Fig. 2 and ML classification performance tables in main manuscript, and the experimental results in Section A and Section F.2. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No data or models are being released in this paper, so there is no risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets used in the paper are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This supplementary material enhances the main manuscript by providing detailed experimental results and additional visualizations.• Appendix A provides additional experimental results on the machine learning classification for imbalanced data, ablation study, and feature correlation analysis.• Appendix B studies the impact of sample size on the machine learning performance and comprehensively analyzes the experiments conducted on the toy dataset.• Appendix C provides comprehensive information on the datasets, baselines, implementation details, and prompt design and examples.• Appendix D provides related work.• Appendix E discusses the broader impacts of our research.• Appendix F presents confusion matrix results and the complete results of Table 1 .Our experiments with synthetic tabular data for imbalanced classes follow three main approaches: (1) Adding synthetic data to both minority and majority classes of the existing dataset and evaluating machine learning classification performance. The results of this approach are presented in Table 1 of the main manuscript. ( 2) Adding synthetic data only to the minority class, with detailed results provided in Appendix A.1. (3) Evaluating classification performance using only synthetic data without the original dataset, as discussed in Appendix A.2. Moreover, Appendix A.3 presents further ablation study results, and Appendix A.4 provides a feature correlation analysis for the sick dataset.One of the major advantages of our method is its ability to generate high-quality synthetic data for minority classes, even with limited samples. Similar to oversampling techniques like SMOTE, we add synthetic dataset only to the minority class and evaluate machine learning classification performance.To balance the ratio of majority to minority classes, we add only minority class samples to the original data and then measure the classification performance of the machine learning model. For this task, we additionally investigate the effectiveness of our method compared to SMOTE and SMOTENC [5] .As shown in Table 6 , adding minority class samples generated from our method leads to higher We compare the classification performance using only synthetic data for training, as shown in Table 7 . For both the Travel and HELOC datasets, the classification model trained on synthetic datasets generated by our model outperforms the one trained on original data in terms of F1 score and sensitivity. This improvement in performance suggests that the data generated by our method accurately represents the distribution of the original data. Furthermore, it highlights the effectiveness of our class-balanced generation approach in improving the performance of classification models.This section presents additional ablation study results on the Sick datasets. We evaluate the token efficiency and generation stability of ablated versions based on 100 inferences, each with 20 random input samples. As shown in Table 8 , for the Sick dataset, due to its repetitive, monotonous values, the number of generated samples is low, even with grouping. Using a unique variable mapper in such cases significantly increases the number of samples generated per attempt from 6.93 to 17.68. This improvement demonstrates that our approach effectively controls the quantity of generated data samples, ensuring the stable production of synthetic tabular data. Moreover, the ability to generate multiple data samples in a single inference can significantly lower generation time and costs.We also compare the efficiency and stability of generation when ablating task specification elements in our prompt design, as shown in Table 9 . The prompt design from Curated LLM [27] , despite requiring the highest number of input tokens, achieves high performance in terms of output samples and success rate. However, the generated data shows relatively low quality, resulting in a low classification performance with an F1 score and balanced accuracy, as discussed in Section 3.4. In contrast, the prompt design of LITO [37] is the most efficient in terms of input tokens but produces the fewest output samples and has the lowest success rate, indicating instability in synthetic data generation.Our proposed method balances efficiency and performance, using approximately half the input tokens compared to CuratedLLM's prompt while generating a high number of output samples and maintaining a high success rate. Additionally, the synthetic data generated by our method achieves the highest classification performance in terms of F1 score, as shown in Section 3.4. When class distinctions are provided, the average number of output samples decreased from 17.68 to 9.70. This result indicates that it is more robust to distinguish groups with simple numbering, such as A and B, rather than explicitly specifying class information in each group. Including variable descriptions slightly reduces the number of output samples and success rate compared to not including them, but the difference is not significant. This suggests that the grouping strategy plays a more crucial role in robust data generation than the presence of variable descriptions.In the main manuscript, we visualize the feature correlation between categorical variables. Here, we analyze the Pearson correlation between numerical variables in the Sick dataset and compare the differences in feature correlation values to the original data, as shown in Fig. 5 . As a result, our methodology consistently shows feature correlations similar to the original data for both minority and majority classes. In contrast, other baselines either show significant deviations in feature correlation for all classes or perform well for majority classes but not for minority ones. These results demonstrate that our methodology shows superior performance compared to the baselines in generating highquality data that accurately represents both minority and majority classes within the original dataset.This section provides an in-depth analysis of the proposed method, EPIC. Appendix B.1 analyzes the effect of sample size on classification performance. Appendix B.2 discusses the significance of the observed performance improvements. Appendix B.3 provides complete qualitative results on the toy dataset. Finally, Appendix B.4 explores tabular data classification with in-context learning methods.Adding synthetic data to the original dataset Data samples in prompts can be selected either with or without replacement. Our method uses sampling with replacement to generate a large and diverse dataset, as sampling without replacement limits the amount of synthetic data to the number of actual data points. While we ensure that there are no overlapping examples within each prompt to maintain diversity, each prompt is constructed using sampling with replacement.We conduct experiments to analyze (1) how much datasets can be enlarged and (2) how this impacts classification performance by comparing sampling with and without replacement. As shown in Fig. 6 , performance improves steadily in both scenarios as the volume of generated data increases. However, the improvement is constrained when sampling without replacement due to the limited number of possible samples. When sampling with replacement, as the dataset size expands, there is a noticeable improvement in the balance between sensitivity and specificity, which contributes to enhanced overall performance, including gains in balanced accuracy and F1 score. Generating up to 40K synthetic data points resulted in even better performance than the 20K synthetic data points reported in Table 1 of our main manuscript. We also observed that as the volume of generated data continues to increase, the gains in balanced accuracy and F1 score eventually plateau, indicating diminishing returns and suggesting that further data generation beyond a certain point offers limited additional benefit.We investigate how varying the number of synthetic samples affects classification performance. This analysis is conducted on the Income and Diabetes datasets, and the results are shown in Fig. 7 . For the Income dataset, sample sizes range from 500 to 10,000, while for Diabetes, the range was 500 to 20,000. For both datasets, a consistent increase in the F1 score is observed as the number of samples grows. This result demonstrates that our method can adequately represent the original data distribution by generating a sufficient number of synthetic data samples.We analyze the significance of performance improvements achieved by EPIC. As shown in Fig. 6 , while the gains in F1 score and balanced accuracy between 0 and 40K samples may seem modest, the sensitivity actually increases significantly from around 50% to 70%. This improvement balances performance across all classes, leading to a more meaningful outcome and greatly enhancing model usability. Baselines often learn biases in the training data, resulting in abnormally high specificity at the expense of sensitivity. This imbalance can inflate balanced accuracy or F1 scores, but such performance is ineffective for real-world tasks. In contrast, our method significantly enhances sensitivity with only a slight reduction in specificity. As a result, our method achieves the most balanced performance across all four metrics, as shown in Fig. 8 .Our method is the only approach among the seven baselines that consistently improves both F1 score and balanced accuracy compared to using only the original data (Table 19 ). Extensive experiments across six real-world datasets from diverse domains (finance, healthcare, marketing, and social 5) comprehensive data distribution coverage, with improvements over its ablated versions and the fine-tuned GReaT model [4] . science) using four classifiers, each tested five times, demonstrate the state-of-the-art performance of our method, with significant gains in the highly imbalanced Travel, Sick, Income, and Thyroid datasets (Fig. 13 ). While GReaT and TabDDPM are the closest baselines, they exhibit inconsistent performance. As highlighted in Table 10 , they reduce original data performance in more than half of the cases (red). For example, both underperform on HELOC and Diabetes.In stark contrast, our method consistently outperforms the original data across all datasets (blue). In the challenging Diabetes dataset, only our method improves over the original data. While specificity decreases in all methods, our method still achieves a superior balance of metrics, leading to greater practicality. These results underscore that our method offers the greatest practical utility and a meaningful performance advantage among the models tested.In Fig. 9 , we illustrate the complete qualitative results on the toy dataset. We assume a challenging multivariate data scenario in an imbalanced binary classification situation, where variables with and without correlations are mixed. Using the scikit-learn library [23] , we created a total of 12 variables. These variables have strong correlations in pairs. There are 11 numerical variables and one categorical variable, and the categorical variable is shown on the x-axis in the last row of Fig. 9 . In the figure, the majority class is shown in blue, and the minority class is shown in red.The generated results are evaluated from four aspects: (1) assessing whether generated data samples correctly belong to their respective classes, ensuring clear class boundaries; (2) examining whether the shapes formed by variable correlations, such as circles or moon patterns, are accurately represented;(3) evaluating whether the generated values fall within an acceptable range for each variable; and (4) analyzing whether relationships between categorical and numerical variables are accurately captured. Here, with 180 input samples from the imbalanced toy dataset per inference, using the ablated method without grouping requires a total 23 inferences to generate 1,000 samples.The baseline method, GReaT [4] , has successfully learned from the training data to generate classseparated samples. However, upon closer inspection, there are samples belonging to different classes. Ours, despite generating samples without training, shows similar or better quality.Overall, CSV-style generation produces better class separation boundaries compared to sentencestyle generation. Regarding class presentation, giving a single class can sometimes result in good generation outcomes, but it fails to accurately capture the correlation with the majority class, resulting in shapes like squares instead of circles, and it also fails to accurately generate the categorical variable. On the other hand, when generating multi-class samples, if the sample numbers are not balanced, the generation results for the minority class can be compromised. Even when balanced, providing grouping leads to clearer class distinctions. Additionally, adding unique variable mapping significantly improves the performance, resulting in a well-distinguished categorical variable.In addition, we visualize the inputs and outputs of both our method and the ablated model without grouping, as shown in Fig. 10 and Fig. 11 . For each inference, 180 class-balanced samples are input into the LLM prompt. To generate a total of 1,000 samples, our method required 23 inferences, whereas the ablated model needed 37 inferences, demonstrating the stability of our method in generating synthetic data samples. In terms of data generation quality, our method successfully generates both majority and minority classes in most inferences, with the distribution of the generated samples closely mirroring that of the actual input data. In contrast, the ablated model exhibits a higher number of inferences where the generated data deviates from the input data distribution.The ability of LLMs to generate high-quality synthetic data for imbalanced classes in our approach highlights their potential to directly address classification tasks, offering a promising avenue for future research. However, the challenge lies in designing effective prompts to fully leverage this potential.The performance of LLMs is highly dependent on prompt design, and numerous studies have shown that small variations in prompt crafting can lead to significant differences in outcomes [18, 25] . Our work shares this motivation, focusing on designing prompts that enable LLMs to effectively generate high-quality tabular data, particularly to address class imbalance. While exploring LLMs for direct classification could enhance performance, this direction is beyond the scope of our current study. Here, with 180 input samples from the imbalanced toy dataset per inference, using the ablated method without grouping requires a total 37 inferences to generate 1,000 samples.We conduct experiments using recent in-context learning-based tabular classification methods, such as TabPFN [14] , which employs a pretrained transformer, and T-Table [33] , which leverages LLMs. We also test TabR [12] , an advanced deep-learning tabular classification model, using its official code. As detailed in Table 11 , we evaluate classification performance on the Travel dataset using the original and synthetic data. For T-Table , we use the GPT-3.5-turbo model, incorporating all original data but limiting synthetic samples to 200 per input prompt due to token constraints. To further enhance performance, we employ voting across five inferences [31] . The results indicate that, while the newly introduced classifiers do not outperform traditional ones, adding synthetic data generated by our method to the original data consistently leads to the significant performance improvements across all classifiers. In contrast, using TabDDPM and GReaT often results in decreased performance. Our method uniquely and consistently enhance the performance of different classifiers, demonstrating superior label matching quality. These findings underscore the value of high-quality synthetic data in enhancing classifier performance across diverse models, highlighting the importance of data generation research as a distinct area from classifier development.Tabular data generation is a critical area of research with significant implications [4, 27, 37] . This task serves two primary purposes: (1) enhancing classification performance in a model-agnostic manner through data augmentation, similar to SMOTE, as shown in Tables 1, 2 , and 6; and (2) generating synthetic data to replace original data in security-critical or privacy-sensitive contexts, as demonstrated in Table 7 . In fields such as healthcare, where collecting new samples or achieving balanced class labels is challenging and data may be noisy or incomplete, generating high-quality synthetic data is crucial.In conclusion, even as more advanced classification models for tabular data are developed in the future, our proposed method could continue to play a crucial role in enhancing performance This section provides the illustration of ablated versions (Appendix C.1), comprehensive information on the datasets (Appendix C. This section visualizes five ablated versions of our proposed prompt design, as shown in Fig. 12 .The ablations involve removing or altering four key components: data format, balanced class ratio, multi-class generation, and class grouping.A detailed description of the datasets is shown in Table 12 . The variable descriptions are directly sourced from dataset platforms like Kaggle or the UCI repository. In our approach, we designate the class with fewer instances as the positive class for binary classification tasks. We also provide the class distribution information for the datasets used in this work, as shown in Fig. 13 . Among the six datasets, the Travel, Income, Sick, and Thyroid datasets exhibit substantial class imbalances. In our research, we utilize implementations from the Synthetic Data Vault [22] for TVAE [35] , CTGAN [35] , and CopulaGAN, while for the GReaT [4] model, we employ code from its official repository. In our experiment, the GReaT model fails to accurately generate column names 'max_glu_serum' and 'A1Cresult' in the Diabetes dataset, even after training for 85 epochs that reported in the original paper. Extending the training to 105 epochs does not rectify this issue, with 'max_glu_serum' and 'A1Cresult' being correctly generated in only approximately 2.8% and 12.1% of instances, respectively. For these columns, failure cases are treated as NaN value.We utilize the official open-source code from the official TabDDPM GitHub repository to reproduce CTAB-GAN [39] , CTAB-GAN+ [40] , and TabDDPM [19] . For the Adult Income and California datasets, the hyperparameters provided are used. For other datasets lacking provided hyperparameters, we apply the hyperparameters from the Wilt dataset, which, according to the corresponding paper, demonstrated the largest performance improvement over the original data.Due to the lack of official code provided by the authors of LITO [37] , we adopt the prompt design described in their original paper. Specifically, we use the prompt for LITO-C to generate minority class data. To generate samples for both majority and minority classes, we also use the modified prompt to generate majority class data. Similarly, for CuratedLLM [27] , we follow the prompt design detailed in their paper. Although LITO and CuratedLLM propose multi-stage techniques for data generation, we focus solely on evaluating their prompt designs.For classification, we utilize the target class of each dataset to divide groups. Specifically, the Diabetes dataset has three groups, and the Travel, Sick, HELOC, Income and Thyroid datasets have two groups. For the process of unique variable mapping, each discrete value is transformed into a combination of three characters, including uppercase letters and digits. Regarding hyperparameter settings of downstream machine learning models: For the gradient boosting classifier, we use the default hyperparameters provided by scikit-learn library. For XGBoost [7] , CatBoost [24] , and LightGBM [17] , we conduct 5-fold cross-validation on the training set of the original data, optimizing two key hyperparameters: learning rate and max depth. These optimized settings are then consistently applied across all experiments, aligning the model's performance closely with the characteristics of the original data. Where possible, we train the classification models using a single NVIDIA GeForce RTX 3090 GPU on an Ubuntu 18.04 system.This section outlines the construction of the proposed prompt. As shown in the prompt examples in Tables 13 14 15 16 17 18 , the final prompt template for synthetic data generation consists of four parts.Descriptions (Optional). The prompt may start with descriptions of the variables in the dataset, provided line-by-line, if precise and available. These descriptions define what each column in the dataset represents, offering contextual understanding to the LLM about the data. It will be located at the top of the prompt (but not repeated to maintain token efficiency) for additional context.Set-level examples. Then, a set-level examples are present. This section begins with a data entry format header, specifying the order and names of each feature. This header acts as a guide for the LLM, indicating the start of an example set, which is crucial for interpreting the structure of the tabular data. Following the header, there's a line specifying a group name, signaling the start of a specific group. Instances belonging to this group are then listed, one per line, in a CSV style. This process is repeated for each group within the set.Repetition of set-Level template. To reinforce the data structure and patterns, this set-level template is repeated several times within the prompt, each time with distinct data samples. This repetition is key in enabling the LLM to recognize, learn, and subsequently replicate the set format in its synthetic data generation process.Trigger for LLM data generation. After all the example sets are presented, the same header that indicates the start of an example set is included at the end of the prompt. This header serves as a signal for the LLM to commence generating synthetic data, maintaining the structure and patterns established in the prompt.By following this structured approach, the LLM is guided to effectively generate synthetic data that matches the structure and characteristics of the original dataset.LLM in-context learning prompting With the remarkable advancements in LLMs, extensive research has been conducted in prompt engineering to maximize their potential for various natural language processing (NLP) tasks, including text summarization and question answering, yielding significant value [9, 28, 36, 41] . Optimizing prompts for specific tasks is inherently a combinatorial challenge, and in the absence of established optimization principles, progress has often been driven by heuristic methods validated through rigorous empirical evaluations [25, 32, 38] . Our study adopts this empirical approach, conducting extensive experiments to develop effective prompts for synthetic tabular data generation, which holds unique challenges that differ from typical NLP tasks. Through these experiments, we introduce a tailored prompt design, EPIC, that employs class-balanced, grouped, and structured formatting and unique variable mapping.Tabular data generation In the field of tabular data synthesis, traditional methods have predominantly focused on interpolation techniques, such as SMOTE [5] . Although useful, these methods often struggle to capture complex relationships between features [2] . Recently, generative models have shown promise in generating realistic tabular data [6, 8, 16, 35, 39, 40] , and diffusion models have also achieved promising results in this area [19] . Building on these advances, GReaT [4] achieved notable success by transforming tabular data into natural text format and fine-tuning LLMs for synthetic tabular data generation. However, this fine-tuning process is resource-intensive, requiring extensive training of large models for each dataset. To address this, LITO [37] and Curated LLM [27] have explored using LLMs through in-context learning for tabular data generation, yet these approaches require additional filtering steps to ensure data quality. In contrast, we demonstrate that with comprehensive prompt exploration, high-quality tabular data generation can be achieved through prompt design alone, without the need for additional steps. Our work is distinguished by the thorough and comprehensive evaluations across six real-world datasets and synthetic toy data, aiming to provide deeper insights into the prompting process. We addressed experiments and analyses often overlooked in prior research on tabular data generation.Our method is generally applicable to tabular data generation and classification tasks with minimal preprocessing, making it accessible for diverse research and industrial Potential positive impacts include enhanced data accessibility and privacy preservation, which can support decisionmaking processes in fields such as healthcare and finance reliance on real data protecting individual privacy. However, potential negative impacts risk of data misuse. Synthetic data generation be exploited maliciously, such as to create deceptive datasets intended to mislead systems or individuals. using closed-source LLMs poses potential risks, including data leakage through API calls, particularly concerning when handling sensitive and limitations on access verify models being used. Ensuring quality, security, and ethical use of generated data these conditions is essential.This section presents the confusion matrix results in Appendix F.1 and compares the classification performance with the baselines in Appendix F.2.present normalized confusion matrix results on binary classification datasets, as shown in Fig. 14 . The results demonstrate that adding the synthetic data generated by our method to the original dataset significantly enhances classification performance, outperforming baselines by a large margin. Additionally, using only the synthetic data generated by our method achieves better performance than augmenting the original dataset synthetic data from baselines. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The paper explores the effectiveness of LLMs in generating synthetic tabular data to address class imbalance in machine learning tasks. It introduces an approach that leverages in-context learning with LLMs to generate realistic and balanced synthetic data, enhancing the performance of machine learning models, particularly for minority classes in imbalanced datasets. The study investigates various prompt design strategies, including data formatting, class presentation, and variable mapping, finding that a CSV-style format and class-balanced grouping significantly improve data generation quality. The approach is validated across multiple real-world datasets, demonstrating its ability to improve classification performance and generate high-quality data without extensive model fine-tuning.

The paper is well-written and easy to follow. The reviewers had some questions and concerns about the experiments and analysis of the paper. During the rebuttal, the authors addressed these issues, and the reviewers were satisfied with the responses and additional experiments. Therefore, I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: The paper investigates how to use LLMs to generate synthetic tabular data for mitigating class imbalance in machine learning tasks. By exploring various prompting methods, the authors aim to identify key design elements that optimize the generation performance. The paper shows that using GPT3.5/Mistral/LLaMA to balancing classes, and employing unique variable mapping produces realistic and reliable data, enhancing classification performance (XGBoost etc) for minor classes in imbalanced datasets.
strength_weakness: strength_weakness: 1. Why not use the LLM itself to perform classification: The designed method uses LLMs to generate data examples for imbalanced classes, which means the LLMs must already have a good ability in modeling the data distribution even for the imbalanced classes. If that's the case, why not use the LLM itself to perform classification directly with the help of in-context learning? I expect that it will be a strong baseline compared to the XGBoost classifiers. The only downside of LLM-based classifiers may be the inference cost, but we should include this baseline.
questions: questions: 1. Could you provide the experiment results of in-context learning LLM-based classifiers on these tasks?
limitations: limitations: Yes in Appendix G
paper_summary: paper_summary: The paper explores the domain of tabular data generation using in-context learning with LLM, in order to improve performance of an ML classifier, especially in imbalanced classes scenarios. The paper explores different prompting techniques, with detailed results on 6 datasets as well as a visualized study on a toy dataset. The paper also compares the result performance boost on several ML classifiers, which is indeed significant. There's a further discussion of efficiency and stability of the generation itself, and how it is affected given the different prompting methods proposed. Results using three different LLMs are presented.
strength_weakness: strength_weakness: The main discussion that I missed in the paper was regarding - how much extra data ends up being generated, and does generating more and more data using this method make any sense? I understand that the method uses actual data points, and samples from the original dataset without replacement. This probably creates a limitation to data creation - a point which is not discussed in the main paper body, and should be added. Furthermore, it might make sense to scramble and re-sample the dataset to create even more samples. I would have liked to see a discussion of this point, both to how much the datasets are actually enlarged using the no-replacement sampling, and also if it makes sense to generate more data with replacement, with the limits of this method discussed - when do we reach repetition and duplication, without any further improvement to the classifiers. 
I'm also missing the original dataset sizes and minority vs majority class distributions - to show more strongly the class imbalance. In fact, no dataset descriptions are given at all, which in my opinion takes away from the paper.
questions: questions: I'd add which model was used in table 1 (I'm assuming it's ChatGPT). 
I would stress better the counts of additional synthetic data and why do you sometimes use 1K and sometimes 10K.
limitations: limitations: Limitations are discussed in the appendix, which is okay. I would perhaps add further discussion on social implications, clarifying the meaning of fig 10 - this method might create more bias in already biased data.
paper_summary: paper_summary: This paper proposed an LLM-driven tabular data class balancing approach, adapting the in-context learning paradigm and trying various formats and templates to explore optimal prompts to mitigate imbalanced tabular data. Experimental results showed that the proposed method alleviated the CSV-format data imbalance.
strength_weakness: strength_weakness: 1. The research purpose of this paper covers merely the CSV-format data, limiting the further adaptivity and impact of the proposed method.
2. This paper merely tried to find plausible optimal prompts by handcraft, lacking theory analysis, making the prompting even a black box.
3. Empirical results are not consistent on all test data, especially in the specificity aspect, indicating the instability of the prompting approach.
4. According to the prompting design, it seems that the prompting process is a bit sophisticated, however, there are no computing cost comparison experiments.
questions: questions: 1. This paper proposed to mitigate the label imbalance issue for tabular data by generating scarce label data, but how to guarantee the generated data and labels were matched?
2. The presented method is basically an instruction prompting approach, and why didn’t the authors analyze the LLM during the prompting process to make it more traceable?
3. How did the proposed method handle missing data or noisy features in the original dataset?
4. Empirical results on the Travel data outperform other data, does that mean that such a prompt is more suitable for it and there exists better prompts for other datasets?
5. The overall performance of the proposed method achieved limited improvement, and have the authors conducted the improvement-computing cost ratio to make the profit more intuitive?
limitations: limitations: The authors have adequately discussed the limitations.
paper_summary: paper_summary: The paper proposes a method by which large language models are utilized to generate synthetic tabular data in order to mitigate class imbalances in existing datasets. The authors provide some tips which they discovered to result in more reliable data being generated, such as enforcing a CSV format. Their methods are validated on 6 real world datasets and one toy dataset.
strength_weakness: strength_weakness: 1. In Table 1, outside of the Travel dataset, the other 5 datasets do not show a significant increase in F1 score or balanced accuracy between the proposed method and the next closest baseline. Given this, it is not clear how to interpret the utility of the proposed method.
2. The motivation for why a large language model would be used to generate synthetic data for tabular settings, at scale, is a bit lacking. In particular, there needs to be discussion about the reasons for why the use of an LLM (with all of the computational and prompt-related overhead) would be preferable to existing data augmentation techniques for tabular data. In particular, it is not clear that the stated results provide enough evidence to persuasively demonstrate the utility of using LLMs for this purpose
questions: questions: 1. In Table 1, for the row "Ours" --- which LLM was used to generate the synthetic data? If it one of the gpt-3.5 models, then that should be stated up front so as to not increase confusion.
limitations: limitations: The limitations are addressed in Appendix G. However, they should be moved up to the main paper for full transparency. Additionally, the provided limitations are quite sparse. It would be helpful if the authors could think a bit more about the real-world usability of their method, as well as the drawbacks associated with using potentially closed-source large language models for their synthetic data generation purposes.

Comments: comment: We are pleased that our clarifications addressed your concerns, and we appreciate your decision to raise the score. Thank you for your insightful feedback and for taking the time to review our work. 

We remain committed to addressing any further suggestions or clarifications that could strengthen our work.
Thank you for the reply, my concerns were addressed, and I will update the rating score correspondingly.
Thank you for your valuable feedback and for taking the time to review our work. We are glad our clarifications addressed your concerns, and we appreciate your consideration in raising the score.

We remain committed to addressing any further suggestions or clarifications that could enhance the quality of our work.
Thanks for the reply! The response resolved my concerns. I will raise my score.
Thank you for taking the time to review our work and for your valuable feedback. We appreciate your consideration and are pleased that our responses have been helpful in updating your review.

We are committed to making further improvements and are prepared to address any additional suggestions or clarifications that could strengthen our work.
Thank you for your valuable comments and for taking the time to review our paper. We are glad that some of your concerns have been addressed, and we appreciate the opportunity to clarify and expand on the additional points.
***
> Q1. Prompt optimization

A1. We appreciate the suggestion, but there are significant differences between the typical applications of these methods and the challenges of synthetic tabular data generation.
Self-discover and OPRO are designed to enhance LLM accuracy in structured tasks like multiple-choice questions or math problems, where correctness is relatively explicit. However, synthetic tabular data cannot be directly evaluated for correctness, making these methods less applicable to our domain.

Recognizing the potential, we adapted OPRO, which provides official code, to our task by optimizing prompts based on the classification accuracy of a robust classifier, CatBoost. Using the Thyroid dataset with GPT-3.5-turbo, we applied OPRO to optimize our prompt. Unfortunately, **the optimized prompts failed to produce valid synthetic tabular data**. For instance, one generated prompt was:

`No,45,F,No,No,No,Euthyroid,Single nodular goiter-right,No,Papillary,Multi-Focal,Intermediate,T3a,N1b,M0,II,Excellent`

This is because OPRO is designed to optimize simple instructions, e.g., `Take a deep breath and work on this problem step-by-step` or `Break this down`, as shown in Table 1 of the OPRO paper. **While effective for question-answering and reasoning tasks, these types of prompts are unsuitable for constructing the complex structures required in synthetic tabular data generation.**

Our experiments indicate that our proposed approach remains the most effective method for generating high-quality synthetic tabular data. Future research could explore prompt optimization methods specifically designed for tabular data generation to further enhance performance.
***
> Q2. Dataset details & Generalizability

A2. As noted in Section 3, we provided dataset details in Appendix I.2. However, **we acknowledge that this information may not be sufficiently highlighted in the main text**.

To address this, **we will revise the manuscript to include a comprehensive introduction to these datasets within the main text, clearly demonstrating the broad applicability and generalizability of our method.** We will also emphasize the importance of tabular data research in enhancing decision-making and efficiency in various real-world applications, better communicating the impact and significance of our work in multiple fields.
***
> Q3. Advanced classifier for label matching

A3. We employed top-performing tabular classifiers, XGBoost, CatBoost, LightGBM, and Gradient boosting classifier, known for their strong performance, often surpassing recent deep learning models. These models have served as robust baselines in tabular classification (TabR, ICLR’24 [1]) and have been used to assess generated data quality in tabular generation studies (the benchmark paper, NeurIPS’23 [2] and TabDDPM, ICML’23). In our work, we evaluated label matching quality by averaging results across 20 runs with these models, five runs each, demonstrating the superiority of our method.

We conducted preliminary experiments using recent in-context learning-based tabular classification methods, TabPFN (ICLR’23) [3] and T-Table (KDD’24) [4], on Travel, as detailed in **Table B of the attached PDF**. We also tested TabR (ICLR’24) [1], an advanced deep-learning tabular classification model, using its official code. 
|Model|Original|+Ours|+TabDDPM|+GReaT|
|-|-|-|-|-|
|TabR|46.41|**60.78**|44.88|32.41|

The F1 score results indicate that **while the advanced models like TabPFN, T-Table, and TabR did not outperform traditional classifiers, adding synthetic data generated by our method consistently led to significant performance improvements, even with advanced tabular classification models.** In contrast, baselines resulted in performance decreases. Our method uniquely and consistently enhanced the performance of different classifiers, demonstrating superior label matching quality.

Furthermore, as discussed earlier, our label matching quality is validated, regardless of the classifier, by:
* Distinct class distribution in the generated data (Fig. 1, Toy set)
* Feature correlation similarity with the original data across classes (Fig. 4, Travel & Fig. 5, Sick)

These findings affirm that our model produces data with the best label matching compared to other baselines.
***
[1] TabR: Tabular Deep Learning Meets Nearest Neighbors, ICLR’24  
[2] Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark, NeurIPS’23  
[3] TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second, ICLR’23  
[4] From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models, KDD’24  
[5] Self-consistency improves chain of thought reasoning in language models, ICLR’23
Thank you to the authors for their detailed feedback and responses. I have updated my review accordingly.
Thank you for the responses, some concerns were addressed, however, there are still some questions.

1. Since the proposed approach is a prompt-based generation, have the authors tried to optimize the fixed prompt with some prompt optimizing methods, such as the Self-discover [1] and OPRO [2]?
2. There was no detailed introduction for various tabular datasets and no case study, which made me confused about the generalizability of the proposed method.
3. For the label matching of generated data, the authors need to employ advanced tabular data classification algorithms to evaluate the quality.

[1]. Self-Discover: Large Language Models Self-Compose Reasoning Structures.

[2]. Large Language Models as Optimizers.