Language-Interfaced Tabular Oversampling via Progressive Imputation and Self-Authentication
Authors: June Yong Yang
Geondo Park
Joowon Kim
Hyeongwon Jang
Eunho Yang
Venue: ICLR 2024 poster (2024)
Keywords: Tabular data
imbalanced learning
language models
Primary area: generative models

Abstract: Tabular data in the wild are frequently afflicted with class-imbalance, biasing machine learning model predictions towards major classes. A data-centric solution to this problem is oversampling - where the classes are balanced by adding synthetic minority samples via generative methods. However, although tabular generative models are capable of generating synthetic samples under a balanced distribution, their integrity suffers when the number of minority samples is low. To this end, pre-trained generative language models with rich prior knowledge are a fitting candidate for the task at hand. Nevertheless, an oversampling strategy tailored for tabular data that utilizes the extensive capabilities of such language models is yet to emerge. In this paper, we propose a novel oversampling framework for tabular data to channel the abilities of generative language models. By leveraging its conditional sampling capabilities, we synthesize minority samples by progressively masking the important features of the majority class samples and imputing them towards the minority distribution. To reduce the inclusion of imperfectly converted samples, we utilize the power of the language model itself to self-authenticate the labels of the samples generated by itself, sifting out ill-converted samples. Extensive experiments on a variety of datasets and imbalance ratios reveal that the proposed method successfully generates reliable minority samples to boost the performance of machine learning classifiers, even under heavy imbalance ratios.

Published as a conference paper at ICLR 2024
LANGUAGE-INTERFACHE TABULAR OVERSAMPLING VIA PROGRESSIVE IMPUTATION AND SELF AUTHENTI- CATION
June Yong Geondo Park Geondo KAIST Seoul National University?, AITRICS [laoconeth, geondopark, kjwispro, eunhoy] .c. Banghw0911@gmalt com
Joowon Joowon Kim', Hyeongwon Jang", Eunho Yang
ABSTRACT
Tabular data in the wild are frequently afflicted with class-imbalance, biasing ma- chine learning model predictions towards major classes. A data-centric solution this problem is oversampling where the classes are balanced by adding synthetic minority samples via generative methods. However, although tabular generative models are capable of generating synthetic samples under a balanced distribution, their integrity suffers when the number of minority samples is low. To this end, pre-trained generative language models with rich prior knowledge are a fitting candidate for the task hand. Nevertheless, an oversampling strategy tailored for tabular data that utilizes the extensive capabilities of such language models is yet to emerge. In this paper, we propose novel oversampling framework for tabular data to channel the abilities of generative language models. By leveraging its con- ditional sampling capabilities, we synthesize minority samples by progressively masking the important features of the majority class samples and imputing them towards the minority distribution. To reduce the inclusion of imperfectly converted samples, utilize the power of the language model itself to self-authenticate the labels of the samples generated by itself, sifting out ill-converted samples. Ex- tensive experiments
variety of datasets and imbalance ratios reveal that the proposed method successfully generates reliable minority samples to boost the performance of machine learning classifiers, even under heavy imbalance ratios.
INTRODUCTION
Tabular data is of the most common forms of data in real-world applications, spanning vast industries such as healthcare (Johnson et al., 2021), marketing (Sakar et al., 2019), and finance (Shah al., 2022). However, due to their idiosyneratic nature and volatility in the data collection process, tabular data in the wild are often ridden with class-imbalance. For example, a financial report dataset such events
curated predict major economic collapse likely be class-inbalanced rare in the real world. In turn, machine learning classifiers trained from such imbalanced data are inclined to be biased, as the model is trained towards classifying an overly significant amount of samples as the majority class. This misalignment in generalization directly impacts the classification performance of minority classes, which are often the critical foci of interest. For instance, cancer prediction classifier would be of limited use if its predictions are substantially inclined towards predicting all patients as cancer-free, as resulting false negatives inevitably incur major liabilities.
Faced with the issue, diverse methodologies have been proposed to address the issue of class imbal- ance. These methods predominantly focus on either modifying the model itself (Kang et al., 2020; Tian al., 2020; Menon al., 2021; Hong al., 2021) adapting its loss function enhance robustness against class imbalance (Cao et al., 2019). However, these approaches fall short in terms of general applicability since they require the end-user directly modify the existing off- fff-tff-sheff machine learning models. Considering the common practice of an end-user to employ a range of readily available models depending on the domain peculiarities of the given data, these model-
"Equal contribution.

---

Published as a conference paper at ICLR 2024
centric strategies bound to incur potential difficulties and friction from the perspective of the end-user. Consequently, a model-agnostic, data-centric approach of minority oversampling (Chawla et al., 2002; Han et al., 2005; He et al., 2008) has been proposed to synthetically generate and add minority samples to balance the class distribution. Through such means, the model itself or its learning mechanism can be kept intact while the data itself becomes balanced remedy the issue.
Recent advances in deep generative models have bestowed the means to generate high-quality syn- thetic data (Kingma & Welling, 2013; Goodfellow al., 2020; Ho et al., 2020; Radford al., 2019). Given their success in other data modalities (Rombach et al., 2022; OpenAl, 2023), they have been successfully adapted to the tabular domain (Choi et al., 2017; Xu et al., 2019; Kim et al., 2023; Kotelnikov et al., 2023; Borisov et al., 2023). By training these generative models on the imbalanced dataset, synthetic minority samples can be drawn from the generative distribution to oversample the minority classes. However, this approach often struggles in practical situations when the number of minority samples is too small to successfully train a generative model. In this situation, the model may fail to learn the minority distribution, or simply memorize the minority samples.
To this end, a stronger generative model equipped with pre-trained knowledge is required to mit- igate overfitting. An intriguing solution to this problem is the usage of generative large language models (LLMs) (Radford et al., 2019; OpenAl. 2023; Touvron et al., 2023a;b) to process tabular data through language interfaces (Dinh et al., 2022). Recent studies have demonstrated that by rep- resenting tabular data as free text and leveraging the knowledge of language models, effective task performance can be achieved even when the amount of data is limited (Hegselmann et al., 2023; Nam et al., 2023). However, this abrupt shift of paradigm makes it challenging to apply conven- tional imbalance handling techniques, such as borderline sampling (Han al., 2005; He et al., 2008), major-to-minor translation (Kim et al., 2020), or style transfer (Kim et al., 2022). Thus, there emerges a need for a systematic framework to effectively perform minority oversampling based on language models.
In this paper, we propose framework for tabular data that comprehensively utilizes the power of Language-interfacee tabular learning. Using the conditional sampling capabilities of the generative language model, samples with minority traits can be synthesized through class label prompting. Based on this functionality, we develop a borderline sampling method that converts a majority sample to a minority by progressively "puncturing" the feature values of the sample and imputing them under minority conditioning, so that they convey the characteristics of the minority class.
Interfaced Tabular Oversampling (LITO), a novel oversampling
However, recent observations on the generative capabilities of language models report the potential for biased (Gallegos et al., 2023) or contradictory (Ji et al., 2023) generation. When supplied with class-imbalanced data, the model might incline towards generating majority classes even with class conditioning, introducing noisy samples. Such risks can be exacerbated during the execution of aggressive techniques such borderline sampling, there is possibility that the sample may not be fully converted. To effectively sample synthetic minorities under these conditions, we propose a simple yet effective rejection sampling procedure to prevent the inclusion of ill sil-syytteeeized minority samples. Motivated by self- aelf-alignment methodologies (Sun et al., 2023) of large language models, we use the language model on itself to predict the labels of its own generated samples, filtering out ill-synthesized suspects. Integrating these components, we then propose a progressive imputation scheme harnessing the power of the language model itself to guide its own oversampling procedure. We validate our method against various tabular datasets from OpenML-CCI8 (Bischl et al., 2019) and UCI machine learning repository (Dua & Graff, 2017), with varying imbalance ratios. Also, we demonstrate the applicability of the LITO framework on black-box chatbot LLMs such as GPT3.5- turbo through in-context learning (ICL).
Our contributions are threefold:
We propose a novel tabular oversampling strategy based on generative language models that converts majority sample to minority sample by inducing missing values in the important columns and filling them through minority-class conditioned imputation.
To mitigate the problem of faulty synthesis, we introduce a simple yet effective technique that utilizes the generative language model itself to filter out defective samples.
Our oversampling strategy enhances the performance of off-the-shelf machine learning models, even for highly imbalanced datasets.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1 OVERSAMPLING FOR MAASS-MBBLLANCED DATA
We formally define the problem setting of our interest. We take into consideration a supervised learning problem given a training dataset D := {2,, (Izz yr )n)] J== where I E m is the feature vector and E [1..... C} is the class label of the data point 2. For each class, we denote the number of training samples that belong to class c as Ne, where the classes are sorted by their number of samples in descending order without the loss of generality
No. A dataset is class-imbalanced the number of samples for each class is not equal and skewed: Ni > No. We define the majority class the class with the largest number of samples (N1) and other classes as minority classes. The imbalance ratio is defined as a Ni An oversampling strategy introduces synthetic minority samples to the dataset until the number of samples for all minority classes becomes equal to the majority class, as in N1 N₂ = No. This process yields a class-balanced dataset D which in turn is employed to train desired machine learning model.
in Ni 2 N2 2
2.2 LANGUAGE- INNGUAAGF-ITTERAFFE TABULAR GENERATION
To the best of our knowledge, clear consensus on the precise definition of tabular data is yet to be reached. In this paper, we temporarily define tabular data as data formatted into N rows (samples) and .M columns (features), which can be cast into the format of comma-separated values (CSV). Unlike other data modalities such as vision, speech, or graphs, a notable characteristic of tabular data is that it can be straightforwarlly represented in free text. This suggests the possibility of applying natural language processing algorithms to process tabular data. Particularly, an intriguing approach is the handling of tabular data using generative language models. Since tabular data can be readily formatted into text, they can be processed by generative language models without the usage of external adapters or representation alignment (Tsimpoukelli et al., 2021). Toward this goal, Dinh et al. (2022); Hegselmann et al. (2023); Borisov et al. (2023) explored the prediction and generation capabilities of generative language models by transforming tabular data into text prompts using textual encoders. Following Dinh et al. (2022), we refer to such a paradigm as language interfacing for tabular data, and refer to the core language model as tabular language models (TLM), for the scope of this paper. Given a tabular dataset, the n-th row of the table can be represented as a textual prompt tn using the following rule:
mm -hhm: "is", Un,ms ","] and t. = [tn,11 2 .... Mn.M]
where n E {I...., N). m E fl...., M). Un,m is the (n, m))th value of the table, and hm is the name of the mth column. The row index may be omitted for brevity. Here, we follow the encoding format proposed in Borisov et al. (2023). However, the syntactic structure or the choice of articles to construct the textual prompt may vary.
In the same vein, the generation of tabular data is also feasible using the language paradigm. The GReaT (Borisov et al., 2023) framework for tabular generation fine-tunes au- toregressive pre-trained transformer with tabular data to generate novel tabular samples. textual encoder encodes the column names and features into text, which is then fed to the language model for fine-tuning. Since the columns of the table do not exhibit spacial locality, a random feature or- der permutation function is employed. To generate synthetic samples, conventional autoregressive sampling is used. Since the generative model is fine-tuned with random feature order permutation. arbitrary conditional sampling is possible by prompting the model with conditioning column names and values to a TLM f:
[ Fog. Eonr fr fry........................ fm]] = f(((( f((tr]: fry fry............................. fr..))
where 4, are column conditional prompts and to are sampled column texts. Such capabilities can be readily utilized for cands-condiiooee sampling by placing the label column and values in the front of the prompt for autoregressive decoding. Although certain deep generative models for tabular data support class conditional sampling, arbitrary not present in other forms of tabular generative models.
sampling capabilities are generally
3

---

Published as a conference paper at ICLR 2024
(6) Language Fine-Tuning
(c) Autl-uuuthntiiiion
Textual Enceding
CSV
TGi
Lecel
le Sex mall, 75k Labells
Synthesized Samples
Age 1.51 Sex sctessile 750 ie 00, Tabel repartor
Fermalle
0.80
Reepetion Neggttvv
Femele
0.30
Sign
Wile
08
Wale
2.2
Minority Condition Sampling Importance-swan Imputation
Semple from Major
Synthespeed Samples
TLM
Age is 28, Bonis mall: FSH to S.O. Cabel eegatioe
Age
TSH
41
lsbel
Accept
Reject
Inpux
45
Feruls
3.1
Sea
Lobel Sox malu. TSM
Male
22
Label conwert
Puncture
Figure I: Overview of our framework (LITO). Using TLM backbone, minority candidates generated via class-conditioned borderline sampling. Then, generated candidates are self-authenticated by re-imputing their labels. For rejected candidates, the process is repeated until the sample is converted as long as the predic- tion confidence increases with each iteration.
3 LANGUAGE-INTERRACE TABULAR OVERSAMPLING (LITO)
In this section, we introduce our Language-Interfann Tabular Oversampling (LITO) framework that synthesizes minority samples by leveraging the conditional sampling capabilities and the rich prior knowledge of pre-trained generative language models. Our framework composed of two compo- nents: minority class-conditionss sampling to generate synthetic minority candidates (Section 3. 3.1), and the self-authentication procedure to filter out defective samples (Section 3.2). The overall frame- work of LITO is described in Figure 1.
3.1 MINORITY- CONDITIOEED SAMPLING WITH IMPORTANCE-AWAR IMPUTATION
The first stage of our framework is to generate minority- -lass conditioned samples via class- conditioned prompting, using arbitrary conditioned sampling (Equation 2) of TLMs. A synthetic minority sample of class be generated from TLM via class-conditioning by prompting the model with the minority class label:
t t= = (((iabet)
where liabet = ["label", "is", "c",, 4. With this functionality, we propose a novel borderline sam- pling strategy tailored to a decoder-only TLM by converting a majority sample to a minority sample through class-conditioned imputation. First, a substrate majority class sample is randomly drawn from the entire training set. Subsequently, we induce missing values by 'puncturing' out & columns, obtaining truncated prompt (f,, f2...... ... t ))) By prompting the sample as the target minority class and performing conditional imputation, we convert the sample to the targeted minority class:
f(tiaatet &1, t2....,tas.kk
(4)
where the conditioned columns are re-indexed without the loss of generality. Note that our approach is equivalent to full condicoodd sampling when & = M.
Considering the heterogeneity of columns, randomly puncturing and imputing the feature values may neglect important columns that are key towards defining sample as majority class or minority class. In other words, imputing unimportant columns may not be sufficient to convert majority class sample to minority class sample. With this in mind, we propose to puncture and impute columns guided by a feature importance criterion. In detail, we utilize the self-attention scores of the TLM to attribute the importance of column features. We input given substrate to the TLM and extract the last-layer attention map to obtain the attention scores with respect to the label token, and calculate the importance score for each tabular column by summing the attention scores of the tokens that consist of the respective column. In the case where the label consists multiple tokens, the mean of the attention scores the label tokens. The importance score of the
4

---

Published as a conference paper at ICLR 2024
m-th column of substrate sample is given as:
dobel) tok(t,.m)
(s,,)) m))
Σ
Σ
Σ Al
(5)
|tok (ts.(ttstabb))
where tok(!..) is the set of tokenized sequence indices that consists of the mn-th tabular column of substrate s, h is the attention head index. and J, .. is the last layer attention score of the i-th token by the j-th token for head h for substrate s. After obtaining the sample-wise column importance scores, we select top-k important columns to be subject to imputation.
3.2 REJECTION SAMPLING VIA SELF-AUTHENTICATINN
Although intuitive, the proposed strategy carries inherent risks due to the uncertainty of whether the samples generated by clndstcondiiional prompting or borderline sampling are indeed minority sam- sampling, there is a possibility that the model may generate a majority
ples. For class-conditions class sample even under such conditioning if the model is inclined towards the majority class. For borderline sampling. if the number of column punctures is insufficient, there remains possibility for the sample to either retain its majority class identity or fall into an ambiguous category between the two. Despite the provision of class-conditional prompting, oversampling through imputation can still yield either ambiguous minority samples or even majority samples, which can adversely affect the formation of decision boundaries. Therefore, it is imperative to devise method to filter out such ill- elllgenerated synthetic samples.
For this purpose, we propose simple yet effective rejection sampling method that utilizes the char- acteristics of language models. Our key intuition is that the generative language model is also capa- ble of imputing the label of the given sample when the label is absent. Thus, generated sample can be verified by the language model itself by imputing the label of the sample:
[Frabet (t,...., tM]] f(ft...................... ................................ (xx))
Based on this functionality, we introduce the Authentttetiiaaooonn procedure to the oversampling process. After a minority sample is generated, we discard the label of the generated sample and re- impute the label using the TL.M. If the label of the synthesized minor indeed predicted as minor, then the sample is accepted. If else, then the sample is deemed faulty and is discarded. This process is akin to the self-alignment process of language models (Sun et al., 2023), where the language model itself is utilized to control the generation process of the language model. We refer to this sample-and-aauthentiaa authhenttteeeee framework as LITO. Note that our self atteentttenti method not only can be combined with borderline sampling, but also full class-conditions sampling (LITO-C).
3.3 ADAPTIVE OVERSAMPLING WITH PROGRESSIVE IMPUTATION
Combining the components described above, we now propose LITO-B, -B,, an adaptive borderline sam- pling strategy. When transforming the features of majority class samples into minority class samples through imputation, it is important to acknowledge that the number of column imputations required successfully convert a majority class sample into the target minority class may vary from one sample to another. In other words, certain samples might not achieve proper translation due to an insufficiently determined number of column punctures. In such cases, despite the potential for these samples to be converted into the minority class by increasing the number of column punctures, they are preemptively excluded during self aiff-authentication.
To synthesize more diverse and probable samples. we propose a progressive imputation sampling method, wherein we incrementally increase the column puncture count while iteratively generat- ing and filtering samples. For each round, we puncture & columns according to the importance ordering and execute conditional imputation. Then, self-authentication is performed to confirm the converted samples. For the samples that have failed the authentication, we puncture the next k- important columns and iterate the process. Additionally, if the prediction probability of the target label (confidence) does not increase after each imputation round, then the substrate sample is dis- carded. Through this process, the number of imputed columns can be adapted dynamically, generat- ing diverse samples. The overall process is described in Algorithm 1.
5

---

Published as a conference paper at ICLR 2024
Algorithm LITO-B
1:
Imput: Training D (zn (In, ] Fine-tuned TLM f, Number of classes C, puncture per round Authenticated 4- for each class G; in C' do
2:
3: while generate amount == Cuajar | do
4:
Sample the substrate in D where y, Compute the attention score of each columns from equation 5
Sort 7, by descending order
7:
o= Index of round r 0
8:
9:
while True do
Puncture ofr (r + 1) indices s change label column to target label G Resulting in Sp = [uubee 11,12, l2...., [M]
11:
b Imputation
12:
Synthesized sample
= [(sp)
13:
Puncture label column of 8y
For self-authentication
14:
pred, logit = f(x)
15:
pred = c; then
16:
pnaeenticated_samplpe <- Break
17:
18:
else
if prob (c,, s.) >= prob(c,, 8,-1) then 8 + 8r r+
D Check confidence increase
20:
21:
22:
else
23:
Break
24:
end if
25:
end if
26:
end while
27:
end while
28:
end for
3.4 EXTENDING BLACK-BOR LANGUAGE MODELS THROUGH IN-CONTEXT LEARNING
Considering the scaling law of LLM performance with respect to their number of parameters (Ka- plan et al., 2020), it can be projected that the effectiveness of the LITO framework will also in- crease with the size of the backbone TLM. However, for powerful but proprictary black-box LLMs, fine-tuning a model to perform tabular oversampling entails practical difficulties for an end-user. Intriguingly, recent variants of LLMs possess the ability to effectively learn from the input data in the inference phase through in-context learning. enabling an end-user to adapt LLMs to their distinctive needs without fine-tuning the language model. By supplying an LLM with the imbal- anced dataset and instruction prompts for feature importance calculation, conditional imputation, and self authentthnnii (Appendix C), the LITO framework can be adapted to black-box L.LMs, as demonstrate in Section 4.5.
EXPERIMENTS
In this section, we demonstrate the effectiveness of our framework by conducting experiments on multiple publicly available datasets. We validate method and its individual components follow- ing the conventions of prior works. Below, we provide a brief overview of our experimental setting in Section 4.1, followed by empirical results and detailed analyses the subsequent sections. We also analyze the sampling quality and diversity in Appendix B.
4.1 EXPERIMENTAL SETUP
Datasets. We validate our method on six tabular benchmark datasets: Default, Shoppers, Sick, and D abetes for binary classification, Obesity and Sat image for multi-class classifi- cation. They exhibit variation in terms of sample sizes and feature dimensions. The detailed charac- teristics of the datasets are provided in Appendix A. We partition the datasets into 80% for training and 20% for the test following the previous works (Kim et al., 2022). For datasets with rela-

---

Published as a conference paper at ICLR 2024
Table 1: Comparison our method with other baselines for binary classification tasks, We conduct experi-
ments on both a mild class-imbalance setting (a = 10) and an extremely dasslimmalanced setting (a 100).
We report the Fl-score and balanced accuracy with the standard errors on four benchmark datasets. The best performance is bolded, and the second-best is underlined.
Extreme imbalance (imbalance ratio
100)
Detault
Shoppers
Sick
Diabetes (c: = 20)
Dataset
F1
BAcc
BAce
bAcc
F1
bAcc
Vanilla
45.16
50.60
=0.04
54.02
=0.11
54.15
=0.06
67.71
+0.70
13 +0.48
49.54 20.23
54.29
=0:00
SMOTE
35.23
5774
20.0s
63.12"
85.40
50.00
66.50
+0.30
65.63
40.45
60.06.0.544
66..88
B-SMOTE
20.14
53.89
20.07
64.65
=0.20
62.15
20.23
48.68
11:45
58.
#1.58
59.77
20.47
60.46
=0:38
CTGAN
55.13 10.18
55.47
10:13
57.25
10.14
56.07
10.00
59.26
10.53
59.00 10.85
55.31
11:40
55.50
11
SOS
36.19+0..0
56.71
+0.19
65.65
+0.52
63.30
+0.45
80.53
83.04 +0.35
49,80
+0.52
54.39
#0:15
GreaT
51.38 =0.09
52.71
=0.00
56.64
=0.10
65.02
#0.10
70.47
#0.62
82.73 +0.60
49.44
#0.68
49.59
=0.04
LITO-C
61.91 +0.07
62.11
+0.08
23.75
+0.18
71.06
+0.06
85.09 +0.51
84.69
#1.06
63.04
+0.46
63.20 +0.41
LITO-B
65.52
67.33
74.58
73.09
20.33
85.63
87.67
64.23
20:42
64.05
Mild imbalonce (imbalance ratio o
10)
Detault
Shoppers
Sick
12:2)
Diabetes
Dataset
F1
bAcc
F1
BAcc
F1
bAcc
F1
bAcc
Vanilla
55.27 =0.07
55.97
=0.05
71.08
=0.10
68.23
#0.17
85.76
+0.77
83.01 #0.80
52.66
+0.50
55.77
=0.27
SMOTE
62.81 +0.20
62.54
#0.19
74.31
=0.10
74.89
+0.28
71.91
+0.68
72.59
#0.79
65.85
+0.51
65.70
=0.27
B-SMOTE
61.74 20
61.37
69.84
=0.16
76.53
#0.46
67.63
69.52
65.48
#0.70
65,20
CTGAN
59.15 10.22
59.34
10.25
72.30
10.18
70.28
10.21
77.15 10.02
83.77 10.58
62.66 10.96
63.43 10.93
TVAE
63.23 20.15
61.70
10.13
71.24
20.22
69.00
#0.26
82.02 40.47
10. 57 86.38
63.17
#1.04
62.77 =0:91
SOS
60,96
60.16
10.20
74.24
10.29
75.19
84.04
90.51
53,06
10:42
55,92
GreaT
65.22 +0.10
66,78
#0.19
68.32
+0.20
75.89
+0.11
82.86
+0.34
91.09
+0.22
64.94 +0.01
65.31
+0.72
LITO-C
67.94 20:05
67.48
20.04
75.85
29.15
77.66
20.12
86.00
10.19
91.13 10.04
64.29 200.5
64.00 20:38
LITO-B
66.86
+0.05
68.85
04
75.59
+0.05
76.86
+0.09
85.83
+0.12
91.92
+0.14
65.94
+0.44
66.12 +0.30
tively small sizes (Diabetes, Sick), we split the dataset into 70% training set and 30% test set. To construct class-imbalanced datasets, we deliberately reduce the number of training samples per class to establish long-tail distribution in the class distribution. Then, given the parameter 7 which
controls the imbalance ratio a 2/ - of datasets, we decide the number of samples for class l following Ni = Ni a and the remainder are discarded. We conduct experiments on mild (a = 10) and extremely class- mass-imbalanced (a= 100) settings.
Baselines. We select two statistical methods and four deep learning- based generative models as baselines. SMOTE (Chawla et al., 2002) and Podderline-SMory (B-SMOTE) (Han et al., 2005) are classical oversampling methods that augment the samples of minor classes. For deep generative models, we use CTGAN (Xu et al., 2019), TVAE (Xu et al., 2019), and SOS (Kim et al., 2022).
CTGAN is a generative adversarial network (GAN) specifically designed to generate synthetic tabu- lar data samples under conditional constraints. Similarly. TVAE is variational autoencoder (VAE) modified for tabular data. We do not compare with TVAE in the case of extremely imbalanced setting behavior, completely failing to generate any minor-
(a=11 (00),0), as the model exhibits ity samples. SOS is a score-basee generative model equipped with imbalance-tailoree techniques, proposed as oversampling method for imbalanced tabular data. GReaT (Borisov et al., 2023) is a generative TLM based on fine-tuning GPT-2 (Radford et al., 2019) and its distilled version (Sanh GReaT based Distill-GPTY as the
et al., 2019). For the experimental evaluations below, baseline. For our experimental evaluations, we adopt the same Distill-GPT2 GReaT model as our backbone TLM. Finally, Vanilla indicates the case where no oversampling is employed and the ma- chine learning classifiers are trained with the imbalanced train set as-is.
Evaluation protocol. Our primary evaluation metric to measure the strength of a given oversam- pling strategy is machine learning (ML) efficiency. detail, ML efficiency quantifies the classifica- tion performance of a given set of machine learning models trained on training set augmented with oversampled data, which includes the original training samples and the synthesized samples evalu- ated on the held-out test set. Following the common evaluation procedure in the literature (Kim et al., 2022; Borisov et al., 2023), we compute the average classification performance of a curated set of ML models. For binary classification, we employ Decision Trees (Loh, 201 1), AdaBoost (Schapire et al., 1999), Logistic Regression (Cox, 1958), and Multi-Layer Perceptron (Bishop & Nasrabadi, 2006), following (Kim et al., 2022). For multi-class classification. add Random Forest (Breiman,
2001) and XGBoost (Chen & Guestrin, 2016) and exclude AdaBoost and MLP due to their ob-
served training instabilities. For evaluation, minority classes generated and added the training
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of our method with other baselines multi-class classification tasks. We conduct ex-
periments on both mild class imbalance settings (a 10) and extremely daassiimbllanced settings (a = 100). We report the Fl-score and balanced accuracy with standard errors on two benchmark datasets. The best per- formance is bolded, and the second-hest is underlined.
Extreme imbolance (a-
100)
Mild imbalance (a
20)
Obesity
Satimage
Obesity
Satimage
Dataset
F1
bAce
bAce
bAcc
F1
bAcc
Vanilla
48.69
52.06
70.43
71.70 10.01
71.39
71.32
80.52
10.07
80.04
SMOTE
34.59 +0.80
39,90 #0.05
77.63 +0.32
77.34
+0.27
49.61
+1.76
54,21
+1.24
84.03
+0.25
83,36
+0:22
B-SMOTE
37.39 #1.15
41.79 #0.85
74.53 20.14
74.15 #0.15
50.58
#0.07
54.22
#1.05
82.21
#0.15
81.69 =0.30
CTGAN
62.67 10:27
63.42
65.24
10.16
68.05
62.43
62.91
78.07
10:41
77.64
TVAE
69.75 +0.94
70,33
t0.95
82.17 +0,30
81.69 +0.50
SOS
47.53 #0.74
50.79
=0.52
75.66
#0.28
76.35
#0.24
68.66 +0.50
69.93
+
83.05
+0.35
82.79 =0.32
GreaT
57.18
58.20
74.32 10.18
74.64
100330
76.73
77.98
82.36
82.43 10:15
LITO-C
66.62 20.47
67,58
20.50 56
81.74 20.13
81.14
10.12
78.13
20.24
79.23
12
84.53
20.05
83,82 =0.05
LITO-B
68.02 20.44
69.08
10. sM
76.81 10.16
78.24
1.0.1
79.74 10.44
80.64
1.036
84,69
1.0.0
83.92 10:31
set using the respective oversampling methods until all the minority classes reach the number of the majority class. To account for the randomness in the sampling procedure, we repeat the over- sampling process times, and train the machine learning models 5 times for a single oversampling instance, resulting in a total of 20 evaluations per method. We measure the Fl-score and balanced accuracy (bAcc) and report their mean and standard error, as the uniform split of the test set is also imbalanced. A more detailed description of the experimental setting is described in Appendix A.
4.2 BINARY CLASSIFICATION
We first evaluate the effectiveness of our method in addressing imbalanced binary classification tasks. For experiments on mild imbalance scenarios, we use a = 10 except for Sick since its 12.2. For extreme imbalance experiments, we for Diabetes where we use a = 20, as all generative models collapse for = 100. The results are presented in Table 1. For both mild imbalance and extreme imbalance scenarios, our methods consistently outperform all baselines, including statistical and deep generative methods, on all four tabular datasets. These results demonstrate the effectiveness of the minority samples synthesized through our oversampling strategy in assisting machine learning algorithms in their formation of more effective decision boundaries. As demonstrated by the significant improvement observed with our method in the case of highly imbalanced cases, our method is able to effectively perform even in extremely -inaalnnddd scenarios with a limited number of samples for the minority class. Notably, comparing method to GreaT, we observe a significant performance difference, implying the importance of self auttentiiitiinnn..
natural imbalance ratio is
100 except
4.3 MULTI-CLASS CLASSIFICATION
To validate the effectiveness of our method on imbalanced multi-class classification tasks, we also conduct experiments on two multi-class tabular benchmark datasets. Note there is more than one minority class in multi-class imbalanced data, a wide variety of one-to-one imbalance ratios exist within the dataset. As shown in Table 2, method brings better imbalance handling performance in most cases compared to other baselines. In the extreme imbalance setting, our methods clearly outperform all baselines by large margins. For mild imbalance scenarios, our method consistently outperforms other baselines in both cases.
4.4 ABLATION STUDY
Here, we conduct an ablation study to verify the effect of the individual components that comprise the LITO framework: importance-aware imputation, seff-authentication, and progressive imputation. For Shoppers and Sick datasets with imbalance ratio a = 100, we compare the MI. efficiency performance by incrementally adding the respective components, shown in Table 3. First, ob- serve that importance-awwer imputation increases the performance over random imputation. Sccond, seff-authentication significantly boosts the performance. Finally, progressive imputation improves over dingle-iteration imputation. These results confirm the contributions of each LITO component.
8

---

Published as a conference paper at ICLR 2024
Table Ablation study investigating the individual components of LITO. We analyze the effects of adding importance-awart imputation, self-authentication, and progressive imputation.
Shoppers
Sick
Sampling rounds
Puncture criterion
Self-authentictiion
bAcc
Fl
bAcc
Random
X
53.65 do.ss
63.32
65.70
t1.0
77.68
1.000
Single iteration
Importance
x
54.31 10.19
64.64 20.26
67.13
10.79
79.64
10.38
Importance
73.39 +0.16
73.08
84,53
+0.70
84,93
+0.00
Random
73.57 10.00
70.58
82.74 20.30
86,29
20.40
Progressive
Importance
74.58 10.20
73.09
-0.23
85.63
10.95
87.67
10.00
4.5 IN-CONTEXT LITO WITH BLACK-BOX LANGUAGE MODELS
Section 3.4 discussed the possibilities of adopting LITO to larger language models via in-context learn- ing (ICL) and prompting. Here, we conduct proof- of-concept experiment to demonstrate the performance of in-context LITO using OpenAl G - API. For the highly imbalanced (a 20) setting of the diabetes dataset, we report the performance of LITO-C and LITO-B against the baselines. Ta- ble 4 shows that oversampling minority class samples through in-context learning is indeed effective. The de- tailed settings and additional experiments are provided in Appendix C.
Table 4: Performance of in-context LITO
for the diabetes dataset (a = 20).
Method
F1
bAcc
Vanilla
49.54
54.29
SMOTE
66.06
61.28
B-SMOTE
59.77
60.46
CTGAN
55.31
55.50
SOS
49.80
54.39
GReaT (Distill GPT2)
49.44
49.59
ICL-LITO-C (GPT3.5)
67.37
66.88
ICL-LITO-B (GPT3.5)
63.68
64.01
RELATED WORK
Class-imbalanced learning. There are principally two overarching strategies address class im- balance. Model-centric approaches try to handle the imbalance problem by modifying the objective function alter the classifier margins (Cao al., 2019; Tan al., 2020; Menon al., 2021), I), reweight minority classes (Japkowicz & Stephen, 2002; Cui et al., 2019), or correct the model in posthoc fashion (Kang al., 2020; Tian al., 2020; Menon al., 2021; Hong et al., 2021) by alter- ing the logits in the inference phase. A data-centric approach is sampling, where synthetic minority samples are introduced into the training data via means such as augmentation (Kang al., 2020; Liu et al., 2020; Ren et al., 2020) generation (Chawla al., 2002; Han et al., 2005; He et al., 2008; Kim et al., 2020; Chu et al., 2020; Zhang et al., 2021; Wang al., 2021; Kim et al., 2022).
Deep generative models for tabular data. Although tabular data is modality where deep learn- ing does not generally excel over classical machine learning models, tabular generative models show distinctive advantages in modeling the generative distribution of the data. TVAE (Xu al., 2019) is variational autoencoder customized for tabular data. MedGAN (Choi et al., 2017), TableGAN (Park al., 2018), CTGAN (Xu et al., 2019) are tabular generative model architectures based on gen- erative adversarial networks. With the recent success of diffusion- baatud generative models (Ho al.. 2020; Song et al., 2021), the methodology has been successfully adapted to the tabular do- main. TabDDPM (Kotelnikov et al., 2023) is based on denoising diffusion probabilistic models, while StaSy (Kim al., 2023) is score-based model. One unexpected tabular generative model GReaT (Borisov et al., 2023), which is based on a generative language model.
6 CONCLUSION
In this paper, presented a language-interfaced oversampling framework for tabular data that com- prehensively utilizes the generation capabilities generative language models. By progressively iterating class-conditioned borderline sampling and rejection sampling through self-authentication. our framework successfully generates synthetic minority samples that benefit the learning of ma- chine learning classifiers. We verified the performance of our framework multiple tabular datasets and imbalance ratios. Also, the proposed framework can be extended to black-box language models through in-context learning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSII) (No.2019-0-00075, Artificial In- telligence Graduate School Program(KAIST), No.2022-0-00713, Meta-learning applicable to real- world problems, No. 2022-0-00984, Development of Artificial Intelligence Technology for Person- alized Plug-and-Play Explanation and Verification of Explanation).
REFERENCES
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. Rijn, and Joaquin Vanschoren. Openml benchmarking suites. (rXiv:1708.0373122 [s), 2019.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https: /ttp:p::///ppeeeeteeeee
net forum ?idm dEygdEyg@e
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with label-distibutiinnaawaae margin loss. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019,
December 8-14, 2019, Vancouver, BC, Canada, pp. 1576,176,, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. volume 16, pp. 321-332133111 2002.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Edward Choi. Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun. Gen- erating multi-labes discrete patient records using generative adversarial networks. In Machine learning for healthcare conference, pp. 286-305. PMLR, 2017.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long- tailed data. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23- 28, 2020, Proceedings, Purt XXIX, volume 12374 of Lecture Notes in Computer Science, pp- 694-710. Springer, 2020.
David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society Series B: Statisticul Methodology, 20(2):215-232, 1958.
Yin Cui, Menglin Jia, Tsung-Y: Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based effective number of samples. In IEEE Conference Computer Vision and Pattern Recog- nition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 9268-9277. Computer Vision Foundation IEEE, 2019.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaeed fine-tuning for non- language machine learning tasks. Advances in Neural Information Processing Systems, 35:11733- 11784, 2022.
Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. http://archive. ics. eddu/m
Isabel O Gallegos. Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dermon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770, 2023.
10

---

Published as a conference paper at ICLR 2024
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Hui Han, Wen-Yuan Wang. and imbalanced data sets learning. In International conference on intelligent computing, pp. 878-887. Springer, 2005.
Mao. Borderline-smote a new sver-iampling method in
Haibo He, Yang Bai, Edwardo A Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling ap- proach for imbalanced learning. In 2008 IEEE international joint conference neural networks (IEEE world congress on computational intelligence), pp. 1322-1325 1328.. leee, 2008
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 5549-5581 PMLR, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Youngkyu Hong. Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Dis- entangling label distribution for long-tailed visual recognition. In IEEE Conference on Computer Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. Vision Foundation / IEEE, 2021.
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: systematic study. Intell. Data Anal,, 6(5):422-44))) 2002.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv, 2021. URL https: http///hysisstee org.
org/ wwt./oontettttoeoooooooooooo.eoe content wtt.ee/tn//iiii//.////////
Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 13896-13905, 2020.
Jayoung Kim, Chaejeong Lee, Yehjin Shin, Sewon Park, Minjung Kim, Noseong Park, and Jihoon Cho. Sos: Score-based oversampling for tabular data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 762-772, 2022.
Jayoung Kim, Chaejeong Lee, and Noseong Park. STasy: Score-based tabular data synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview net/ for.mmid=inmmm sididd
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv: 2013.
arXiv preprint
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models. In International Conference on Machine Learning, pp. 17564- 17579. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2024
Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas. Imbalanced- A python tool- box to tackle the curse of imbalanced datasets machine learning. Journal of Machine Learning Research, 18(17):1-5, 2017. URL http:// /mrr.. wwtpg//appsseeeeee
365..6666... html.
Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. MESA: boost ensemble imbalanced learning with meta-sampler. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020.
Wei Loh. Classification and regression trees. Wiley interdisciplinary reviews: data mining and knowledge discovery. 1(1):14-23 2011.
Leland Mclnnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference Machine Learning, pp. 7176-7185. PMLR, 2020.
Jaehyun Nam, Woomin Song, Seong Hyeon Park, Jihoon Tack, Sukmin Yun, Jaehyung Kim, and Jinwoo Shin. Semi-supervised tabular classification via in-context learning of large language models. In Workshop on Efficient Systems for Foundation Models@ ICMI,2023, 2023.
OpenAl. Gpt-4 technical report, 2023.
Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Young- min Kim. Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, (110), 2018
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. The synthetic data vault. In IEEE Interna- tional Conference on Data Science and Advanced Analytics (DSAA), pp. 490, Oct 2016. doi: 10.1109/DSAA.2066.499
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2888 2011
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma. Haiyu Zhao Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. December 6-12, 2020, virtual, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
C Okan Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and Istm recurrent neural networks. Neural Computing and Applications, 31:6893-69088 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
Robert E Schapire et al. A brief introduction to boosting. In Ijcai, volume 99, pp. 1401-1406. Citeseer, 1999.
12

---

Published as a conference paper at ICLR 2024
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. When flue meets flang: Benchmarks and large pretrained language model for financial domain. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLPP). Association for Computational Linguisties, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021. URL
Forum?id=PXTIG12RE.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305. 03047, 2023.
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp-
11659-11666 11668. IEEE, 2020.
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Jianfeng Wang. Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. RSG: A simple but effective module for learning imbalanced datasets. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 3784-3793. Computer Vision
Foundation / IEEE, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: Sfate-of-the-art natural language processing. arXiv preprint 1910.37110 2019.
Lei Xu, Maria Skoularidou, Alfredo Cuesta- and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. Advances in neural information processing systems, 32, 2019.
Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long- longetailed visual recognition with deep convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(4):3447-3455 May 2021.
13

---

Published as a conference paper at ICLR 2024
A DETAILED EXPERIMENTAL SETTINGS
In this section, we describe the details of the experiments conducted. We implement method with the PyTorch deep learning framework and Huggingface Transformers package (Wolf et al,, 2019). For the SMOTE and B-SMOTE baselines, we use the Imbalanced-Learn (Lemaître al., 2017) package. For CTGAN and TVAE, we use the Synthetic Data Vault (Patki et al., 2016) package. For SOS and GReaT, we use their respective official Github repositories. For the machine learning models involved in measuring the Machine Learning Efficiency (MLE) metric, we use the Scikit- Learn (Pedregosa et al., 2011) package.
A.1 DATASET DETAILS
In this subsection, describe the 6 tabular benchmark datasets used in our experiments.
Default: A binary classification dataset that provides information about default payment status among credit card clients Taiwan.
Shoppers A binary classification task which involves predicting the intent of online shoppers.
Sick A binary classification task of medical domain that involves classifying thyroid disease as either negative or positive. For this, we exclude the rows which have the missing value.
Diabetes A binary classification dataset which contains information related to diabetes.
Obesity: A multi-class classification task aiming to categorize obesity levels based on individuals' eating habits and physical condition.
Sat Image A multi-class classification dataset, produced by the Australian Centre for Remote Sensing, designed for categorizing different types of land use.
Table 5: Details of dataset statistics used in our experiments.
Datasets
#train
#test
#columns
#continuous
#categorical
Default
24K
6K
24
13
11
Shoppers
9.8K
2.4K
18
10
8
Sick
1.7K
0.8K
29
7
22
Diabetes
0.6K
0.2K
10
3
Obesity
1.6K
0.4K
25
24
1
SatImage
4.4K
1.3K
37
37
0
A.2 HYPERPARAMETERS FOR EVALUATING MACHINE LEARNING EFFICIENCY
For reproducibility, we describe the hyperparameters we used for measuring machine learning effi- ciency.
Decision Tree Max depth is 32, criterion is gini.
AdaBoost: The number of ensemble estimators is 100, and the learning rate is 1.0.
Logistic Regression: We "Ibfgs" solver and 1000 iteration. Penalty for training is L2 normalization.
MLP Used hidden layer size is (100, 100), max iteration is 200 and weight for penalty (alpha) is 0.0001.
Random Forest The number of estimators is 100 and the max depth is 8.
XGBoost : We use the muti-label softmax objective, 5 max depth and 1.0 learning rate.
The number of ensemble estimators is 100.
14

---

Published as a conference paper at ICLR 2024
A.3 HYPERPARAMETERS USED FOR FINE-TUNING GENERATIVE LANGUAGE MODELS.
First, we utilize pre-trained generative language models from the established HuggingFace frame- work (Wolf et al., 2019). We fine-tune the Distill-GPT-2 model for each data set for 200 300 epochs according to convergence rate. If the loss curve is not converged, we fine-tune Distill-GPT-2 for 300 epochs, if not we fine-tune for 200 epochs. We use the constant 5e-5 learning rate following Borisov et al. (2023).
15

---

Published as a conference paper at ICLR 2024
B SAMPLING QUALITY AND DIVERSITY
Here, we analyze the sampling diversity and quality of LITO and other competing baselines. First, conduct qualitative analysis by reporting the column-wise distributions of synthesized minority samples w.r.t. the ground truth minority samples. Also, we provide visualizations of the synthetic minority samples via UMAP (MeInnes et al., 2018). For quantitative analysis, we report the coverage score (Naeem et al., 2020) to assess the quality and diversity of the synthesized samples.
B.1
QUALITATIVE ANALYSIS
B.1.1 COLUMN-WISE HISTOGRAMS
We first visualize the histogram of values for individual feature columns of synthetic minority sam- ples generated by each method, compared the real minority distribution. Figure 3, shows the histogram of column values datasets used in our experiments. The results indicate that the samples generated by LITO are most close the ground truth minority class distribution.
Real
Real
Real
Density
Denstry
Density
LIMIT .BAL
LIMIT BAL
LIMIT BAL
Real
Real
Real
Denpity
PAY
Columns [.LM) [PAY 22] of the De fafuutt dataset.
GreaT Real
SOS
UTOO
Real
Real
Density
Density
Density
ExitRates
GreaT Reall
SOS Real
LITO
Real
Density
Denstty
Density
(b) Columns [xxi (Rates),, [Month] shoppers dataset.
Figure 2: Histogram of column values of synthetic minority samples compared the ground truth minority distribution.
16

---

Published as a conference paper at ICLR 2024
SOS
UTOO
Real
Real
Real
Density
Density
Denaity
age
age
age
Great Real
SOS
LIOO
Real
Real
Density
Density
Denaity
TUU
TAU
TAU
(a) Columns [Age], [T4u] of the S ck dataset.
GreaT Real
Real
Real
Dersity
Dencity
BMI
8MI
8MI
Real
Real
Real
Dersity
Density
Insulin
Insulin
Insulin
(b) Columns [BMI], [Insulin] of the Di abetes dataset.
Figure 3: Histogram column values synthetic minority samples compared ground truth minority distribution.
17

---

Published as a conference paper at ICLR 2024
GreaT
SOS
UTO
Density
Density
Density
GreaT Real
SOS
LTOO
Real
Density
Density
Density
Height
Height
Height
(a) Columns [CH20], [Height] the Obesity dataset.
GreaT
SOS
LITO
Density
Density
Density
D10attr
GreaT
SOS
LITO
Density
(b) Columns [D10attr], [Cattr] of the Sat image dataset.
Figure 4: Histogram of column values of synthetic minority samples compared to the ground truth minority distribution.
18

---

Published as a conference paper at ICLR 2024
B.1.2 UMAP VISUALIZATIONS
Here, visualize the manifold occupied by the synthetic minority samples via UMAP. Figure 5, 6 shows that the samples generated by LITO are most close to the ground truth minority class distribution.
Class
Class
Class
(a) UMAP of ground truth training data.
(b) UMAP of SOS.
Class
Class
Class
(c) UMAP of GReaT.
(d) UMAP of LITO.
Figure 5: UMAP visualization results for the diabetes dataset. For comparison, ground truth train- ing data is superimposed as transparent points. UMAP results show that while LITO successfully generates synthetic minority samples, while other baselines fall short in generating samples near the ground-truth minor- ity distribution and overlap into the majority class distribution.
19

---

Published as a conference paper at ICLR 2024
Cass 2.0
Cass 2.0
Coss 4.0
Cast
Clss 3.0
Cass 3.0
(a) UMAP ground truth training data.
(b) UMAP SOS.
Class 2.0
Class
Class 1.0
Class
Class 4.0
Class
Class 0.0
Class
Cless 3.0
Class 3.0
Cless 5.0
Class
(c) UMAP of GReaT.
(d) UMAP LITO.
Figure 6: UMAP visualizations for the satimage dataset. For comparison, the ground truth training data is
superimposed transparent points. UMAP results show that LITO successfully generates synthetic minority samples that resemble the ground truth training data.
20

---

Published as a conference paper at ICLR 2024
B.2 QUANTITATIVE ANALYSIS
To quantitatively analyze the quality and diversity of generated samples, we measure the coverage score (Naeem al., 2020), which measures whether at least generated sample is contained in the k-nn manifold of a real data sample. Although a high coverage score reflects the diversity of the generated samples, it does not guarantee the quality generated samples, as our main focus is in minority oversanpling. For example, sampling distribution with high coverage score may invade the majority class distribution, resulting degraded downstream performance. In this sense, we devise and report the spillage score which is the coverage score of the sampled minority data w.r.t. the majority class samples. For minority oversampling, the generative model should obtain high coverage score while maintaining low spillage score. In Table 6 and Table 7, we report the respective scores of our method and recent competing baselines on binary and multi-class datasets for imbalance ratio 100.
Table Coverage and spillage scores for binary classification tasks.
Imbalance ratio
100
Default
Shoppers
Sick
Diabetes
20)
Dataset
Coverage
Spillage
Coverage
Spillage
Coverage
Spillage
Coverage
Spillage
Real
0.533
0.345
0.074
0.599
SOS
0.608
0.203
0.769
0.213
0.929
0.466
0.375
0.553
GreaT
0.859
0.883
0.969
0.884
0.943
0.457
0.798
0.931
LITO-C
0.945
0.629
0.891
0.324
0.950
0.466
0.761
0.338
LITO-B
0.951
0.656
0.870
0.440
0.936
0.485
0.676
0.376
Table 7: Coverage and spillage scores multiclass classification tasks.
Imbalance ratio = 100
Obesity
Satimage
Dataset
Coverage
Spillage
Coverage
Spillage
Real
1
0.325
0.126
SOS
0.657
0.161
0.338
0.045
GreaT
0.847
0.392
0.630
0.400
LITO-C
0.912
0.370
0.672
0.108
LITO-B
0,926
0.386
0.509
0.125
21

---

Published as a conference paper at ICLR 2024
C DETAILED SETTINGS FOR IN-CONTEXT LEARNING
For in-context learning oversampling experiments, we use the OpenAl chatbot API to access GPT-3. 5-turro--t-- 0613. For the generation step, provide majority samples and minority sam- ples as context and prompt the model to generate synthetic minority samples, Due to token limits, supply 100 majority-class samples and 17 minority-clase samples (all of them) the context and request to generate samples or impute a given sample. The prompt structure used is described Figure 7, For LITO-B, For self-authentication, we use mn-ontext few-shot (17 shots) learning to predict the labels of the generated samples. The prompt format used is described in Figure 9. In Table we provide further experiments on ICL-LITO with and without aeff-authentication. Results show that self-authentication effectively increases the oversampling performance.
Table 8: In-context LITO results with and without self-authentication.
Method
bAcc
Vanilla
49.54
54.29
SMOTE
61.06
61.28
B-SMOTE
59.77
60.46
CTGAN
55.31
55.50
SOS
49.80
54.39
GReaT (Distill-PPT2)
49.44
49.59
TCL-LITÖ-C Tw7o SA)
64.70
65.42
ICL-LITO-C (with SA)
67.37
66.88
ICL-LITO-B B (w/o SA)
58,32
59.76
ICL-LITO-B (with SA)
63.68
64.01
Role
Con tent
SY3 TEM
You ase disbetes expert with nedica degree.
This is disbetes record cav file. These people are label 0.
Actro,Actrl,Attrd, Att At
2,,
23, 33, 0,,
1,,,,,,,0,0, 0,3.6,,,,,,444,,,1111,,,,
USER
This
paople
Attro,Aterl, Att Attr, Attrr,, Attz7, label 0,125,68, 42.3, 0,336,, 5,122,64,32,155 35. 0, 30,1
129,1110, 16.7............33....33
Read
filea. you synthesize label records
Figure 7: In-context learning prompt format for LITO-C.
22

---

Published as a conference paper at ICLR 2024
Role
Con tent
SYSTEM
You diabetes expest Nith nedical deqree.
record
people
Pregnanctes, Slocose, Aloooo
Texal RMT, Featteeseed Penct Tabel
2, 6, 27, 25, 3. clacs0
1, 117, 60, 23, 106, 3,,66,, olaco
128,
class0
disbesex secord car file. These peoplee new clesdl.
Pregnancles, ocooee, Ai Boddd reseore, Skinthi okness, Traul RMT, DlabetesFedi weteeteseedgggeeeaaeeee greeeennet io Age, label 1,53 68, 25,, 3,365, claze1 1, 35,,35.1,, 6.682, olacsl
129, 67.1, 2.9.9,26,,11......
Kox, letta look wt the fo lowling claest record. To has wissing values, indicated by question narks:
Pregnancies, (lonoee/BLoodreessree Skinthickness, Inculin, praeeeeeeeeerreeettttttieeeee 7,0, 0,262, 62,
Cencralizing from the previous records, impute the nicsing volues and ourput dizectly into CEV fozzat, starsing with the column rases. No talking.
Figure 8: In-context learning prompt format for LITO-B
Role
Con ten
expert
degree.
given information and questians. Faink stop by stop, prodiet shether Laxe on classl, You must chonew in (classd, class-).
bataset has Prequancien, Glucose, sloodizeszaze,, skinthicknes,, Insulin, BaL, siabetennederreetuse fon, Age as il iepet varlables and 1sbel as catpat. Lonttinn Peagnencies DiabetestedigreeSine
Slanose 88, aloodereeere is 62, Seinthiukkwws is 24, iosulin is the label?
0.422, Age
questicn: Whon Peoenancics 6, BHT 15 26.3, Distetesfedigree sazettsseddgrreePuntta i5 0.178, Age is 50, then what is the label? You nust choo se in (class0, clasai], Anaversclasal
clusosa
BloodPesssure 62, Skinthickacsa 6, Insclin
USER
Pregnanctes
BloodPreseure 0,147, Age 21, then what is the Label? must choo
Seinthhiekeess
nwI is 18,2, biabetessedlggeeneee so in (classo, classl), Ansvericlaest Ooestionikhen Pregnanstes 1s 1, Stucese is 150, RloodPreesure is 72, sedethickness is 35, Tneulin BHI Lo 35.5, deteettaeodoooooooattteette in 0.627, Age in 50, then what is the label? you must chaose (olassd, classi]. Answer:
Figure 9; In-context learning prompt format self-authentication.
23


Reviews:
[{"review_id": "68NGuovFxK", "reviewer": "Reviewer_puAc", "summary": "This paper targets at the problem that tabular data usually struggles with class-imbalance problem.\nTo deal with this issue, this paper introduces an algorithm which oversamples the tabular data in a self-authentication manner.\nSpecifically, the algorithm synthesizes the minority class samples by progressively masking the important features of the majority class samples and imputing them towards the minority distribution.\nThe experiments show that the algorithm is able to generate reliable minority which can boost the ML classifier performance.", "questions": "- As mentioned in the weakness section, have you tried to analysis the label distribution between real negatives, synthesized negatives, and real positives? Can this algorithm bring the dataset diversity which can bring the benefit to the downstream task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing is good enough to make readers easily understand the paper. For example, Figure 1 and algorithm covers most of the details in the algorithm, which can make readers briefly know the whole algorithm in a short period.\"\n- The targeted problem, minority in the tabular data, is interesting and still an under-presented problem in the community. Improving this problem can make a large improvement on the ML community.", "weaknesses": "- The main concern is that the oversampling algorithm is based on masking. While some of the information is changed during masking and self-authentication, most of the information still remain and suggests the algorithm just repeat some similar data and limit the diversity of data. Have you tried to analysis the label distribution between real negatives, synthesized negatives, and real positives?\n- Another concern s about that the novelty of the proposed method. While the claimed contribution in this paper include \"data generation\" with \"self-authentication\", the similar ideas have appeared in some literature, like [1] and [2]. Could the authors state the difference between this work and the previous?\n\n[1] Language Models are Realistic Tabular Data Generators, ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aId060Mwe8", "reviewer": "Reviewer_gyRK", "summary": "This paper studies the class imbalance problem in tabular data processing. Specifically, one major drawback of class imbalance in tabular data is that the minority data is not sufficient to help learn a good predictor. In order to solve this problem, the authors propose to leverage the language model to synthesize minority data. Specifically, there are three major parts, first, the minorities are generated by conditioning on their class labels; then, some incorrect examples are rejected through self-authentication; finally, the authors propose an adaptive oversampling by progressively imputation. Through extensive quantitative experiments, the proposed method shows a larger performance improvement than other baseline methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper is well-written and well-organized.\n- Experimental validations are extensive and effective over other methods.\n- The minority generation can effectively solve the class imbalance problem.", "weaknesses": "- There have already been many methods that leverage language models to augment data. Using a similar strategy to solve the class imbalance problem seems to be a trivial contribution to me. Could the authors identify why the proposed method could be a novel contribution?\n- Whether the generation of minority data could help the learning performance is still unclear to me. It would be better if the quality of the synthesized data could be carefully evaluated. Moreover, how many examples are rejected during self-authentication? It would further improve the quality of this paper if a proper ablation study were conducted.\n- How can different language models influence the performance of the proposed method? Could the author try it on GPT-3 or ChatGPT?\n- Minor: The Algorithm is taking up too many blank spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJhgMjNFBO", "reviewer": "Reviewer_tYE6", "summary": "This paper proposes a way to synthesize minority tabular samples using large language models in order to handle class-imbalance. A self-authentication framework is desiged that uses language models to predict the label of the generated sample and sifts out ill-converted samples. Experiments are conducted on a variety of datasets and imbalance ratios.", "questions": "- To understand LLM as a self-authenticator, I wonder the accuracy of directly using LLM to make classification on original data.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Class-imbalance learning for tabular data is a under-exploited area. This paper makes a meaningful attempt.\n- Synthesizing minor classess with large language models seems to be new.\n- The proposed method is simple to implement. The conversion from tabular to textual representation and the self-authentification with LLM prediction can generalize to different LLMs.", "weaknesses": "- My biggest concern is on the reliability of those minor samples generated by LLMs. LLMs are shown to easily generate counter-factual texts. For the tabular data, although the authors use a self-authentification framework to filter out samples with incorrect prediction, there is no guarantee that the remaining samples are semantically meaningful. Considering the different feature columns, I doubt that current LLMs can truly understand the numeric feature values and their relative relation. \n- The language interfacing requires column feature names. In many tabular data, feature names may not have meaningful semantics such as an abbreviation. It would be hard for LLMs to understand those features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)