Small Models are LLM Knowledge Triggers for Medical Tabular Prediction
Authors: Jiahuan Yan
Jintai Chen
Chaowen Hu
Bo Zheng
Yaojun Hu
Jimeng Sun
Jian Wu
Venue: ICLR 2025 Poster (2025)
Keywords: tabular data
prompt learning
classification
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent development in large language models (LLMs) has demonstrated impressive domain proficiency on unstructured textual or multi-modal tasks. However, despite with intrinsic world knowledge, their application on structured tabular data prediction still lags behind, primarily due to the numerical insensitivity and modality discrepancy that brings a gap between LLM reasoning and statistical tabular learning. Unlike textual or vision data (e.g., electronic clinical notes or medical imaging data), tabular data is often presented in heterogeneous numerical values (e.g., CBC reports). This ubiquitous data format requires intensive expert annotation, and its numerical nature limits LLMs' capability to effectively transfer untapped domain expertise. In this paper, we propose SERSAL, a general self-prompting method by synergy learning with small models to enhance LLM tabular prediction in an unsupervised manner. Specifically, SERSAL utilizes the LLM's prior outcomes as original soft noisy annotations, which are dynamically leveraged to teach a better small student model. Reversely, the outcomes from the trained small model are used to teach the LLM to further refine its real capability. This process can be repeatedly applied to gradually distill refined knowledge for continuous progress. Comprehensive experiments on widely used medical domain tabular datasets show that, without access to gold labels, applying SERSAL to OpenAI GPT reasoning process attains substantial improvement compared to linguistic prompting methods, which serves as an orthogonal direction for tabular LLM, and increasing prompting bonus is observed as more powerful LLMs appear. Codes are available at https://github.com/jyansir/sersal.

Published as a conference paper at ICLR 2025
SMALL MODELS ARE LLM KNOWLEDGE TRIGGERS FOR MEDICAL TABULAR PREDICTION
Jiahuan Yan', Jintai Chen Chaowen Hu', Bo Zheng', Yaojun Hu',
Jimeng Sun", Jian Wu
"College of Computer Science and Technology. Zhejiang University
TThrutt of Artificial Intelligence, Information Hub, HKUST (GZ) Computer Science Department, University of Illinois Urbana-Chann Obbana-Champaign
"The Second Affiliated Hospital Zhejiang University School of Medicine
*Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence
[jyansir, cha 2juzhengbo, yeojunhu, wuji an000 ju. edu.cn, j imengei1 Linois. edu, jintaichen@hkust
ABSTRACT
Recent development in large language models (LLMs) has demonstrated impres- sive domain proficiency on unstructured textual or multi-modal tasks. However, despite with intrinsic world knowledge, their application on structured tabular data prediction still lags behind, primarily due to the numerical insensitivity and modal- ity discrepancy that brings gap between LLM reasoning and statistical tabular learning. Unlike textual or vision data (e.g., electronic clinical notes or medical imaging data), tabular data is often presented in heterogeneous numerical values (e.g.. CBC reports). This ubiquitous data format requires intensive expert an- notation, and numerical nature limits LLMs' capability to effectively transfer untapped domain expertise. In this paper, propose SERSAL, general self- prompting method by synergy learning with small models to enhance LLM tab- ular prediction in unsupervised manner. Specifically, SERSAL utilizes the LLM's prior outcomes as original soft noisy annotations, which are dynamically leveraged teach better small student model. Reversely, the outcomes from the trained small model are used to teach the LLM to further refine its real capabil- ity. This process can be repeatedly applied gradually distill refined knowledge for continuous progress. Comprehensive experiments on widely used medical do- main tabular datasets show that, without access to gold labels, applying SERSAL to OpenAl GPT reasoning process attains substantial improvement compared to linguistic prompting methods, which serves an orthogonal direction for tabu- lar LLM, and increasing prompting bonus is observed as more powerful LLMs appear. Codes are available at
hit.ttt.........o/gg com/ hwwnsiyyannnsaaaaaaee
INTRODUCTION
The advancement of large language models (LLMs) ZZaoo et al. 2023) has made waves in both research and industry communities. Through friendly natural language interaction and powerful in- context reasoning ability, LLMs have shown their remarkable knowledge generalization to language processing (Wei et al., 202T Wang et 2022). complex planning (Qin et. 2023 Zan
2023)
and even vertical domain (e.g., healthcare (Cascella et al.) 2223). law (Deroy et al. 2023 chem-
istry (Guo et al. 2023)) tasks compared to past supervised pre-trained language models (Kenton & Toutanova 2019 Radford et al., (2019), all achieved with suitable prompting and no fine-tuning, yet they are still struggling for the numeric tabular data.
For example, GPT-4 achieves 81.7 % accuracy with zero-shot prompting the United States Med- ical Licensing Examination (USMLE), which metric will be increased to 90.2 % when meticulous prompts are designed (Nori al. 2023). In the left part of Fig. a preliminary experiment exhibits performances of GPT-3.5, GPT4 and the fully supervised BERT on top-5 ICD coding for "The corresponding author.

---

Published as a conference paper at ICLR 2025
MIMIC-III discharge summaries. Even with simple zero-shot prompting. GPT-3.5 has already sur- passed the fine-tumed ClinicalBERT (Huang et al. 2019) and can obtain further improvement with linguistic prompting tricks (e.g., zero-shot CoT (Kojima et al., 2022)). However, when handling medical tables with numerical value features, the trend becomes totally different, in the right part of Fig. [a), such significant prompting bonus disappears, suggesting undeniable void in the current
LLM prompting taxonomy tailored for tabular prediction. There are two key points causing the gap:
(i) Existing competitions for general-purpose LLMs predominantly focus on their capabilities in processing unstructured data (Zhang et al. 2024a|by b). which is naturally different from structured tabular data characterized by dense heterogeneous numerical features (Borisov et al., 2022] Yan et al. 2023] Chen et al.] (2022), and the prevailing technical landscape of LLMs neglects nuanced sensitivity and understanding for numerical values (Qian et al. 2023 Yan et al., 2024b).
(ii) Most LLM tasks of interest can be formulated as sample-level data understanding then reasoning by generation, the input semanties are unstructured and detailed, while the tabular prediction (e.g., disease diagnosis with numerical metrics from medical examination and testing) requires overall statistical relation between numerical features and targets on the whole dataset or a specific task, which is difficult to access using a single tabular instance in high-level and constrained contexts.
Based on these observations, a straightforward question is, how to harness world knowledge of existing versatile LLMs, especially these commercial and blackbox (users cannot access the logit) ones [OpenAl| 2022 2023). to empower tabular prediction like disease diagnosis using medical testing results, which serves potential breakthrough for LLMs on statistical learning tasks.
To fill the aforementioned technical gap and extend LLM's capability to tabular prediction, we pro- pose SERSAL, synergy learning pipeline between small models and LLMs, requiring no gold labels. Different from existing prompting techniques designing hard or soft prompts to augment in- puts for unstructured data understanding,
SERSAL contributes from a brand new perspective that improves existing LLMs on statistical prediction for numeric tabular data, providing an inter- face to adapt LLM untapped knowledge such structured tabular data. SERSAL helps a blackbox LLM trigger and refine its vertical capabilities for domain tabular data in an unsupervised manner with the following steps: (1) Using simple zero-shot prompting to gather the LLM's s output confi- dence as initial coarse annotations of the whole dataset; (2) Teaching better small tabular model (e.g., FT-Transformer (Gorishniy et al., (02T)) from scratch based on the LLM's confidence like semi-supervised learning with noisy labels; (3) Reversely fine-tuning the LLM using the outcomes of the trained small model to further update LLM's annotations in the next loop: The process from step (1) to (3) be repeatedly applied to the LLM for continuous progress dataset. Essentially, SERSAL presents L.M prior knowledge on all tabular samples as "indicators" to small model, which serves "probe" during learning correct patterns from the well-expressed clean part to feedback for LLM self-improvemmnt.
specific tabular
In this paper, the main experiment is based on the well-known online blackbox LLMs, OpenAl GPT-3.5 (OpenAl 2022) & GPT-4 (OpenAT 2023), and as a prompting counterpart, our SERSAL can be directly transterred to other [.LMs once the fine-tuning APIs are given. In nutshell, our main contributions are:
For the first time, we bring the common challenge of existing general-purpose LLMs on numeric tabular prediction, a statistical learning featured task, to the spotlight that has not been covered by prevailing prompting techniques.
We propose SERSAL, novel unsupervised self- tromppronting method to adapt LLM's 's capa- bility to tabular data prediction, which leverages synergy learning with small models to capture and feedback correct patterns from LLM intrinsic knowledge.
Comprehensive experiments reveal that SERSAL is consistently more effective than com- mon textual prompting methods on medical tabular datasets, with general fcasibility in other vertical domains discussed.
2 SERSAL: AN LLM SELF-PROMG LOOP FOR TABULAR PREDICTION
We propose SERSAL, a synergy learning process using small models to trigger LLM's knowledge on tabular data, which is a fundamentally distinct prompting method and serves novel inter-
2

---

Published as a conference paper at ICLR 2025
Text Classification
UNUSABLE
Tabular Prediction
Clinical Nota Classification
Prediction
90
Asking for LLM's confidence
Tenching new small model
(MIMIC-II diecharge summary)
[Cleveland database table)
noisy soft labels
75
85
Zoro Shot CoT
Supervised
Raceunn (%0
Mapro-FF
SELF- ELF-PROMPTING IMPROVEMENT
Reversely updating teacher LLM
Quality Control
student's s labels
55
55
GPT-3.5
GPT-4
BEAT
GPT-3.5
GPT-4
FT-T
(a) Linguistic prompting gap
(b) SERSAL loop prompting
Figure 1: (a) Comparison of prompting effectiveness on unstructured textual data (Mullenbach et al, 2018) and structured tabular data (Detrano al. 1989) from medical domain, it clearly seen, with surprising medical expertise (Nori et al. (223), GPT-4 still struggles to catch up fully super- vised small models (ClinicalBERT (Huang et al.) (2019) for textual tasks and FT-Transformer (Gor-) ishniy et al. ( for tabular ones) on tabular data, implying essential task discrepancy that makes incompatible to rely on typical prompting techniques unlock the potential of LLMs for tabular prediction. (b) Unsupervised SERSAL triggers LLM's knowledge using small model.
face extend current LLMs tabular data prediction. Principally, SERSAL is inspired by the semi-supervised learning with noisy labels (LNL) and teacher-studens training, while several key differences exist: (1) LNL setting requires certain proportion of gold labels as the starting point, while SERSAL only access the LLM's soft pseudo labels (i.c., per-sample confidence) on the whole dataset; (2) teacher-student paradigm the student model is primarily considered to be compara- ble to the teacher, while SERSAL conservatively teaches a better student model by dynamically learning from the relatively clean LLM's outputs and regularizing on noisy ones to avoid misleading confirmation bias (Tarvainen & Valpola 2017), which produces better small model the target task to form a co-teaching manner. The overall framework of SERSAL is outlined in Fig. bb) and formulated in Algorithm Each part is detailed in the following subsections.
2.1 SOFT LLM PSEUDO LABELING
To access the prior knowledge of the LLM on specific tabular dataset, we first query its confidence on each sample using simple zero-shot prompt template. Specifically, the prompt consists of task description and listed feature specifications, for example, "You are professional doctor, here are some clinical metries of a patient, please give a likelihood between 0 to of suffering from a heart discase: [Age] 47 (years old); [Gender] Male; [Systolic Blood Pressure] 138 (mmHg); [Blood Lipid] 240 (mg/dL): ...", In this way the LLM's per-sample confidence on the whole dataset gathered, though the initial zero-shot performance is often far away from the one of supervised small tabular model (see Fig. [a) and Table 2), we can dig into such fine-grained LLM confidence by separately judging then using underlying clean and noisy supervision signal to teach robust small model.
2.2 SMALL MODEL TEACHING WITH NOISY LLM'S ' LABELS
This step aims to teach better small model with the collected soft outputs from the LLM. Intuitively, such LLM confidence is kind of noisy labels, thereby straightforward insight is to reformulate the teaching process learning with noisy labels (LNL). To sufficiently exploit LLM's 's prior knowledge, we adopt a "semi-supen superiiueeeiiee"" learning strategy after dividing training samples into a more reliable labeled set and another unlabeled set, then the small model is fitted with the soft LLM's labels in the labeled set and regularized on the ones of the unlabeled set, the data partition is based on per-sample loss since deep neural networks tend fit samples with clean labels faster than one with wrong
3

---

Published as a conference paper at ICLR 2025
labels according to the LNL theory (Arpit al. 2017), thus lower loss often indicates relatively cleaner labels (Chen et al. 2
In implementation, supervised LNL algorithm for image classification that dynamically fits a Gaussian Mixture Model on per-sample losses to distinguish probably clean and noisy LLM's labels and trains a pair of neural networks simultaneously to keep them diverged to avoid confirmation bias in single-model self- training (Tarvainen & Valpola 2017). 7). Apart from adapting DivideMix tabular data prediction, the used soft labels naturally apply label smoothing guided by the LLM. Besides, we leverage the pseudo labels with extreme confidence for early stopping with underlying assumption that annotations with extreme LLM's confidence is tend to be more accurate, which is observed in Fig. 22 and Fig. [4] and discussed in Sec. 3.3 Specifically, we divide a training subset as the early stopping set Des (:)
adapted version of DivideMix (Li ct al. 2019). common semi-
} to perform early stopping and hyper-parameter selection for the teaching process, where for the i-th sample, y; is its LLM's confidence vector, and y; is the corresponding hard labels (i.e., y; argmax(y:)) samples with maximum label confidence larger than threshold T (we uniformly set r = 0.9 in the experiment) are considered to be accurate enough for early stopping. During semiisuperrvssee some domain (e.g., medicine) tabular datasets suffer from data inadequacy, and the reduction on training subsets may distort data distribution. We formulate this step in the line 3-5 of Algorithm and conduct related ablations in Sec. 3.3
teaching, samples in the early stopping
are also used since
In summary, this teaching step adopts semi-supervised LNL process aggregate and distill prior knowledge into a small model to extend the LLM's real capabilities to tabular data prediction.
2.3 QUALITY CONTROL
Since SERSAL operates iteratively, it requires a termination mechanism to control the loop exit. Here we provide three heuristic strategies, users can also define their own control flow in practice.
Metric-based Control. In Sec. 2.2 we define the high-confidence training subset as the early stopping set Des which pseudo labels relatively more accurate (see Fig. ( There- fore, users can inspect metrics (e.g., AUC or accuracy scores for classification) by treating these pscudo labels as the "ground truth" to control whether to end the loop.
External Validation Control. If budget permits, human experts can collect and annotate appropriate external data as a validation set, e.g., in hospitals, regular medical data quality inspection needs to sample and label small part of data, and learning quality can be assessed with such external labeled set.
Rule-based Control. For example, users can define fixed iteration time.
For simplicity, in the main experiment uniformly report one-loop SERSAL performances in med- ical and other domain datasets (5) which has significantly surpassed the ones prevailing prompting methods, and further discuss the effectiveness of multi-loop SERSAL in Sec. 3.4
2.4 REVERSE LLM TUNING
The final step is to reversely teach the LLM using the well-trained small model to feedback the ag- gregated knowledge. Similar to using LLM's soft confidence to teach the small model in Sec. 2.2 we also use soft confidence from the small model to fine-tune the LLM (fine-tune the online black- box GPT's through their APIs in experiment). Specifically, the training samples are re-labeled by the small model with its guessed probabilities (line 7-8 in Algorithm []. the same prompt templates in Sec. 2. are used to construct the training corpus for the LLM. To avoid the excessive memorization of the LLM on the small model outputs (Bordt et al. 2023), we employ conservative tuning strat- egy that sets the maximum training epoch to 3 with proper early stopping (the fine-tur mee-tuning APIs of GPT-3.5 & GPT-4 provide automatic early stopping in default), making the LLM slightly fitted on the guessed labels while keeping a non zero minimum training loss. Then the updated LLM initiates the next SERSAL loop, forming an iterative process.
non-zero
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Unsupervised SERSAL. Line 2: LLM pseudo labeling (Sec. 2.1 Line 3-5: Small model teaching (Sec. 2.2; Line 6: Quality control (Sec. 2.3p; Line 7-9: Reverse tuning (Sec. (2.4).
Input: Unlabeled training set Xtrain and test set test. large language model fLLM Parameter: Confidence threshold т. quality control function fote
Output: Improved zero-shot tabular prediction Viest
1: Let 1. // Initialize iteration number
2: Softly labeled dataset D = (Xtraiu ((())))f(()))))))))))))))))))))) p((1) by current c(t) ULM
3: Randomly initialize a small tabular model ((t)
4: Select early stopping set D = {X,, (X y mmax 2 27 S
((1) train
5: (r(+) DivideMix (D train D g(0), -). Adapted DivideMix (Li et al. 2019
6: while fete (0r(1) X) do
(1) Predict(X..) train; (*())) // Soft label guessing by the small model
7:
$(0) Sharpen (ysm, temperature = 0.1). // Simple temperature sharpening (()) (())
8:
sm
9:
LLM
(l-1)
Finetune(Xueas train: am LLM) // Reversely tune the LLM with guessed labels
10:
f =t+1 1.
11: Repeat Line 2-5. // Self-prompting loop
12: end while
13: Viest Predict (rrdict(Xtest::s ge(t)). // Final prediction with the taught small model
14: return Frest.
3 EXPERIMENTS
In this section, first compare SERSAL with prevailing prompting techniques (using GPT-3.5 & GPT-4) and the fully supervised small tabular models on extensive medical tabular datasets in Sec. 3.2 Next, we conduct ablation on several key adaptations semi-supervised learning with noisy labels (LNL) in Sec. 2.2.222 inspect the effectiveness of multi-loop SERSAL in Sec. Also, we discuss the general adaptability of SERSAL on tabular data from other non-medical domains in Sec. 3.5 Besides, we explore the method interpretability by visualizing Shapely Value variation
during SERSAL process Sec. 3.6
3.1
EXPERIMENTAL SETUP
Datasets We evaluate
widely recognized medical diagnosis tabular datasets various dis-
eases: Heart Failure Prediction (HF, [Dtrano et al. (1989). Lung Cancer Prediction (LCC Ahmad Mayya 2020)), Early Classification of Diabetes (ECD, Islam al. (2020)), Indian Liver Patient
Records (LI, Raaaana et al.) (2012)). Hepatitis C Prediction (HE, Hoffmann et al. (2018)) ). Pima
Indians Diabetes Database (PID, [Smith et al. (1988). Framingham Heart Study (FH, O O'Donnell & Elosua 2008)), Stroke Prediction (ST. Fedesoriano (2020). COVID-19 Presence(CO) Heman- thhari (2020)) and Anemia Disease (AN, Kilicarslan et al. (202T). Besides, datasets clinical trail (Wang& & Sun 2022) and open domains (Gorishniy et al. 2021) are added to further inspect the effectiveness of SERSAL in difficult tasks and general data domains respectively. We split each tabular dataset (80 % for training and 20 % for testing), and keep the same label distribution in each split. Statistics of medical diagnosis datasets are given in Table All evaluated datasets are binary
classification tasks.
Dataset
HF
L.C
FCD
1.I
HE
PID
FH
ST
CO
AN
features
13
15
16
10
12
8
15
20
24
samples
303
309
520
583
615
768
4238
5110
5434
15300
P/N
0.80
6.92
1.60
2.51
0.11
0.54
0.18
0.04
4.17
0.57
disease
Heart
Lung
Diabetes
Liver
Hepatitis C
Diabetes
Heart
Stroke
COVID-19
Anemia
Table 1: Dataset statistics of ten medical diagnosis datasets for binary classification on various diseases. "P/N" denotes the amount ratio of positive samples and negative ones.
5

---

Published as a conference paper at ICLR 2025
Compared Methods Since SERSAL serves as an unsupervised self- promptin method for LLM tabular prediction, we compare with existing linguistic prompting methods for LLM usage in gen- eral textual and tabular tasks, which focus meticulously designed prompt texts: (1) Zero-Shot Prompting (0-shot) is the straightforward prompt that contains no examples: (2) Zero-Shot CoT Prompting (Kojima et al., (022) (CoTT is a popular prompting method which asks the LLMS to answer with intermediate reasoning steps to enable complex reasoning capabilities; (3) 8-shot Prompting (8-shot) is common few-shot prompt setting in standard prompting studies (Wei al. 2022 Kojima et al., 2022, Nori et al. 2023 it provides eight labeled samples (exemplars) to enrich prompt contexts and steer the LLM to the better outputs, in the experiment we randomly sample eight training examples and control the same positive-negative ratio (i.e., "P/N" in Table [T with at least one example for each class; (4) TabLLM (Hegselmann et al. 2023) and (5) LIFT (Dinh) et al. 2022) are two recent known linguistic prompt schemes for textualizing tabular data to fine- tune LLMs with gold labels, though TabLLM was additionally evaluated in zero-shot settings, none of them are originally proposed for unsupervised tabular scenarios, here we use their zero- shot schemes for comparison. Additionally, we provide a fully supervised small tabular model (FSSM) group using FI-Transformer (Gorishniy al. (02T)) for reference representing traditional supervised learning paradigm by fine-tuning dataset-specific small models.
Implementation Details All experiments are conducted with PyTorch on Python 3.8 and run on NVIDIA RTX 3090. For the small model, we uniformly use FT-Transformer with the default model and training configurations provided in the original paper (Gorishniy et al. (002)). For SERSAL, the only adjustable hyper-parameter is the temperature of DivideMix (Li al.) 2019) with choices of 0.5, 5.0 and 10.0 in line 5 of Algorithm [ (DL0 in line 4 of Algorithm 1p. The LLMs in the experiment includes OpenAI GPT-3.5 & GPT-4 to inspect the effectiveness of SERSAL across different LLM capabilities.
which is selected by the metric of the early stopping set
3.2 WIYY WE NEED SERSAL?
HF
LC
ECD
LI
HE
PID
ST
CO
Random guessing
37.22
40.18
46.25
50.28
62.73
63.24
50.39
41.76
71.55
51.28
(supervied FT-T)
88.19
86.61
99.60
78.94
100.00
84,72
66.25
82.98
99.91
99.92
0-shot (GPT-3.5)
71.88
78.87
85.71
76.81
68.51
73.12
60.32
63.01
82,60
90.43
8-shot" (GPT-3.5)
73.65
78.87
87.68
76.81
68.51
73.12
58.27
60.85
77.63
87.19
CoT (GPT-3.55
71.88
78.87
82.36
76.81
68.51
70.83
60.32
63.01
82.60
90.43
TabLLM (GPT-3.5)
76.37
78.87
87.06
78.24
74.39
75.69
61.78
68.48
85.78
89.11
LIFT (GPT-3.5)
78.23
80.69
83.92
73.60
72.57
73.12
60.32
70.92
87.93
90.43
SERSAL (GPT-3.5)
91.39
85.42
86.40
79.39
85.14
78.97
63.97
76.36
96.85
98.37
TabLLM+SERSAL (GPT-3.5)
93.82
85,42
88.39
80.71
89.27
82.54
65.02
81.74
97.51
98.16
SERSAL (GPT-4)
94.18
86.93
92.68
82.51
92.76
82.39
67.14
81.23
97.96
98.82
The AUC scores (%) of different tabular prediction schemes on 10 medical diagnosis datasets. The top part the traditional supervised small models, the middle one is compared LLM prompting methods (the top performances are marked in bold), the bottom part is additional com- binations. Here the results of SERSAL are only based on a single loop. "*" denotes the groups use gold labels. "FSSM" is the fully supervised FT-Transformer. The results more difficult clinical trial datasets are given in Table
Table 2:
Main Results Analysis The performances of different LLM prompting baselines are reported in the middle part of Table 2 An overall trend is that, when the GPT-3.5 meets medical domain tab- ular prediction tasks, the results using common prompting methods are consistently better than the ones of random guessing, demonstrating the general-purpont expertise inherently, but they are still far from the traditional supervised small models (see group "FSSM"), and further performance enhancement can not be achieved through usual prompting tricks as in textual tasks (see Fig. [a)) Specifically, we observe 8-shot prompting slightly benefits the per- formances in small-scale datasets (e.g., HF and ECD) but hurts in the larger datasets (c.g., FH, ST, CO and AN) compared to the 0-shot prompting, which may be explained by the representativeness of the used examples, since the distribution of the smaller datasets more likely to be covered by
LLMs indeed contain medical domain

---

Published as a conference paper at ICLR 2025
few examples, thus 8-shot performs better data scale decreases, and vice versa. For O-shot CoT prompting, it does not affect the overall results in most cases, but we find slight performance decline in two diabetes datasets (i.e., ECD and PID), this may be caused by the over-consideration of CoT on noisy features since diabetes can be diagnosed with several prominent features (e.g.. blood sugar and lipid). Although carefully crafted prompt templates from recent L.MM in-context tabular learn- ing studies (i.e., TabLLM and LIFT) show modest improvement, they still follow the linguistic nature to process numeric tabular data, and are primarily designed for LLM in-context few- shot learning or supervised fine-tuning. Our SERSAL explores a fundamentally novel prompting mechanism exploiting the information gain in the LLM's noisy outputs, which breaks through the predicament from an orthogonal perspective and serves as an interface to effectively adapt the LLM's domain knowledge to numeric tabular data. After applying SERSAL, without access to gold labels, the GPT-3.5 is able to achieve significantly better reasoning these medical domain tasks, with many cases competitive with the supervised small models.
Orthogonal Technical Contribution Based on the above analysis, SERSAL works in distinct underlying mechanism, and we can jointly adopt SERSAL and previous linguistic prompting meth- ods for better combined performances (see group "TabLLM+SERSAL" in Table 2
Continuous Performance Growth We additionally apply SERSAL to OpenAl GPT-4 on med- ical diagnosis datasets (the bottom part of Table 2 and more difficult clinical trial datasets (see Table 7 It can be seen SERSAL can further realize substantial performance gains as the capability of used LLMs becomes more powerful, which can even surpass the traditional supervised paradigm (N00041 19 and N003 12008 datasets in Table (7). indicating ample room for continuous prompting bonus in SERSAL alongside the emergence of more advanced LLMs.
3.3 SEVERAL KEY ADAPTATIONS
HF
LC
ECD
LI
HE
PID
FH
ST
CO
AN
SERSAL
91.39
85.42
86.40
79.39
85.14
78.97
63.97
76.36
96.85
98.37
w/o soft pseudo
84.58
76.58
87.24
78.25
75.79
75.93
62.58
75.05
93.97
97.53
w/o
84.03
74.11
75.92
59.39
47.41
68.43
57.08
74.70
90.57
97.57
Table The AUC scores of ablation on two key adaptations. "w/o soft pseudo" means replacing the LLM's soft outputs with hard ones during teaching the student model, "w/o ES" denotes no early stopping during DivideMix (line 5 in Algorithm
In Sec. 2.2 to adapt the LLM's outputs to semi-supervised LNL process to teach small model, gather per-sample confidence from the LLM as soft noisy annotations and heuris- tically select samples with extreme confidence for early stopping. In this section, we will ana- lyze the effect of the two designs which distin-
ECD
90
AUC
Acouracy
80
Performance
guish our SERSAL from traditional LNL set- tings.
70
60
The Effect of using Soft Labels We query soft confidence from the LLM (see Sec. [2.1] rather than directly using hard outputs for small model teaching. The prediction probabilities inherently reflect the LLM's prior knowledge as well as uncertainty on the domain tabular data and can be naturally treated as kind of la- bel smoothing. Besides, the probability values can be used to select relatively reliable labels to early stop the teaching process and avoid over- fitting. In Table 3 we compare the effect of using soft labels by replacing it with hard ones during
50
40
LLM confidence range
Figure 2: Performances in different LLM's confi- dence ranges on ECD and LI datasets. Extreme- confidence samples are relatively more reliable.
7

---

Published as a conference paper at ICLR 2025
SERSAL reasoning (group "w/o soft pseudo"). We find that using hard ones is usually suboptimal since it loses both prediction uncertainty and label smoothing, which is unable to exploit fine- grained LLM's knowledge.
The Effect of Early Stopping In addition to using LLM's soft outputs. relatively clean training subset is selected by threshold clipping on the per-sample confidence (line 4 in Algorithm D for early stopping. Table B] report the ablation results by directly training 100 epochs (group "w/o ES"). can be clearly seen, simply following the original DivideMix is far from the desired results, since tabular features are heterogeneous and high-level compared the well-patterned pixels of images (Chen et al.] 2023] Yan et al.] 2024b), and in medical tabular domain the typically limited available data further makes it prone overfit without early stopping, for example, except large AN dataset, all other tabular datasets appear to be significantly impacted by removing the early stopping mechanism. The heuristic design of selecting extreme-confidence sample is inspired from the empirical assumption that confident predictions from the LLM are more likely to be accurate, which is supported by the performance variation of different confidence ranges in Fig. 2 2nd Fig.
3.4 EFFECTIVENESS OF MULTI-LOOP SERSAL
Since SERSAL can be iteratively applied to the LLM (see Fig. [b)), we further inspect the effec- tiveness of multi-loop SERSAL for GPT-3.5 reasoning. Specifically, we repeat the pipeline three times on ECD and I.I datasets, the result variations are reported in Table
During three loops, progressive improvement
on both the small model (SERSAL outputs are
ECD
LI
from the well-trained small model of each loop)
Loop
and the GPT-3.5 is observed. Surprisingly, even inferior the 8-shot prompting baseline on ECD dataset after the first loop (see Ta- 2 we find SERSAL can reduce the gap and even outperform few-shot baselines after sev- eral loops. Such continuous progress probably comes from the synergy learning between the small model and the LLM that shares a similar underlying mechanism of co-teaching (Han al., 2018). i.e. both sides dynamically learn from part of reliable pseudo labels from each other and makes them diverged avoid con-
SERSAL
LLM 0-shol
SERSAL
LLM 0-shot
86.40
85.71
79.39
76.81
2
87.00
86.42
82.47
80.26
3
89.00
87,81
84.07
82.91
Table 4: The AUC score variation of SERSAL outputs and zero-shot prompting of the tuned GPT-3.5 (LLM 0-shot) on LI and ECD datasets during three loops. "# Loop" is the same as the variable / in line I of of Algorithm [ LLM 0-shot group at the first loop is the original LLM.
firmation bias, forming mutual improvement manner to aggregate and refine LLM's untapped domain knowledge for tabular prediction.
3.5 GNERALIZED DATA ADAPTABILITY ON OTHER DOMAINS
In this section, we further explore the data adaptability of SERSAL other non-medical domains. We evaluate on three classic bi- nary classification datasets: Churn Model- ing (Iyyer) 2019), Credit (Credit Fusion| [201T) and Adult (Kohavi et al.] 1996). which are widely included in general tabular prediction studies (Gorishniy et al. 202T Yan et al., 2023 Grinsztajn et al., (022). Additionally, we build a dataset "Fake" by randomly generating fea- tures and binary labels to emulate an extreme case where the LLM has no relevant knowledge at all. The data information and the results are given in Table 5 As in the medical domain, the 3.5 indeed holds the world knowledge and can directly achieve the considerable results with simple zero-shot prompting, and SERSAL further enhances the performance significantly. However, when facing the tabular data from an unknown
Chum
Credit
Adult
Fake
domain
Business
Finance
Seciology
N/A
# features # sumples
10
10
14
10000
16714
48842
1000
Random guessing FSSM
66.35
43.80
58.73
53.85
86.27
84.88
91.39
55.31
0-shot (GPT-3.5)
77.81
69.05
75.10
46.28
SERSAL (GPT-3.5)
83.29
79.36
88.72
38.72
Table 5: The dataset statistics and AUC scores on other non non-medical domains. "Fake" is generated dataset with random labels and features. The de- notations follow the ones in Table [ and Table 2
8

---

Published as a conference paper at ICLR 2025
domain (i.e., the Fake dataset), the LLM outputs high confidence wrong labels, SERSAL is unable to recognize such totally misleading bias. Therefore, our SERSAL shares the same basic limitation other linguistic prompting methods that the applied LLMs require certain level of knowledge in the target domain.
3.6 INTERPRETABILITY OF SERSAL
High
Aspariate
Age
Total Biliubii
Aspariate Aminnorranseerase
Alanine Aminotransferase
Alkaline Phosphatase
Feature value
Direct Bilirbin
Total Bilirubin
Alkaline Phosphatase
Alanine Aminotransterase
Albumin
Direct Bilrubin
Total Proteins
Total Proteins
1st SERSAL loop
2nd SERSAL loop
Age
Albumin
AUC: 79.39
AUC: 82.47 %
Albumin and Globulin Ratio
Accuracy: 65.51
Gender
Accuracy: 70.69
Gender
Albumin Giobulin
Low
0.1 02 0.3 0.4
0.005 0.000 0.005 0.010 0015 SHAP value (impact on model output)
SHAP value (impact on model output)
Figure 3: Interpretability visualization from feature importance perspective: the variation of the Shapley Values (treat SERSAL outputs the targets) and performances Indian Liver Patient Records (IJI dataset) after one and two SERSAL loops using GPT-3.5.
In Fig- 3 we visualize the variation of Shapely Values on Indian Liver Patient Records (I.I) dataset after one (left) and two (right) SERSAL loops by treating the predictions (i.c., Algorithm D as tar- gets. It can be clearly seen the feature "Age" is adequately considered after one loop self pol--promp,ing. which highlights strong and reasonable positive correlation between age and the incidence liver diseases that aligns with the medical expertise. Besides, negative correlation with "Total Proteins", a guiding clinical metric reflecting the liver's synthetic function, captured in the right figure to con- tribute the prediction, since lower total protein level indicates risk of liver eirrhosis. These two reasonable changes of feature importance interpret the SERSAL prompting is able to iteratively re- fine the domain expertise in the LLM, calibrating the statistical feature-target relationship for better reasoning results during the process.
4 RELATED WORK
Prompt Engineering for In-Context Learning Prompt engineering is flourishing discipline for better LLM reasoning through meticulously designed linguistic input contexts or interaction process. The most common and straightforward prompting the single-round instruction that directly asks
with zero (zero-shot) or several (few-shot) demonstrations (Brown et al. 2020 Wei et al. (021). but such prompt style fails work in more complex reasoning tasks (Wei et 2022). To tackle this deficiency and improve the L.MM's capacity vanced prompting methods are emerging, such as chain-of-thought (CoT) (Wei et al., 2002 Kojimal ct al. 2022 Zhang et al. 2023a), tree-of-thought (ToT) (Yao et al. (2023) and self-consistency (Wang ctt al. 2023). However, current prompting methods are mostly designed to serve unstructured data tasks (Zhang al. 2023b). Although recent studies on LLM in-context learning for tabular data
wide range of tasks, recently, studies on more ad-
(e.g.. TabLLM (Hegselmann et al.) 2023), LIFT (Dinh et al, 2022) ) propose table-friendly prompt- ing strategies, their linguistic nature still hinders the numeric table understanding (Yan et al. 2024b).
Semi-supervised Learning with Noisy Labels Semi-supervised learning treats the unlabeled samples as regularization for better model generalization (Lee et al.] 20133] Tarvainen & Valpola 2017 Miyato et al.) 2019 Berthelot al. (2019). Recently, the related theory has been introduced to noisy label learning scenarios (Song et al. 2022) that dynamically divide samples into clean labeled group and noisy unlabeled one (Li al.
2019 achieve robust learning from noisy labels.

---

Published as a conference paper at ICLR 2025
LLMs for Tabular Data Prediction As machine learning task in tabular data applications, tabu- lar prediction has gained increasing attention from the research community due to the heterogeneous nature and numerical features of tabular data compared to other unstructured modalities. Previous studies in tabular prediction models focus on designing tailored neural networks (Arik & Pfister 2021 Gorishniy al.] |Ya et al.] 2024a Chen et al. 2024) emulate and surpass traditional tree-based models (Chen & Guestrin 2016 Ke et al. (0117) under fully supervised paradigm. More recently, motivated by the widespread success pre-trained language models (Brown et al. 2020
OpenAT, 2022, 2023), the unique bonus of neural networks is exploited in tabular model develop- ment, such as pre-training (Wang Sun 2022) and in-context learning Hollmann al. (2023), and open-sourced LLMs have been popular base models to be adaptively pre-trained for better tabular prediction since their inherent knowledge (Yan et al. 2024b, Wen et al. (024). However, current ex- ploration tabular prediction L.LMs involves costly pre-training on large-scale tabular data, which requires access to LLM codes and parameters, and heavy adaptation to tabular data may impact the original usability of the LLMs on other unstructured data tasks.
CONCLUSIONS
This paper revealed the common challenge of existing general-purge sanrra--prppsse LLMs tabular prediction and proposed SERSAL, a novel unsupervised self-prompting method in non-linguistic mechanism that triggers the LLM's domain knowledge for better tabular prediction. This is achieved through a co-teaching process between the LLM and a well-taught small tabular model which learn from the other's noisy outputs to aggregate and refine the LLM's untapped capabilities. Extensive ex- periments on medical and non-medical domain tabular datasets demonstrate that, as an orthogonal prompting landscape, SERSAL is consistently suitable for extending the potential of LLMs to nu- meric tabular data.
ACKNOWLEDGMENTS
This research was partially supported by National Natural Science Foundation of China under grant No. 12326612, Zhejiang Key R&D Program of China under grant No. 2023C03053, and Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence.
REFERENCES
Ahmad S Ahmad and Ali M Mayya. A new tool to predict lung cancer based on risk factors. Heliyon, 2020.
Sercan Ö Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In AAAI, 2021.
Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In ICML, 2017.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, 2019.
Sebastian Bordt, Harsha Nori. and Rich Caruana. Elephants never forget: Testing language mod- els for memorization of tabular data. NeurlPS 2023 Second Table Representation Learning Workshop, 2023.
Vadim Borisov, Tobias Leemann, Kathrin Sessler, Johannes Haug, Martin Pawelezyk, and Gjergji Kasneci. Deep neural networks and tabular data: survey. TNNLS, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NewrIPS, 2020.
Marco Cascella, Jonathan Montomoli, Valentina Bellini, and Elena Bignami. Evaluating the feasi- bility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of Medical Systems, 2023.
10

---

Published as a conference paper at ICLR 2025
Jintai Chen, Kuanlun Liao, Yao Wan, Danny Z Chen, and Jian Wu. DANets: Deep abstract networks for tabular data classification and regression. In Proceedings of the AAA! Conference on Artificial Intelligence, 2022.
Jintai Chen, KuanLun Liao, Yanwen Fang, Danny Z. Chen, and Jian Wu. TabCaps: A capsule neural network for tabular data classification with BoW routing. In ICLR, 2023.
Jintai Chen, Jiahuan Yan, Qiyuan Chen, Danny Z Chen, Jian Wu, and Jimeng Sun. Can a deep learning model be sure bet for tabular prediction? In SIGKDD, pp. 288-296, 2024.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In ICML, 2019.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.
Will Cukierski Credit Fusion. Give me some credit, 2011. URL https://kaggle.r com/
competitions
Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and Iims for legal case judgement summarization? arXiv preprint arXiv:2306.0122 01244, 2023.
Robert Detrano, Andras Janosi, et al. International application of a new probability algorithm for the diagnosis of coronary artery disease. The Americun journal of cardiology, 1989.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfacee fine-tuning for non-
language machine learning tasks. In NeurlPS, 2022.
Fedesoriano. Stroke prediction dataset.
https:/ /wwww http///aaageee..o///aaaaaaaaeeee Eedesoriano/ http://tttoeepeeiiiiiiii..ooo dat dataset 2020.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. NeurlPS, 2021.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In NeurlPS, 2022.
Taicheng Guo, Kehan Guo, et al. What indeed can gpt models do in chemistry? a comprehensive benchmark on eight tasks. arXiv preprint arXiv:2305.1 18365, 2023.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurIPS, 2018.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models. In AISTATS, 2023.
Hemanthhari, Symptoms and covid presence (may 2020 data). https:///// http:::///www../ http://wwwwwwww..e/e// kwggla. com/ datasets hataasttss/eemmtthhaaee hhariiatthhaii /swpptttssssssspp..o -nw-ooiiidddddddodaooooo-oo----d wwwovddppreeencce 2020.
Georg Hoffmann, Andreas Bietenbeck, Ralf Lichtinghagen, and Frank Klawonn. Using machine
learning techniques to generate laboratory diagnostic pathways-a case study. Lab Precis Med, 2018.
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. In ICLR, 2023.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.0533 2019.
MM Islam, Rahatara Ferdousi, Sadikur Rahman, and Humayra Yasmin Bushra. Likelihood predic- tion of diabetes at early stage using data mining techniques. In Computer Vision and Machine Intelligence in Medical Image Analysis, 2020.
11

---

Published as a conference paper at ICLR 2025
Shruti lyyer. Churn modelling. 2019.
URL
saxutimechleaee mazttt/meheaanncchrr3 waw.aaann/chuu-ooeeeeeaaee
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 2016.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: highly efficient gradient boosting decision tree. In NeurlPS, 2017.
Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HLT, 2019.
Serhat Kilicarslan, Mete Celik, and Şafak Sahin. Hybrid models based on genetic algorithm and deep learning algorithms for nutritional anemia disease classification. Biomedical Signal Processing and Control, 2021.
Ron Kohavi al. Scaling up the accuracy of Naive-Bayer Bayes classifiers: A decision-tree hybrid. In KDD, 1996.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022.
Dong-Hyur Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, 2013.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: supervised learning. In ICLR, 2019.
Learning with noisy labels as semi-
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supi-super learning. TPAMI, 2019.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. In NAACL, 2018.
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purposs tuning? case study in medicine. arXiv preprint arXiv:2311.16452, 2023.
Christopher J O O'Dnnelll and Roberto Elosua. Cardiovascular risk factors. insights from framingham heart study. Revista Española de Cardiología (English Edition)) 2008
OpenAl. Openai: Introducing chatgpt. http://s//gggeaaa..cc////////aap 2022.
OpenAl. Gpt-4 technical report. http:///cdchhopeeai.co/moaaeeeee
pdf 2023.
Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction. In ACL, 2023.
Yujia Qin, Shihao Liang, et al. Toolllm: Facilitating large language models to master 16000+ real- world apis. arXiv preprint arXiv:2307. 16789, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Bendi Venkata Ramana, M Surendra Prasad Babu, and NB Venkateswarlu. critical comparative study of liver patients from usa and india: an exploratory analysis. IJCSI, 2012.
Jack W Smith, James E Everhart, et al. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, 1988.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- fae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
12

---

Published as a conference paper at ICLR 2025
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve supi-supe deep learning results. NeurIPS, 2017.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy,
Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In ICML, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
Zifeng Wang and Jimeng Sun. Transtab: Learning transferable tabular transformers across tables. In NeurlPS, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. In ICLR, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
Xumeng Wen, Han Zhang, Shun Zheng, Wei Xu, and Jiang Bian. From supervised to generative: A novel paradigm for tabular deep learning with large language models. In SIGKDD, pp. 3323- 3333, 2024.
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Chen, and Jian Wu. T2G-Former: Organizing tabular features into relation graphs promotes heterogeneous feature interaction. In AAAI, 2023.
Jiahuan Yan, Jintai Chen, Qianxing Wang, Danny Z Chen, and Jian Wu. Team up gbdts and dnns: Advancing efficient and effective tabular prediction with tree-hybrid mips. In SIGKDD, pp. 3679- 3689, 2024a.
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. In ICLR, 2024b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurlPS, 2023.
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large language models meet nl2code: survey. In ACL, 2023.
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm- llms: Recent advances in multimodal large language models. arXiv preprint 24X1v:2401.13601 2024a.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-languas models for vision tasks: A survey. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024b.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In ICLR, 2023a.
Zhuosheng Zhang, Aston Zhang. Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923. 2023b.
Wayne Xin Zhao, Kun Zhou, et al. A survey of large language models. arXiv:2303. 18223, 2023.
arXiv preprint
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS & IMPACTS
As discussed in Sec. 3.5 though our SERSAL is distinguished from traditional prompting methods by its non-linguistic mechanism, it still requires the LLMs with latent knowledge in the target do- main to be effective, Therefore, in practice the user should have prior understanding of the used LLM's capability advantageous application fields. SERSAL contributes to the progress in both LLM prompting and tabular data community through providing a novel interface to adapt untapped knowledge LLMs the tabular prediction tasks zero-shot manner, which is particularly useful in the regime where limited data or annotation is available.
B DATASETS AND EXPERIMENT DETAILS
We provide detailed data information of the experiment tabular datasets in Table 6 We drop the samples with missing features and adopt the same preprocessing as Gorishniy al. (202T) before training. For MIMIC-III discharge summary dataset (Johnson et al. [016] Mullenbach et al. 2018) used in Fig- aa), we retain the most frequent labels (medical codes) since our goal is just to demonstrate the prompting effectiveness medical textual tasks and conducting validation on the full label version (several thousands labels) inconvenient, During conducting zero-shot prompting for GPT-3.5v and GPT-4v the MIMIC-III dataset, we follow the PhysioNet Credentialed Data Use Agreement and enroll the Azure OpenAl service without human review the data to protect the data from third-party access.
Sumplele
Stutee
Indian Patient Reconds
hetpst/ /eww, keggle, comm www.://ttttte/ecce/////oooaoaaae
Lveerpett recorox
Pina Indians Diabeses Databose
PID
768
Re44 WAy 2030 comeeeen te Deric rigine aire He MOVO KANS
Tramingham Heart Study
F11
4238
0.18
neteps 7W9. /AA. 75 com con. ooetece Ho0h rehurihe RoSt wow.----------- Pay
Stroke Prediction
5110
0.04
m-4p51 7WW /WW. 753018, com
Heponitis C Prediction
HE
6115
12
0.11
0- 77ay Zagule coom La L97 Tell por.eeoooee poriaho here a aal
COVID-19 19
LU
5433
20
4.17
79W 7.3ere comm on seicasees Ee derlee auporoie and 2e011 2esenee
Lung Cancer Prodiction
I.C
509
15
K.92
7999 /2.. Zane le Jomm dal avelts 878 ant shiaa 1ir Talioe
Heart Falluc Prediction
HF
503
13
0000
aar 199 199. cento
Early Classification of Diabetes
DCD
530
16
1660
444 1715//999 2430 con cont MAraz o. and rASmaar Eay ye
Anemia Discose
AN
15300
24
0.57
(1357//80 Kaggge co00 0ara55ee7
Chum Madeling
10000
10
0.26
Give Me Some Credit
16714
10
1.00
bctes:/ Pcttps://ww //wgy. Kogale comte e c/Civcct-ennccree
US Adult Incontic
45842
0.31
Table Detailed data information of used tabular datasets (10 from the medical domain and from others). "P/N" denotes the amount ratio of positive samples and negative ones.
C RESULTS ON CLINICAL TRIAL DATASETS
We evaluate SERSAL clinical trail mortality datasets, which require specialized scientific knowl- edge for clinical trials. Although SERSAL prompting with GPT-3.5 cannot directly achieve good performance such vertical tasks, further performance gains are still observed once more powerful GPT-4, indicating room for continuous improvement as more advanced L.MSs appear.
N00041119
N00174655
N00312208
N00079274
N00694382
FSSM (superviie FT-T)
62.38
89.20
77.83
71.78
73.89
0-shot (GPT-3.5)
56.79
73.08
63.49
59.85
62.70
CoT (GPT-3.5)
56.79
73.08
60.73
59.85
62.70
SERSAL (GPT-3.5)
58.31
82.64
71.92
64.17
66.31
SERSAL (GPT-4)
65.08
88.62
78.39
67.94
71.47
The AUC scores (%) of different tabular prediction schemes clinical trail mortality datasets used in Wang & Sun (002) (see ClinicalTrials.gov The similar denotations are used as Table 2 No gold labels are used for prompting methods here. It can be seen SERSAL achieve continuous improvement and even perform comparably with the traditional supervised paradigm once more powerful base LLMs are applied.
Table 7:
https::/ppyysllnnet.orrg/eeee
14

---

Published as a conference paper at ICLR 2025
D MECHANISM EXPLANATION OF DIVIDEMIX IN SERSAL
To make the paper friendly to the audiences from different background, in this section we provide detailed mechanism explanation of learning with noisy labels (LNL) and how to learn better small (neural network) model from LLM noisy annotations using DivideMix.
DivideMix mechanism in SERSAL In the traditional noisy data learning field, was theoreti- cally proved and empirically observed that the "memorization" behavior of neural networks leads to different optimization behavior on real data and noisy ones that neural networks tend to learn simple patterns first before fitting label noise (Arpit et al. ). Based on this theoretical foundation, typical group of LNL methods (Berthelot et al. 2019; Li et al. 2019) exploit per-sample training loss judge the noisy labels, for example, in paper adopt DivideMix (Li al. (2019) to learn a small model using LLM noisy annotations, which models the noise probabilities of each sample by dynamically fitting Gaussian Mixture Model (GMM) on per-sample losses, all training samples are divided into a clean set and a noisy set based on a probability threshold r. During the DivideMix training process, samples in the clean set are used for supervised learning (using their soft LLM annotations), while ones in the noisy set is used in an unsupervised manner (only using their fea- tures), e.g., learn with regularization loss or reconstruction task. The process will be ended until the average loss of heuristically selected early stopping subset NiighLLMM-coffdennn samples Des Algorithm 1) is converged, i.e., the loss of early stopping subset not decreased for m epochs. Notably, clean sample is not equivalent to high-LLMMcooffdeece sample, but the sample which LLM annotation is easier to by the small tabular model. Since the small model (i.e., FT- Transfoffrrm here) is only supervised by clean data and regularized noisy data, all data is sufficiently and reasonably exploited to acquire a better pattern.
DivideMix hyperparameters in SERSAL We refer the original hyperparameter settings in DivideMix paper [4] and only search the temperature (T') in {0.5, 10.0). with fixed regular- ization loss weight L,, 25, clean probability 7 to 0.9, and the learning rate of the small model (FT-Transformer) to lc-4. Additionally, we uniformly introduce the early stopping patience m to 5. The best temperature selected based on the training loss of early stopping subset Des-
HF
LC
ECD
LI
HE
PID
ST
AN
0-shot GPT-3.5 #1
71.88
78.87
85.71
76.81
68.51
73.12
60.32
63.01
82.60
90.43
SERSAL #1
91.39
85.42
86.40
79.39
85.14
78.97
63.97
76.36
96.85
98.37
0-shot GPT-3.5 #2
87.58
83.74
86.42
80.26
86.18
79.26
63.86
73.62
91.29
93.62
SERSAL #2
92.03
86.15
87.00
82.47
87.32
80.61
65.27
79.58
97.20
98.93
0-shot GPT-3.5 43
89.26
85,39
87.81
82.91
86.87
81.47
64.12
76.37
93.65
94.13
SERSAL #3
93.58
85.42
89.00
84.07
89.57
81.83
65.27
80.93
97.02
98.60
Table 8: The AUC score variation of SERSAL outputs and zero-shot prompting of the tuned GPT- 3.5 on all datasets from Table 2 during three loops.
15

---

Published as a conference paper at ICLR 2025
Heart Failure (HF)
Lung Cancer (LC)
Hepatitis C (HE)
Pima Indians Diabetes
(PID)
AUC
AUC
Accuracy
90
Accuracy
Accurary
Accuracy
20
Perfommenee D00
Rerfomanca NN
70
80
50
50
30
40
40
LLM confidence
LLM confidience range
confidence
LLM confidence
Framingham Heart Study
Stroke (ST)
COVID-19 (CO)
Anemia Disease (AN)
(FH)
AUC
AUI
Accunry
Accuracy
90
50
Accuracy
20
Peformere (N)
Pefforunce 283
Perforunce
70
80
50
ENDAIII 3. FORIIIES (518)) LLM confideace rarge
JP.. MAATITTEEGGASIIS..... LIM conndence range
PARAEEAAARE UPNBAC RRIN0-
LUM confidance narge
LLM confidence sange
Figure 4: Performances in different LLM confidence ranges other eight datasets. The overall trend of high-confidence samples being relatively more reliable still holds.
16


Reviews:
[{"review_id": "eUdiRcAPG9", "reviewer": "Reviewer_m9i1", "summary": "The authors propose a framework for refining a pretrained LLM for zero-shot tabular prediction. In particular, the authors propose to iteratively refine a given LLM's tabular prediction capability via a small \"student\" tabular prediction model (FT-Transformer), where at each iteration, the LLM is prompted to generate noisy labels for an unlabeled training dataset, the tabular prediction model is trained on the noisy labels based on the DivideMix approach, and the predictions from the trained tabular prediction model is used to fine-tune the LLM. Focusing primarily on tabular prediction datasets from the medical domain, the authors conduct several experiments using GPT-3.5 and GPT-4, and compare the resulting performance of their approach against those of alternative prompting methods.", "questions": "Please see my comments in the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The limited generalizability/transferability of LLMs to the tabular domain, which is discussed as the main motivation for the paper, is a well-known issue and an active area of research.\n- The idea of using an LLM as noisy labelers and combining it with semi-supervised learning methods designed to handle noisy labels is  interesting.\n- Several ablation experiments are considered to investigate the impact of different design choices (e.g., soft vs. hard labels from the LLM, early stopping), and empirically validate the assumption that high LLM confidence scores tend to correlate with the accuracy of the LLM-generated labels.", "weaknesses": "- Overall, there appears to be a misalignment in the motivation for the paper (i.e., improving the zero-shot generalizability of LLMs for tabular prediction) and what is proposed in the paper. My interpretation of what is really being addressed in the paper is \"How can we leverage LLMs as noisy labelers to train a downstream *tabular prediction model* (e.g., FT-Transformer) when we only have access to an *unlabeled* training set. In particular, the following are my specific concerns:\n    - While the proposed iterative process also involves fine-tuning the LLM, predictions on any test set are generated using the downstream tabular prediction model, as presented in line 13 of Algorithm 1 ($\\theta^{*(t)}$ denotes the parameter of the downstream model and not the LLM).\n    - The proposed approach is not really \"zero-shot\" in that while the training dataset does not include the ground-truth labels, the dataset is still being used to explicitly train the tabular prediction (and also the LLM for that matter).\n    - The Introduction discusses limitations of LLMs in handling heterogeneous numerical features, but the paper does not directly address how to resolve this problem for LLMs. \n    - As such, the title and the overall framing of the work appear ill-conceived, as the work is not really enhancing the \"zero-shot tabular prediction for LLMs\".\n- In Section 2.2, the overall description of how the DivideMix approach is being applied to the considered context would benefit from better clarity and details (even if pushed to the Appendix given space constraints). For example, how is the early stopping being performed within the adapted DivideMix algorithm? What are the hyperparameters in the phrase \"hyper-parameter selection for the teaching process\" and how are they being selected in the absence of a validation set?\n- In the experiments, the proposed approach is compared against alternative zero-shot (vanilla zero-shot, zero-shot CoT, TabLLM, LIFT) and few-shot prompting strategies for generating tabular predictions from off-the-shelf LLMs, which do not seem to be the most relevant baselines here. The proposed approach involves directly *fine-tuning* the LLM as opposed to being a new \"prompting\" strategy (not to mention that it's not even really the LLM that's generating the final predictions), and while TabLLM and LIFT were both originally proposed for fine-tuning the LLM for tabular prediction, these baselines are only considered in the zero-shot prompting regime.\n    - On a related note, since TabLLM and LIFT were considered in the zero-shot setting, this would mean that the only difference between them and the vanilla zero-shot prompting baseline is really in the prompt format and the set of instructions. The AUC values in Table 2 seem to indicate that the prediction results are highly sensitive to such choices. It is well-known in the literature that LLMs are highly sensitive to the choices of the prompt [1] and the tabular feature serialization method [2,3], so the results should appropriately account for these concerns to ensure a fair comparison.\n- The datasets considered in the main evaluations (Section 3.1) are publicly available datasets, with most of them having been around for a long time. As such, there is a potential risk that these datasets have been part of the LLM pretraining corpus, which limits the generalizability of the findings. While the authors do conduct an ablation study in Section 3.6 involving a randomly generated dataset (denoted as \"Fake\") and demonstrate that the proposed approach can fail if the model does not have relevant domain knowledge, it seems important that the evaluations are also carried out on either (i) private datasets or (ii) public datasets that have been released after the pretraining-data cutoff dates for the LLMs, which still come from a domain that the LLMs would carry relevant domain knowledge. For example, would the LLM still generate high-quality confidence scores if we were to test on a new medical diagnosis dataset released in the future? It appears important that the LLM confidence scores are reasonably well-calibrated for this approach to work in the first place; otherwise, the error will only compound over several iterations of the proposed algorithm.\n- In Figure 2 and Table 4, why are the results shown only for the ECD and LI datasets? Do similar trends hold for the other datasets?\n\nReferences:\n\n[1] Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (Sclar et al., 2024)\n\n[2] TabLLM: Few-shot Classification of Tabular Data with Large Language Models (Hegselmann et al., 2023)\n\n[3] Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey (Fang et al., 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fZkL3o9jcy", "reviewer": "Reviewer_diWx", "summary": "This study focused on training a small model to effectively promote LLMs to enhance their zero-shot predictions performance on healthcare tabular data. The study was motivated by the observations that, in the literature, most of the related studies have been primarily focused on either the medical imaging data or the text data. While, healthcare tabular data may pose their unique challenges, where it is common to have widely heterogenous distributions across different variables. The proposed idea of synergy learning is to first deploy LLM to generate some noisy annotations to allow the small model to learn and generate tuning promotes to refine LLM for the task; and we iterate this \"synergy learning\" to continuous improve the model's final performance. The authors conducted detailed experiments with some publicly healthcare tabular datasets to validate the performance of their proposed approach. Additionally, the authors have also investigated the generalizability of the proposed approach over other domains via evaluations with 3 classification datasets and 1 generated dataset, aiming to simulate the scenario for a unknown domain.", "questions": "1/ Point 1 under weaknesses\n\nIt is interesting to aggregate LLM knowledge in a small tabular model in SERSAL process, could you give a detailed mechanism explanation on why a better performed small model can be learned from LLM outputs?\n\n2/  Point 2 under weaknesses\n\nWonder how sensitive the effectiveness of SERSAL prompting is to the initial LLM performance on the target dataset, or how to apply SERSAL on the data domain where the initial performance is not good?\n\n3/ Point 3 under weaknesses\n\nCould provide the detailed computational complexity analysis of SERSAL in its real-world applications?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1/ The manuscript is, in general, well written, Objectives and motivations of the study were clearly explained. Details of the experiments were provided. Key findings were appropriately summarized in tables and figures.\n\n2/ I share the same belief that the size of the LLMs continues to go. It will become infeasible to retrain or fine tune the entire LLMs for specific domains. The use of small models to learn from LLMs and/or to promote LLMs is indeed a promising direction to explore.\n\n3/ The study proposed an interesting synergy learning mechanism. In the proposed mechanism, small model first learn from the noisy annotations from LLMs with low confidence, then, after quality controls, the learned small model generate promotes aiming to refine LLMs' predictions; while the refined LLMs will again generate slightly more confident labels to further train the small model. The proposed synergy learning process is interesting and sound.\n\n4/ The team has evaluated the performance of the proposed approach over 10 healthcare datasets focusing on different disease. This helps to show the stability of the proposed approach's performance over healthcare tabular data.\n\n5/ Also, to demonstrate the potential generalizability of the proposed approach to other domains (other than healthcare), it was also evaluated over 3 binary classification datasets and also 1 simulated dataset.\n\nOverall the quality of the manuscript is good. Nevertheless, there are still some areas for improvements as mentioned next.", "weaknesses": "1/ Better explanation on background knowledge for our general LLM audiences\n\nThe proposed SERSAL prompting for LLMs on tabular prediction task bears an essentially different mechanism compared to prevailing textual prompting methods, requiring relevant knowledge on noisy label learning theory for general audiences in LLM research fields, it will be better to detail the inherent mechanism on why a better small model can be obtained by learning from the LLM noisy outputs.\n\n2/ Dependency on initial LLM performance?\n\nAs pointed out in the Limitation Section (Appendix A), the proposed SERSAL prompting requires the LLMs with latent knowledge in the target domain to be effective, though a randomly generated dataset (Fake dataset in Table 5) is evaluated to partially answer SERSAL unusability on unreasonable data domain, but the case of reasonable data with very low initial accuracy and how to handle such datasets is not discussed (the fault tolerance limit or the least initial LLM performance that SERSAL can be accepted).\n\n3/ Lack of detailed computational complexity analysis \n\nThe current manuscript did not analyze the computational complexity in detail which may be important for the usability of SERSAL on large-scale datasets. As one of the key novelty of the study is to enable the use of small models to enhance the learning efficiency of large models. Do think that detailed computational complexity studies are needed to justify this point.\n\n4/ Some reference formatting issue\n\nThere are several instances of the misuse of LaTeX citation commands `\\citep` and `\\cite` in the paper (e.g., Line 65, 76, 85 and the caption of Fig. 1).\n\nPlease carefully correct the relevant LaTeX commands during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ASbyHXnzv2", "reviewer": "Reviewer_jQZe", "summary": "The paper proposes a new method, SERSAL, using synergy learning to improve LLMs for zero-shot tabular prediction. SERSAL designs a loop which includes (1) soft LLM pseudo labelling (2) small models training with DivideMix (3) quality control (4) LLM fine-tuning. With one-loop training, SERSAL demonstrates new state-of-the-art zero-shot performance on 9 out of 10 tabular prediction datasets in the medical domain and 3 other binary classification datasets in other domains. Experiments also demonstrate improvements with more (3 loops) training by SERSAL.", "questions": "1. Transformer is still a relatively large model. It will be useful to see how small models like CNN, RNN, or even RF, SVM may work and accelerate the loops.\n\n2. Table 4, please consider changing Loop 0,1,2 to Loop 1,2,3 to avoid confusion. \n\n3. Table 4, should the performance of N-loop SERSAL be the same as (N+1)-loop 0-shot LLM, because (N+1)-loop 0-shot LLM is the fine-tuned LLM from N-loop SERSAL?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is overall very well written with clear methodology, experiments, and strong results.\n\nThe proposed method SERSAL designs a loop to enhance LLM capabilities in zero-shot tabular prediction. The idea of incorporating the semi-supervised learning DivideMix to address/mitigate noisy labels from LLMs and improve the performance of LLMs with small models (which are trained by DivideMix) is very interesting. Experiments also demonstrate the superior performance of SERSAL and the potential for further improvement with more loops of training, and better foundation LLMs (e.g. GPT-3.5 -> GPT4) across multiple domains.\n\nSuperior zero-shot performance across multiple datasets across multiple domains. Reasonable design and setup of experiments.", "weaknesses": "The type of tasks is restricted to binary classification. The method may have potential in more complicated tasks and more types of input data (beyond just tabular data) but these are not demonstrated or discussed in this paper.\n\nThe experiments on the effectiveness of multi-loop SERSAL are quite light (compared to experiments with one-loop SERSAL). Experiments with more loops and more datasets will be useful.\n\nElaboration on DivideMix in Section 3 will help improve the readability and understanding of the methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCYP8v9CGD", "reviewer": "Reviewer_uFdg", "summary": "The authors propose SERSAL, a general loop of thought prompting method by synergy learning with small models to unconditionally enhance tabular prediction for LLMs in an unsupervised way. This paper mainly solves the medical tabular prediction tasks.", "questions": "1. Does the paper mainly to solve traditional tabular prediction tasks or medical prediction tasks? If it's former, it seems that the used tabular data does not contain regression and multi-class classification, as discussed in [1]. So how does it behave on traditional tabular data? If it's latter, how do the authors ensure the safety of processing medical data with Large Language Models (LLMs), especially when adapting to real-world healthcare scenarios where data security is crucial [2], and the data is used for fine-tuning the LLMs? In addition, while the paper seems to position medical tabular prediction tasks as the central focus, the proposed method appears to be more suited for general tabular learning rather than specialized medical prediction. This perception arises from the lack of incorporation of specific medical domain knowledge within the paper. Could the authors give more explanations?  \n[1] Revisiting Deep Learning Models for Tabular Data. NeurIPS 2021.   \n[2] Large language models encode clinical knowledge. Nature 2023.  \n\n2. I am uncertain whether it would be more appropriate to refer to the setting as unsupervised rather than zero-shot. In the context of zero-shot learning, as illustrated in TabLLM [3] and GTL [4], zero-shot implies that in target dataset, no training instances (x) that are similar to test instances are provided. In this paper, in target dataset, training instances from the same distribution without labels are provided for training. What is the definition of zero-shot used in this paper? Does 'no label' equate to 'zero-shot'?  \n[3] TabLLM: Few-shot Classification of Tabular Data with Large Language Models.   \n[4] From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models.   \n\n\n3. Is the annotation of samples by LLMs done on a sample-by-sample basis? If so, this process seems to require a high cost. Could you provide the time cost?  \n\n4. Is TabLLM in Table 2 be tested in a zero-shot scenario? If so, it seems that TabLLM is better than GPT-3.5. Could you please explain why this might be the situation, given that GPT 3.5 has a more extensive knowledge base compared to TabLLM?     \n\n5. What is the meaning of TabLLM + SERSAL (GPT-3.5) in Table 2, what is the role of each component?    \n\n6. Recently, generative models pertained on a large set of tabular data have also demonstrated zero-shot capabilities on tabular prediction tasks [4]. I suggest that these models should be considered as baseline models since they can be seen as powerful tools for zero-shot performance on new tasks.   \n[4] From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The direction of applying synergy learning with small models and LLMs to enhance tabular prediction in an unsupervised way is promising and important.  \n2. The paper is easy to follow.  \n3. The proposed method is novel.", "weaknesses": "Please refer to questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)