LLM Embeddings Improve Test-Time Adaptation to Tabular $Y|X$-Shifts

## Abstract

Abstract For tabular datasets, the change in the relationship between the label and covariates ( Y | X conditional 𝑌 𝑋 Y|X italic_Y | italic_X -shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples.
We focus on building more informative representations of tabular data that can mitigate Y | X conditional 𝑌 𝑋 Y|X italic_Y | italic_X -shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies.
The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts .

### 1 Introduction

Predictive performance degrades when the distribution of target domain shifts from that of source (training) [, , , , ].
Distribution shifts can be categorized into shifts in the marginal
distribution of covariates ($X$-shifts) or changes in the relationship between the label and covariates ($Y|X$-shifts).
In computer vision, $X$-shifts are prevalent since high-quality human labels are consistent across different images [, , ];
in contrast, $Y|X$-shifts are prevalent in tabular data due to missing variables and hidden confounders.
There is a large body of work addressing $X$-shifts
due to its dominance in vision and language [, , ], yet the work on $Y|X$-shifts remain relatively limited [].

The main challenge with addressing $Y|X$-shifts in tabular tasks
is that the source data may provide little insight on the target distribution.
Since it is impossible to generalize to a completely new and unknown domain [, ], we focus on leveraging few labeled target examples (on the order of 10 to 100) to address small $Y|X$-shifts
that negatively impact model performance.
Our goal is to build a feature representation $\phi(X)$ such that
the difference between
$\mathbb{E}_{\rm source}[Y|\phi(X)]$ and $\mathbb{E}_{\rm target}[Y|\phi(X)]$
are learnable even based on a few target data.

Using the wealth of world knowledge learned during pre-training, LLMs have the potential to build representations
that mitigate the impact of confounders whose distribution changes across source and target.
Specifically, we use a recent and advanced LLM encoder (e5-Mistral-7B-Instruct []) to featurize tabular data—which we referred to as LLM embeddings—and fit a shallow neural network (NN) on these embeddings for tabular prediction (Figure 1).
In contrast to classical numerical encoding of tabular data,
our approach automatically incorporates the semantics of each covariate using off-the-shelf LLMs, and can include additional contextual domain-level information that can help account for missing variables whose distribution shifts from source to target.

Throughout our investigation, we use the same LLM encoder to extract the LLM embeddings, and we only “finetune” the shallow NNs we use as the main prediction model across target domains. Investigating how different LLM encoders, e.g., LLMs specialized for tabular data [], affect tabular $Y|X$ shifts is left as a future work.
Comparison to other LLM-based tabular classification is discussed in related work below.

Figure: Figure 1: Overview of methods incorporating LLM embeddings.
Refer to caption: x1.png

For rigorous empirical evaluation, we consider 7,650 natural spatial shifts (source$\rightarrow$target) based on three real-world tabular datasets (ACS Income, Mobility, Pub.Cov []). Our testbed serves as a *large-scale* benchmark for $Y|X$-shifts on tabular data, offering a standardized protocol for training, validation, testing, and finetuning, as well as a consistent hyperparameter selection process.
Compared to previous benchmarks on tabular distribution shifts [], this paper not only explores a significantly greater variety of shift settings but also introduces a series of novel approaches to incorporate LLM embeddings as features.
Such a comprehensive evaluation ensures the robustness and adaptability of our findings across diverse scenarios, setting a new standard for future work in this domain.
We compare our proposed approach with: (i) typical methods on Tabular features, including basic models (LR, SVM, NN), gradient-boosting trees (GBDT; XGB, LGBM, GBM), distributionally robust methods (DRO; KL-DRO, $\chi^{2}$-DRO, Wasserstein DRO, CVaR-DRO, and Unified-DRO); and (ii) in-context learning [] using recent and advanced pre-trained LLMs, including TabPFN [] and GPT-4o mini []. See Table 2 for further details.
In total, we consider 22 algorithms and 261,000 model configurations.

Figure: (a) ACS Income
Refer to caption: x2.png

Since it is unrealistic to expect any single method to uniformly dominate over large number of source$\rightarrow$target settings, we complement traditional average-case metrics using the *fraction of times each method performs best*. For each method $\mathcal{M}$,

$$ $\text{FractionBest}(\mathcal{M};\Delta):=\frac{|\mathcal{S}(\Delta)\cap \mathcal{S}_{\mathcal{M}}|}{|\mathcal{S}(\Delta)|},$ (1.1) $$

where $\mathcal{S}(\Delta)$ contains all source$\rightarrow$target settings where the performance between the best and second best model is larger than $\Delta$ (we set $\Delta=1\%$ in this paper), and $\mathcal{S}_{\mathcal{M}}$ contains all source$\rightarrow$target settings where model $\mathcal{M}$ performs the best.
FractionBest calculates the proportion of source→target settings where (i) $\mathcal{M}$ outperforms all other methods and (ii) the improvement over the second-best model is meaningful (and significant).

First, we consider LLM embeddings without any adaptation to labeled target data(^†^††We do not conduct any target adaptation; however, we use 32 labeled target samples for validation (selection of hyperparameters, etc.).). Shallow networks based on LLM embeddings (LLM$|$NN) outperform all other methods on tabular features in 85% settings in the ACS Income dataset, and in 78% in the ACS Mobility dataset. However, for the ACS Pub.Cov dataset, the FractionBest drops to 45%, which indicates that LLM embeddings do not always offer a perfect solution (see Figure 2 (a)-(c)). We conclude *LLM embeddings sometimes improve robustness*, but *do not consistently surpass* state-of-the-art tree-ensemble methods.

However, we find that *finetuning* the prediction model (shallow NN)
*with few target samples* can *make a big difference* even when using identical LLM embeddings. When finetuning with just 32 target samples for finetuning, the FractionBest ratio (Equation 1.1) improves from 85% to 86% on ACS Income, from 78% to 86% on ACS Mobility, and from 45% to 56% on ACS Pub.Cov (see Figure 2 (d)-(f)).
We find this improvement *surprising*:
although the shallow NN has numerous parameters, finetuning with only 32 target samples surprisingly improve target performance by a relatively large margin. More importantly, such improvement is observed under $Y|X$ shifts and holds across a wide range of distributional shift settings.
Furthermore, our approach significantly outperforms in-context learning using advanced pre-trained models like TabPFN [] and GPT-4o mini [], while being much more lightweight. We only need to finetune shallow NNs, rather than engage in extensive engineering efforts required for in-context learning, such as prompt engineering, or the costly end-to-end updates involved in few-shot fine-tuning of LLMs (see further comparison in related work below).
This suggests a promising new research direction for further investigation of using LLM embeddings in tabular classification tasks.
Theoretical insights are discussed in Section 4.

Our method also implies multiple additional benefits (See Section 3.2).
*The performance gain brought by target samples is larger under stronger $Y|X$-shifts*, where the level of $Y|X$-shifts is measured by DISDE []. Finetuning with 32 target samples yields an average performance gain of 5.4 percentage points across the worst 500 settings on ACS Pub.Cov, compared to no finetuning. This is notably higher than the 1.2% average gain observed across all 2550 settings (4.5 times).

Beyond our primary findings, we also conduct ablation studies to better understand our approach in Section 3.3.
First, incorporating the *“right” domain information has an outsize impact* on tabular $Y|X$-shifts. As shown in Figure 6 (f), for ACS Pub.Cov with finetuning, adding additional domain information from Wikipedia improve F1 scores by 1.4pp on average across all 2550 settings and 1.7pp across the worst 500 settings when compare to using LLM embeddings alone. However, identifying the best domain information requires non-trivial engineering effort, which we left as a future work. Moreover,
given the large number of model parameters and limited labeled target samples, one might expect parameter-efficient methods like Low-Rank Adaptation (LoRA) [] and Prefix Tuning [] to offer a clear advantage for target adaptation. However, we find *the specific finetuning approach has small impact* on target adaptability under tabular $Y|X$-shifts.
In Figure 7, all target adaptation methods significantly outperform the non-finetuned version, both when using LLM embeddings alone and using LLM embeddings with domain information from Wikipedia.

Another practical question is how to allocate a fixed number of labeled target samples between target adaptation and validation (selection of finetuning method, hyperparameters, etc).
In Figure 8 to come,
we compare two allocation schemes of $64$ labeled target samples: (i) using all 64 samples for validation (solid bar), and (ii) dividing the samples into 32 for validation and 32 for finetuning (shaded bar). For ACS Mobility and ACS Pub.Cov, *target adaptation provides significant gains over the validation-only approach*, highlighting the need for further investigation
into sample allocation.

##### Related work

Tabular data is a common modality in electronic health records, finance, and social and natural sciences.
Unlike other modalities like images and text,
gradient-boosted trees (GBDT; GBM [, ],
XGBoost [], LGBM []) remain the
state-of-the-art [, , ]
even when compared to
neural networks specifically designed for tabular data [, , , ].
GBDTs have recently been observed to provide strong performance under distribution shifts, which forms the basis of its use as a main baseline [, ].
We train GBDTs using only tabular features and do NOT use LLM embeddings because GBDTs are known to be slow to train on high-dimensional inputs [].

Recently, there exists another line of work that uses LLMs for tabular data classification. Unlike our approach that performs *lightweight* probing using an encoder LLM, most work use a decoder LLM to pose the classification task as a question and predict the next token (e.g., “Yes” for the positive class and “No” for the negative class [, ]). These work can be categorized into three groups: (1) few-shot finetuning that uses labeled target to finetune or lightweight finetune the LLM [, , ]; (2) in-context learning [, , ] that uses few labeled target data as the prompt; (3) zero-shot learning [, ], which assumes no access to the target data. The decoder can be a general LLM (e.g., T0 as in []) or finetuned on general tabular data [].
In this paper, due to the large-scale testbed we consider, we do NOT consider expensive end-to-end updates [, ] to LLMs using source or target data, which requires full weight access and weight updates to the LLM. This approach is more computational expensive than finetuning a shallow NN, and we find it infeasible across thousands of source-target pairs we consider.
We leave few-shot finetuning as potential future work.

More importantly, existing LLM work for tabular classification does not explicitly address distributional shifts. In particular, zero-shot learning struggles to generalize across target domains with varying levels of $Y|X$ shifts, making it impractical in our setting. Moreover, while in-context learning adapts to labeled target samples, it generally cannot utilize the large amount of labeled source data due to token limits(^‡^‡‡An alternative can be finetuning the decoder on source data and then apply in-context learning with labeled target samples. However, to our best knowledge, this has NOT been explored in the literature. Also, end-to-end updates to the LLM are costly and impractical for our large-scale testbed.). Since the $Y|X$ distribution in the source domain may be relevant to the target domain, this limits the suitability of in-context learning for tabular $Y|X$ shifts.
As shown in Figure 2, we consider two baselines for this line of research: (i) TabPFN(^§^§§TabPFN [] can handle in-context learning with a relatively large number of samples, up to the level of 1000. Therefore, we use it for in-context learning on the source data, even though this is much smaller than the 20,000 source samples available.) with 1024 labeled source data [], which shows strong performance for tabular data classification with no distribution shift and outperform 18 other algorithms across 176 dataset in [], and (ii) in-context learning using the recent, advanced decoder GPT-4o mini and 32 labeled target sample.

A wide range of methods have been proposed to address distribution shifts, notably robust learning methods, balancing methods, and invariant learning methods.
Distributionally robust optimization (DRO) construct an uncertainty set around the training distribution and optimizing for the worst-case distribution within this set, thereby mitigating the impact of potential distribution shifts.
Variants of DRO methods have been developed using different distance metrics, such as $\chi^{2}$-divergence [, ], KL-divergence [], and Wasserstein distance [, , , ].
However, these approaches have recently been observed to be ineffective in addressing real-world tabular distribution shifts [].
On the other hand, invariant learning [, , ] seeks to learn causally invariant relationships across multiple *pre-defined* environments. In this work, we include 5 typical DRO methods but do not consider invariant learning methods as they require multiple training environments, which is not the focus of this work.
Some statistical works [, ] provide theoretical guarantees for simple linear models in transfer learning, but these guarantees often do not extend to more complex models like decision trees or neural networks used in real-world applications.
This version improves clarity and flow while retaining the original meaning.Additionally, many studies have explored domain adaptation [, , ].
However, most of these focus on $X$-shifts in image data, whereas our work addresses $Y|X$-shifts in tabular data.

### 2 Methods

In this section, we introduce a series of methods utilizing LLM embeddings for tabular prediction, as well as different choices to incorporate additional domain information, different model architectures, and target adaptation techniques
using a small amount of labeled target samples.
To the best of our knowledge, this work is the first to comprehensively explore the impact of LLM embeddings on tabular $Y|X$-shifts.

#### 2.1 LLM Embeddings for Tabular Prediction

We first introduce how we transform tabular data into LLM embeddings, where the
key idea is to serialize each sample into a natural language format that the
LLM can process.
There is a substantial body of research on serialization, including using another LLM to rewrite tabular data into natural language [], adding descriptions of the classification task, training and test examples [, ], etc.
Among these methods, Hegselmann et al. [] demonstrate that using
a straightforward text template with a task description consistently
achieves the best empirical performance.

Using an income prediction problem to illustrate, consider a simple task description such as “Classify whether US working adults’ yearly income is above $50000 in 2018.” along with a simple serialization template that enumerates all features in the format
“The [feature name] is [value]”.
Adopting this serialization approach, we employ the encoder model e5-Mistral-7B-Instruct [] to generate the LLM embedding. Formally, the encoder takes the serialization Serialize($X$) of sample $X$ as input and outputs its corresponding embedding $\Phi(X)$ as

$$ $X\xrightarrow{\text{serialization}}\text{Serialize}(X)\xrightarrow{\texttt{e5- Mistral-7B-Instruct}}\Phi(X).$ $$

Since e5-mistral-7b-instruct requires input data to be formatted in the following template:

$$ Instruct: description of the classification task \n Query: $\displaystyle\texttt{description of the data},$ $$

we provide task description in the “Instruct” part, and use the serilization template to format the tabular data in the “Query” part.
An illustrative example is provided in Part A-D of Figure 1,
with additional details available in
Appendix A.1.
Analyzing the impact of different LLM encoders, task descriptions, and serialization methods is left for future work.

#### 2.2 Additional Domain Information

Another advantage of using LLM embeddings is their ability to incorporate additional domain information or prior knowledge, denoted by $C$.
As demonstrated in Section 1, incorporating domain-specific information can help address $Y|X$-shifts and improve generalization performance in the target domain.

In this work, we propose a simple yet effective approach for integrating domain knowledge into tabular predictions.
Rather than combining the domain information with serialized tabular data and generating a single LLM embedding, we generate separate LLM embeddings for the domain knowledge and the serialized tabular data, and then concatenate them together.
The benefits of this approach are twofold: (a) although the domain information may contain significantly more words than the serialized tabular features, our concatenation method ensures a balanced 1:1 ratio between the two, preventing a single embedding that disproportionately focuses on the longer domain information; (b) by separating the tabular features from the domain information, we can efficiently update the domain information without having to regenerate all the embeddings for the entire dataset.

We explore three sources of domain information: Wikipedia, GPT-4 [], and labeled target samples.
Given that our experiments (see Table 1 and Section 3) focus primarily on socioeconomic factors, we collected “Economy” data for each U.S. state from Wikipedia as $C$.
For GPT-4, we prompt it to provide background knowledge relevant to each prediction task in each state as $C$.
For labeled target samples, we serialize 32 labeled samples from the concerned domain as the prior knowledge $C$.
Further details can be found at Appendix A.2.
After obtaining domain information $C$, we use e5-mistral-7b-instruct to generate an LLM embedding for $C$.
As illustrated in Parts E.2 and F.1 of Figure 1, this embedding is then concatenated with the LLM embeddings of the tabular data, which serve as input to the backend neural network models (NN).
This approach allows us to generate the LLM embedding for the dataset *just once*, and subsequently concatenate it with embeddings from different prompts as needed.
In Section 3, we study whether and how this
additional domain information can enhance generalization under $Y|X$-shifts.

In addition, recent works on prompt engineering have focused on incorporating additional domain information to enhance prediction tasks, often through detailed instructions [, ].
Our proposed framework introduces a novel approach to leveraging such information and remains fully compatible with these existing methods.

#### 2.3 Model Training and Target Adaptation

##### Model architecture

For the backend model, we use a vanilla neural network (NN) classifier on both tabular features and LLM embeddings for tabular data classification.
The NN is a simple feedforward neural network with several hidden layers, dropout layer, and ReLU activation functions.

When adding additional domain information via an embedding layer, the same embedding is applied to all samples from the same domain.
Since the output of e5-mistral-7b-instruct is a 4096-dimensional vector,
we simply concatenate the LLM embeddings with the embeddings of the domain information. This concatenated vector is then
passed through the hidden layers, dropout layer, and ReLU activation functions. For all NNs, the final linear layer an output dimension of 2, followed by a softmax layer for binary classification.
During training, we use cross-entropy as the loss function, batch size as 128,
and use the Adam optimizer.
Detailed model architecture and hyper-parameters are provided in the
Appendix A.3 and A.4,
with a discussion on hyperparameter selection provided in
Section 3.1.

##### Target Adaptation

Even with the incorporation of LLM embeddings and domain information, our model may still experience $Y|X$-shifts.
In practice, it is common to have a small set of samples from the target domain, which can be leveraged to better adapt the model to the target domain.

For each (source domain, target domain) pair, we begin by selecting the best
training hyperparameter based on a validation criterion, which will be
discussed in the Section 3.1.
Using this model trained on the source domain, we explore four primary
methods for target adaptation: in-context domain info, full-parameter fine-tuning,
low-rank adaptation (LoRA), and prefix tuning for domain information.

For in-context domain info (F.1 of Figure 1),
we keep the trained model frozen and only update the
domain information, switching it from natural language description of labeled sample from the source domain during training to that of target domain during inference phase.
For the other three methods, we conduct further training of the model. In full-parameter fine-tuning (F.2 of Figure 1), the entire neural network is fine-tuned using the target samples. For LoRA,
we introduce a low-rank adaptation layer to each linear layer by incorporating two smaller matrices, $A$ and $B$, both with a rank of 16. Specifically, matrix $A$ has dimensions corresponding to the input size and the rank, while matrix $B$ has dimensions corresponding to the rank and the output size. Matrix $A$ is initialized with a mean of 0 and a standard deviation of 0.02, whereas matrix $B$ is initialized with zeros. These matrices are then multiplied together and added to the original weight matrix.
We then fine-tune only these LoRA parameters, while keeping the rest of the
model unchanged.
In prefix tuning (F.3 of Figure 1), the initial domain information embedding serves as a starting point for further refinement. During training, both the the NN and the domain information embedding of the source domain are trained. For target adaptation,
we switch the domain information embedding from the source to the
target domain.
The NN is kept frozen, and only the domain information embedding of the target domain is updated using these samples from the target domain.
We refer to this process as prefix tuning.

As shown in Table 1, we use different hyperparameters for
target adaptation. Detailed hyperparameters are provided in
Appendix A.4, and the
hyperparameter selection process is discussed in Section 3.1.

### 3 Numerical Experiments

**Table 1: Details of datasets used in this work. “# Source$\rightarrow$Target Pair” denotes the number of distribution shift pair for each dataset, and we consider the natural *spatial* shift between US states.**
| #ID | Dataset | #Samples | #Features | Outcome | #Source Domains | #Target Domains | #Source$\rightarrow$Target Pair |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | ACS Income | 1.60M | 9 | Income$\geq$50k | 51 (US States) | 50 (US States) | 2550 |
| 2 | ACS Mobility | 621K | 21 | Residential Address | 51 (US States) | 50 (US States) | 2550 |
| 3 | ACS Pub.Cov | 1.12M | 18 | Public Ins. Coverage | 51 (US States) | 50 (US States) | 2550 |

Figure: Figure 3: Shift pattern analysis. For the 2550 source$\rightarrow$target distribution shift pairs in ACS Income dataset, we attribute the performance drop for each source$\rightarrow$target pair into $Y|X$-shifts (red curve) and $X$-shifts (blue curve), and sort all pairs according to the drop introduced by $Y|X$-shifts. We draw the *worst-500* settings in each dataset, and the decomposition method used here is DISDE [] with XGBoost as the reference model. Results on other datasets are in Figure 9.
Refer to caption: x8.png

In this section, we conduct a thorough investigation of 7650 natural shift settings (source $\rightarrow$ target domain) in 3 tabular datasets over 261,000 model configurations and summarize the observations.
Our findings highlight the potential of incorporating LLM embeddings to enhance the generalization ability in tabular data prediction tasks.

#### 3.1 Testbed Setup

##### Dataset

In this work, we use the ACS dataset [] derived from the US-wide ACS PUMS data, where the goal is to predict various socioeconomic factors for individuals.

- •
ACS Income: The goal is to predict whether an individual’s income is above $50K based on individual features. We filter the dataset to only include individuals above 16 years old with usual working hours of at least 1 hour per week in the past year, and an income of at least $100.
- •
ACS Mobility: The goal is to predict whether an individual has the same residential address as one year ago.
We filter the dataset to only include individuals between the ages of 18 and 35, which increases the difficulty of the prediction task.
- •
ACS Public Coverage (abbr. as ACS Pub.Cov):
The goal is to predict whether an individual has public health insurance. We focus on low-income individuals who are not eligible for medicare by filtering the dataset to only include individuals under the age of 65 and with an income of less than $30,000.

The details of datasets are summarized in Table 1.

##### Shift Pattern Analysis

Before benchmarking, we first analyze the shift patterns among the 2550 source$\rightarrow$target pairs in each dataset.
Specifically, we utilize DISDE [] (reference model as XGBoost) to decompose the performance degradation from the source domain to the target into two parts: (a) $Y|X$ (concept)-shifts and (b) $X$ (covariate)-shifts.
By utilizing tailored shift patterns, we can conduct an in-depth analysis of where the strength of LLM embeddings lies.
As shown in Figure 3, we sort all pairs according to the strength of $Y|X$-shifts, where we find that the natural spatial shifts are mainly comprised of $Y|X$-shifts.
These findings broaden the scope of the analysis in WhyShift [] by examining 7,650 shift pairs, a significant increase from the 169 pairs studied in the original work.

##### Algorithms

As introduced in Section 2, we compare various methods that incorporate LLM embeddings into tabular data prediction, including different finetuning methods (no finetuning, finetuing on full parameters, and low rank adaptation (LoRA)) and different embeddings (w/ or w/o extra information).
Besides, in order to fully compare the performances, we also include a wide range of learning strategies that perform on Tabular features, including basic models (LR, SVM, NN), tree ensembles (XGB, LGBM, GBM), robust methods (KL-DRO, $\chi^{2}$-DRO, Wasserstein DRO, CVaR-DRO, and Unified-DRO), and in-context learning methods (TabPFN, GPT-4o-mini).
All methods are summarized in Table 2.

##### Experiment Setup

We conduct experiments with more than 261,000 model configurations on 2550 source$\rightarrow$target shift pairs in ACS Income, ACS Mobility, and ACS Pub.Cov datasets respectively (7650 settings in total).
For each source$\rightarrow$target shift pairs, we randomly sample $20,000$ labeled data from the source and target domain respectively, as the training and test dataset.
We evaluate the model trained on the source domain, with or without target adaptation, and report the *Macro F1 score* on the testing dataset.

Given the numerous training hyperparameters—learning rate, number of training epochs, hidden layer dimension, dropout ratio—we use a validation set of 32 randomly sampled labeled target domain samples to choose the optimal training hyperparameters, based on the highest F1 score in the validation dataset.
Since our metric is Macro F1 score, the validation set is set as balanced between positive and negative classes.
Note that the hyperparameter selection is *near*-oracle, as it leverages target samples, albeit in a limited quantity.
When doing finetuning, we sample another 32 labeled target samples to finetune the model.
And we use the same 32-sample validation dataset (for training hyperparameter selection) to select the target adaptation hyperparameters that yield the best Macro F1 score.
Note that our testbed allows flexible sample sizes for training, validation, testing, and finetuning.
Additionally, we perform an ablation study on different allocations of the overall target samples in validation and finetuning (see Figure 8).
See details on hyperparameters in the Section A.4.

#### 3.2 Primary Findings

We begin by presenting the key observations from our results.
In addition to the metric (1.1) introduced in Section 1, we report performance metrics averaged over the source-target pairs.

##### LLM embeddings improve performance, but when applied alone do not consistently outperform tree-ensembles.

To better assess the generalization ability when incorporating LLM embeddings, we select the worst 500 settings (out of 2,550 total settings per dataset) based on the severity of $Y|X$-shifts and report the average Macro F1 Score in Figure 4. Each bar represents the average result across these worst 500 settings, characterized by the most severe $Y|X$-shifts.
Thus, even a 1pp improvement is significant, as it implies consistent gains of about 1pp across each of the worst 500 settings.

Comparing “NN on LLM embeddings” to “NN on tabular features” (with the backbone model fixed as NN), we observe LLM embeddings significantly enhance generalization under distribution shifts on the ACS Income and ACS Mobility datasets, with average improvements of 2.4pp and 9.9pp. Notably, “NN on LLM embeddings” even outperforms XGBoost under strong distribution shifts on these datasets. This demonstrates the potential of LLM embeddings in tabular data prediction, where they can contribute to more generalizable models.

A different trend is observed on the ACS Pub.Cov dataset, where the inclusion of LLM embeddings results in a performance drop for NN models. This suggests that simply incorporating LLM embeddings does not always resolve distribution shift issues; their effectiveness may vary across datasets, particularly depending on whether the LLM embeddings provide additional relevant information for the specific prediction task.

Figure: (a) ACS Income
Refer to caption: x9.png

Figure: (a) ACS Income
Refer to caption: x12.png

Figure: (a) ACS Income (no extra)
Refer to caption: x15.png

##### A small number of target samples can make a big difference.

While incorporating LLM embeddings doesn’t always yield improvements, we find that even a small number of target samples can have a significant impact.
As shown in Figure 4, finetuning the “NN on LLM embeddings” model with just 32 target samples significantly improves the average performance across the worst 500 settings for both ACS Mobility and ACS Pub.Cov. Similar trends are observed for other worst-$K$ settings, as shown in Figure 6 (a)(b)(c) to come.
Notably, for the ACS Pub.Cov dataset, where LLM embeddings alone provided no improvement, finetuning with only 32 target samples leads to a 5.4pp gain, even surpassing XGBoost by 2.2pp.
This highlights the adaptability of LLM embeddings, making them a promising tool for harnessing their power across various downstream real-world tasks.

Furthermore, in Figure 5, we illustrate how the performance of finetuning methods varies with different numbers of target samples used for finetuning.
Our conclusions hold true regardless of the number of target samples used for finetuning.

Figure: Figure 7: Comparison between full-parameter finetuning, LoRA, and prefix tuning. We show the average Macro F1 Score over the worst-500 settings.
Refer to caption: x27.png

Figure: Figure 8: Comparison between different allocation of target samples into validation and finetuning. We report the average Macro F1 Score over the worst-500 settings.
Refer to caption: x28.png

##### The performance gain brought by finetuning with target samples is larger under stronger Y | X conditional 𝑌 𝑋 Y|X italic_Y | italic_X -shifts.

As shown in Figure 6 (a)-(c) (LLM $|$ NN v.s. LLM $|$ NN (finetuning)), for NN using LLM embeddings, finetuning with 32 target samples yields an average performance gain of 5.8pp across the worst-100 settings on ACS Pub.Cov, compared to no finetuning. This is notably higher than the 1.2pp average gain observed across all 2550 settings (4.8 times).
Similarly, for ACS Income, it is about 9 times.

#### 3.3 Auxiliary Findings

In addition to the primary findings, we have several other noteworthy observations.

##### “Right” domain information matters.

In Figure 6 (d)-(l), we study how additional domain information — from Wikipedia, GPT-4, or 32 target samples (via in-context domain information, F.1 of Figure 1)
impacts F1 scores in both non-adaptation and target adaptation scenario. We do NOT observe a consistent trend across all three datasets: additional domain information can either improve or reduce F1 scores. In the non-adaptation case, both LLM&wiki $|$ NN and LLM&GPT $|$ NN outperform LLM $|$ NN in ACS Income (d)(g), but underperform in ACS Mobility (e)(h). When finetuning NNs, LLM&wiki $|$ NN (finetuning) and LLM & GPT $|$ NN (finetuning) beat LLM $|$ NN (finetuning) in ACS Pub.Cov (f)(i), but underperform in ACSMobility. Moreover, LLM& Target Samples $|$ NN shows comparable performance with LLM $|$ NN (finetuning) in ACSIncome (j), but underperform LLM $|$ NN (finetuning) in ACS Mobility and ACS Pub.Cov (k)(l), and even underperform LLM $|$ NN that has no target adaptation in ACS Mobility (k).
Our results indicates that “right” domain information can indeed improve tabular data classification under $Y|X$ shifts, yet identifying the best domain information requires non-trivial engineering efforts, which we leave for future work.

For simplicity, we only compare LLM embeddings (without extra domain information) to LLM&Wiki in the sequel, as LLM&Wiki and LLM&GPT exhibit similar trends.

##### Specific finetuning approaches are less crucial than expected.

Given the large number of model parameters and limited labeled target samples, one might expect parameter-efficient methods like Low-Rank Adaptation (LoRA) and Prefix Tuning (F.3 of Figure 1) to offer a clear advantage. However, as shown in 7,
all methods significantly outperform the non-finetuned version when using LLM embeddings,
and the choice of finetuning method appears less significant in our setting.

An exception is prefix tuning on the ACS Pub.Cov task, where performance was several percentage points lower. While this requires further investigation, our key takeaway is that under $Y|X$ shifts, 1) finetuning models using LLM embeddings can greatly enhance classification performance; 2) popular finetuning methods yield comparable results.

##### Allocation of labeled target samples matters.

In Figure 8, we compare two allocation schemes of $64$ labeled target samples. For the solid bars, all 64 samples are utilized as the validation dataset for hyperparameter selection. For the shaded bars, we allocate 32 samples for validation and the remaining 32 samples for finetuning.
Based on this, we initially explore and understand the impact of sample allocation.
For ACS Mobility and ACS Pub.Cov, target adaptation using LLM embeddings (the shaded bar) significantly outperforms the validation approach (the solid bar). However, for ACS Income, the improvement from target adaptation is only marginal.
This shows that although target adaptation is effective, its improvement is highly dependent on the specific distribution and $Y|X$ shift level. This raises further questions about how to optimally allocate resources. Although our findings endorse considering target adaptation, identifying the best allocation strategy is left as future work.

### 4 Discussion based on theoretical insights

We take a brief examination of the theoretical insights that may lie behind our empirical findings. While standard generalization bounds are vacuous for neural networks, we nevertheless find that theoretical results from domain adaptation provide a useful starting point for understanding why finetuning LLM embeddings with small target samples can result in superior generalization performance under substantial $Y|X$ shifts, particularly in comparison to tabular features.

Let $\phi(X)$ denote a feature map and $Y$ a binary label. We let $P$ and $Q$ be the source and target distributions. We consider a model class $\mathcal{H}$ of VC dimension $d$. For any model $h\in\mathcal{H}$, we use $\epsilon_{P}(h):=\mathbb{E}_{P}[\mathbb{I}(h(\phi(X))\neq Y)]$ to denote the expected 0-1 loss on the source domain and $\hat{\epsilon}^{m}_{P}(h)$ its empirical counterpart based on $m$ i.i.d. samples from $P$. We define $\epsilon_{Q}(h)$ and $\hat{\epsilon}^{m}_{Q}(h)$ on the target.

Although in practice we finetune on the target, we use a mixture problem as a rough approximation. Suppose that we have $(1-\beta)m$ i.i.d. samples from source domain $P$ and $(\beta m)$ i.i.d. samples from target domain $Q$. Let $\hat{h}_{\alpha,\beta}$ be the minimizer of the $\alpha$-weighted empirical error

$$ $\hat{h}_{\alpha,\beta}:=\mathop{\rm argmin}_{h\in\mathcal{H}}\left\{\alpha\hat {\epsilon}^{\beta m}_{Q}(h)+(1-\alpha)\hat{\epsilon}^{(1-\beta)m}_{P}(h)\right\}.$ $$

The following classical result from Ben-David et al. [, Theorem 3] bounds
the generalization error on the target.

###### Proposition 1 .

For any $\delta\in(0,1)$,
with probability at least $1-\delta$,

$$ $\displaystyle\epsilon_{Q}(\hat{h}_{\alpha,\beta})-\inf_{h\in\mathcal{H}} \epsilon_{Q}(h)\leq 4$ $\displaystyle\sqrt{\frac{\alpha^{2}}{\beta}+\frac{(1-\alpha)^{2}}{1-\beta}} \sqrt{\frac{2d\log(2(m+1))+2\log\frac{8}{\delta}}{m}}$ (4.1) $\displaystyle+2(1-\alpha)\underbrace{d_{\mathcal{H}\Delta\mathcal{H}}(P_{X},Q_ {X})}_{X\text{-shifts}}+2(1-\alpha)\underbrace{\inf_{h\in\mathcal{H}}\left\{ \epsilon_{P}(h)+\epsilon_{Q}(h)\right\}}_{Y|X\text{-shifts}},$ $$

where $d_{\mathcal{H}\Delta\mathcal{H}}(\cdot,\cdot)$ denotes the $\mathcal{H}\Delta\mathcal{H}$-distance between two (marginal) distributions.

Since we use limited (32) labeled target samples for finetuning, $\beta$ is small.
Also, we use a smaller learning rate for finetuning than for training on the source, implying that $\alpha$ is even smaller than $\beta$.
Thus, we expect the constants $\sqrt{\frac{\alpha^{2}}{\beta}+\frac{(1-\alpha)^{2}}{1-\beta}}$ and $(1-\alpha)$ to both be close to $1$.
Although the VC-dimension $d$ can be vacuously large for neural networks (a well-known defect in statistical learning theory), having a large enough sample size $m$ can generally make the first term comparable to the next two terms.
For the raw features $\phi_{\rm tabular}(X)=X$, we have observed empirically that
$Y|X$-shifts are salient, implying that the third term (related to $Y|X$-shifts) dominates the second term (related to $X$-shifts), and it can also dominate the first term when $Y|X$ shifts are particular significant.
In comparison,
we conjecture that LLM embeddings $\phi_{\rm LLM}$ can reduce
the gap between $\mathbb{E}_{P}[\mathbb{I}(Y\neq h(z)\mid\phi_{\rm LLM}(X)=z]$
and $\mathbb{E}_{Q}[\mathbb{I}(Y\neq h(z))\mid\phi_{\rm LLM}(X)=z]$
by *incorporating prior knowledge* encoded during LLM pre-training. Since $\epsilon_{P}(h)=\int\mathbb{E}_{P}[\mathbb{I}(Y\neq h(z))\mid\phi(X)=z]~{}
\mathrm{d}P_{\phi(X)}(z)$ (and similarly for $Q$), we thus expect the third term to be
much smaller than using raw features $\phi_{\rm tabular}(X)$.
This suggests that when using LLM embeddings $\phi_{\rm LLM}$, the generalization bound can be smaller than that of raw features $\phi_{\rm tabular}(X)$, especially when $Y|X$ shift is more significant under raw features $\phi_{\rm tabular}(X)=X$; recall Figure 6.

We hope our empirical findings spur future theoretical investigations into the foundations of LLM-based target adaptation.

###### Proposition 1 .

## Appendices

### Appendix A Model Training Details

In this section, we outline the serialization scheme, the generation of additional domain information, our model architecture, and the hyperparameters used for training and target adaptation.

#### A.1 Serialization Scheme

As discussed in Section 2.1, serializing a row of tabular data, such as $X$, requires two components: a task description and a description of the data.

##### Task description

For the ACS Income, ACS Mobility, and ACS Public Coverage datasets, we consider the same binary classification task as
described in Ding et al. [].
We adopt concise task descriptions as suggested by [], as follows:

- •
ACS Income: Classify whether US working adults’ yearly income is above $50000 in 2018.
- •
ACS Mobility: Classify whether a young adult moved addresses in the last year.
- •
ACS Public Coverage : Classify whether a low-income individual, not eligible for Medicare, has coverage from public health insurance.

##### Description of the data

For all three ACS datasets, we utilize features as shown in [, Appendix B], along with the domain name (state) to characterize the data.
Specifically, for data from the source state, we apply the source domain name, and for data from the target state, we use the target domain name.

As illustrated in Figure 1 and recommended by [], we adopt a straightforward text template: ”The feature name is value.” However, for certain features with less common or more complex feature names, we opt for a template as “The person (appropriate verb) value” to more clearly convey the data description. For example, for the feature “Gave birth to child within the past 12 months” with the value “No”, the serialization would be “The person did not give birth to a child within the past 12 months.” The features that use this alternative template are as follows:

- •
ACS Income: class of worker;
- •
ACS Mobility: class of worker, disability, employment status of parents, citizenship, military service, hearing difficulty, vision difficulty, cognitive difficulty, grandparent living with grandchildren, employment status;
- •
ACS Public Coverage: disability, employment status of parents, citizenship, mobility, military service, hearing difficulty, vision difficulty, cognitive difficulty, gave birth to child within the past 12 months, employment status.

For features without associated values, we omit them during serialization.

#### A.2 Additional domain information

As shown in Section 2.2, we study three sources of domain information: Wikipedia, GPT-4, and labeled target samples. As we use the
e5-mistral-7b-instruct to generate an LLM embedding for the domain information, we need to specify both the “Instruct” and “Query” components.

For the “Instruct” part, we apply the same task description as outlined in Section A.1. For the “Query” part, we utilize various descriptions of the additional domain information:

- •
Wikipedia: For all three datasets, we use the “Economy” section of each state’s Wikipedia page as the additional domain information.
- •
GPT-4: For each state, we pose the following question to GPT-4, and use its response as the additional domain:
–
ACS Income: “We aim to develop a classifier to determine whether U.S. individuals earned over $50000 in 2018, using features such as age, sex, educational attainment, race, class of worker, marital status, occupation, and hours worked per week over the past 12 months. Given the unique economic and demographic profile of state_name, how might these factors influence income levels differently compared to other U.S. states? Please provide a 2000-word summary detailing these differences.”
–
ACS Mobility: “We aim to develop a classifier to determine whether a young adult moved addresses in the last year, using features such as age, sex, educational attainment, race, class of worker, marital status, occupation, total income, and hours worked per week over the past 12 months. Given the unique economic and demographic profile of state_name, how might these factors influence mobility levels differently compared to other U.S. states? Please provide a 2000-word summary detailing these differences.”
–
ACS Public Coverage: “We aim to develop a classifier to determine whether a low-income individual, not eligible for Medicare, has coverage from public health insurance, using features such as age, sex, educational attainment, race, disability, marital status, occupation, citizenship status, mobility status, military service, nativity, total income, and employment status. Given the unique economic and demographic profile of state_name, how might these factors influence public coverage levels differently compared to other U.S. states? Please provide a 2000-word summary detailing these differences.”
- •
labeled target samples: We use the following template for 32 labeled target samples:
Here are some examples of the data: \n
description of one target sample \n
Answer: (Yes or No). \n \n
description of another target sample \n
Answer: (Yes or No). \n \n
…
We use the same data descriptions as in Section A.1, with the exception that the state/domain name is omitted, as it is represented by the labeled target samples provided.

#### A.3 Model Architecture

We detail the baselines used in our paper.

##### Fully-connected Neural Networks (NN)

Given the varying input dimensions for Tabular features, LLM embeddings, and LLM embeddings with additional domain information, we employ three neural networks with similar architectures. We then train these networks and conduct target adaptation using Empirical Risk Minimization (ERM).

For all three datasets using Tabular features, we use a hidden layer with output dimension being
hidden layer dim as a hyperparameter, a dropuput layer with dropout ratio being a hyperparameter, ReLu activation. We then have another hidden layer with input and output dimension both being hidden layer dim, and then softmax layer with dimension 2 as the output.

For datasets using Tabular features, the network includes a hidden layer where the output dimension is set by the hyperparameter hidden layer dim. It is followed by a dropout layer (dropout ratio as a hyperparameter), ReLU activation, another hidden layer the input and output dimensions are both equal to
hidden layer dim. The network concludes with a softmax layer with an output dimension of 2.

For datasets using only LLM embeddings, the input dimension is 4096, which is the output dimension of e5-mistral-7b-instruct. The network consists of three hidden layers with fixed dimensions, where the input and output dimensions are (4096, 1024), (1024, 256), and (256, 128), respectively. Each layer uses ReLU activation. Next, there’s a hidden layer with an input dimension of 128 and an output dimension set by the
hidden layer dim hyperparameter. This is followed by a dropout layer (with
dropout ratio as a hyperparameter), ReLU activation, and another hidden layer where both the input and output dimensions are
hidden layer dim. A ReLU activation follows this final hidden layer, and the network concludes with a softmax layer that outputs a dimension of 2.

For datasets using LLM embeddings concatenated with additional domain information, the input dimension is 8192. We slightly update the first three hidden layers, with input and out dimensions being (8192, 2048), (2048, 512), and (512, 128), respectively. All other neural network structure and hyperparameters are the same as NNs with LLM embeddings only.

When applying low-rank adaptation (LoRA) for target adaptation, we introduce a low-rank adaptation layer to each linear layer by incorporating two smaller matrices, $A$ and $B$, both with a rank of 16. Specifically, matrix $A$ has dimensions corresponding to the input size and the rank, while matrix $B$ has dimensions corresponding to the rank and the output size. Matrix $A$ is initialized with a mean of 0 and a standard deviation of 0.02, whereas matrix $B$ is initialized with zeros. These matrices are then multiplied together and added to the original weight matrix.

##### Tree Ensemble Models

Gardner et al. [] show that several tree-based methods are competitive on tabular datasets.
And gradient-boosted trees (e.g., XGB [], LGBM [], GBM []) are widely considered as the state-of-the-art methods on tabular data.
Therefore, we compare XGB, LGBM, and GBM in this work.
For GBM, we use the standard implementations in scikit-learn [].
For XGB and LGBM, we use the standard implementations in the xgboost package(^¶^¶¶[https://pypi.org/project/xgboost/](https://pypi.org/project/xgboost/)) and the lightgbm package(^∥^∥∥[https://pypi.org/project/lightgbm/](https://pypi.org/project/lightgbm/)).
All these methods are trained on CPUs.

##### DRO Methods

Distributionally robust optimization (DRO) methods optimize the worst-case loss over an ambiguity set $\mathcal{P}$:

$$ $\min_{f\in\mathcal{F}}\sup_{Q\in\mathcal{P}}\mathbb{E}_{Q}[\ell(f(X),Y)].$ (A.1) $$

The ambiguity set is typically chosen as a “ball” around the training distribution $P_{\text{tr}}$

$$ $\mathcal{P}(d,\epsilon)=\{Q:d(Q,P_{\text{tr}})\leq\epsilon\},$ (A.2) $$

where $d(\cdot,\cdot)$ is a distance metric between probability measures and $\epsilon$ is the radius of set.
When $d$ is set as the generalized $f$-divergence (including CVaR) as:

$$ $d(P,Q)=E_{Q}\bigg{[}f\big{(}\frac{dP}{dQ}\big{)}\bigg{]},$ (A.3) $$

for the KL-DRO method [], we use $f(x)=x\log x-(x-1)$;
for the $\chi^{2}$-DRO method [], we use $f(x)=(x-1)^{2}$;
for the CVaR-DRO problem [],
we use $f(x)=0$ if $x\in[\frac{1}{\alpha},\alpha]$ and $\infty$ otherwise, and $\alpha$ controls the worst-case ratio.

For Wasserstein DRO [], we choose $d(\cdot,\cdot)$ as the Wasserstein distance.
Unified-DRO [] combines Wasserstein distance and KL-divergence as $d(\cdot,\cdot)$, and we follow their initial Github codebases and hyperparameter selection when implementing these methods.

**Table 2: Summary of methodologies: As baselines, we use basic models (LR, SVM, and NN), GBDTs (XGB, LGBM, and GBM), robust learning methods (KL-DRO, $\chi^{2}$-DRO, etc.), and recent advanced LLM in-context learning models (TabPFN and GPT-4-mini). For methods utilizing LLM embeddings, we consider different ways to incorporate domain information and different model architectures and finetuning techniques.**
| Name | Feature | Domain Info | Model | Adaptation | # of HPs | Part of Fig. 1 |
| --- | --- | --- | --- | --- | --- | --- |
| LR | Tabular | - | LR | No Finetuning | 34 | - |
| SVM | Tabular | - | SVM | No Finetuning | 34 | - |
| GBDT | Tabular | - | XGB, LGBM, GBM | No Finetuning | 200 | - |
| KL-DRO | Tabular | - | SVM | No Finetuning | 138 | - |
| $\chi^{2}$-DRO | Tabular | - | SVM | No Finetuning | 138 | - |
| Wasserstein-DRO | Tabular | - | SVM | No Finetuning | 17 | - |
| Unified-DRO | Tabular | - | SVM | No Finetuning | 150 | - |
| CVaR-DRO | Tabular | - | NN | No Finetuning | 200 | - |
| Tabular $|$ NN | Tabular | - | NN | No Finetuning | 96 | - |
| Finetuning | 12 | - |  |  |  |  |
| LoRA | 15 | - |  |  |  |  |
| LLM In-Context<br>Learning | Tabular | - | TabPFN | No Finetuning | - | - |
|  | State Name | GPT-4-mini | No Finetuning | - | - |  |
| LLM $|$ NN | LLM<br>Embeddings | - | NN | No Finetuning | 96 | E.1 |
| Finetuning | 12 | F.2 |  |  |  |  |
| LoRA | 15 | F.2 |  |  |  |  |
| LLM & Wiki/<br>GPT-4 $|$ NN | LLM<br>Embeddings | Wikipedia or GPT-4 | NN | No Finetuning | 48 | E.2 |
| Finetuning | 12 | F.2 |  |  |  |  |
| LoRA | 15 | F.2 |  |  |  |  |
| Prefix Tuning | 18 | F.3 |  |  |  |  |
| LLM & In-Context<br>Domain Info $|$ NN | LLM<br>Embeddings | Labeled<br>Target Samples | NN | No Finetuning | 96 | F.1 |

#### A.4 Hyperparameters for Training and Target Adaptation

For each algorithm, we maintain a grid of candidate hyperparameters as shown in
Tables 3 and 4.
and perform hyperparameter selection as described in Section 3.1.
When the number of hyperparameter configurations exceeds 200, we randomly select 200 configurations to reduce computational cost and maintain fairness in the comparison across all algorithms.

**Table 3: Training hyperparameter grids used in all experiments. $\diamond:$ for methods with the total grid size above 200, we randomly sample 200 configurations for fair comparisons.**
| Model | Feature | Domain Info | # of HPs | Hyperparameter | Value Range |
| --- | --- | --- | --- | --- | --- |
| SVM | Tabular | - | 96 | C | $\{1e^{-2},1e^{-1},1,1e^{1},1e^{2},1e^{3}\}$ |
| Kernel | $\{\text{linear},\text{RBF}\}$ |  |  |  |  |
| Loss | Squared Hinge |  |  |  |  |
| $\gamma$ | $\{0.1,0.3,0.5,1,1.5,2,\text{scale},\text{auto}\}$ |  |  |  |  |
| LR | Tabular | - | 23 | $L_{2}$ penalty | $\{1e^{-3},3e^{-3},5e^{-3},7e^{-3},1e^{-2},$ |
| $3e^{-2},5e^{-2},\dots,1.3,1.7,5\}$ |  |  |  |  |  |
| $1e^{1},5e^{1},1e^{2},5e^{2},1e^{3},5e^{3},1e^{4}\}$ |  |  |  |  |  |
| Tabular $|$ NN | Tabular | - | 96 | Learning Rate | $\{0.001,0.003,0.005,0.01\}$ |
| Hidden Layer Dim | $\{16,32,64,128\}$ |  |  |  |  |
| Dropout Ratio | $\{0,0.1\}$ |  |  |  |  |
| Train Epoch | $\{50,100,200\}$ |  |  |  |  |
| GBM | Tabular | - | 1680^⋄ | Learning Rate | $\{1e^{-2},1e^{-1},5e^{-1},1\}$ |
| Num. Estimators | $\{32,64,128,256\}$ |  |  |  |  |
| Max Depth | $\{2,4,8,16\}$ |  |  |  |  |
| Min. Child Samples | $\{1,2,4,8\}$ |  |  |  |  |
| LGBM | Tabular | - | 1680^⋄ | Learning Rate | $\{1e^{-2},1e^{-1},5e^{-1},1\}$ |
| Num. Estimators | $\{64,128,256,512\}$ |  |  |  |  |
| $L_{2}$-reg. | $\{0,1e^{-3},1e^{-2},1e^{-1},1\}$ |  |  |  |  |
| Min. Child Samples | $\{1,2,4,8,16,32,64\}$ |  |  |  |  |
| Column Subsample Ratio (tree) | $\{0.5,0.8,1.\}$ |  |  |  |  |
| XGB | Tabular | - | 1944^⋄ | Learning Rate | $\{0.1,0.3,1.0,2.0\}$ |
| Min. Split Loss | $\{0,0.1,0.5\}$ |  |  |  |  |
| Max. Depth | $\{4,6,8\}$ |  |  |  |  |
| Column Subsample Ratio (tree) | $\{0.7,0.9,1\}$ |  |  |  |  |
| Column Subsample Ratio (level) | $\{0.7,0.9,1\}$ |  |  |  |  |
| Max. Bins | $\{128,256,512\}$ |  |  |  |  |
| Growth Policy | $\{\textrm{Depthwise},\textrm{Loss Guide}\}$ |  |  |  |  |
| KL-DRO | Tabular | - | 117 | Uncertainty Set Size $\epsilon$ | $\{1e^{-4},\dots,0.01,\dots,0.99\}$ |
| $\chi^{2}$-DRO | Tabular | - | 117 | Uncertainty Set Size $\epsilon$ | $\{1e^{-4},\dots,0.01,\dots,0.99\}$ |
| Wasserstein-DRO | Tabular | - | 138 | Uncertainty Set Size $\epsilon$ | $\{1e^{-4},\dots,0.01,\dots,0.99,\dots,3\}$ |
| CVaR-DRO | Tabular | - | 1620^⋄ | Worst-case Ratio $\alpha$ | $\{0.01,0.1,0.2,0.3,0.5,1.0\}$ |
| Underlying Model Class | NN |  |  |  |  |
| Unified-DRO | Tabular | - | 180 | Distance Type | $L_{\text{inf}}$ |
| Uncertainty Set Size $\epsilon$ | $\{1e^{-3},\dots,9e^{-1}\}$ |  |  |  |  |
| $\theta_{1}$ | $\{1.001,1.01,1.1,1.5,2,3,5,10,50,100\}$ |  |  |  |  |
| LLM $|$ NN | LLM | - | 48 | Learning Rate | $\{0.001,0.01\}$ |
| Hidden Layer Dim | $\{32,64,128\}$ |  |  |  |  |
| Dropout Ratio | $\{0,0.1\}$ |  |  |  |  |
| Train Epoch | $\{100,200,300,500\}$ |  |  |  |  |
| LLM & In-Context Domain<br>Info/Wiki/GPT-4 $|$ NN | LLM | Labeled<br>Target Samples<br>or Wikipedia<br>or GPT-4 | 48 | Learning Rate | $\{0.001,0.01\}$ |
| Hidden Layer Dim | $\{32,64,128\}$ |  |  |  |  |
| Dropout Ratio | $\{0,0.1\}$ |  |  |  |  |
| Train Epoch | $\{100,200,300,500\}$ |  |  |  |  |

**Table 4: Target adaptation hyperparameter grids used in all experiments.**
| Model | Target Adaptation Method | # of HPs | Hyperparameter | Value Range |  |
| --- | --- | --- | --- | --- | --- |
| Tabular $|$ NN<br>or LLM $|$ NN<br>or LLM & Wiki/GPT-4 $|$ NN | Finetuning | - | 12 | Learning Rate | $\{10^{-7},10^{-6},10^{-5},10^{-4}\}$ |
| Target adaptation Epoch | $\{25,50,100\}$ |  |  |  |  |
| Tabular $|$ NN<br>or LLM $|$ NN<br>or LLM & Wiki/GPT-4 $|$ NN | LoRA | - | 12 | Learning Rate | $\{10^{-6},10^{-5},10^{-4},10^{-3},0.01\}$ |
| Target adaptation Epoch | $\{25,50,100\}$ |  |  |  |  |
| LLM & Wiki/GPT-4 $|$ NN | Prefix Tuning | - | 18 | Learning Rate | $\{10^{-5},10^{-4},10^{-3},0.01,0.05,0.1\}$ |
| Target adaptation Epoch | $\{25,50,100\}$ |  |  |  |  |

### Appendix B Additional Figures

##### Shift Patterns

The shift patterns of all datasets are shown in Figure 9.

Figure: Figure 9: Shift pattern analysis. For the 2550 source$\rightarrow$target distribution shift pairs in ACS Income, ACS Pub.Cov, ACS Mobility datasets respectively, we attribute the performance drop for each source$\rightarrow$target pair into $Y|X$-shifts (red curve) and $X$-shifts (blue curve), and sort all pairs according to the drop introduced by $Y|X$-shifts. We draw the *worst-500* settings in each dataset, and the decomposition method used here is DISDE [] with XGBoost as the reference model.
Refer to caption: x29.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The proposed method appears limited to datasets where all features have available descriptions. Expanding this approach to include datasets with features lacking descriptions would increase its versatility. One possible solution could involve learning embeddings for such features from scratch, similar to methods like TabTransformer, which can effectively handle mixed data types by embedding categorical features without requiring explicit descriptions.

It would be beneficial for the authors to discuss this limitation in the manuscript and potentially compare their method to approaches like TabTransformer that support feature embedding without descriptions. This addition would provide a clearer understanding of the method’s scope and suggest pathways for future improvements in handling various types of tabular data.
soundness: 3
strengths: This paper excels in addressing a pressing issue in machine learning: adapting models to distributional shifts in tabular data, specifically Y|X-shifts. The authors take an innovative approach by leveraging LLM embeddings to encode tabular data, which allows the model to incorporate broader contextual knowledge. This strength is significant because it enables the model to adapt to changes in label-covariate relationships, even with minimal (32 examples) labeled data from the target domain. The comprehensive evaluation of 7,650 source-target pairs across three datasets (ACS Income, Mobility, and Public Coverage) highlights the robustness of their approach. By testing against 20 algorithms (traditional GBDT and NN based models) and exploring a vast configuration space, the authors provide a solid foundation for the efficacy of LLM embeddings in generalizing across diverse tabular data environments. This thorough evaluation not only strengthens the empirical findings but also sets a new benchmark in evaluating distributional shifts in tabular datasets.
summary: This paper investigates the challenges posed by shifts in tabular data, specifically shifts in the relationship between labels and covariates (Y|X-shifts). These shifts often occur due to missing variables or confounders, making it difficult for models trained on a source domain to generalize to a target domain. 

The authors propose a solution using embeddings generated from large language models (LLMs) to encode tabular data. They demonstrate that LLM embeddings can improve adaptation to target domains even with limited labeled target samples, as the embeddings incorporate a broader contextual knowledge that helps mitigate the effects of these Y|X-shifts. Their study includes a comprehensive evaluation across 7,650 source-target pairs and comparisons with 20 algorithms, revealing that shallow neural networks (NNs) trained on LLM embeddings outperform other models in many cases.

The study's results highlight that while LLM embeddings alone do not consistently surpass state-of-the-art tree-based models, their performance significantly improves when adapted with a small number of target samples. Specifically, using 32 labeled target samples for fine-tuning yields substantial gains across multiple datasets, especially under scenarios of strong Y|X-shifts. The authors also explore different methods of adaptation, including in-context learning, low-rank adaptation, and prefix tuning, finding that fine-tuning with LLM embeddings provides a practical approach for generalizing across tabular Y|X-shifts.
weaknesses: This method is only applicable when there is a description available for all features of the tabular data. It would be good to expand this method to datasets which include features without any description. A possible solution can be by learning embeddings of such features from scratch and utilizing them as is done in TabTransformer. This will provide a comprehensive solution for different types of tabular data.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Figure 3: Recent study shows that label shift in tabular data causes performance degrade [1], why not take label shift into account? 
2. Are Y|X shifts and concept shift the same in meaning?
3. How does proposed compare to state-of-the-art methods designed for tabular TTA [2]?
4. How to evaluate Y|X shifts degree?

[1] Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. "Benchmarking distribution shift in tabular data with tableshift." Advances in Neural Information Processing Systems 36 (2024).

[2] Ren, Weijieying, et al. "TabLog: Test-Time Adaptation for Tabular Data Using Logic Rules." Forty-first International Conference on Machine Learning.
soundness: 3
strengths: 1. The motivation is clear and the algorithm is sensible.
2. The proposed method is tested on several benchmarks.
3. The proposed method is easy to understand and simple. It does not require a large number of LLM calls and the computational complexity is not high.
summary: This paper proposes a method to solve the Y|X shift problem in the tabular data domain using LLM embeddings. Under the LLM+NN model architecture, it uses concatenation to combine the domain information of LLM with the original tabular data information for improving prediction robustness and performance. The paper study models that are easy to adapt to the target domain even with few labeled examples so that fine-tuning the classification model with a small number of samples can further improve performance. Experiments can prove the effectiveness of the proposed method.
weaknesses: 1. On adaptation stage: The method requires the target's true label to guide the model parameter update. This setting is different from the basic test-time adaptation (TTA) setting, which is generally an unsupervised objective function [1]. The setting in this article is more like a fine-tuning setting.
2. The experiments in this article are mainly based on the ACS dataset. In recent studies, there are some other benchmarks like TableShift [2] contains Y|X shifts datasets. In the report they provided, ACS Income and ACS Pub.Cov dataset seem not have severe Y|X shifts. In order to prove the robustness to severe shifts, I think experiments on those datasets like ACS Unemployment and ICU Length of Stay are necessary.

[1] Wang, Dequan, et al. "Tent: Fully test-time adaptation by entropy minimization." arXiv preprint arXiv:2006.10726 (2020).

[2] Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. "Benchmarking distribution shift in tabular data with tableshift." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the "Weaknesses" section and answer the following questions. 
1. Why does this paper lack a conclusion section?
2. How are prompt templates tailored for different datasets? Typically, datasets with varying columns and feature types require significantly different prompts to optimize the effectiveness of the LLM.
soundness: 2
strengths: 1. The paper is well-structured and easy to follow, with the three technical components clearly and accessibly presented.
2. The experiments are comprehensive, which can demonstrate the claims of this paper.
summary: This paper tackles an exciting problem by utilizing LLM embeddings for tabular classification. The authors conduct comprehensive experiments to assess the effectiveness of LLM embeddings in tabular tasks, with promising results that highlight a new and potentially impactful direction for advancing tabular data analysis.
weaknesses: 1. The dataset variety is limited, as experiments are conducted on only three datasets within the same domain, which restricts the generalizability of the results and may affect the robustness of the findings.
2. The proposed method incorporates additional domain knowledge to enhance classification performance. However, it is unclear whether comparisons with tree-based methods are entirely fair, given that such methods may not effectively leverage domain knowledge. A thorough discussion on the source of performance improvements: whether from domain knowledge integration or specific technical innovations—would add valuable clarity.
3. The theoretical analysis may not be entirely appropriate for this context. Although the paper introduces a few-shot prompt tuning approach to maximize the use of limited labeled data, the theorem presented assumes a large  $m$  to achieve a small generalization bound, which contradicts the few-shot setting.
4. The term “test-time adaptation” in the title may be misleading, as it typically refers to model tuning using only unlabeled test data, which does not align with the approach described in this paper.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: I suggest the authors to provide a conclusion of the paper summarising the key findings and discussing the limitations.
soundness: 3
strengths: 1.  The authors provide a broad empirical evaluation across thousands of source-target pairs, diverse model configurations, and multiple algorithms, which strengthens the credibility of their findings. The paper includes a variety of performance metrics to assess the effectiveness of LLM embeddings under different conditions.  

2. By highlighting different scenarios and distributional changes, the authors clarify the conditions under which the approach excels or may face limitations. This transparency in results provides a balanced view of the method's robustness and allows researchers to see where it might or might not be effective.

3. The authors include a focused analysis on the effectiveness of few-shot adaptation, showing how minimal labeled data in the target domain can significantly improve performance.
summary: This paper studies distribution shifts in tabular data by using embeddings from large language models (LLMs) to enhance test-time adaptation. The paper uses a LLM encoder  to featurize tabular data and fit a shallow neural network on these embeddings for tabular prediction. The paper investigates different configurations including various training algorithms, incorporating additional domain information or not, fine-tuning the shallow neural network with few target domain samples or not. The paper provides comprehensive experiments on these configurations.
weaknesses: 1. The approach essentially applies existing LLMs as embeddings for tabular data without introducing substantial new methods for embedding generation or adaptation. Serializing tabular data into texts and using LLMs to extract features/make predictions have been studied in previous works. The adaptation technique described (using few-shot labeled samples for fine-tuning) is a well-known concept in transfer learning and domain adaptation. Few-shot learning techniques have been applied across various fields, and using minimal labeled data to adapt models to new domains is not unique to this work.

2. Although the paper provide comprehensive experiments on the configurations of the proposed method, it is still unclear to position how the proposed method performs in the literature. Specifically, the paper compares with baselines of Tabular Data + NN, where NN is the fully-connected neural networks. However, there are rich literatures on more advanced neural networks tailored for tabular data (e.g., FT-Transformer and many others). In addition, LLMs have been used for tabular data recently [1]. Comparisons with state-of-the-art methods on neural networks for tabular data and LLMs for tabular data are expected. Having said this, I'm not asking the authors to add more experiments in the rebuttal stage.

[1] Fang, Xi, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Jane Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. "Large language models (LLMs) on tabular data: Prediction, generation, and understanding-a survey." (2024).

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: As discussed above.
soundness: 2
strengths: 1) The paper focuses on the test-time adaptation for tabular data with Y|X shift. The problem is important and has been rarely studied.
2) The proposal is clear and expected to be effective.
summary: This paper focuses on the test-time adaptation for tabular data with Y|X shifts. Specifically, the proposal converts the raw tabular data into a new embedding with LLM and trains a shallow neural network on the embedding. When facing the distribution shift, the authors propose to fine-tune the network for the adaptation. Experimental results on three datasets are reported.
weaknesses: 1) The technical contribution is limited. The paper is more like an engineering work. The adopted techniques are all widely-adopted techniques for tabular data learning and the conclusions are also not surprising. It is not clear what technical problems have been addressed or what new insights have been proposed by this paper.
2) For the experimental results, the authors only adopted three datasets and the feature dimensions are small-scale.
3) How many times about the experiments repeated and how about the performance variance?
4) The authors claim that the LLM embeddings improve performance. However, the LLM embeddings are influenced by the serialization method and the choices of different LLMs. The influence of these should be discussed.
5) It is impractical to know the distribution shift in advance. In more practical scenarios, we need to detect the distribution shift first and then adapt.

Meta-review: This paper proposes a method to solve the Y|X shift problem in the tabular data domain using LLM embeddings. Under the LLM+NN model architecture, it uses concatenation to combine the domain information of LLM with the original tabular data information for improving prediction robustness and performance. The content of the paper is well written and intellectually challenging. However, the paper only conducts experiments on a limited number of datasets, which restricts the generalizability of the results. What’s more, the approach presented in the paper does not seem to be particularly novel or convincing and its theoretical results do not align well with the proposed method.   Therefore, I recommend the rejection of this paper.

Decision: Reject