<figure><img src="image_1.png" /></figure>

HAL
open science

# Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings

Lucas Mattioli, Youness Ait-Hadichou, Sabrina Chaouche, Martin Gonzalez

<figure><img src="image_2.png" /></figure>

## To cite this version:

Lucas Mattioli, Youness Ait-Hadichou, Sabrina Chaouche, Martin Gonzalez. Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings. Workshop "DataWorld: Unifying Data Curation Frameworks Across Domains" at International Conference on Machine Learning (ICML), Jul 2025, Vancouver (Canada), Canada. hal-05124551

HAL Id: hal-05124551

https://hal.science/hal-05124551v1

Submitted on 22 Jun 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Data Curation Matters: Model Collapse and Spurious Shift Performance
Prediction from Training on Uncurated Text Embeddings

Lucas Mattioli*¹ Youness Ait-Hadichou*¹ Sabrina Chaouche¹ Martin Gonzalez¹

**Abstract**

Training models on uncurated Text Embeddings (TEs) derived from raw tabular data can lead to a severe failure mode known as *model collapse*, where predictions converge to a single class regardless of input. By comparing models trained with identical hyper-parameter configurations on both raw tabular data and their TE-derived counterparts, we find that collapse is a consistent failure mode in the latter setting. We introduce a set of metrics that capture the extent of model collapse, offering a new perspective on TE quality as a proxy for data curation. Our results reveal that TE alone does not effectively function as a curation layer—and that their quality significantly influences downstream learning. More insidiously, we observe that the presence of model collapse can yield artificially inflated and *spurious Accuracy-on-the-Line correlation*. These findings highlight the need for more nuanced curation and evaluation of embedding-based representations, particularly in out-of-distribution settings.

# 1. Introduction

Tabular data is a central modality to critical domains like finance, healthcare, and the social sciences (Sirignano & Cont, 2019; Johnson et al., 2016; Athey, 2017; Shwartz-Ziv & Armon, 2022), encoding structured, heterogeneous information vital for decision-making. Despite the rise of deep learning across other modalities, gradient-boosted decision trees (GBDTs) (Friedman, 2001; 2002; Chen & Guestrin, 2016; Ke et al., 2017) remain state-of-the-art for tabular data, both in prediction accuracy (Arik & Pfister, 2021; Huang et al., 2020; Kadra et al., 2021; Katzir et al., 2020), and robustness under distribution shifts (Gardner et al., 2022;

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Illustration of our results. We train models with shared hyper-parameter configuration on raw tabular data and their LLM-based embedded counterparts. We exhibit model collapse to single class prediction both in ID & OOD test-sets and observe a rise of spurious correlation between ID-OOD accuracy (i.e. Accuracy-on-the-Line), as a consequence of model collapse severity.</figcaption></figure>

Liu et al., 2023; Gorishniy et al., 2021; Shwartz-Ziv & Armon, 2022). However, they face limitations in low-data regimes and struggle with high-dimensional or unstructured inputs like text (Ke et al., 2017), often requiring external embeddings. They also perform poorly under conditional distribution shifts of $P(Y|X)$, which we denote $Y|X$-shifts (Gardner et al., 2023).

To address these issues, large language models (LLMs) and their text embeddings (TEs) have emerged as a complementary modality. By converting mixed-type tabular fields into dense representations, TEs act as a form of data curation—organizing and filtering inputs to create a more model consumable format. Recent work has even shown some positive results of this approach for out-of-distribution (OOD) robustness against $Y|X$-shifts.

We investigate the use of text embeddings as a form of data curation in tabular classification. A central theme of our study concerns Accuracy-on-the-Line (ACL): a striking phenomenon in ML where in-distribution (ID) accuracy positively correlates with OOD accuracy across varying model configurations.

<footnote>*Equal contribution †IRT SystemX, France. Correspondence to: Martin Gonzalez <martin.gonzalez@irt-systemx.fr>.</footnote>
Our primary contribution is the identification of a critical and previously overlooked failure mode: a new kind of model collapse, where models trained on uncurated text embeddings (TEs), degenerate their predictions to a single class regardless of inputs. This not only compromises performance but can also spuriously inflate the ACL correlation, naively giving a false signal of OOD performance prediction. We introduce a set of dedicated metrics to quantify collapse severity and evaluate on 4615 model configurations, tested on numerous domain-shift scenarios, picked to exemplify our findings rather than to propose exhaustive numerical experiments. In doing so, we also uncover notable inconsistencies between MTEB leaderboard rankings (Muennighoff et al., 2023) and performance in tabular classification tasks—highlighting the limitations of benchmark-driven model selection. Shifting from traditional tabular training to LLM-based embedding pipelines introduces domain-specific challenges in representation and alignment, which our work addresses through generalizable metrics for evaluating embedding quality and model robustness. Moreover, these findings emphasize the need for human oversight on fully automated pipelines, as relying on metrics like $R^2$ can fail silently. Without human intervention, uncurated TE may inject bias in a worst-case fashion which further highlights the ethical dimensions of representation learning, where embedding quality, fairness, and robustness intersect.

## 2. Background and Related Work

**Robust OOD generalization.** Generalizing under OOD conditions remains a central challenge in ML. Benchmarks — especially in vision often focus on covariate shift ($X$-shifts), offering a narrow view of robustness (Hendrycks & Dietterich, 2019; Taori et al., 2020). Yet, in many real-world scenarios, particularly with tabular data, $Y|X$-shifts are far more prevalent and problematic, arising from missing variables, hidden confounders, or evolving social-economic contexts (Zeng et al., 2024; Gardner et al., 2023; Alvarez-Melis & Fusi, 2020).

Methods to address distributional shifts include distributionally robust optimization (DRO) (Duchi et al., 2021), balancing techniques, and invariant learning. DRO constructs worst-case distributions within uncertainty sets using metrics like $\chi^2$ divergence (Duchi et al., 2021), KL divergence (Hu & Hong, 2013), and Wasserstein distance (Blanchet et al., 2019; 2023). However, these approaches often underperform on real-world tabular data (Liu et al., 2023). Invariant learning (Arjovsky et al., 2019; Koyama & Yamaguchi, 2020; Peters et al., 2016) assumes access to multiple training environments, which limits practical applicability. While some theoretical results exist for linear transfer models (Li et al., 2022; Tian & Feng, 2023), they rarely extend

to complex models like GBDTs or deep networks. Additionally, most domain adaptation studies (Iwasawa & Matsuo, 2021; Chen et al., 2023; Liang et al., 2023) focus on covariate $X$-shifts in images, whereas we address conditional $Y|X$-shifts in tabular contexts.

Inference-time adaptation methods like TENT (Wang et al., 2021), struggle in tabular settings, while recent approaches like FTAT are specifically designed to handle both $X$ and $Y|X$-shifts effectively (Zhou et al., 2025). New metrics such as DISDE help decompose performance drops into $X$ and $Y|X$-shift contributions (Cai et al., 2023), while Accuracy-on-the-Line (ACL) (Miller et al., 2021) measures how ID vs. OOD performances correlate, and Agreement-on-the-Line (Baek et al., 2022), an analog phenomenon of ACL which, if valid, allows to induce ACL without the need of labeled test-set data.

**LLM Embeddings.** LLMs have recently been applied to tabular classification in two main ways. One strategy treats classification as a text generation task using decoder-based models with prompts like “Is this patient diabetic? Yes/No,” though this approach is sensitive to prompt design and yields variable results (Hegselmann et al., 2023; Slack & Singh, 2023). Recent work (Zeng et al., 2024) shows that LLM embeddings derived from tabular data can improve OOD generalization. However, their setting is established using embeddings from a single LLM, leaving open the question of whether such generalization holds across model families and LLM variants. Several recent LLM variants aim to enhance embeddings quality through fine-tuning or architectural enhancements: Linq-Embed-Mistral (Choi et al., 2024), is fine-tuned specifically to focus on hard negatives, SFR-Embedding-Mistral (Meng et al., 2024) leverages transfer learning to improve generalization across domains, e5-Mistral-7B-Instruct (Wang et al., 2024) serves as a general-purpose model, while Zeta-Alpha-E5-Mistral (Câmara, 2024) uses LoRA adapters for efficiency under resource-constrained conditions. While these approaches diversify embedding strategies, their robustness under distribution shift in tabular settings is still poorly understood.

**Benchmark-Based Assumptions.** Benchmarks like MTEB (Muennighoff et al., 2023) have become standard tools for assessing LLM performance across NLP tasks, offering scalable comparisons in fast-evolving model ecosystems. However, a growing reliance on rankings only raises concerns about their performance on domain-specific applications—such as tabular data under distribution shift. Recent work has highlighted that spurious correlation benchmarks often disagree in their evaluations, underscoring the need to revisit benchmark assumptions and develop task-sensitive evaluation protocols (Bell et al., 2024).
**Model collapse of Embedding-Based Models.** Recent studies have uncovered diverse failure modes across ML settings. (Herman et al., 2024) show that indiscriminate use of LLM-generated content during training leads to irreversible model collapse, where tails of the original data distribution disappear. Modality collapse in multimodal models occurs when noisy features from one modality overshadow useful signals from others; the authors propose Explicit Basis Reallocation (EBR) to mitigate this by disentangling shared representations (Chaudhuri et al., 2025). Strong collapse from synthetic data emerges when models overfit even small amounts of synthetic inputs, degrading generalization—particularly in large models—despite increased training size (Dohmatob et al., 2024). Finally, non-iterative model collapse in LLMs arises from single-phase exposure to synthetic data, narrowing coverage and inflating frequent patterns; the authors propose ToEdit, a token-level data editing method, to alleviate this (Zhu et al., 2024).

# 3. Evaluation Methodology

To define and evaluate model collapse, we build on two key ideas: using LLMs to generate embeddings as a data curation layer for downstream models, and analyzing ID vs. OOD performance across model families to uncover hidden patterns. This section first outlines the two underlying procedures, then introduces our notion of model collapse along with metrics to categorize and quantify its severity.

## 3.1. Text Embeddings as Data Curation

Our method begins by collecting tabular rows from both a source and a target domain. These rows are converted into natural language through a serialization process (Tab2Text), which reformulates each row into a sentence, while a task-specific instruction is appended. This serialized input is then passed to a pre-trained LLM that serves as an encoder. It will generate vector representations, called embeddings. More specifically, let LLM be an LLM encoder (such as e5-Mistral-7b-Instruct) and let S be a tabular dataset. The overall procedure for transforming S into a set, denoted LLM|S, of LLM embeddings of S based on LLM has been described in detail in (Zeng et al., 2024). In short, it consists on instructing an LLM to render plain text Tn out of the information of the nth row sn of the table S, incorporating a task specific prompt I and use both (Tn, I) as input for LLM to produce a vector encoding Φ sn of sn. When repeating this procedure for the whole dataset S we obtain a vector set of LLM embeddings that we will denote LLM|S. We choose LLMs with same embedding vector dimension (4096) to rule out possible biases or impacts which might directly be associated to vector dimensionality and we do not make any changes to the chosen off-the-shelf LLMs. In other words, all LLMs are used as a form of

automated data curation strategy.

## 3.2. On-the-line Analysis & Model Collapse

According to the data-centric approach of robustness, well-curated data representation should exhibit a certain degree of resilience to distribution shifts. Although the usual way of evaluating robustness is simply by testing against a specific dataset, a less known line of work is to analyze the behavior of models both on In-Distribution (ID) and OOD test sets as a way to understand the shifted data in question. In this second line of work, many examples of distribution shifts arising from Computer Vision have exhibited a specific linear correlation pattern arises for such pairs of test sets, a phenomenon coined as Accuracy-on-the-line (ACL), and which, if present, can serve to predict the OOD performance of a model if one knows its ID performance and the parameters of the linear fit arising from ACL. Determining validity and strength of ACL is usually deduced by the value of the R² coefficient of the linear fit. Our method picks a set of training Hyper-Parameter Configurations (HPCs) for different models, which are trained on a source domain of tabular data. Let fc be a model trained with HPC c and let S be a test-set. We denote $A^S(f_c)$ for the prediction accuracy of fc on S. Given ID and OOD test sets S and Q, and a HPC set HP of cardinality N, we plot all N models fc in what we call the *ACL-plane*, for which each model fc has coordinates ($A^S(f_c)$, $A^Q(f_c)$). Recognizing that shifts often co-occur with class imbalance, and that accuracy may be insufficiently sensitive to such cases, we call *F1L-plane* the plane where each fc has coordinates its corresponding macro F1 scores over S and Q respectively.

## 3.3. Binding Models across Modalities

Once we have a specified LLM encoder LLM, we bind model couples ($f_{c,1}$, $f_{c,2}$) with shared hyper-parameter configuration where $f_{c,1}$ has been trained over a set $D_{train}$ of raw tabular data and $f_{c,2}$ was trained over the curated data LLM|$D_{train}$ by means of the procedure described in Subsection 3.1. Then, given two tabular test sets S and Q, and we plot the coordinates of for each model $f_{c,1}$ both in the *ACL-plane* and the F1L-plane with respect to S and Q according to the procedure described in Subsection 3.2. Next, we use the same LLM to encode both S and Q and evaluate $f_{c,2}$ accordingly. Namely, for $f_{c,2}$, we plot a point with coordinates ($A^{LLM|S}(f_{c,2})$, $A^{LLM|Q}(f_{c,2})$) in the ACL-plane. Now that both $f_{c,1}$ and $f_{c,2}$ have their respective coordinates in the ACL-plane and the F1L-plane, we bind the points corresponding to $f_{c,1}$ and $f_{c,2}$ by means of a straight line attaching them. We highlight the fact that our method is primarily focused on examining the patterns of the bindings of all pairs of models with shared hyper-parameter configuration, and incidentally then focused on the partic-
ular patterns of the set of models trained on each of the two modalities (tabular and embeddings). Nevertheless, one should not be overlooked the fact that the coordinates of $f_{c,1}$ and $f_{c,2}$ are obtained from $S$ and $Q$ and from $LLM|S$ and $LLM|Q$ respectively. Rigorously, the straight line representing the bind between $f_{c,1}$ and $f_{c,2}$ is actually a line binding two different points of two different planes that we have overlapped. As a consequence, we report the $R^2$ coefficient associated to each of these two sets of models separately. Our overall method can be illustrated in Fig 1, where we can already see what a model collapse looks like : numerous models trained on the tabular modality are bound to what seems to be a “single” collapse point un the upper-left part of the ACL-plane. Measuring the severity of model collapses along different pairs of source and target domains, as well as their different kinds, brings us to propose several model collapse metrics.

### 3.4. Model collapse metrics on binary classification

In this subsection, we define model collapse metrics that we will use to distinguish between different kinds of collapses as well as to measure their severity. Our approach consists on considering, for a given model $f$ (such as MLP, LR, GBM, XGB, etc.), a set $HP_f$ of training Hyper-Parameter combinations for which models trained on raw tabular datasets provide no single-class predictors, and then ask how many of those incur into single-class predictors when trained under same HP configurations for over LLM embeddings.

Let $f_c$ be a model trained with hyper-parameter configuration $c$ and let $S$ be a test-set. We denote $PP^S(f_c)$ (resp. $PN^S(f_c)$) for the number of predicted positives (resp. predicted negatives) of $f_c$ on $S$. We say that the model $f_c$ is collapsed over $S$ if $PN^S(f_c) = 0$ or $PP^S(f_c) = 0$.

Let $f$ be a method, $D$ be a domain with training (resp. test) tabular data $D_{train}$ (resp. $D_{test}$) and a test-set $S$. We will say that a hyper-parameter set $HP_f$ is well-defined w.r.t. $S$, and we denote it $HP_{f,S}$, if for all hyper-parameter combinations $c \in HP_{f,S}$, the trained model $f_c$ on $D_{train}$ does not collapse over $D_{test}$ and $S$. For simplicity, we denote it $HP_f$ in the special case when $S = D_{test}$.

Let LLM be an LLM Encoder. For a model $f_{c,D}$ trained over a tabular dataset $D_{train}$, we denote $LLM|f_{c,D}$ for the model $f$ trained with same HP configuration $c$ but over the associated embeddings $LLM|D_{train}$ obtained according to procedure 3.1. While the context is clear, we will simply denote it $f_c$.

Given a well-defined set $HP_f$, we define metrics to measure model collapses arising from switching from standard tabular training on $D_{train}$ to representation training over LLM embeddings $LLM|D_{train}$. In other words, we present

metrics that will allow to measure how the set $HP_{LLM|f}$ is far from being well-defined over $LLM|S$. The rationale is that well-curated training data do not incur into single-class predictors on binary classification tasks, so that the ratio of combinations in $HP_{LLM|f}$ incurring into model collapse over different test-sets $LLM|S$, is an informative indicator on the quality of the text embeddings in their function as a data curation layer.

In what follows, we fix a method $f$, we simply denote HP for a well-defined set $HP_{f,D}$ as long as the context is clear. Let $S$ be a test-set. We define the Collapse Ratio of HP over $S$ as the ratio of hyper-parameter combinations resulting in models collapsed into single class prediction on $S$ as

$$CR_{HP}^{S} = \frac{1}{|HP|} \sum_{c} \mathbb{I}_{\{\text{Cond}_{T}^{S}(c)\}} + \mathbb{I}_{\{\text{Cond}_{F}^{S}(c)\}},$$

where $\text{Cond}_{T}^{S}(c) = (PN^{S}(f_c) = 0)$ and $\text{Cond}_{F}^{S}(c) = (PP^{S}(f_c) = 0)$. Notice that condition $PN^{S}(f_c) = 0$ implies $A^{S}(f_c) = \text{TR}^{S}$ and condition $PP^{S}(f_c) = 0$ implies $A^{S}(f_c) = \text{FR}^{S}$, where

$$FR^{S} = \frac{|\text{cl}(\text{FALSE})|}{|S|}, \quad \text{TR}^{S} = \frac{|\text{cl}(\text{TRUE})|}{|S|}$$

are the proportions of each of the two classes {FALSE, TRUE} in $S$. Also, as we will see in short, models not always collapse to the same class so $CR_{HP}^{S}$ computes the ratio of configurations inducing model collapse on either one of the two classes. Let $S$ and $Q$ be two test sets. We define the pair-wise strong collapse ratio $CR_{s,HP}^{S,Q}$ of HP over $S$ and $Q$ as the ratio of hyper-parameter combinations resulting into model collapse simultaneously for both test sets. For simplicity, in what follows we assume that our procedure exhibits model collapses on the positive classes in $S$ and $Q$, then $CR_{s,HP}^{S,Q}$ is formalized as

$$CR_{s,HP}^{S,Q} = \frac{1}{|HP|} \sum_{c} \mathbb{I}_{\{\text{Cond}_{T}^{S}(c) \cap \text{Cond}_{T}^{Q}(c)\}}.$$

The pair-wise projection ratio as the ratio of hyper-parameter combinations resulting into model collapse in either one of the test-sets

$$CR_{p,HP}^{S,Q} = \frac{1}{|HP|} \sum_{c} \mathbb{I}_{\{\text{Cond}_{T}^{S}(c) \cup \text{Cond}_{T}^{Q}(c)\}}.$$

When the context is clear, we will denote this metrics simply $CR_s$ and $CR_p$.

Finally, we define a metric to quantify near-collapse phenomena. Experiments show that there are HP configurations entailing models that are almost collapsed but not completely, as they have a small fraction of false or true negatives. On the one hand, the above metrics are too rigid to account such near-collapses. On the other hand, such
predicted negatives can only be considered to be negligible relative to the size of their associated test-set and the proportion of ground-truth negatives among the latter. In what follows, all collapse ratios will include near-collapses and we defer the formal definition of this more involved metrics to Subsection A.2 in the appendix.

# 4. Experiments

We make experiments according to the following questions:

1. How prevalent is model collapse across different LLMs, and what impact do they have on the validity and strength of the ACL phenomenon?

2. For a fixed model family and source and target domains, how does model collapse vary in kind and intensity over the ACL & F1L planes across different LLMs?

3. Does valid ACL for models trained on tabular data imply valid ACL for their representation learning counterparts? Do current Test-Time Adaptation methods for domain shifts remedy any lack of ACL validity?

4. Is there a correlation between LLM-based Text Embedding quality benchmarking as present in MTEB and their quality as data curation layers for real-world domain shifts?

5. Are LLM embeddings transferable between LLMs? More specifically, how do models trained on embeddings from one LLM behave when tested on test sets encoded by other LLMs?

To the best of our knowledge, this is the first evaluation on the model collapse phenomenon as framed in this work. It consists of an initial investigation of the concept and we prioritize qualitative insights as primary findings, hoping to open the door to further benchmarking and systematic analysis such as the one performed in (Zeng et al., 2024). Nevertheless, we include a re-implementation and extension of part of their experiments as a reference point. Where appropriate, we include results using the same LLM and MLP configuration they use, to allow direct comparison with their findings.

We focus on a binary classification task to predict whether US working adults' yearly income exceeds $50,000 in 2018, based on the ACS Income dataset (Ding et al., 2021). We take into consideration six states: Alabama (AL), Alaska (AK), Arizona (AZ), Arkansas (AR), California (CA), Puerto Rico (PR). As we will already see, this very restricted context already provides many insightful answers to the above questions. In all our experiments, the "<50k" class is going to be the positive class.

Four LLMs are identified and chosen from the MTEB Leaderboard, which ranks LLMs based on the performance of the text embeddings they generate on a variety of tasks. The selection accounted for the LLM used in all experiments reported in (Zeng et al., 2024) and, for the remaining LLMs, they were restricted only to models with a fixed embedding dimensionality of 4096 to improve the rigor of our comparative analysis. Tables 2 and 1 show the number of model configurations and MTEB ranks, as consulted on March 1, 2025. Notice that 3 LLMs were chosen out of the top-10 best ranked models while we chose one LLM ranked way below the 100th place.

<table><caption>Table 1. Name & Ranking of LLMs according to MTEB.</caption><thead><tr><td>Name</td><td>LLM Full Name</td><td>Rank</td></tr></thead><tbody><tr><td>Ling</td><td>Ling-Embed-Mistral</td><td>2</td></tr><tr><td>SFR</td><td>SFR-Embedding-Mistral</td><td>5</td></tr><tr><td>e5</td><td>e5-Mistral-7B-Instruct</td><td>9</td></tr><tr><td>Zeta</td><td>Zeta-Alpha-E5-Mistral</td><td>177</td></tr></tbody></table>

We organized our results into two sets of experiments: the first addressed the initial three questions, while the second provided evidence relevant to the final two.

## 4.1. Model Collapse arising from On-the-Line Analysis

**Experiment setup.** We fix CA as our training domain for this set of experiments and the remaining states as test domains. We select 7 families of models with varying number of training hyper-parameter configurations as shown in Table 2 providing a total of 934 model configurations.

<table><caption>Table 2. Number of configurations per model family.</caption><thead><tr><td>Family</td><td>MLP</td><td>CVaR-DRO</td><td>LR</td><td>XGB</td><td>GBM</td><td>RF</td><td>SVM</td></tr></thead><tbody><tr><td>Configs</td><td>96</td><td>203</td><td>100</td><td>200</td><td>200</td><td>101</td><td>34</td></tr></tbody></table>

The configurations match those reported in (Zeng et al., 2024), with the exception of LR, where we add additional configurations to obtain a total of 100 models and RF, which is new. All model configurations are then trained on the tabular data CA and on each of the four LLM embeddings of CA. We highlight the fact that the sets *S* and *LLM|S* have different modalities, but have the same positive class ratio.

**Model collapse is ubiquitous over all tested LLM Embeddings and almost all model families.** We illustrate our findings directly by providing the 140 pairs of ACL and F1L planes with plotted bindings among models with shared HP configurations, for each combination of test domain, model family and LLM encoder. Notice that in this setup, the LLM used to encode the training set from which models are trained is the same LLM used to encode the test sets.
<figure><img src="image_5.png" /><figcaption>Figure 2. Different patterns as exemplified by different combinations of LLMs, models and target states.</figcaption></figure>

All figures are reported in Subsection A.5 in the Appendix. All tested LLMs induce model collapse and, while some combinations of LLMs, model family and (source, target) states show no model collapse, there is no model family not exhibiting model collapse for all tested states for any tested LLM. Moreover, such collapses do not only happen on states with strong $Y|X$-Shifts but are rather systematically appearing on all the tested target states. Additionally, there are configurations inducing model collapse simultaneously on all states. For instance, in Figure 10 one exhibits that, out of the 96 $e_5$|MLP configs, there are 34 are collapsed on $e_5$|CA and $e_5$|PR (these are the pair-wise strong collapses). It turns out that there are 12, 33, and 31 configs that induce strong model collapse for Linq, SFR and Zeta respectively, and that there are 7 that induce strong collapses for the pair (CA, PR) for all LLMs (see Table 3 for details).

**Model collapse varies in kind and severity across LLMs.**
An attentive look of all figures in Subsection A.5 in the Appendix reveals that there are several kinds of patterns arising from analyzing the bindings in the ACL and F1L planes. We showcase 3 such distinguished patterns that we report in Figure 2. Detailed calculations of all metrics presented here can be found in the appendix.

First, Figure 2a shows both the ACL and F1L planes for the LR model family, with target domain AL and with base LLM encoder LLM=SFR. Blue points in the ACL plane correspond to models $f_{c,1}$, for $c \in$

$HP_{LR}$, with coordinates $(\mathcal{A}^{CA}(f_{c,1}), \mathcal{A}^{AL}(f_{c,1}))$ while green points correspond to models $f_{c,2}$ with coordinates $(\mathcal{A}^{LLM|CA}(f_{c,2}), \mathcal{A}^{LLM|AL}(f_{c,2}))$. Analyzing the bindings show that there is no model collapse in this scenario. This fact is confirmed by computing the confusion matrix of each point, and in particular showing that the green points close to the ratio of the class “<50k” (i.e. the positive class ratio) are no near collapses. More importantly, the bindings between points seems to follow some logic of geometric nature, a fact that is only observed for the LR family but that is present in all pairs of tested states and all 4 LLMs.

Second, Figure 2b exhibits the frequent kind of model collapse, where a high number of blue points are bound to multiple overlapping green points exactly at the intersection of the positive class ratio of both the LLM|CA and LLM|PR (which also equals the intersection of the positive class ratio of the corresponding CA and PR tabular test-sets). The specific values of such collapsed models correspond *exactly* to the population statistics found in Table 4. Analyzing the confusion metric of the rest of the green points close to the positive class ratio of CA confirms that there are *only* pair-wise strong collapses in this scenario, so that $CR_s = CR_p = 61,58\%$.

Third, Figure 2c exhibits another frequent kind of model collapse, where blue points are bound to green points which are projected into at least one of the two corresponding positive class ratios for CA and AL. In this case, there
<figure><img src="image_6.png" /><figcaption>Figure 3. (a) An example of a valid ACL and F1L; (b,c,d) Test-Time adaptation methods fail to strengthen ACL or F1L.</figcaption></figure>

are both pair-wise strong collapses as well as projection collapses so that $CR_s = 18, 23\%$ and $CR_p = 43, 84\%$.

Last, but not least, all figures for RF in the appendix show that the set $HP_{RF}$ is not well-defined w.r.t. CA. in the sense of this article. In other words, there are already models collapsing over tabular data under our choice of HP configurations. Notice that collapsed models in tabular training do not imply collapsed models in rep training. In the case of the scenario of Figure 13c, we have 26 strong collapses of models trained on tabular data, 11 such collapses on the corresponding SFR embeddings and only 3 bindings between pairs of collapsed models. In other words, collapses in the tabular modality do not imply collapses in the embedding modality, which is why we restrict our attention in this article on model collapses arising in *well-defined* HP configuration sets (i.e. those that do not present model collapse on the tabular modality).

**Strong Model collapse induces spurious ACL.** When analyzing both the ACL and F1L planes in Figure 2b, one can notice that the $R^2$ coefficient of the green points is significantly higher than the one for the blue points. In appearance, one would be tempted to say that LLM Embeddings might improve the correlation between ID and OOD performance of models but this is quickly shown to be due to the fact that the strong collapse ratio is high, which means that the fact that many green points are overlapping creates a misleading increase of the $R^2$ parameter while it is clear that the green points that are not collapsed are nowhere near forming some linear correlation both on the ACL and the F1L planes. Worse yet, because of the high class imbalance presented in PR, single prediction models are in appearance among the best performing in terms of accuracy while they

are systematically predicting the same class both in CA and PR. Hence the importance of the F1L plane, which resolves part of the issue but still shows spurious correlations and cannot in any way be used for OOD performance prediction.

**Delimiting the Validity and Correlation Perimeter of Accuracy/F1-on-the-Line Phenomena.** Although we have used the ID vs OOD F1-plane to get a more nuanced view on how LLMs behave, it is interesting to hint that there is a scope of validity when ACL and F1L are correlated when they are valid. When observed, as shown in Figure 3a, this is analog to the experimentally exhibited correlation between ACL and the so-called Agreement-on-the-line (AGL) phenomenon. Now, AGL is particularly useful to be able to predict OOD performance without then need of labeled OOD data because the linear fit has shown to match the one obtained for ACL. Although we do not expect that the linear fits match between ACL and F1L, our findings suggest that on-the-line phenomena and their scope of validity should be studied with different metrics of interest beyond accuracy. In the original ACL paper, the F1-score was used in one experimental setup but it was not tied to a reflection about imbalanced data but rather as the standard metric associated to that specific task. We stress out that studying on-the-line phenomena for different metrics induced by data-centric approaches may yield novel insights about OOD Robustness.

Finally, Figure 9 in the Appendix shows that when ACL is valid on the basis of standard training, **it does not imply ACL on the basis of LLM Embeddings.** As such, our experiments exhibit a case of a correlation between valid ACL and valid F1L and a counter-example of a valid ACL at the standard training level that fails to correlate with a
valid ACL on the representation training counterpart.

**Available TTA methods do not induce stronger ACL.**
The paper (Kim et al., 2024), the authors reveals that Test-Time Adaptation (TTA) methods not only improve model performance on out-of-distribution data but also significantly strengthen the linear correlation between in-distribution and out-of-distribution metrics. Inspired by this approach, we employ TTA methods for MLP only, and we select Representation Finetuning as described in (Zeng et al., 2024) (F.2), TENT (Wang et al., 2021), and FTAT (Zhou et al., 2025) giving a total of 288 adapted models. Our findings in Figures 3b, 3c and 3d show that current test-time adaptation methods, including TENT or its more recent analog designed for tabular data FTAT, fail to provide any help if one seeks to use these methods to widen the scope of validity of ACL even when dealing with X-shifts. Concerning FTAT, a new pattern arises as a second horizontal line emerges in both the ACL and F1L lines, and which does not match either class ratio.

<figure><img src="image_7.png" /><figcaption>Figure 4. LLM Embeddings In-Distribution and OOD Generalization Upper-Bounds.</figcaption></figure>

**LLM Embeddings Push Toward ID/OOD Generalization upper bounds.** In Figure 4a we report all 923 model combinations along all families of models in the ACL plane for standard tabular training. This is a partial reproduction of the ACL plane reported in (Liu et al., 2023). Figure 4b reports the combinations on the basis of training over e5 embeddings. Although the collapses are not directly visible in the figure, one can see that the overall movement of models with same configurations as those used to plot Figure 4a, tends to approach an upper bound in terms of in-distribution accuracy, with heavily varying levels of accuracy on the target domain, reaching a generalization upper-bound.

## 4.2. Benchmarking of LLM Embeddings' Robustness

**Experiment setup.** In this set of experiments we choose two MLP configurations: the first, $c_1$, being one that we know results in a collapse for LLM=e5 and the (CA → PR)

pair; the second, denoted $c_2$, being the optimal HP configuration found in (Zeng et al., 2024), where such configuration was the one giving the best average macro F1 score among MLPs for all pairs of US states (including PR).

<figure><img src="image_8.png" /><figcaption>Figure 5. Pattern behavior across different LLMs for MLP.</figcaption></figure>

**The MTEB ranking of LLMs is not correlated with their OOD Generalization capabilities.** To address Question 4, we train MLPs $f_{c_2}$ for the optimal reported configuration from (Zeng et al., 2024) over the 4 LLM embeddings of CA separately. We test each of the four models obtained on all states encoded by the same LLM used to train each model. As such, we evaluate the performance of each LLM by means of how each model trained along a choice of LLM embeddings performs when tested on embeddings of other domains but encoded by the same LLM. This could be reframed as a particular evaluative task for benchmarking LLM embeddings consisting on OOD robustness beyond pure performance on classification tasks. The results are reported in Figure 5.

Surprisingly, Linq, the second highest ranked LLM for MTEB, is never in the first position. But most importantly, Zeta (ranked 177th in MTEB) shows surprisingly superior performance on this task in comparison to all other LLMs. In order to make an extended assessment of our insight, we use the FractionBest introduced in (Zeng et al., 2024), and
<figure><img src="image_9.png" /><figcaption>Figure 6. Model behavior across LLM Embeddings. (a) Results for $f_{c1}$; b) (resp. c)) Accuracy (resp. F1-score) for $f_{c2}$.</figcaption></figure>

we conducted an exhaustive pairwise comparison across all LLMs, hyper-parameter configurations for all families, and all combinations of train and target domains which we report see Figure 7. We find that Zeta outperforms competing LLMs in 31% of the pairwise comparisons based on accuracy, and in 22% based on F1-score. We stress that this experiment takes into account models trained over all different states and not just the CA state.

Second, the ranking agreement between our 4 LLMs across states is not universally valid. The ranking is consistent for the couples (PR,AZ) and (AL,AK) w.r.t accuracy, while presenting different degrees of variability across other states. Also, when testing on CA, Accuracy and F1-score present a slight ranking switch between e5 and SFR.

Third, different states present different metric variability: on the source domain CA, Accuracy and F1-score values are fairly comparable for all 4 LLMs but on the target domain PR, there is a much more pronounced metric variability across all LLMs. Lastly, domain gap w.r.t. accuracy does not universally penalize models on target domains due to class imbalance: 3 out of 4 LLMs provide improved performance in PR compared to the CA, with only Linq actually performing worse. Nevertheless, domain gap w.r.t. F1-score does penalize all models on all states.

**Train/Test LLM mismatch significantly increases Model Collapse.** In order to address question 5, we proceed to train two MLPs along configurations $c_1$ and $c_2$ for e5 embeddings only, which we denote $f_{c1}$ and $f_{c2}$. We then test both models on all 4 LLM embeddings of all states. The rationale here is to study how a model that has shown to collapse (resp. has good performance) over states (e.g. $f_{c1}$, resp. $f_{c2}$) behaves on those same states but encoded by other LLMs. The results are reported in Figure 6.

On the one hand, Figure 6a shows that the model $f_{c1}$ which we knew was collapsed over the e5|CA and e5|PR domains, remains collapsed along all e5 embeddings of the rest of the states. More strikingly, $f_{c1}$ collapses on the embeddings of all states for all 4 LLMs and not just the one

used to train it. This suggests that model collapses transfer across LLM embeddings for different LLMs than the one used to train the model. In other words, changing the LLM does not induce a positive behavior on collapsed models.

On the other hand, Figure 6b shows that performance of the model $f_{c2}$ for the optimal configuration from (Zeng et al., 2024) not only does not maintain its performance across embeddings, but it even often results in model collapse over most states encoded by a different base LLM encoder. Notice that this model, when evaluated on the all Linq embeddings of all states, exhibits collapse. We report in Table 5 the confusion matrix values for each model. For $f_{c2}$, we observe:strict collapse on Linq|PR, SFR|PR, Zeta|PR, Zeta|CA, Zeta|AL, Zeta|AZ, Zeta|AR; near collapse on SFR|CA, SFR|AR, SFR|AZ, SFR|AL, Zeta|AK. Finally, $f_c$ is not a near collapse on SFR|AK: we have $\text{TNR}(f_c, \text{SFR}|\text{AK}) + \text{FNR}(f_c, \text{SFR}|\text{AK}) = 0, 142 > \varepsilon$. Additionally, the model trained on the linq embeddings with these same hyper-parameters does not collapse in any of the linq embeddings of the 6 states. Last, but not least, **models already collapse on the source domain for different LLM embeddings**. Figure 8 exhibits model collapse even when the target test set is CA but encoded by different LLMs. This suggest that the conclusions of (Zeng et al., 2024) may differ across LLMs.

**Conclusion.** Training on uncurated text embeddings (TEs) from raw tabular data can lead to model collapse, where models predict a single class regardless of input. We show this is a consistent failure mode in TE-based models, unlike those trained on raw data. To capture this effect, we introduce metrics that assess collapse severity and reveal TE quality as a key factor in downstream performance. Notably, strong model collapse can induce spurious ACL correlations, distorting evaluation and misleading OOD performance prediction. We also find no correlation between MTEB rankings and an LLM's effectiveness as a data curation layer. We hope that our experimental setup, far from being comprehensive, is already rich enough to offer a foundation for more robust OOD evaluation for text embedding benchmarks.
**Acknowledgments.** This work has been supported by the French government under the "France 2030" program, as part of the SystemX Technological Research Institute. M.G. thanks Martin Royer for his several insightful comments, which greatly improved the quality of this paper and to Jiashuo Liu for many fruitful conversations about methodologies for modelling distribution shifts.

## References

Alvarez-Melis, D. and Fusi, N. Robustness and Generalization via Algorithmic Stability. *arXiv preprint arXiv:2002.08791*, 2020.

Arik, S. Ö. and Pfister, T. Tabnet: Attentive Interpretable Tabular Learning. In *Proceedings of the AAAI conference on artificial intelligence*, volume 35, pp. 6679–6687, 2021.

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant Risk Minimization. *arXiv preprint arXiv:1907.02893*, 2019.

Athey, S. Beyond Prediction: Using Big Data for Policy Problems. *Science*, 355(6324):483–485, 2017.

Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift. *arXiv preprint arXiv:2206.13089*, 2022.

Bell, S. J., Bouchacourt, D., and Sagun, L. Reassessing the Validity of Spurious Correlations Benchmarks. *arXiv preprint arXiv:2409.04188*, 2024.

Blanchet, J., Kang, Y., Murthy, K., and Zhang, F. Data-driven optimal transport cost selection for distributionally robust optimization. In *2019 winter simulation conference (WSC)*, pp. 3740–3751. IEEE, 2019.

Blanchet, J., Kuhn, D., Li, J., and Taskesen, B. Unifying Distributionally Robust Optimization via Optimal Transport Theory. *arXiv preprint arXiv:2308.05414*, 2023.

Cai, T. T., Namkoong, H., and Yadlowsky, S. Diagnosing Model Performance Under Distribution Shift. *arXiv preprint arXiv:2303.02011*, 2023.

Chaudhuri, S., Yang, Y., Yan, J., Khan, S., Shah, M., and Liu, Z. Mitigating Modality Collapse via Explicit Basis Reallocation. *arXiv preprint arXiv:2505.22483*, 2025.

Chen, L., Zhang, Y., Song, Y., Shan, Y., and Liu, L. Improved test-time adaptation for domain generalization. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 24172–24182, 2023.

Chen, T. and Guestrin, C. XGBoost: A Scalable Tree Boosting System. In *ACM SIGKDD International Conference on Knowledge Discovery*, pp. 785–794. ACM, 2016.

Choi, C., Kim, J., Lee, S., Kwon, J., Gu, S., Kim, Y., Cho, M., and yong Sohn, J. Linq-Embed-Mistral Technical Report. *arXiv preprint arXiv:2412.03223*, 2024.

Câmara, A. Fine-tuning an LLM for State-of-the-Art Retrieval: Zeta Alpha's Top-10 Submission to the MTEB Benchmark. Zeta Alpha Blog, 2024.

Ding, F., Hardt, M., Miller, J., and Schmidt, L. Retiring Adult: New Datasets for Fair machine Learning. *Advances in Neural Information Processing Systems* 34, 34, 2021.

Dohmatob, E., Tarres, P., and Thomas, V. Strong Model Collapse from Weak Synthetic Data. *arXiv preprint arXiv:2410.04840*, 2024.

Duchi, J. C., Glynn, P. W., and Namkoong, H. Statistics Of Robust Optimization: A Generalized Empirical Likelihood Approach. *Mathematics of Operations Research*, 46:946–969, 2021.

Friedman, J. H. Greedy Function Approximation: A Gradient Boosting Machine. *Annals of statistics*, pp. 1189–1232, 2001.

Friedman, J. H. Stochastic Gradient Boosting. *Computational statistics & data analysis*, 38(4):367–378, 2002.

Gardner, J., Popovic, Z., and Schmidt, L. Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation. *Advances in Neural Information Processing Systems*, 35:9939–9954, 2022.

Gardner, J., Popovic, M., and Schmidt, M. TableShift: Quantifying Distribution Shift in Tabular Data. *arXiv preprint arXiv:2306.11638*, 2023.

Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting Deep Learning Models For Tabular Data. *Advances in Neural Information Processing Systems*, 34: 18932–18943, 2021.

Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., and Sontag, D. TabLLM: Few-shot Classification Of Tabular Data With Large Language Models. In *International Conference on Artificial Intelligence and Statistics*, pp. 5549–5581. PMLR, 2023.

Hendrycks, D. and Dietterich, T. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In *iclr19*, 2019.
Herman, L., Geirhos, R., Tversky, A., Bowers, J., and Firestone, C. Language Models Can Learn Spurious Correlations Like Humans. *Nature*, 629:769-774, 2024. doi: 10.1038/s41586-024-07566-y.

Hu, Z. and Hong, L. J. Kullback-Leibler Divergence Constrained Distributionally Robust Optimization. *Optimization Online*, 1(2):9, 2013.

Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tabtransformer: Tabular Data Modeling Using Contextual Embeddings. *arXiv preprint arXiv:2012.06678*, 2020.

Iwasawa, Y. and Matsuo, Y. Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization. In *Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), *Advances in Neural Information Processing Systems*, volume 34, pp. 2427–2440. Curran Associates, Inc., 2021.

Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G. MIMIC-III, A Freely Accessible Critical Care Database. *Scientific data*, 3:160035, 2016.

Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well-Tuned Simple Nets Excel On Tabular Datasets. *Advances in neural information processing systems*, 34: 23928–23941, 2021.

Katzir, L., Elidan, G., and El-Yaniv, R. Net-Dnf: Effective Deep Modeling Of Tabular Data. In *International conference on learning representations*, 2020.

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. *Advances in neural information processing systems*, 30, 2017.

Kim, E., Sun, M., Baek, C., Raghunathan, A., and Kolter, J. Z. Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line. In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.

Koyama, M. and Yamaguchi, S. When Is Invariance Useful In An Out-Of-Distribution Generalization Problem? *arXiv preprint arXiv:2008.01883*, 2020.

Li, S., Cai, T. T., and Li, H. Transfer Learning For High-Dimensional Linear Regression: Prediction, Estimation And Minimax Optimality. *Journal of the Royal Statistical Society Series B: Statistical Methodology*, 84(1):149–173, 2022.

Liang, J., He, R., and Tan, T. A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts. *International Journal Of Computer Vision*, 2023.

Liu, J., Wang, T., Cui, P., and Namkoong, H. On the Need for a Language Describing Distribution Shifts: Illustrations on Tabular Datasets. *Advances in Neural Information Processing Systems*, 36, 2023.

Meng, R., Liu, Y., et al. SFR-Embedding-Mistral: Enhance Text Retrieval with Transfer Learning. *Salesforce Blog*, 2024.

Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the Line: on the Strong Correlation Between Out-of-distribution and In-distribution Generalization. In *Proceedings of the 38th International Conference on Machine Learning*, 2021.

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. In *Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pp. 2014–2037, 2023.

Peters, J., Bühlmann, P., and Meinshausen, N. Causal Inference By Using Invariant Prediction: Identification And Confidence Intervals. *Journal of the Royal Statistical Society Series B: Statistical Methodology*, 78(5):947–1012, 2016.

Shwartz-Ziv, R. and Armon, A. Tabular Data: Deep Learning Is Not All You Need. *Information Fusion*, 81:84–90, 2022.

Sirignano, J. and Cont, R. Universal features of price formation in financial markets: perspectives from deep learning. *Quantitative Finance*, 19(9):1449–1459, 2019.

Slack, D. and Singh, S. Tablet: Learning from Instructions for Tabular Data. *arXiv preprint arXiv:2304.13188*, 2023.

Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring Robustness To Natural Distribution Shifts In Image Classification. In *Advances in Neural Information Processing Systems 20*, 2020.

Tian, Y. and Feng, Y. Transfer Learning Under High-Dimensional Generalized Linear Models. *Journal of the American Statistical Association*, 118(544):2684–2697, 2023.

Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully Test-Time Adaptation by Entropy Minimization. In *International Conference on Learning Representations*, 2021.

Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving Text Embeddings with Large Language Models. *arXiv preprint arXiv:2401.00368*, 2024.
Zeng, Y., Liu, J., Lam, H., and Namkoong, H. LLM Em-
beddings Improve Test-time Adaptation to Tabular Y|X-
Shifts. In *NeurIPS 2024 Third Table Representation
Learning Workshop*, 2024.

Zhou, Z., Yu, K.-Y., Guo, L.-Z., and Li, Y.-F. Fully Test-Time Adaptation for Tabular Data. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 23027–23035, 2025.

Zhu, P., Dai, X., Yao, Z., Dou, D., and Yang, D. ToEdit: Token-Level Editing Prevents Non-Iterative Model Collapse on Synthetic Data. *arXiv preprint* arXiv:2412.14689, 2024.
# A. Appendix

## A.1. Auxiliary findings

This subsection consists of supporting auxiliary findings that are mentioned in the main part of this paper but whose details are delegated due to space constraints.

### A.1.1. FRACTIONBEST METRIC FOR DIFFERENT LLMs

We examine the FractionBest metric introduced in (Zeng et al., 2024) (both with respect to accuracy and macro F1-score) to compare LLM|f for f one of our chosen model families, using four different encoders (e5, SFR, Linq and Zeta) against typical methods on tabular features. This will help us understand how different embedding models perform across all possible combinations of our distribution shift scenarios.

<figure><img src="image_10.png" /><figcaption>Figure 7. Fraction Best across all states with $\Delta = 0\%$.</figcaption></figure>

The results of FractionBest w.r.t. accuracy reveal striking differences in performance across encoders. All embedding based models outperform traditional methods like XGBoost (6%), while traditional methods like Logistic Regression (17%) remain competitive. The results of FractionBest w.r.t. macro F1-score consolidate Zeta as the best performing LLM in terms of providing good quality text embeddings while Linq shows to fall below all embedding-based methods and even tabular training methods such as LR and XGBoost.

### A.1.2. MODEL COLLAPSE ON THE CA DOMAIN FOR DIFFERENT LLM EMBEDDINGS

Figure 8 shows the pattern behavior of different LLMs embeddings of the same state CA for the ACL and the F1L planes. In this experiment, we choose a test set from the state of California (CA) as our source domain, we plot on the x-axis accuracy and macro F1-score for the e5-Mistral-7B-Instruct embeddings and in y-axis we have the same metrics but for the CA embeddings obtained by the other 3 LLMs (namely Linq, SFR, and Zeta).

### A.1.3. ACCURACY AND F1 ON-THE-LINE DO NOT HOLD FOR REPRESENTATION TRAINING

Figure 9 shows the different patterns behaviors of the embeddings stemming from the four models for the pair (California (CA), Arizona (AZ)) in contrast to the behavior of a standard MLP trained on the same pair. In this setting, we trained the MLP on the embeddings of the source (CA) for the four LLMs and evaluated its performance embeddings of the target (AZ). We report as before for the two phenomena: accuracy on the line and F1 on the line.

### A.1.4. CONFIGURATIONS INDUCING COLLAPSE ON ALL STATES (FOR FIXED BASE LLM ENCODER)

Table 3 displays the different configurations we have selected for the MLP that lead to a systematic collapse across all the LLM-based embeddings.

For each configuration, the MLP was trained on the embeddings produced by the LLMs on the source CA then tested on the
<figure><img src="image_11.png" /><figcaption>Figure 8. Pattern behavior across different LLMs Embeddings for CA on the MLP HP Configuration Set.</figcaption></figure>

other states; for instance, for cfg-1, the MLP with a hidden size of 16 was trained for 200 epochs on the embeddings of CA with a $lr = 0.01$ and a dropout ratio of 0.1 then was tested on the embeddings of the remaining states.

<table><caption>Table 3. Shared MLP configurations that systematically lead to collapse across all LLM-based embeddings.</caption><thead><tr><th>Collapse Configs</th><th>cfg-1</th><th>cfg-2</th><th>cfg-3</th><th>cfg-4</th><th>cfg-5</th><th>cfg-6</th><th>cfg-7</th></tr></thead><tbody><tr><td>lr</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td></tr><tr><td>hidden_size</td><td>16</td><td>16</td><td>32</td><td>32</td><td>64</td><td>128</td><td>128</td></tr><tr><td>dropout_ratio</td><td>0.1</td><td>0.1</td><td>0.1</td><td>0.1</td><td>0.1</td><td>0.1</td><td>0.1</td></tr><tr><td>train_epochs</td><td>200</td><td>100</td><td>200</td><td>50</td><td>200</td><td>200</td><td>100</td></tr></tbody></table>

## A.2. Detailed Model Collapse Metrics

Let $S$ and $Q$ be two test-sets. We define the pair-wise strong collapse ratio as the ratio of hyper-parameter combinations resulting into model collapse both test-sets

$$ \mathrm{CR}_{\mathrm{strong}, \mathrm{HP}}^{S,Q} = \frac{1}{|\mathrm{HP}|} \sum_{c} \mathbb{I}_{\{\mathrm{Cond}_{K_1}^S(c) \cap \mathrm{Cond}_{K_2}^Q(c)\}}, \quad \text{for } K_1, K_2 \in \{T, F\} $$

$$ = \frac{1}{|\mathrm{HP}|} \sum_{c} \mathbb{I}_{\{\mathrm{Cond}_{T}^{S}(c) \cap \mathrm{Cond}_{T}^{Q}(c)\}} + \mathbb{I}_{\{\mathrm{Cond}_{F}^{S}(c) \cap \mathrm{Cond}_{F}^{Q}(c)\}} + \mathbb{I}_{\{\mathrm{Cond}_{T}^{S}(c) \cap \mathrm{Cond}_{F}^{Q}(c)\}} + \mathbb{I}_{\{\mathrm{Cond}_{F}^{S}(c) \cap \mathrm{Cond}_{T}^{Q}(c)\}} $$

Let us denote TNR$^S(f_c)$ (resp. FNR$^S(f_c)$) for the true negative (resp. false negative) rate of $f_c$ on $S$.
<figure><img src="image_12.png" /><figcaption>Figure 9. Pattern behavior across different LLMs for (CA,AZ). ACL and F1L are valid in tabular training but do not transfer into valid ACL or F1L in LLM-based representation training.</figcaption></figure>

Let $\epsilon > 0$. We define the positive $\epsilon$-near collapse ratio of HP w.r.t. $S$ is defined as

$$PCR_{HP}^{S}(\epsilon) = \frac{1}{|HP|} \sum_{c} \mathbb{I}_{\{\text{Cond}_{T}^{S}(c,\epsilon)\}}, \text{ where}$$

$\text{Cond}_{T}^{S}(c, \epsilon) = [(\mathcal{A}^{S}(f_c) \approx \text{TR}^{S}) \cap (\text{TNR}^{S}(f_c) + \text{FNR}^{S}(f_c) < \epsilon)]$. Here the $\approx$ symbol refers to the resulting approximation of $\text{TR}^{S}$ depending on the $\text{TNR}^{S}(f_c)$ and $\text{FNR}^{S}(f_c)$ values. The value of $\epsilon$ varies from one experiment to the next, and in particular should be chosen with consideration to the overall shape of the ID vs. OOD scatter of $f_c$ performance variations for all $c \in \text{HP}$. The definition of negative $\epsilon$-near collapse ratio is analogously defined. Finally, let $\epsilon = (\epsilon_1, \epsilon_2)$. The strong positive $\epsilon$-near collapse ratio of HP w.r.t. $S$ is defined as

$$PCR_{s,HP}^{S,Q}(\epsilon) = \frac{1}{|HP|} \sum_{c} \mathbb{I}_{\{\text{Cond}_{T}^{S}(c,\epsilon_1) \cap \text{Cond}_{T}^{Q}(c,\epsilon_2)\}}.$$

The definition of the remaining combinations of the above metrics are not used in this article, their formalization is left to the reader.

<table><caption>Table 4. Selection of US States, with their respective test-set size and class balance.</caption><thead><tr><th>State</th><th>Test-set size</th><th>cl(&lt; 50k) size</th><th>cl(&gt; 50k) size</th><th>cl(&lt; 50k)%</th><th>cl(&gt; 50k)%</th></tr></thead><tbody><tr><td>CA</td><td>50000</td><td>31011</td><td>18989</td><td>62.022%</td><td>37.978%</td></tr><tr><td>PR</td><td>9071</td><td>8109</td><td>962</td><td>89.395%</td><td>10.605%</td></tr><tr><td>AZ</td><td>33277</td><td>21977</td><td>11300</td><td>66.043%</td><td>33.957%</td></tr><tr><td>AR</td><td>13929</td><td>10196</td><td>3733</td><td>73.200%</td><td>26.800%</td></tr><tr><td>AK</td><td>3546</td><td>2175</td><td>1371</td><td>61.337%</td><td>38.663%</td></tr><tr><td>AL</td><td>22268</td><td>15344</td><td>6924</td><td>68.906%</td><td>31.094%</td></tr></tbody></table>

Let us exemplify the implementation of these metrics on some basic examples. First of all, in Table 4, we report the population statistics of the test sets in our experiments. These will be the base reference to compute our model collapse metrics.
**Detailed metric computation for Figure 2b.** Let $f$ be $\in \mathbb{E}|\text{CVaR} - \text{DRO}$, $S$ be $\in \mathbb{E}|\text{CA}_{\text{test}}$ and $Q$ be $\in \mathbb{E}|\text{PR}_{\text{test}}$. We have $\text{HP}_{\in \mathbb{E}|\text{CVaR}-\text{DRO}} = 203$. There are 123 configurations $c \in \text{HP}_{\in \mathbb{E}|\text{CVaR}-\text{DRO}}$ satisfying $\text{PN}^S(f_c) = 0$ and there are two configurations inducing near collapses. As such, we have

$$ \text{PCR}_{s, \text{HP}}^{S,Q}(\varepsilon) = \text{PCR}_{s, \text{HP}_{\in \mathbb{E}|\text{CVaR}-\text{DRO}}}^{\in \mathbb{E}|\text{CA}_{\text{test}}, \in \mathbb{E}|\text{PR}_{\text{test}}}(0.1, 0.1) = \frac{125}{203} = 61.58\% $$

An important subtle point about our metrics for near collapse is that the underlying configurations that induce them might contribute to different metrics and need to be handled with care. For instance, if we denote $c_1$ and $c_2$ for the above near collapse configurations, we have,

$$ \begin{aligned} \text{PN}^S(f_{c_1}) &= 0 \quad ; \quad \text{PN}^Q(f_{c_1}) = 1 \\ \text{PN}^S(f_{c_2}) &= 2 \quad ; \quad \text{PN}^Q(f_{c_2}) = 8 \\ \text{TNR}^S(f_{c_1}) + \text{FNR}^S(f_{c_1}) &= 0.00187 < \varepsilon = 0.1 \\ \text{TNR}^S(f_{c_2}) + \text{FNR}^S(f_{c_2}) &= 0.02314 < \varepsilon = 0.1 \end{aligned} $$

In particular, configuration $c_1 \in \text{HP}$ induces a model $f_{c_1}$ that is both a strong positive $\varepsilon$-near collapse with respect to $S$ and $Q$ (i.e. it contributes to $\text{PCR}_{s, \text{HP}}^{S,Q}(\varepsilon)$) but is also a projection (strict) collapse with respect to $S$ (i.e. it contributes to $\text{CR}_{\text{HP}}^S$).

**Detailed metric computation for Figure 2c.** Let $f$ be $\text{Linq}|\text{CVaR} - \text{DRO}$, $S$ be $\text{Linq}|\text{CA}_{\text{test}}$ and $Q$ be $\text{Linq}|\text{AL}_{\text{test}}$. We have $\text{HP}_{\text{Linq}|\text{CVaR}-\text{DRO}} = 203$, and in this particular case, there are no near collapses but only strict collapses:

$$ \begin{aligned} \text{PCR}_{\text{HP}}^{S} &= \text{PCR}_{\text{HP}}^{S}(\varepsilon) = \text{PCR}_{\text{HP}_{\text{Linq}|\text{CVaR}-\text{DRO}}}^{\text{Linq}|\text{CA}_{\text{test}}}(0.1) &= \frac{61}{203} \approx 30.05\% \\ \text{PCR}_{\text{HP}}^{Q} &= \text{PCR}_{\text{HP}}^{Q}(\varepsilon) = \text{PCR}_{\text{HP}_{\text{Linq}|\text{CVaR}-\text{DRO}}}^{\text{Linq}|\text{AL}_{\text{test}}}(0.1) &= \frac{65}{203} \approx 32.02\% \\ \text{PCR}_{s, \text{HP}}^{S,Q}(\varepsilon) &= \text{PCR}_{s, \text{HP}_{\text{Linq}|\text{CVaR}-\text{DRO}}}^{\text{Linq}|\text{CA}_{\text{test}}, \text{Linq}|\text{AL}_{\text{test}}}(\varepsilon) &= \frac{37}{203} \approx 18.23\% \\ \text{CR}_{p, \text{HP}}^{S,Q}(\varepsilon) &= \text{PCR}_{p, \text{HP}_{\text{Linq}|\text{CVaR}-\text{DRO}}}^{\text{Linq}|\text{CA}_{\text{test}}, \text{Linq}|\text{AL}_{\text{test}}}(\varepsilon) &= \frac{89}{203} \approx 43.84\% \end{aligned} $$

**On models which are not near collapses but have an accuracy close to a class ratio.** In many of the experiments of this paper, there are examples of points that seem to be near collapses by the fact that their accuracy is very close to a certain class ratio but where the sum $\text{TNR}^S(f_c) + \text{FNR}^S(f_c)$ is far bigger than 0.1.

Next, the following table reports the number of non zero predicted negatives corresponding to results in Figure 6b.

<table><caption>Table 5. True Negatives (TN) and False Negatives (FN) by State for Optimal $\in \mathbb{E}|\text{HP}$. We only report for LLMs inducing non-zero Predicted Negatives.</caption><thead><tr><th rowspan="2">State</th><th colspan="2">SFR</th><th colspan="2">Zeta</th></tr><tr><th>TN</th><th>FN</th><th>TN</th><th>FN</th></tr></thead><tbody><tr><td>CA</td><td>21</td><td>1</td><td>-</td><td>-</td></tr><tr><td>AK</td><td>179</td><td>27</td><td>9</td><td>0</td></tr><tr><td>AR</td><td>1</td><td>0</td><td>-</td><td>-</td></tr><tr><td>AL</td><td>107</td><td>7</td><td>-</td><td>-</td></tr><tr><td>AZ</td><td>2</td><td>0</td><td>-</td><td>-</td></tr></tbody></table>

Together with the class statistics from Table 4, we readily check that, for $f_c$ the MLP trained over the optimal configuration $\text{lr}=0.001$, $\text{hidden_size}=32$, $\text{dropout_ratio}=0$, and $\text{train_epochs}=500$ only on $\in \mathbb{E}|\text{5}$ embeddings of CA, and tested on the rest of embeddings of all states, we have:

* $\text{TNR}(f_c, \text{SFR}|\text{AL}) + \text{FNR}(f_c, \text{SFR}|\text{AL}) = 0,0004562 + 0,0154535 < \varepsilon$;

* $\text{TNR}(f_c, \text{Zeta}|\text{AK}) + \text{FNR}(f_c, \text{Zeta}|\text{AK}) = 0 + 0,00656455 < \varepsilon$;

* $\text{TNR}(f_c, \text{SFR}|\text{AK}) + \text{FNR}(f_c, \text{SFR}|\text{AK}) = 0,01241379 + 0,13056163 > \varepsilon$.
### A.3. Shared Hyperparameter Configuration Sets for Experiment 4.1

For each model, we report the associated set of hyperparameters as shown in Table 6, along the same logic from the HP grids reported in (Liu et al., 2023) and (Zeng et al., 2024).

<table><caption>Table 6. Sets of shared hyper-parameter combinations used in all experiments. ° for methods with the total grid size above 200, we randomly sample 200 configurations for fair comparisons. The column "Well-defined" refers to empirical observation that the corresponding HP sets do not collapse for the tabular modality in our setting.</caption><thead><tr><th>Model</th><th># of HPs</th><th>Well-Defined</th><th>Hyperparameter</th><th>Value Range</th></tr></thead><tbody><tr><td rowspan="3">SVM</td><td rowspan="3">96</td><td rowspan="3">Yes</td><td>C</td><td>{1e-2, 1e-1, 1, 1e1, 1e2, 1e3}</td></tr><tr><td>Kernel</td><td>{linear, RBF}</td></tr><tr><td>Loss</td><td>Squared-Hinge</td></tr><tr><td></td><td></td><td></td><td>γ</td><td>{0.1, 0.3, 0.5, 1, 1.5, 2, scale, auto}</td></tr><tr><td rowspan="2">LR</td><td rowspan="2">100</td><td rowspan="2">Yes</td><td rowspan="2">L<sub>2</sub> penalty</td><td>{1e-4, 2e-4, ..., 1e-3, 1.5e-3, ..., 1e-2, 1.5e-2, ..., 1e-1, 1.5e-1, ..., 1, 1.3, 1.5, 1.7, 2, 2.5, ..., 5, 6, 1e1, 1.5e1, 2e1, 3e1, ..., 1e2, 2e2, 3e2, 5e2, 7e2, 1e3, 2e3, 3e3, 5e3, 1e4, 5e4}</td></tr><tr><td></td></tr><tr><td rowspan="4">MLP</td><td rowspan="4">96</td><td rowspan="4">Yes</td><td>Learning Rate</td><td>{0.001, 0.003, 0.005, 0.01}</td></tr><tr><td>Hidden Layer Dim</td><td>{16, 32, 64, 128}</td></tr><tr><td>Dropout Ratio</td><td>{0, 0.1}</td></tr><tr><td>Train Epoch</td><td>{50, 100, 200}</td></tr><tr><td rowspan="4">GBM</td><td rowspan="4">1680°</td><td rowspan="4">Yes</td><td>Learning Rate</td><td>{1e-2, 1e-1, 5e-1, 1}</td></tr><tr><td>Num. Estimators</td><td>{32, 64, 128, 256}</td></tr><tr><td>Max Depth</td><td>{2, 4, 8, 16}</td></tr><tr><td>Min. Child Samples</td><td>{1, 2, 4, 8}</td></tr><tr><td rowspan="6">XGB</td><td rowspan="6">1944°</td><td rowspan="6">No</td><td>Learning Rate</td><td>{0.1, 0.3, 1, 2}</td></tr><tr><td>Min. Split Loss</td><td>{0, 0.1, 0.5}</td></tr><tr><td>Max Depth</td><td>{4, 6, 8}</td></tr><tr><td>Column Subsample (tree)</td><td>{0.7, 0.9, 1}</td></tr><tr><td>Column Subsample (level)</td><td>{0.7, 0.9, 1}</td></tr><tr><td>Max Bins</td><td>{128, 256, 512}</td></tr><tr><td></td><td></td><td></td><td>Growth Policy</td><td>{Depthwise, Loss Guide}</td></tr><tr><td rowspan="5">RF</td><td rowspan="5">640°</td><td rowspan="5">No</td><td>Num. Estimators</td><td>{32, 64, 128, 256, 512}</td></tr><tr><td>Min. Samples Split</td><td>{2, 4, 8, 16}</td></tr><tr><td>Min. Samples Leaf</td><td>{1, 2, 4, 8}</td></tr><tr><td>Max. Features</td><td>{sqrt, log2}</td></tr><tr><td>CCP Alpha</td><td>{0, 0.001, 0.01, 0.1}</td></tr><tr><td rowspan="2">CVaR-DRO</td><td rowspan="2">1620°</td><td rowspan="2">Yes</td><td>Worst-case Ratio α</td><td>{0.01, 0.1, 0.2, 0.3, 0.5, 1}</td></tr><tr><td>Underlying Model Class</td><td>MLP</td></tr></tbody></table>
## A.4. Supporting Tables for Experiment 4.2

### A.4.1. DETAILS FOR FIGURE 5

Table 7 presents the accuracy and F1 scores for LLM|MLP model evaluated under these parameters: lr=0.001, hidden_size=32, dropout_ratio=0, and train_epochs=500.

<table><caption>Table 7. Detailed Results for Figure 5.</caption><thead><tr><th rowspan="2">LLM\State</th><th colspan="6">Accuracy</th><th colspan="6">F1-macro</th></tr><tr><th>CA</th><th>PR</th><th>AL</th><th>AZ</th><th>AK</th><th>AR</th><th>CA</th><th>PR</th><th>AL</th><th>AZ</th><th>AK</th><th>AR</th></tr></thead><tbody><tr><td>e5</td><td>81.85</td><td>90.07</td><td>79.23</td><td>80.88</td><td>74.64</td><td>81.02</td><td>81.13</td><td>56.74</td><td>76.71</td><td>77.73</td><td>74.52</td><td>75.06</td></tr><tr><td>linq</td><td>78.71</td><td>66.59</td><td>76.04</td><td>72.29</td><td>74.25</td><td>71.44</td><td>77.60</td><td>55.54</td><td>73.96</td><td>71.63</td><td>73.84</td><td>68.75</td></tr><tr><td>sfr</td><td>82.41</td><td>86.49</td><td>80.68</td><td>76.70</td><td>77.15</td><td>80.15</td><td>80.95</td><td>69.80</td><td>76.16</td><td>68.44</td><td>76.18</td><td>74.28</td></tr><tr><td>zeta</td><td>82.84</td><td>88.00</td><td>80.40</td><td>77.38</td><td>76.90</td><td>80.11</td><td>81.61</td><td>69.64</td><td>75.68</td><td>69.84</td><td>76.29</td><td>68.16</td></tr></tbody></table>

### A.4.2. DETAILS FOR FIGURE 6B

<table><caption>Table 8. Accuracy of an MLP trained on CA and evaluated on each state with different LLM embeddings (Fig. 6b).</caption><thead><tr><th>LLM\State</th><th>CA</th><th>PR</th><th>AL</th><th>AZ</th><th>AK</th><th>AR</th></tr></thead><tbody><tr><td>e5</td><td>81.85</td><td>90.07</td><td>79.23</td><td>80.88</td><td>74.64</td><td>81.02</td></tr><tr><td>linq</td><td>62.02</td><td>89.39</td><td>68.90</td><td>66.042</td><td>61.33</td><td>73.19</td></tr><tr><td>sfr</td><td>62.06</td><td>89.39</td><td>69.35</td><td>66.048</td><td>65.62</td><td>73.20</td></tr><tr><td>zeta</td><td>62.02</td><td>89.39</td><td>68.90</td><td>66.042</td><td>61.59</td><td>73.19</td></tr></tbody></table>

## A.5. Supplementary Figures for Experiment 4.1

### Target State: Puerto Rico

<figure><img src="image_13.png" /><figcaption>Figure 10. Pattern behaviour across different LLMs for MLP.</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Figure 11. Pattern behaviour across different LLMs for CVaR-DRO.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 12. Pattern behaviour across different LLMs for LR.</figcaption></figure>
<figure><img src="image_16.png" /><figcaption>Figure 13. Pattern behaviour across different LLMs for RF.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 14. Pattern behaviour across different LLMs for SVM.</figcaption></figure>
<figure><img src="image_18.png" /><figcaption>Figure 15. Pattern behaviour across different LLMs for GBM.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 16. Pattern behaviour across different LLMs for XGB.</figcaption></figure>
Target State: Alabama

<figure><img src="image_20.png" /><figcaption>Figure 17. Pattern behaviour across different LLMs for MLP.</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Figure 18. Pattern behaviour across different LLMs for CVaR-DRO.</figcaption></figure>
<figure><img src="image_22.png" /><figcaption>Figure 19. Pattern behaviour across different LLMs for LR.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 20. Pattern behaviour across different LLMs for RF.</figcaption></figure>
<figure><img src="image_24.png" /><figcaption>Figure 21. Pattern behaviour across different LLMs for SVM.</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Figure 22. Pattern behaviour across different LLMs for GBM.</figcaption></figure>
<figure><img src="image_26.png" /><figcaption>Figure 23. Pattern behaviour across different LLMs for XGB.</figcaption></figure>
Target State: Arizona

<figure><img src="image_27.png" /><figcaption>Figure 24. Pattern behaviour across different LLMs for MLP.</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 25. Pattern behaviour across different LLMs for CVaR-DRO.</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>Figure 26. Pattern behaviour across different LLMs for LR.</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Figure 27. Pattern behaviour across different LLMs for RF.</figcaption></figure>
<figure><img src="image_31.png" /><figcaption>Figure 28. Pattern behaviour across different LLMs for SVM.</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>Figure 29. Pattern behaviour across different LLMs for GBM.</figcaption></figure>
<figure><img src="image_33.png" /><figcaption>Figure 30. Pattern behaviour across different LLMs for XGB.</figcaption></figure>
Target State: Alaska

<figure><img src="image_34.png" /><figcaption>Figure 31. Pattern behaviour across different LLMs for MLP.</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>Figure 32. Pattern behaviour across different LLMs for CVaR-DRO.</figcaption></figure>
<figure><img src="image_36.png" /><figcaption>Figure 33. Pattern behaviour across different LLMs for LR.</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>Figure 34. Pattern behaviour across different LLMs for RF.</figcaption></figure>
<figure><img src="image_38.png" /><figcaption>Figure 35. Pattern behaviour across different LLMs for SVM.</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Figure 36. Pattern behaviour across different LLMs for GBM.</figcaption></figure>
<figure><img src="image_40.png" /><figcaption>Figure 37. Pattern behaviour across different LLMs for XGB.</figcaption></figure>
Target State: Arkansas

<figure><img src="image_41.png" /><figcaption>Figure 38. Pattern behaviour across different LLMs for MLP.</figcaption></figure>

<figure><img src="image_42.png" /><figcaption>Figure 39. Pattern behaviour across different LLMs for CVaR-DRO.</figcaption></figure>
<figure><img src="image_43.png" /><figcaption>Figure 40. Pattern behaviour across different LLMs for LR.</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>Figure 41. Pattern behaviour across different LLMs for RF.</figcaption></figure>
<figure><img src="image_45.png" /><figcaption>Figure 42. Pattern behaviour across different LLMs for SVM.</figcaption></figure>

<figure><img src="image_46.png" /><figcaption>Figure 43. Pattern behaviour across different LLMs for GBM.</figcaption></figure>
<figure><img src="image_47.png" /><figcaption>Figure 44. Pattern behaviour across different LLMs for XGB.</figcaption></figure>