Causes and Consequences of Representational Similarity in Machine Learning Models

## Abstract

Abstract Numerous works have noted similarities in how machine learning models represent the world, even across modalities. Although much effort has been devoted to uncovering properties and metrics on which these models align, surprisingly little work has explored causes of this similarity. To advance this line of inquiry, this work explores how two factors—dataset overlap and task overlap—influence downstream model similarity. We evaluate the effects of both factors through experiments across model sizes and modalities, from small classifiers to large language models. We find that both task and dataset overlap cause higher representational similarity and that combining them provides the strongest effect. Finally, we consider downstream consequences of representational similarity, demonstrating how greater similarity increases vulnerability to transferable adversarial and jailbreak attacks.

## 1 Introduction

A large body of work has considered ways in which AI model representations—models’ mathematical depictions of real-world inputs—align (e.g.  Sucholutsky et al. (2024); Kornblith et al. (2019); Klabunde et al. (2023); Raghu et al. (2017); Morcos et al. (2018); Laakso & Cottrell (2000); Nguyen et al. (2021); Dravid et al. (2023); Zou et al. (2023), among many others). Although much of this work has focused on neural networks, recent work found evidence of “feature universality” (a.k.a. representational alignment) across generative models like large language models (LLMs) (Huh et al., 2024). Although representational alignment across AI models is well-documented and novel methods for measuring it are developed frequently, only limited work works has investigated its root causes. Understanding the causes and consequences of this phenomenon may enable more principled interactions with the AI models now pervasive in critical arenas like healthcare and national security (Wang et al., 2022; Caballero & Jenkins, 2025; Ma et al., 2024; Steinberg et al., 2021).

Huh et al. (2024) postulated that task alignment among large-scale AI models will increase their similarity, since as models grow larger they will perform more tasks, and this set of tasks will eventually converge. While compelling, this argument lacks empirical evidence. Another, yet-unexplored factor could also drive model alignment: dataset overlap. Many of today’s generative AI models are trained on huge swaths of the internet, since AI “scaling laws” suggest that training on more data produces a better model (Kaplan et al., 2020).
While cheap to acquire, internet-sourced model training datasets may overlap significantly. For example, many models are trained on subsets of Common Crawl, Reddit, and Wikipedia (Naveed et al., 2025; Bommasani et al., 2022). This overlap could drive downstream model similarity.

Since limited work explores causes of representational similarity, these two competing hypotheses for model alignment remain unexplored. The task alignment hypothesis of Huh et al. (2024) is only ever argued-for theoretically, so no experiments exist. Similarly, the dataset overlap hypothesis not been explored, since the AI research community focuses more on issues of privacy and copyright in large-scale internet data use, not downstream effects on model behaviors.

Our contribution. We propose empirical methods to explore of causes of representational similarity. Building on hypotheses of Huh et al. (2024) and known realities of model training, we explore the relative effects of dataset overlap and task overlap on downstream representation similarity. To do this, we propose two novel methods, *dataset splitting* and *task splitting*, that generate datasets with controlled overlap at both the data point and task level, allowing us to quantify how overlap on these properties affects representation similarity. Along the way, we create a new dataset, called ColorShapeDigit800K, to train models with nuanced task differences. We run dataset and task splitting experiments on models ranging from classifiers to small language models and find that:

- •
Both dataset and task overlap are positively correlated with increased representational similarity in models. We observe this trend in nearly all models and datasets we evaluate, from image classifiers to diffusion to text generation models.
- •
Increasing dataset and task overlap together induces strongest representational alignment between models, as measured by mutual information. We compare the mutual information between different overlap types and measured representational similarity in models.
- •
Higher dataset/task overlap cause higher vulnerability to transfer adversarial and jailbreaking attacks, indicating a possible downstream consequence of unexamined model similarity.

The rest of the paper proceeds as follows. §[2](https://arxiv.org/html/2505.13899v2#S2) situates our study in the broader landscape of representational similarity. §[3](https://arxiv.org/html/2505.13899v2#S3) motivates the two causal factors we explore—dataset and task overlap. §[4](https://arxiv.org/html/2505.13899v2#S4) provides details on our experimental setup. §[5](https://arxiv.org/html/2505.13899v2#S5) and §[6](https://arxiv.org/html/2505.13899v2#S6) present key findings. §[7](https://arxiv.org/html/2505.13899v2#S7) discusses limitations and broader impacts of our work. Please see code(^2^22[https://anonymous.4open.science/r/ReprSimCauses-5D26](https://anonymous.4open.science/r/ReprSimCauses-5D26)) and dataset ColorShapeDigit800K(^3^33[https://drive.google.com/file/d/1N1xWKNWm3rD-a7EOmMwQn86pyus8Xn0P/view?usp=share_link](https://drive.google.com/file/d/1N1xWKNWm3rD-a7EOmMwQn86pyus8Xn0P/view?usp=share_link)).

## 2 Related Work

Observations of model similarity. Numerous prior works have demonstrated that AI models, even those trained for different tasks, can exhibit similar properties, either in their outputs (functional similarity) or their internal behaviors (representational similarity, the focus of this work) (Mehrer et al., 2020; Nguyen et al., 2021; Räuker et al., 2023; Kornblith et al., 2019; Li et al., 2015; Hermann & Lampinen, 2020; Sucholutsky et al., 2024). Prior work on representational similarity has considered similarities in neuron activation patterns and feature space representations of inputs (Li et al., 2015; Hermann & Lampinen, 2020; Nguyen et al., 2021; Dravid et al., 2023; Gurnee et al., 2024), and numerous metrics have been proposed to measure similarities between models (Kornblith et al., 2019; Klabunde et al., 2025; Huh et al., 2024; Morcos et al., 2018; Geirhos et al., 2020; Hacohen et al., 2020; Bansal et al., 2021; Moschella et al., 2023; Hermann & Lampinen, 2020). Extensions of this phenomenon, including alignment between AI model and human representations of the world, is an ongoing area of research (see Sucholutsky et al. (2024) for a detailed overview).

Causes of model similarity. Prior work has explored factors affecting similarity of computer vision models, such as increases in model and dataset size (Kornblith et al., 2019; Roeder et al., 2021; Huh et al., 2024), overall training objectives (Ciernik et al., 2025; Lindsay et al., 2022), multi-modal task generalization (Huh et al., 2024), and model initialization (Mehrer et al., 2020). The main findings show that representational alignment scales with model and dataset size and that differences in training objectives and tasks cause divergent model representations (Sucholutsky et al., 2024). However, these factors have only been explored at a high level, and few efforts have been made to untangle direct causes of representational similarity in models. This motivates our work, which empirically studies how two possible causal factors—dataset overlap and task overlap—affect representational similarity.

Downstream effects of model similarity. Beyond curiosity about causes of model similarity, our work is motivated by concerns about negative effects of unexpected model similarity. Sometimes, similarity is desired and explicitly constructed—e.g. via techniques like model distillation and transfer learning that explicitly pass knowledge from one model to another to expedite learning (Phuong & Lampert, 2019; Cho & Hariharan, 2019; Sucholutsky et al., 2024; Roth et al., 2024; DeepSeek-AI et al., 2025). However, representation alignment between models may not always be desirable. A significant body of work has highlighted the risk of “transferable” adversarial attacks, in which a malicious input designed to mislead one model can also mislead a different model with similar feature representations (Zou et al., 2023; Shan et al., 2020; Demontis et al., 2019; Carlini & Wagner, 2017). Finally, Wenger & Kenett (2025) showed that large language models exhibit homogeneous behaviors on creative tasks. If independently trained models converge to similar representations, this could cause pervasive bias or consistent errors across models. This motivates our §[6](https://arxiv.org/html/2505.13899v2#S6) investigation of possible consequences from representational similarity.

## 3 Setup and Intuition

In this work, we consider two models, $f_{\hat{\theta}_{1}}$ and $f_{\hat{\theta}_{2}}$, trained on datasets $D_{1}$ and $D_{2}$ respectively. Each dataset consists of samples drawn from an unknown true data distribution $\mathbb{D}$. Each model is optimized to minimize a loss function on samples from their respective datasets:

$$ $\hat{\theta}_{i}=\arg\min_{\theta}\mathbb{E}_{(z,y)\in D_{i}}\left[\mathcal{L}(h_{\theta}(f_{\theta}(z)),y)\right]$ (1) $$

where $\mathcal{L}$ is the training objective. The model consists of $f$ and $h$ parameterized by $\hat{\theta}_{i}$, where $f$ is the backbone that generates a rich latent-space representation of input $z$ and $h$ is a small head module that transforms the latent vector into human-usable outputs, perhaps classification decisions or generated text. This work considers the output of $f$, where $f_{\hat{\theta}_{i}}:\mathbb{D}\rightarrow\mathbb{R}^{n}$ maps an input $z\in\mathbb{D}$ to an $n$-dimensional latent representation of input $z$. Given an input $z\in\mathbb{D}$ and model backbones $f_{\hat{\theta}_{1}}$ and $f_{\hat{\theta}_{2}}$, we can measure the similarity in the latent representations of $f_{\hat{\theta}_{1}}(z)$ and $f_{\hat{\theta}_{2}}(z)$. Figure [1](https://arxiv.org/html/2505.13899v2#S3.F1) provides an overview of this end-to-end framework, which forms the backbone of our methodology.

Figure: Figure 1: Overview of our experimental framework. Two models are trained on dataset splits with controlled dataset or task overlap, and we measure their representational similarity.
Refer to caption: x1.png

### 3.1 Dataset overlap

Our exploration of this factor is motivated by the reality that today’s large-scale AI models are trained on overlapping datasets. Due to ever-increasing demands for training data (Kaplan et al., 2020), model trainers often turn to internet scrapes as a rich and cheap source of data. From whitepapers, one can easily discern that the scraped training dataset of large models often overlap. For example, GPT (Brown et al., 2020), Jamba (Team et al., 2024), Llama (Touvron et al., 2023), PaLM (Chowdhery et al., 2023), and Phi (Abdin et al., 2023) are all trained on subsets of CommonCrawl (Crawl, 2025), while GPT, Llama, and PaLM are all trained on Wikipedia and Books datasets. This reality motivates our exploration of how dataset overlap drives downstream representational similarity.

Intuitively, a large amount of overlapping content in training datasets $D_{1}$ and $D_{2}$ (drawn from the same underlying data distribution $\mathbb{D}$) should lead to high similarity in models’ latent representations. Suppose two datasets overlap significantly,
i.e., $D_{1}\bigcap D_{2}\approx D_{1}\bigcup D_{2}$,
then solving [Equation˜1](https://arxiv.org/html/2505.13899v2#S3.E1) for $D_{1}$ and $D_{2}$ produces models optimized toward the similar input-label patterns in the two datasets. As a result, the backbones should output similar latent representations for the same data.
Prior work shows that models trained on disjoint datasets drawn from the same true distribution (e.g. faces) also have similar latent space representations (e.g.  (Shan et al., 2020)), bolstering this hypothesis.

### 3.2 Task overlap

Prior work speculates that overlap in model tasks, e.g. specific desired model behaviors, will increase downstream model similarity (Huh et al. (2024), Section 3.1). Intuitively, this makes sense: two models trained to answer science and math questions will likely be more similar than a model trained to answer science and math questions and a model trained for general-purpose text completion. However, the claims of Huh et al. (2024) are not explored empirically. This motivates our quantitative exploration of task overlap as a second potential cause of representational similarity.

In our work, we define a model task as a specific type of data that the model is trained to process within a fixed learning paradigm (e.g. supervised learning, self-supervised learning). For example, each class in the CIFAR-10 dataset constitutes a task; in a similar vein, next token prediction on the Shakespeare and Tiny-Codes datasets are two distinct tasks.
Our key insight in evaluating task overlap is that a single training data element can be used for multiple learning tasks based on how it is labeled. For example, in computer vision, an image containing both a digit and shape can be used to train one model to recognize digits and another to recognize shapes. Similarly, in language models, the same sequence of text can be used to generate text in different styles. Within the same learning paradigm, the underlying data remains the same, but the task partition determines the type of representation the model learns by assigning different objectives to the same or overlaping datapoints. We leverage this insight to design our task overlap experiments.

## 4 Methodology

To test the relative effects of dataset/task overlap on representational similarity, we trained numerous pairs of models with varying data and task overlap in their training datasets, following Figure [1](https://arxiv.org/html/2505.13899v2#S3.F1). This section describes our techniques to induce dataset and task overlap and methods for model training and evaluation. Figure [2](https://arxiv.org/html/2505.13899v2#S4.F2) gives an overview of our task and dataset splitting approaches.

### 4.1 Dataset and Task overlap

Figure: Figure 2: Our task and dataset splitting strategies. Task splitting organizes partitions (e.g. classes in image datasets or text data with a specific style) in the original dataset into two new datasets. Dataset splitting produces new datasets with certain amounts of per-class overlap.
Refer to caption: x2.png

To control task/dataset overlap, we first subdivide training datasets of interest $D_{1},D_{2}$ into partitions. Partitions are formed contextually based on the dataset domain. For example, in image classification, datasets are partitioned by classes, while in text generation, datasets are partitioned by text style (e.g. code generation vs. science question answering vs. conversational text completion).

Dataset overlap. To control dataset overlap, we fix a certain amount of partition-level data point overlap between models. Practically, this means we consider two datasets $D_{1}$ and $D_{2}$, each containing data points drawn from the same set of partitions (e.g. classes in image classification). Given a fixed overlap proportion $\alpha\in[0,1]$, we modify the data points in each partition such that for every partition $p$, a proportion $\alpha$ of the data points in $D_{1}$ and $D_{2}$ are shared (i.e., identical), and the remaining proportion $1-\alpha$ of data points are unique to each dataset.

Task overlap with varying dataset overlap. Controlling task overlap requires varying models’ training objective rather than the data points seen during training. As a baseline, we control task similarity between models trained on otherwise-identical datasets $D_{1}$ and $D_{2}$ by varying the number of overlapping partitions $p$ between the two datasets. Specifically, we split the data into partitions $P=\{p_{0},p_{1},\dots,p_{n}\}$, where each partition corresponds to a class (or, in the case of language datasets, tokens from different corpora). We construct $D_{1}$ and $D_{2}$ by selecting $K$ total partitions for each, with $\alpha K$ partitions shared between them and $(1-\alpha)K$ partitions unique to each.

Task splitting varies the training objective between datasets $D_{1}$ and $D_{2}$ by controlling the number of shared partitions ($\alpha$), unlike dataset splitting which shares data points directly. As $\alpha$ decreases, the models train on increasingly different subsets of the original dataset $\mathbb{D}$, potentially influencing their representation similarity. However, since dataset partitions contain distinct data points, increasing overlap in partitions also increases the amount of shared training data, making it hard to isolate the effect of task similarity from data overlap. To address this, the next section introduces a task splitting approach where partition overlap is varied but data overlap is constant.

Figure: Figure 3: Illustration of task partitioning. Datasets with varying levels of task overlap (they all share the same images). The task overlap progressively increases from left to right. Example images and the corresponding labels are shown for each dataset at a particular task overlap.
Refer to caption: x3.png

Task splitting with constant dataset overlap.
We construct a new dataset, ColorShapeDigit800K, which maintains a fixed set of data points while varying the task definition. This approach allows us to control training objective similarity simply by modifying dataset labels. ColorShapeDigit800K is constructed from two datasets: geometric shape images from rivaldo (2022) and digit images from Kapoor (2021). We choose these base datasets because both contain large collections of visually simple and distinct shapes and digits on clean backgrounds, which enable compositional task construction.

**Table 1: Composition of task partitions in ColorShapeDigit800K. The labels of $D_{1},D_{2}$ and the number of classes in $D_{1},D_{2}$ for each partition is provided as <info for $D_{1}$>/<info for $D_{2}$>. S, D, and C refer to shape, digit, and color, respectively. [Figure˜6](https://arxiv.org/html/2505.13899v2#S5.F6) combines partitions 2A, 2B, and 2C.**
| Partition | 1 | 2A | 2B | 2C | 3 |
| --- | --- | --- | --- | --- | --- |
| Labels | S/D | S+D/S+C | S+D/D+C | S+C/D+C | S+D+C/S+D+C |
| Number of classes | 8/10 | 80/80 | 80/100 | 80/100 | 800/800 |

For our experiments with this task overlap approach, we use identical data points in both $D_{1}$ and $D_{2}$ and progressively adjust the labelling scheme to define increasingly specific tasks (e.g., based on shape, digit, or color). This is illustrated in Figure [3](https://arxiv.org/html/2505.13899v2#S4.F3). As more attributes are incorporated into the task definition, the training objectives across $D_{1}$ and $D_{2}$ become more aligned, although the data points never change. This progression of the labelling scheme to consider more attributes is summarized in Table [1](https://arxiv.org/html/2505.13899v2#S4.T1), which shows how the complexity in the number of classes increases as more attributes are considered. When all data attributes are used in labelling, the models are trained on identical tasks. This approach allows us to examine how increasing task similarity without the confounding variable of increasing dataset overlap affects model representations.

### 4.2 Experimental setup

We run our task and dataset splitting experiments on a variety of settings, ranging from image classifiers to large language models to diffusion models.

Model training. To understand how our different overlap approaches affect representational similarity across modalities, we finetune image classifiers and LLMs, and train small language models and diffusion models from scratch. Table [2](https://arxiv.org/html/2505.13899v2#S4.T2) summarizes models and training datasets used in our experiments.
All ResNet/ViT (He et al., 2016; Dosovitskiy et al., 2021) models used cross entropy loss with label smoothing at 0.1. The small language models are a GPT-Style transformer (Karpathy, 2023) with optimizations such as mixed precision, gradient accumulation & clipping, and early stopping, with task overlap and/or data overlap varied across runs. The diffusion UNet (von Platen et al., 2022) is trained using the score matching loss. We LoRA finetuned (Hu et al., 2022) Llama-3.2-Instruct (Touvron et al., 2023) models on 16 million tokens. Refer to §[A.1.1](https://arxiv.org/html/2505.13899v2#A1.SS1.SSS1) for more details. Table [5](https://arxiv.org/html/2505.13899v2#A1.T5) and Figure [8](https://arxiv.org/html/2505.13899v2#A1.F8) list baseline model performance.

**Table 2: Models and datasets used in our experiments. ${}^{*}=$ ColorShapeDigit800K was only used to train ResNet/ViT models for task overlap experiments, since it is an image classification task; all other models/datasets were used for both task and dataset overlap experiments. We did not have enough resources to train ColorShapeDigit800K on Diffusion UNets, which are slow to converge.**
| Models | Training Data | Overlap Methods<br>Dataset Task |
| --- | --- | --- |
| ResNet/ViT | CIFAR-100, TinyImagenet, | ✓  ✓^∗ |
|  | ColorShapeDigit800K^∗ (ours) |  |
| UNet (diffusion) | CIFAR-10 | ✓  ✓ |
| nanoGPT and | Tiny-{Stories, Codes}, Shakespeare, | ✓  ✓ |
| Llama-3.2-Instruct | Tiny-{WebText, TextBooks}, WikiText |  |

Measuring representational similarity.
After training, we evaluate the downstream effects of the various overlap methods by measuring the representational similarity between pairs of models trained with a specific dataset or task overlap. Let $f_{\hat{\theta}}(z)$ denote the representation of an input $z$ for all $z\in\mathbb{D}$. We use a kernel matrix $\mathbf{K}:\mathbb{D}\times\mathbb{D}\rightarrow\mathbb{R}$ of size $n\times n$ to measure similarity between representations, where $\mathbf{K}(z_{i},z_{j})=\langle f_{\hat{\theta}}(z_{i}),f_{\hat{\theta}}(z_{j})\rangle$ and $\langle\cdot,\cdot\rangle$ is the inner product. Representation similarity between two models is often evaluated via kernel matrix comparison through a technique known as Centered Kernel Alignment (CKA) (Kornblith et al., 2019). Other common metrics include nearest-neighbor scores (Klabunde et al., 2025), mutual KNN (Huh et al., 2024), and SVCCA (Raghu et al., 2017). We use implementations of these from the codebase of Huh et al. (2024), and report CKA in our main results (§[5](https://arxiv.org/html/2505.13899v2#S5)) due to its widespread in both vision and language models (Huh et al., 2024; Klabunde et al., 2023). Additional metrics are evaluated in Appendix [A.2](https://arxiv.org/html/2505.13899v2#A1.SS2).

Splitting method implementation and evaluation. We implement dataset and task splitting differently for the different modalities we explore—see §[A.1](https://arxiv.org/html/2505.13899v2#A1.SS1) for details. For each setting, we train pairs of models with various task/dataset overlap across 4 different seeds and evaluate the representational similarity (using CKA) for model pairs on held-out validation sets from the initial training datasets.

## 5 Experimental Results

Overall, we find that increasing both task and dataset overlap drives higher representation alignment between models, and that combining them produces the strongest trend.

### 5.1 Effect of dataset overlap on representation similarity

Figure: (a) ResNets - TinyImageNet
Refer to caption: x4.png

As described previously, we control training data overlap while holding all other variables (dataset size, task, model architecture) constant. We report CKA scores between models versus models’ training dataset overlap in Figure [4](https://arxiv.org/html/2505.13899v2#S5.F4). As this figure shows, when datasets overlap more, the CKA score computed between model pairs steadily rises, even across different model types, sizes, and modalities. We fit a linear model to these result, using dataset overlap as the independent variable and CKA score as the dependent and report these results in Figure [4](https://arxiv.org/html/2505.13899v2#S5.F4). This trendline further confirms our results, as the fitted regressions generally have positive slope and high $R^{2}$ values. There are some exceptions in the Llama results. As described in §[4.2](https://arxiv.org/html/2505.13899v2#S4.SS2), we fine-tuned Llama models rather than training from scratch due to compute limitations. We hypothesize that the relatively small set of trainable LoRA parameters and the small fine-tuning dataset result in insignificant changes to the model’s representations, yielding the observed lack of trend. The consistent trend in ViT/ResNet, for which we do end-to-end training, suggests that training data overlap drives representational alignment in models trained from scratch.

### 5.2 Effect of task overlap on representation similarity

Next, we evaluate how our two different methods of task splitting affect representational similarity. First, we consider method (1), task splitting with increasing dataset overlap, which closely mirrors the setup of our dataset splitting experiments. Then, we analyze method (2), task splitting with constant dataset overlap, which eliminates the confounding variable of varying dataset size at each task split.

(1) Task splitting with increasing dataset overlap. When both dataset and task overlap simultaneously increase, we observe a positive trend between task overlap and model CKA, see Figure [5](https://arxiv.org/html/2505.13899v2#S5.F5).

Figure: (a) ResNet - TinyImageNet
Refer to caption: x7.png

Figure: (a) ViTs
Refer to caption: x10.png

(2) Task splitting with constant data overlap. The first task splitting approach allows us to evaluate the same datasets/models as in the dataset splitting experiments but introduces a confounding variable since dataset overlap increases as task overlap increases. Therefore, we isolate the effect of scaling task overlap through experiments on ColorShapeDigit800K, as described in §[4.2](https://arxiv.org/html/2505.13899v2#S4.SS2). Results from these experiments are shown in Figure [6](https://arxiv.org/html/2505.13899v2#S5.F6). We find that, even when images in $D_{1}$ and $D_{2}$ are identical, increased task overlap positively correlates with increased CKA similarity scores in the models we evaluate. Linear regression models fit to these data (independent variable is task overlap, dependent variable is CKA) have a strong positive slope and reasonable $R^{2}$ values. These results empirically show that shared tasks among models, independent of data overlap, drive representational alignment.

### 5.3 Comparing the effects of dataset and task splitting on representational similarity

Finally, we evaluate which of our overlap methods—dataset alone, task alone, or dataset and task together—most strongly influences downstream representational similarity. To measure this, we use mutual information (MI) (Ross, 2014), an information-theoretic measure that quantifies the amount of shared information between two random variables. We compute the weighted mean mutual information for each overlap setting, where each model architecture evaluated contributes equally. Results are shown in [Table˜3](https://arxiv.org/html/2505.13899v2#S5.T3). In this table, “task overlap” refers to experiments with ColorShapeDigit800K in which we vary task while holding dataset constant, while “task + dataset overlap” refers to experiments in which task and dataset overlap increase together. As [Table˜3](https://arxiv.org/html/2505.13899v2#S5.T3) shows, the dataset and task overlap combined method has the highest MI score. This indicates that datasets with many shared characteristics (e.g. shared data points and task objectives) yield models with the most similar feature representations.

**Table 3: Mutual information between dataset/task overlap and representational similarity is highest when both dataset and task overlap. We report average mutual information between overlap and CKA metric (we use CKNNA for diffusion UNet) across overlap types, weighted by model architecture. “Task overlap” refers to task splitting method (2) with ColorShapeDigit800K, while “task + dataset overlap” refers to task splitting method (1).**
|  | Dataset Overlap | Task Overlap | Task + Dataset Overlap |
| --- | --- | --- | --- |
| Mutual Information | 0.584 | 0.771 | 0.891 |

## 6 Downstream Consequences of Representational Similarity

Finally, we explore downstream effects from dataset/task overlap, and consider whether one overlap method more strongly influences behavioral similarity. Here, we consider two downstream behaviors: adversarial example transferability and jailbreak transferability. This exploration is motivated by prior work highlighting the possibility of transferable adversarial examples and jailbreaks (Zou et al., 2023; Demontis et al., 2019; Carlini & Wagner, 2017). While intriguing, these works do not investigate how (or if) varying levels of model representational similarity affect transfer attack success rates.

Transferable Adversarial Examples. We use the MI-FGSM attack (Dong et al., 2018) to perform transfer attacks between ResNet and ViT models using TinyImageNet for task splitting method (1) and dataset splitting. For each pair of models trained using the same seed and overlap, we generate adversarial images using the MI-FGSM attack with whitebox access to one model and evaluate the transferable attack success rate (ASR) by computing if this example fools the other model in the pair. As Figure [7](https://arxiv.org/html/2505.13899v2#S6.F7) (b) shows, we observe a strong positive trend between dataset/task overlap, providing empirical evidence that high training set overlap between models may increase their susceptibility to transfer attacks. See [A.3](https://arxiv.org/html/2505.13899v2#A1.SS3) for attack settings, whitebox ASR, example images, and full results.

Transferable Jailbreaks. To implement jailbreak attacks, we follow the methodology of Andriushchenko et al. (2025), which uses a random search algorithm to construct an adaptive jailbreak attack for arbitrary LLMs by finding an adversarial suffix that can be appended to prompt $P$ to elicit unsafe responses. We use the input prompt template and random search algorithm from Andriushchenko et al. (2025) to generate jailbreak prompts for one model in a pair, then test the jailbreak on the other model. We evaluate jailbreak attack success using the LLM-as-a-judge technique from (Chao et al., 2024; 2025), in which a secondary LLM evaluates if the jailbreak was successful. We report transferable jailbreak attack success rates (ASR) on pairs of Llama models fine-tuned with various dataset and task overlap percentages and use Llama-70B as the judge. Specifically, let $LLM_{split0}$ and $LLM_{split1}$ be models fine-tuned on the splits described in section [4.1](https://arxiv.org/html/2505.13899v2#S4.SS1). We evaluate cross-split attack transferability by testing how adversarial suffixes optimized for models trained on one split perform when applied to models trained on the other.

As Figure [7](https://arxiv.org/html/2505.13899v2#S6.F7)(a) shows, transferable jailbreak ASR remains high across various overlap percentages. We hypothesize this is because the large LLMs were fine-tuned and not trained from scratch, which resulted in smaller changes in similarity scores between models across various overlap percentages as shown in Figure [4](https://arxiv.org/html/2505.13899v2#S5.F4)(b). Refer to [A.4](https://arxiv.org/html/2505.13899v2#A1.SS4) for full jailbreaking result.

Summary. Our exploration here highlights how representation similarity can be a useful indicators of security risks between a pair of machine learning models. We provide experimental evidence that high similarity scores correspond to higher transferability of both adversarial examples and jailbreak attacks, which to the best of our knowledge has not been explored in the literature.

Figure: (a) Llama 3.2, Llama-70B judge
Refer to caption: x12.png

## 7 Discussion

Our work provides the first empirical investigation of potential causes of observed representational similarity in models. Through a broad empirical study, we find that both task and data overlap in model training data drive downstream representational similarity in models, and that combining these yields the highest effect. Here, we discuss broader implications and limitations of our work.

Broader impacts. Today’s large-scale AI models, particularly generative models, are often trained on overlapping datasets (§[3.1](https://arxiv.org/html/2505.13899v2#S3.SS1)) for similar purposes, e.g. serving as general purpose chatbots. Our findings that dataset and task overlap drive model similarity then suggest these models may have unexpectedly similar feature space representations. Recent work has surfaced concerns about how feature space alignment between models could result in homogeneous model behaviors. For example, Wenger & Kenett (2025) showed that a broad set of generative AI models returns a narrow set of responses—much narrower than human responses—when responding to creative prompts. Homogeneity in AI models, in creative domains and beyond, could negatively impact AI users, who may find their AI-influenced behaviors unexpectedly narrowed and/or driven towards those of other AI use (Peterson, 2025).

Limitations and future work. Our work has several limitations. First, we primarily use CKA to measure representational similarity. Prior work has pointed out downsides of CKA relative to other metrics (Bansal et al., 2021; Sucholutsky et al., 2024). CKA scores are also difficult to interpret—they have maximum theoretical value of 1, but there is no established threshold for what score indicates strong alignment between models. Second, the size of models with which we experiment is limited due to compute constraints. This prevents us from making statements about how dataset and task overlap might affect similarity of today’s production-scale AI models. Third, we only compare models with the same architecture to avoid introducing confounding variables in our analysis, but future work could design controlled experiments to address this. Finally, we consider two possible factors affecting model similarity, but other factors could exist, leaving much room for future exploration.

## References

- Abdin et al. (2023)
Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu
Chen, et al.
Phi-2: The surprising power of small language models, 2023.
[https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/).
- Andriushchenko et al. (2025)
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.
Jailbreaking leading safety-aligned llms with simple adaptive
attacks.
In *The Thirteenth International Conference on Learning
Representations*, 2025.
- Bansal et al. (2021)
Yamini Bansal, Preetum Nakkiran, and Boaz Barak.
Revisiting model stitching to compare neural representations.
*Proceedings of NeurIPS*, 34:225–236, 2021.
- Bommasani et al. (2022)
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, et al.
On the opportunities and risks of foundation models, 2022.
[https://arxiv.org/abs/2108.07258](https://arxiv.org/abs/2108.07258).
- Brown et al. (2020)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan,
Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,
Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray,
Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford,
Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In *Proceedings of the 34th International Conference on Neural
Information Processing Systems*, NIPS ’20, Red Hook, NY, USA, 2020. Curran
Associates Inc.
ISBN 9781713829546.
- Caballero & Jenkins (2025)
William N. Caballero and Phillip R. Jenkins.
On large language models in national security applications.
*Stat*, 14(2):e70057, 2025.
doi: https://doi.org/10.1002/sta4.70057.
URL [https://onlinelibrary.wiley.com/doi/abs/10.1002/sta4.70057](https://onlinelibrary.wiley.com/doi/abs/10.1002/sta4.70057).
e70057 sta4.70057.
- Carlini & Wagner (2017)
Nicholas Carlini and David Wagner.
Towards evaluating the robustness of neural networks.
In *Proc. of IEEE Security & Privacy*, 2017.
- Chao et al. (2024)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko,
Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J
Pappas, Florian Tramer, et al.
Jailbreakbench: An open robustness benchmark for jailbreaking large
language models.
*Advances in Neural Information Processing Systems*,
37:55005–55029, 2024.
- Chao et al. (2025)
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas,
and Eric Wong.
Jailbreaking black box large language models in twenty queries.
In *2025 IEEE Conference on Secure and Trustworthy Machine
Learning (SaTML)*, pp. 23–42. IEEE, 2025.
- Cho & Hariharan (2019)
Jang Hyun Cho and Bharath Hariharan.
On the efficacy of knowledge distillation.
In *Proceedings of the IEEE/CVF international conference on
computer vision*, pp. 4794–4802, 2019.
- Chowdhery et al. (2023)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
et al.
Palm: Scaling language modeling with pathways.
*Journal of Machine Learning Research*, 24(240):1–113, 2023.
- Ciernik et al. (2025)
Laure Ciernik, Lorenz Linhardt, Marco Morik, Jonas Dippel, Simon Kornblith, and
Lukas Muttenthaler.
Objective drives the consistency of representational similarity
across datasets.
In *Forty-second International Conference on Machine Learning*,
2025.
URL [https://openreview.net/forum?id=va3zmBXPat](https://openreview.net/forum?id=va3zmBXPat).
- Crawl (2025)
Common Crawl.
Common Crawl - Open Repository of Web Crawl Data., 2025.
[https://commoncrawl.org/](https://commoncrawl.org/).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via
reinforcement learning, 2025.
[https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Demontis et al. (2019)
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio,
Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli.
Why do adversarial attacks transfer? explaining transferability of
evasion and poisoning attacks.
In *Proc. of USENIX Security*, 2019.
- Dong et al. (2018)
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and
Jianguo Li.
Boosting Adversarial Attacks with Momentum .
In *2018 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)*, pp. 9185–9193, Los Alamitos, CA, USA, June 2018. IEEE
Computer Society.
doi: 10.1109/CVPR.2018.00957.
URL
[https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00957](https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00957).
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at
scale.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=YicbFdNTTy](https://openreview.net/forum?id=YicbFdNTTy).
- Dravid et al. (2023)
Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher.
Rosetta Neurons: Mining the Common Units in a Model Zoo .
In *2023 IEEE/CVF International Conference on Computer Vision
(ICCV)*, pp. 1934–1943, Los Alamitos, CA, USA, October 2023. IEEE Computer
Society.
doi: 10.1109/ICCV51070.2023.00185.
URL
[https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.00185](https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.00185).
- Geirhos et al. (2020)
Robert Geirhos, Kristof Meding, and Felix A Wichmann.
Beyond accuracy: quantifying trial-by-trial behaviour of cnns and
humans by measuring error consistency.
*Proceedings of NeurIPS*, 33:13890–13902, 2020.
- Gurnee et al. (2024)
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun,
Will Hathaway, Neel Nanda, and Dimitris Bertsimas.
Universal neurons in GPT2 language models.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=ZeI104QZ8I](https://openreview.net/forum?id=ZeI104QZ8I).
- Hacohen et al. (2020)
Guy Hacohen, Leshem Choshen, and Daphna Weinshall.
Let’s agree to agree: Neural networks share classification order on
real datasets.
In *Proceedings of International Conference on Machine
Learning*, pp. 3950–3960. PMLR, 2020.
- He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)*, pp. 770–778, 2016.
doi: 10.1109/CVPR.2016.90.
- Hermann & Lampinen (2020)
Katherine Hermann and Andrew Lampinen.
What shapes feature representations? exploring datasets,
architectures, and training.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin
(eds.), *Advances in Neural Information Processing Systems*, volume 33,
pp. 9995–10006. Curran Associates, Inc., 2020.
URL
[https://proceedings.neurips.cc/paper_files/paper/2020/file/71e9c6620d381d60196ebe694840aaaa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/71e9c6620d381d60196ebe694840aaaa-Paper.pdf).
- Hu et al. (2022)
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
In *International Conference on Learning Representations*, 2022.
URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9).
- Huh et al. (2024)
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.
Position: the platonic representation hypothesis.
In *Proceedings of the 41st International Conference on Machine
Learning*, ICML’24. JMLR.org, 2024.
- Kaplan et al. (2020)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, et al.
Scaling laws for neural language models.
*arXiv preprint arXiv:2001.08361*, 2020.
[https://arxiv.org/abs/2001.08361](https://arxiv.org/abs/2001.08361).
- Kapoor (2021)
Karnika Kapoor.
Chars74k, 2021.
[https://www.kaggle.com/datasets/karnikakapoor/digits](https://www.kaggle.com/datasets/karnikakapoor/digits).
- Karpathy (2023)
Andrej Karpathy.
nanogpt: The simplest, fastest repository for training/finetuning
medium-sized gpts.
[https://github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT), 2023.
- Klabunde et al. (2023)
Max Klabunde, Mehdi Ben Amor, Michael Granitzer, and Florian Lemmerich.
Towards measuring representational similarity of large language
models.
In *UniReps: the First Workshop on Unifying Representations in
Neural Models*, 2023.
URL [https://openreview.net/forum?id=gG5UGgUzDR](https://openreview.net/forum?id=gG5UGgUzDR).
- Klabunde et al. (2025)
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich.
Similarity of neural network models: A survey of functional and
representational measures.
*ACM Computing Surveys*, April 2025.
ISSN 1557-7341.
doi: 10.1145/3728458.
[http://dx.doi.org/10.1145/3728458](http://dx.doi.org/10.1145/3728458).
- Kornblith et al. (2019)
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.
Similarity of neural network representations revisited.
In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.),
*Proceedings of the 36th International Conference on Machine Learning*,
volume 97 of *Proceedings of Machine Learning Research*, pp. 3519–3529. PMLR, 09–15 Jun 2019.
URL [https://proceedings.mlr.press/v97/kornblith19a.html](https://proceedings.mlr.press/v97/kornblith19a.html).
- Laakso & Cottrell (2000)
Aarre Laakso and Garrison Cottrell.
Content and cluster analysis: assessing representational similarity
in neural systems.
*Philosophical psychology*, 13(1):47–76,
2000.
- Li et al. (2015)
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft.
Convergent learning: Do different neural networks learn the same
representations?
In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar (eds.),
*Proceedings of the 1st International Workshop on Feature Extraction:
Modern Questions and Challenges at NIPS 2015*, volume 44 of *Proceedings
of Machine Learning Research*, pp. 196–212, Montreal, Canada, 11 Dec 2015.
PMLR.
URL [https://proceedings.mlr.press/v44/li15convergent.html](https://proceedings.mlr.press/v44/li15convergent.html).
- Lindsay et al. (2022)
Grace W. Lindsay, Josh Merel, Tom Mrsic-Flogel, and Maneesh Sahani.
Divergent representations of ethological visual inputs emerge from
supervised, unsupervised, and reinforcement learning, 2022.
[https://arxiv.org/abs/2112.02027](https://arxiv.org/abs/2112.02027).
- Ma et al. (2024)
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, et al.
Segment anything in medical images.
*Nature Communications*, 15(1), January 2024.
ISSN 2041-1723.
doi: 10.1038/s41467-024-44824-z.
[http://dx.doi.org/10.1038/s41467-024-44824-z](http://dx.doi.org/10.1038/s41467-024-44824-z).
- Mehrer et al. (2020)
Johannes Mehrer, Courtney J Spoerer, Nikolaus Kriegeskorte, and Tim C
Kietzmann.
Individual differences among deep neural network models.
*Nature communications*, 11(1):5725, 2020.
- Morcos et al. (2018)
Ari S. Morcos, Maithra Raghu, and Samy Bengio.
Insights on representational similarity in neural networks with
canonical correlation.
In *Proceedings of the 32nd International Conference on Neural
Information Processing Systems*, NIPS’18, pp. 5732–5741, Red Hook, NY,
USA, 2018. Curran Associates Inc.
- Moschella et al. (2023)
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco
Locatello, and Emanuele Rodolà.
Relative representations enable zero-shot latent space communication.
In *The Eleventh International Conference on Learning
Representations*, 2023.
URL [https://openreview.net/forum?id=SrC-nwieGJ](https://openreview.net/forum?id=SrC-nwieGJ).
- Naveed et al. (2025)
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad
Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian.
A comprehensive overview of large language models.
*ACM Trans. Intell. Syst. Technol.*, 16(5), August
2025.
ISSN 2157-6904.
doi: 10.1145/3744746.
URL [https://doi.org/10.1145/3744746](https://doi.org/10.1145/3744746).
- Nguyen et al. (2021)
Thao Nguyen, Maithra Raghu, and Simon Kornblith.
Do wide and deep networks learn the same things? uncovering how
neural network representations vary with width and depth.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=KJNcAkY8tY4](https://openreview.net/forum?id=KJNcAkY8tY4).
- Peterson (2025)
Andrew J Peterson.
Ai and the problem of knowledge collapse.
*AI & SOCIETY*, pp. 1–21, 2025.
- Phuong & Lampert (2019)
Mary Phuong and Christoph Lampert.
Towards understanding knowledge distillation.
In *Proceedings of the International conference on machine
learning*, pp. 5142–5151. PMLR, 2019.
- Raghu et al. (2017)
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein.
Svcca: singular vector canonical correlation analysis for deep
learning dynamics and interpretability.
In *Proceedings of the 31st International Conference on Neural
Information Processing Systems*, NIPS’17, pp. 6078–6087, Red Hook, NY,
USA, 2017. Curran Associates Inc.
ISBN 9781510860964.
- Räuker et al. (2023)
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell.
Toward transparent ai: A survey on interpreting the inner structures
of deep neural networks.
In *Proceedings of IEEE Conference on Secure and Trustworthy
Machine Learning (satml)*, pp. 464–483. IEEE, 2023.
- rivaldo (2022)
rivaldo.
Geometric shapes - mathematics, 2022.
[https://www.kaggle.com/datasets/reevald/geometric-shapes-mathematics](https://www.kaggle.com/datasets/reevald/geometric-shapes-mathematics).
- Roeder et al. (2021)
Geoffrey Roeder, Luke Metz, and Durk Kingma.
On linear identifiability of learned representations.
In *Proceedings of the International Conference on Machine
Learning*, pp. 9030–9039. PMLR, 2021.
- Ross (2014)
Brian C. Ross.
Mutual information between discrete and continuous data sets.
*PLOS ONE*, 9(2):1–5, 02 2014.
doi: 10.1371/journal.pone.0087357.
URL [https://doi.org/10.1371/journal.pone.0087357](https://doi.org/10.1371/journal.pone.0087357).
- Roth et al. (2024)
Karsten Roth, Lukas Thede, A. Sophia Koepke, Oriol Vinyals, Olivier J Henaff,
and Zeynep Akata.
Fantastic gains and where to find them: On the existence and prospect
of general knowledge transfer between any pretrained model.
In *The Twelfth International Conference on Learning
Representations*, 2024.
URL [https://openreview.net/forum?id=m50eKHCttz](https://openreview.net/forum?id=m50eKHCttz).
- Shan et al. (2020)
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y
Zhao.
Fawkes: Protecting privacy against unauthorized deep learning models.
In *Proceedings of the 29th USENIX security symposium (USENIX
Security 20)*, pp. 1589–1604, 2020.
- Steinberg et al. (2021)
Ethan Steinberg, Ken Jung, Jason A. Fries, Conor K. Corbin, Stephen R. Pfohl,
and Nigam H. Shah.
Language models are an effective representation learning technique
for electronic health record data.
*Journal of Biomedical Informatics*, 113:103637, 2021.
ISSN 1532-0464.
doi: https://doi.org/10.1016/j.jbi.2020.103637.
URL
[https://www.sciencedirect.com/science/article/pii/S1532046420302653](https://www.sciencedirect.com/science/article/pii/S1532046420302653).
- Sucholutsky et al. (2024)
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu,
et al.
Getting aligned on representational alignment, 2024.
[https://arxiv.org/abs/2310.13018](https://arxiv.org/abs/2310.13018).
- Team et al. (2024)
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak
Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al.
Jamba-1.5: Hybrid transformer-mamba models at scale.
*arXiv preprint arXiv:2408.12570*, 2024.
[https://arxiv.org/abs/2408.12570](https://arxiv.org/abs/2408.12570).
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
[https://arxiv.org/abs/2302.13971](https://arxiv.org/abs/2302.13971).
- van der Maaten & Hinton (2008)
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-SNE.
*Journal of Machine Learning Research*, 9:2579–2605,
2008.
URL [http://www.jmlr.org/papers/v9/vandermaaten08a.html](http://www.jmlr.org/papers/v9/vandermaaten08a.html).
- von Platen et al. (2022)
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert,
Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi
Xu, Steven Liu, and Thomas Wolf.
Diffusers: State-of-the-art diffusion models.
[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers), 2022.
- Wang et al. (2022)
Xinyi Wang, Jianteng Peng, Sufang Zhang, Bihui Chen, Yi Wang, et al.
A survey of face recognition, 2022.
[https://arxiv.org/abs/2212.13038](https://arxiv.org/abs/2212.13038).
- Wenger & Kenett (2025)
Emily Wenger and Yoed Kenett.
We’re different, we’re the same: Creative homogeneity across llms.
*arXiv preprint arXiv:2501.19361*, 2025.
[https://arxiv.org/abs/2501.19361](https://arxiv.org/abs/2501.19361).
- Wightman (2019)
Ross Wightman.
Pytorch image models.
[https://github.com/rwightman/pytorch-image-models](https://github.com/rwightman/pytorch-image-models), 2019.
- Zou et al. (2023)
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt
Fredrikson.
Universal and transferable adversarial attacks on aligned language
models.
*arXiv preprint arXiv:2307.15043*, 2023.

## Appendix A Appendix

### A.1 Details on task/dataset splitting method implementations

For both splitting methods, we experiment with 3 training paradigms. We train and finetune image classifiers (ResNet, ViT architectures) on CIFAR-100, and TinyImagenet; train diffusion UNets using CIFAR-10; and train small language models using the nanoGPT architecture, and finetune Llama models. We evaluate the representational similarity of models the validation set of datasets.

For language data with dataset overlap, we train nanoGPT models with either Shakespeare or TinyStories and we finetune Llama models with TinyCodes; language models with task overlap are trained with three randomly chosen datasets in [Table˜2](https://arxiv.org/html/2505.13899v2#S4.T2). For each setting, we finetune pairs of models using a few different seeds for image data and evaluate the representational similarity for model pairs on validation data.

For both task and dataset splitting across all modalities, we always use two datasets of equal size regardless of the value of $\alpha$. Under task splitting, entire partitions (in image datasets, a partition is a class; in text datasets, a partition is a contiguous text from one corpus) are assigned to one dataset or the other. For $\alpha>0$, this approach means some partitions from the original training distribution $\mathbb{D}$ are excluded from both datasets ($D_{1}$ and $D_{2}$).

Dataset splitting implementation. For image classifiers, dataset partitions $P$ are the classes of the training dataset, while for language models, partitions are different blocks of texts—e.g. a tokenized text corpus divided into 80 evenly-sized blocks, with each block assigned randomly to one of $D_{1},D_{2}$. We experiment with dataset overlap proportions $\alpha$ ranging from 0.0 to 1.0.

Task splitting implementation. As described in §[4.1](https://arxiv.org/html/2505.13899v2#S4.SS1), we have two methods of controlling task overlap. The first, (1) task splitting with increasing dataset overlap leverages the same datasets as in our dataset splitting but changes the number of overlapping classes. The second method, (2) task splitting with constant dataset overlap, only works with the ColorShapeDigit800K dataset.

For task splitting method (1), we experiment with task overlap levels ranging from 0.0 to 1.0 and evaluate models on the validation set. For task overlap experiments with text models, we construct overlapping datasets by combining multiple text corpora, rather than simply splitting one dataset between two models. In our case, each dataset includes three three corpora, meaning the models see text from different writing styles during training. The task splitting is controlled by the number of overlapping corpora across the two models’ datasets.

For task splitting method (2), in which we only trained models on ColorShapeDigit800K, partitions 1, 2, 3 correspond to task overlap $\alpha=0,0.5,1$, respectively.

#### A.1.1 Configuration of models

In our experiments, we finetune ViT/ResNet and Llama LLMs, and train diffusion UNet, nanoGPT, and ResNet18 from scratch. We give a brief description of the setup used for each model type.

- 1.
ViT/ResNet finetuning: we load pretrained models directly from timm (Wightman, 2019) and finetune all models for 5 epochs. The training batch size is 256. We used the AdamW optimizer for ViT at 3e-4 maximum learning rate and the SGD optimizer for ResNet at 0.2 maximum learning rate. The learning rate schedule is a cyclic LR schedule with a single triangle. Please refer to Table [4](https://arxiv.org/html/2505.13899v2#A1.T4) for the model sizes. All images used are upsampled to be $224\times 224$.
- 2.
ResNet-18 training from scratch: we train ResNet-18 architectures from scratch using the SGD optimizer at maximum 0.5 learning rate for 50 epochs. We use a cyclic LR schedule with a single triangle.
- 3.
nanoGPT training from scratch: we use a 6-layer model with 384 as the embedding dimension, 256 as the context size, and 6-headed multihead self attention. We employ early-stopping based on validation loss to reduce overfitting. We train with the AdamW optimizer at 5e-4 learning rate with a cosine annealing learning rate schedule.
- 4.
LLM finetuning: we use the Llama-3.2-Instruct series, specifically Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.2-11B-Vision-Instruct. We use LoRA (rank $=32$, dropout $=0.05$, targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) and approximatley 16.4 million tokens for finetuning.
- 5.
Diffusion UNet training from scratch: we use the diffusers (von Platen et al., 2022) UNet2DModel to generate the UNet backbone, which consists of 3 downsampling and 3 upsampling blocks. The model has 15.47 million parameters. We use a batch size of 1280 and trained for 30k iterations with 5e-4 as the maximum learning rate on a cosine annealing learning rate scheduler.

**Table 4: Model sizes for ViT/ResNet**
| Model Name | ResNet18 | ResNet50 | ResNet101 | ResNet152 | ViT-T/16 | ViT-S/32 | ViT-B/32 | ViT-L/32 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameters (M) | 11.28 | 23.71 | 42.71 | 58.35 | 5.56 | 22.53 | 87.61 | 305.72 |

**Table 5: Accuracy of finetuned image classification models. The mean accuracy and standard deviation of the accuracy over 4 seeds is given for each splitting method. The task splitting experiments use our dataset (ColorShapeDigit800k) for training and evaluation. For brevity, we show the largest and smallest ResNet/ViT.**
| Dataset (Model: ViT-tiny) | Dataset Splitting | Task Splitting | Task + Dataset Splitting |
| --- | --- | --- | --- |
| CIFAR-100 | 84.21 $\pm$ 0.34 | NA | 90.73 $\pm$ 1.06 |
| TinyImageNet | 77.04 $\pm$ 0.25 | NA | 85.10 $\pm$ 1.09 |
| P1: shape labels | NA | 99.62 $\pm$ 0.02 | NA |
| P1: digit labels | NA | 99.90 $\pm$ 0.01 | NA |
| P2: shape & color labels | NA | 99.62 $\pm$ 0.03 | NA |
| P2: digit & color labels | NA | 99.85 $\pm$ 0.01 | NA |
| P2: shape & digit labels | NA | 99.44 $\pm$ 0.01 | NA |
| P3: shape, digit, & color labels | NA | 99.37 $\pm$ 0.01 | NA |

Figure: (a) Task overlap
Refer to caption: plotting_appendix_iclr2026/diffusion_task_overlap.png

### A.2 Comparison of representational similarity metrics

In section §[A.2](https://arxiv.org/html/2505.13899v2#A1.SS2), we present the full results for all models evaluated on $7$ major representational similarity metrics (from codebase of Huh et al. (2024)). We include similarity metrics evaluated on the validation set of the training data for all dataset and task splitting methods.

#### A.2.1 Full ResNet-18 Results (Training from Scratch)

Figure: (a) Task splitting
Refer to caption: x14.png

Figure: (a) Task splitting
Refer to caption: x16.png

Figure: Figure 11: All similarity scores for task overlap while keeping dataset overlap constant (ColorShapeDigit800K, ResNet-18 trained from scratch). Representational similarity measured using metrics in Huh et al. (2024) for ResNet-18 under different task overlap with constant dataset overlap with error bars representing deviation across different splitting and training seeds. Please refer to [Table˜1](https://arxiv.org/html/2505.13899v2#S4.T1) for information on each partition. Evaluated on validation set.
Refer to caption: x18.png

Figure: (a) $\alpha=0.2$
Refer to caption: x19.png

#### A.2.2 Full nanoGPT Results (Training from Scratch)

Figure: (a) Shakespeare
Refer to caption: x21.png

For all sample text, the starting prompt is Do you not hear.

Sample text generated by a Transformer trained on Tiny-Stories (dataset overlap):

Sample text generated by a Transformer trained on Shakespeare (dataset overlap):

Figure: Figure 14: All similarity scores for task splitting (nanoGPT, trained from scratch). Representational similarity measured using metrics in Huh et al. (2024) for nanoGPT under task splitting with error bars representing deviation across different splitting and training seeds.
Refer to caption: x23.png

Sample text generated by a Transformer with a dataset constructed using task splitting:

#### A.2.3 Full ResNet/ViT Finetuning Results

Figure: Figure 15: All similarity scores for dataset splitting (CIFAR100, finetuned ResNet/ViT). Representational similarity measured using metrics in Huh et al. (2024) for finetuned ResNets and ViTs using dataset splitting with error bars representing deviation across different splitting and training seeds. Evaluated on validation set.
Refer to caption: x24.png

Figure: Figure 16: All similarity scores for task splitting method (1) (CIFAR100, finetuned ResNet/ViT). Representational similarity measured using metrics in Huh et al. (2024) for finetuned ResNets and ViTs using task splitting method (1) with error bars representing deviation across different splitting and training seeds. Evaluated on validation set.
Refer to caption: x25.png

Figure: Figure 17: All similarity scores for dataset splitting (TinyImageNet, finetuned ResNet/ViT). Representational similarity measured using metrics in Huh et al. (2024) for finetuned ResNets and ViTs using dataset splitting with error bars representing deviation across different splitting and training seeds. Evaluated on validation set.
Refer to caption: x26.png

Figure: Figure 18: All similarity scores for task splitting method (1) (TinyImageNet, finetuned ResNet/ViT). Representational similarity measured using metrics in Huh et al. (2024) for finetuned ResNets and ViTs using task splitting method (1) with error bars representing deviation across different splitting and training seeds. Evaluated on validation set.
Refer to caption: x27.png

#### A.2.4 Full LLM Finetuning Results

Figure: (a) Task splitting
Refer to caption: x28.png

#### A.2.5 Full Diffusion Results (Training from Scratch)

Figure: (a) Task splitting
Refer to caption: x30.png

### A.3 Adversarial transfer attacks on ViT/ResNet

Attack settings. For the MI-FGSM (Dong et al., 2018) attack, we use $\epsilon=1/255,2/255$ for dataset splitting and task splitting method (1), respectively. We use a larger $\epsilon=2/255$ for task splitting because the white-box ASR at $\epsilon=1/255$ is too low. We perform 3 steps of MI-FGSM for both task and dataset splitting.

**Table 6: Whitebox attack success rate (ASR) on ViT and ResNet, dataset splitting. The whitebox ASR is calculated by applying the MI-FGSM attack on models trained on dataset splitting.**
| ResNet18 | ResNet50 | ResNet101 | ResNet152 | ViT/tiny | ViT/small | ViT/base | ViT/large |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $0.89\pm 0.02$ | $0.71\pm 0.02$ | $0.53\pm 0.03$ | $0.49\pm 0.02$ | $0.90\pm 0.01$ | $0.68\pm 0.02$ | $0.56\pm 0.02$ | $0.49\pm 0.02$ |

**Table 7: Whitebox attack success rate (ASR) on ViT and ResNet, task splitting. The whitebox ASR is calculated by applying the MI-FGSM attack on models trained on task splitting method (1).**
| ResNet18 | ResNet50 | ResNet101 | ResNet152 | ViT/tiny | ViT/small | ViT/base | ViT/large |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $0.95\pm 0.01$ | $0.72\pm 0.02$ | $0.53\pm 0.04$ | $0.47\pm 0.02$ | $0.92\pm 0.01$ | $0.68\pm 0.02$ | $0.58\pm 0.03$ | $0.47\pm 0.03$ |

Figure: (a) ResNet dataset splitting
Refer to caption: x32.png

Figure: (a) Original image
Refer to caption: plotting_appendix_iclr2026/example_0_clean_resnet152.png

### A.4 Full Jailbreaking attack results

Figure: Figure 23: Transfer attack success rate (ASR) for LLMs of various sizes. Results for dataset splitting using Llama-70B as judge. We report transfer ASR of adversarial samples (prompt for LLM) between pairs of models with the same overlap. Please refer to the following tables for more granular reporting of ASR.
Refer to caption: x35.png

Table: Table 8: Jailbreaking attack success rate and standard deviation on Llama-3.2-1B-Instruct fine-tuned with various dataset overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.

Table: Table 9: Jailbreaking attack success rate and standard deviation on Llama-3.2-3B-Instruct fine-tuned with various dataset overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.

Table: Table 10: Jailbreaking attack success rate and standard deviation on Llama-3.2-11B-Instruct fine-tuned with various dataset overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.

Table: Table 11: Sample adversarial suffixes found when employing random search algorithm from Andriushchenko et al. (2025) on $\textbf{Llama-3.2-11B-Instruct}_{split0}$ and $\textbf{Llama-3.2-11B-Instruct}_{split1}$ fine-tuned using dataset splitting. Response to a harmful request are included along with the outputs of the Llama-70b and GPT-5 judges. Note that the jailbroken responses are cut short due to space.

**Table 12: Jailbreaking attack success rate and standard deviation on Llama-3.2-1B-Instruct fine-tuned with various task overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.**
| Method | Judge | $\textbf{Llama-3.2-1B-Instruct}_{split0}$ | $\textbf{Llama-3.2-1B-Instruct}_{split1}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 0.00 | 0.33 | 0.66 | 1.00 | 0.00 | 0.33 | 0.66 | 1.00 |
| Prompt | Llama | 94.0<br>(3.65) | 95.0<br>(4.76) | 93.5<br>(5.00) | 91.5<br>(4.43) | 94.5<br>(3.42) | 99.0<br>(1.15) | 94.5<br>(6.19) | 94.5<br>(3.42) |
| Prompt + RS(split0) | Llama | 82.0<br>(10.83) | 83.5<br>(7.55) | 85.5<br>(4.12) | 72.0<br>(18.76) | 77.5<br>(11.70) | 86.0<br>(10.46) | 83.5<br>(13.89) | 79.0<br>(10.89) |
| Prompt + RS(split1) | Llama | 81.5<br>(17.99) | 73.5<br>(15.61) | 85.0<br>(5.29) | 83.0<br>(8.08) | 86.5<br>(5.97) | 84.5<br>(8.39) | 81.5<br>(17.31) | 87.5<br>(5.97) |
| Prompt | GPT-5 | 31.0<br>(14.28) | 30.5<br>(16.52) | 38.0<br>(11.66) | 37.5<br>(8.54) | 26.0<br>(6.73) | 30.0<br>(7.12) | 27.0<br>(18.07) | 38.0<br>(10.71) |
| Prompt + RS(split0) | GPT-5 | 16.5<br>(11.70) | 20.5<br>(16.84) | 21.5<br>(6.61) | 21.0<br>(11.60) | 15.0<br>(5.03) | 12.5<br>(6.81) | 20.5<br>(26.45) | 13.5<br>(14.18) |
| Prompt + RS(split1) | GPT-5 | 18.0<br>(8.33) | 15.0<br>(14.09) | 19.0<br>(7.39) | 18.5<br>(11.36) | 12.0<br>(8.64) | 9.0<br>(6.22) | 19.5<br>(22.11) | 22.0<br>(14.79) |

**Table 13: Jailbreaking attack success rate and standard deviation on Llama-3.2-3B-Instruct fine-tuned with various task overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.**
| Method | Judge | $\textbf{Llama-3.2-3B-Instruct}_{split0}$ | $\textbf{Llama-3.2-3B-Instruct}_{split1}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 0.00 | 0.33 | 0.66 | 1.00 | 0.00 | 0.33 | 0.66 | 1.00 |
| Prompt | Llama | 86.0<br>(7.12) | 85.0<br>(16.69) | 83.5<br>(9.00) | 89.0<br>(10.39) | 84.5<br>(1.91) | 87.0<br>(5.03) | 85.5<br>(5.51) | 90.0<br>(6.93) |
| Prompt + RS(split0) | Llama | 90.5<br>(12.58) | 87.5<br>(12.26) | 88.5<br>(12.48) | 91.0<br>(10.52) | 89.0<br>(3.46) | 86.5<br>(4.73) | 84.0<br>(10.95) | 90.0<br>(8.33) |
| Prompt + RS(split1) | Llama | 87.5<br>(10.63) | 88.5<br>(9.15) | 88.0<br>(12.33) | 86.5<br>(12.90) | 91.0<br>(5.29) | 93.5<br>(3.00) | 82.5<br>(13.40) | 90.5<br>(9.85) |
| Prompt | GPT-5 | 41.5<br>(15.95) | 38.5<br>(21.99) | 42.0<br>(17.66) | 42.5<br>(19.42) | 41.0<br>(12.91) | 30.5<br>(12.37) | 41.0<br>(12.91) | 51.0<br>(20.03) |
| Prompt + RS(split0) | GPT-5 | 58.5<br>(11.70) | 60.5<br>(16.52) | 69.0<br>(20.94) | 59.5<br>(20.09) | 53.5<br>(4.43) | 40.5<br>(21.44) | 55.0<br>(26.20) | 59.5<br>(16.60) |
| Prompt + RS(split1) | GPT-5 | 55.5<br>(27.20) | 54.0<br>(9.93) | 63.0<br>(24.95) | 60.0<br>(25.03) | 57.0<br>(10.65) | 55.5<br>(12.37) | 51.5<br>(20.42) | 62.0<br>(12.96) |

**Table 14: Jailbreaking attack success rate and standard deviation on Llama-3.2-11B-Instruct fine-tuned with various task overlap splits according to the Llama-3-70B and GPT-5 judges averaged across 4 seeds using random search (RS) attack method from Andriushchenko et al. (2025) for finding adversarial suffixes. The success rate is measured using a set of 50 distinct and harmful requests from AdvBench Zou et al. (2023). Highlighted rows are the adversarial suffix transfer success rates.**
| Method | Judge | $\textbf{Llama-3.2-11B-Instruct}_{split0}$ | $\textbf{Llama-3.2-11B-Instruct}_{split1}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 0.00 | 0.33 | 0.66 | 1.00 | 0.00 | 0.33 | 0.66 | 1.00 |
| Prompt | Llama | 96.0<br>(2.83) | 97.0<br>(2.00) | 98.5<br>(1.91) | 96.0<br>(2.31) | 96.5<br>(1.91) | 95.0<br>(5.03) | 97.5<br>(2.52) | 97.5<br>(1.91) |
| Prompt + RS(split0) | Llama | 92.0<br>(6.53) | 89.5<br>(5.97) | 93.5<br>(3.42) | 96.0<br>(3.27) | 93.0<br>(6.22) | 96.5<br>(3.42) | 97.0<br>(3.46) | 93.0<br>(4.76) |
| Prompt + RS(split1) | Llama | 92.0<br>(5.66) | 94.0<br>(5.16) | 97.0<br>(2.00) | 96.0<br>(1.63) | 97.0<br>(2.58) | 94.5<br>(7.19) | 97.5<br>(1.91) | 95.5<br>(5.26) |
| Prompt | GPT-5 | 69.5<br>(15.78) | 64.0<br>(14.33) | 71.5<br>(6.61) | 63.5<br>(13.60) | 65.0<br>(16.12) | 47.5<br>(10.50) | 61.5<br>(9.57) | 70.5<br>(11.12) |
| Prompt + RS(split0) | GPT-5 | 65.5<br>(9.71) | 60.5<br>(16.60) | 68.0<br>(9.09) | 66.0<br>(12.75) | 63.0<br>(10.13) | 50.5<br>(9.15) | 56.0<br>(13.56) | 66.0<br>(15.49) |
| Prompt + RS(split1) | GPT-5 | 59.0<br>(16.12) | 68.0<br>(14.70) | 66.5<br>(12.90) | 72.0<br>(11.78) | 63.0<br>(12.81) | 55.0<br>(10.52) | 64.5<br>(12.26) | 64.0<br>(19.53) |

Table: Table 15: Sample adversarial suffixes found when employing random search algorithm from Andriushchenko et al. (2025) on $\textbf{Llama-3.2-11B-Instruct}_{split0}$ and $\textbf{Llama-3.2-11B-Instruct}_{split1}$ fine-tuned using task splitting. Response to a harmful request are included along with the outputs of the Llama-70b and GPT-5 judges. Note that the jailbroken responses are cut short due to space. We picked a different request to show negative ratings from the LLM judges.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - Novel causal interventions on causes of model representation alignment. The paper systematically modifiies dataset overlap and task overlap, and measures how these correlate CKA/other alignment metrics for both ViTs and small LLMs.
- Isolate task vs data overlap in representation similarities. Perform experiments on a number of different measures (CKA, nearest-neighbor scores, mutual KNN).
summary: This paper examines the extent that dataset and task overlap correlate with representation similarity. They use a new image dataset that fixes the images and varies labels and objectives; this isolates task overlap. They also perform some experiments on the correlation between jailbreaks and representation similarity.
weaknesses: - Concerns about generality. In general, the results on language modeling seem to be in tension with the stated takeaways of the paper, and the authors do not properly explain this discrepancy. Namely, they note that for the Llama fine-tuning experiments, the models in general have low correlation between their CKA scores and dataset overlap. This is comparatively true for task overlap as well. This lower correlation is hypothesized to be due to the fact that the LLMs were fine-tuned, so a lower proportion of their total training tokens are explained by the new task/dataset training. The authors note in the main body that they also run nanoGPT from scratch; however, these results seem to only be discussed/provided in the appendix (e.g. Figure 13). While these results are not discussed in the paper and the actual correlations between overlap and similarity are not provided, it visually it appears in Figure 13/14 that the correlation between dataset/tasks is pretty weak. The authors should both provide these correlations address this discrepancy. This also goes for the jailbreak correlation results in section 6.
- This paper has some conceptual overlap with [1, 2, 3]. These are discussed briefly in the paper at the moment, but it would be great to see a bit more discussion here.


[1] Ciernik, Laure, et al. "Objective drives the consistency of representational similarity across datasets." Forty-second International Conference on Machine Learning.

[2] Klabunde, Max, et al. "Towards measuring representational similarity of large language models." arXiv preprint arXiv:2312.02730 (2023).

[3] Brown, Davis, et al. "Wild comparisons: A study of how representation similarity changes when input data is drawn from a shifted distribution." ICLR 2024 Workshop on Representational Alignment. 2024.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: * How were the representations exactly extracted for the different models?

* Why didn’t you use the CKNNA measure from the The Platonic Representation Hypothesis when you already point out downsides of CKA in the Limitations section?
soundness: 3
strengths: * The paper studies an interesting question that can have large implications for building and evaluating foundation models.

* The paper is well written and the experiments and results are described clearly.

* The authors make an effort to make the underlying factors configurable and do a good job of holding one property constant (e.g. data overlap) while varying the other (e.g. task overlap).
summary: The authors systematically study how the two different factors (task and data overlap) contribute to the representational similarity of models trained from scratch. They created a new dataset called ColorShapeDigit800K which maintains a fixed set of data points while varying the task definition. With this dataset, they show that both task and dataset overlap cause higher representational similarity and that combining them provides the strongest effect. They further show that a high similarity between model representations also increases the vulnerability to transferable adversarial and jailbreak attacks between the one model to the other.
weaknesses: * The different tasks that are studied for ColorShapeDigit800K are all simple character/digit/color classification tasks while the difference between different SSL objectives (multi-view, masking), image/text, supervised models, text models seem to be larger in my view. I’m not sure if it is possible to approximate this with these simple tasks.

* The results that more tasks and data overlap increase representationals similarities feel a bit trivial. Although it is nice that the authors confirm that in a controlled way, I’m missing a novel insight from these results.

* It would be interesting to investigate whether these are the only factors that contribute to representational similarity or whether other factors (architecture, model size, training duration) also have a significant impact.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - Please see questions already stated in weaknesses :)
- Could you provide cross-architecture comparisons (ResNet18 vs ResNet152, ResNet vs ViT)? While same-architecture controls for confounds, demonstrating the trend across architectures would significantly strengthen the claim that dataset/task overlap drives similarity generally.
- Figure 4: ResNet101 shows much larger variance than ResNet18/50/152. What causes this outlier behavior? Why do other ResNets show such tight distributions?
- Figure 4: How many CKA values are in one box?
- CKA values for ViTs fall in the 0.4-0.6 range where the metric has known interpretability issues (high variability in this region, Figure 4). How confident are you in the regression lines fitted to these mid-range values? Would bootstrapping over data subsets provide more reliable similarity estimates?
- Figure 6: The high R² values seem driven by clustering at extremes rather than a true linear relationship. The level 2 combinations appear to form a blob in the middle. Can you explain this pattern? Why aggregate all level 2 combinations rather than separating them?
- Section 5.2 conclusion (2) states task splitting influences representation alignment "independent of data-overlap" at 100% data overlap. Wouldn't showing results at 50% data overlap make this argument stronger?
- The task overlap metric only applies to vision models, making Section 5.3 comparisons across modalities questionable. Can acknowledge this limitation more explicitly?
- Could you elaborate on why you focus primarily on CKA when 7 metrics were computed? What do the other metrics reveal that CKA (with a linear kernel) doesn't? CKA can also be used with different kernels, allowing to measure similarity more locally.
soundness: 3
strengths: - Clearly motivated work addressing an important gap: trying to understand *causes* rather than just observing representational similarity
- Well-designed methodology to isolate dataset and task overlap effects through controlled splitting
- Broad empirical evaluation across multiple modalities (vision, language, diffusion)
- Creative ColorShapeDigit800K dataset enabling task manipulation without dataset variation
- Novel connection to downstream security implications (adversarial transferability)
- Extensive appendix with multiple similarity metrics and detailed experimental settings
- Well written and enjoyable to read!
summary: Recent work hypothesizes that as AI models scale up and train on more tasks, their representations will converge because their task sets increasingly overlap. This paper empirically tests what drives representational similarity by isolating two factors: dataset overlap (shared training datapoints) and task overlap (shared objectives/classes). The authors develop controlled splitting methods to vary these factors independently and train model pairs (ResNets, ViTs, LLMs, diffusion models) with different overlap levels, measuring representational similarity via CKA. They introduce ColorShapeDigit800K, a vision dataset enabling task variation while keeping images identical. For vision models, the results are clear: both dataset and task overlap increase representational similarity, with combined overlap showing the strongest effect. Language model results (Llama fine-tuning with LoRA) are largely inconclusive, showing weak or absent correlations—whether this reflects LoRA limitations or fundamental differences remains unclear. The authors explore downstream consequences by testing vulnerability to transfer attacks: models with higher representational similarity show increased susceptibility to adversarial examples (vision), suggesting that unintended similarity between models may create shared security vulnerabilities. 

The work is rigorous and provides valuable empirical evidence for the vision domain, though language results and restriction to same-architecture-and-size comparisons can be limiting.
weaknesses: - Language model results are inconclusive and undermine the paper's central claims of dataset and task overlap strongly influencing the representational similarity:
    - Figure 19 (Llama fine-tuning): Only small positive correlations appear with CKA for task splitting; data splitting and local similarity measures show no effect from increasing overlap. Authors attribute this to LoRA without further hypothesis. Can the authors hypothesize beyond "LoRA limitations"? Is this a fundamental difference in how language vs vision models respond to fine-tuning, or purely a methodological artifact?
    - NanoGPT (trained from scratch): No linear trends provided, making assessment difficult
    - Critical: Since Llama results show no positive correlation with overlap increase across most metrics, the abstract/conclusion claims about "both task and dataset overlap" driving similarity might be overstated. The evidence supports this claim for vision models but not language models. The paper could either remove language claims or explicitly limit conclusions to vision models.
- Data vs Task vs Data+Task overlap analysis unfairly compares vision and language models when task overlap metrics only apply to vision (Section 5.3).
- As mentioned in the discussion, the work is limited to same-architecture comparisons; making the same observations cross-architecture results (ResNet vs ViT or small vs large architectures) would strengthen claims about the influencing factors of representational alignment.
- Figure 4:
    - ResNet101 shows anomalous high variance compared to other ResNets (Figure 4).  Also visible when using other similarity metrics.
    - Limited discussion of why ViTs show surprisingly low similarity values despite similar training protocols to ResNets. I assume that the ViTs have also been pretrained with ImageNet. Comparing two models after finetuning on TinyImageNet, shouldn’t they have much higher CKA values?
    - Could the authors comment on these?
- Some figures raise questions about correlation validity (e.g., Figure 6's high R² when points cluster at extremes). Can the authors comment on this?
- Presentation of key results could be improved: The paper uses boxplots with regression lines throughout, which becomes repetitive across dozens of panels with small fonts. While 7 similarity metrics are computed (appendix), the main text focuses on CKA minimally explaining this choice or what different metrics reveal. Better aggregation (summary tables, effect size comparisons across conditions) and/or diverse (aggregated) visualization approaches would significantly improve clarity.
- Missing minor details: which pretrained weights were used? Assumed ImageNet.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: - How does this work provide new insights beyond prior studies [1–3] that already link dataset/task overlap and attack transferability to representational similarity? (see W1)

- Since the study relies mainly on a single similarity metric (CKA) and simple regression analysis, how can the authors ensure that the observed trends are robust across different similarity measures or evaluation methods? (see W2)
soundness: 2
strengths: [S1] The paper addresses an interesting and timely question related to model alignment and generalization.

[S2] The experimental framework is well-designed and systematic, covering multiple model types and data modalities.
summary: This paper investigates the causes of high representational similarity among machine learning models across both vision and language domains.
The authors hypothesize that dataset overlap and task overlap are the main drivers of this phenomenon.
They introduce dataset and task splitting techniques to systematically control these overlaps and evaluate their effects across multiple architectures, including ResNets, ViTs, diffusion models, and large language models.
Empirical results show that greater overlap leads to higher representational similarity, and that models with more similar representations are more susceptible to transferable adversarial and jailbreak attacks.
weaknesses: [W1] Limited novelty of findings. The main results of this paper, that overlapping datasets or tasks yield more similar representations and higher attack transferability, are intuitive and have already been shown in prior literature.
For example, [1] treats training dataset differences as a key cause of representational dissimilarity, and [2] explores model similarity through adversarial attack transferability across different architectures and training regimes. Similar patterns have also been reported in [3], which links network similarity directly to attack transferability.
Thus, the contributions of this paper are largely confirmatory and show low novelty beyond well-understood phenomena.

[1] What shapes feature representations? Exploring datasets, architectures, and training (NeurIPS 2020)
[2] Similarity of Neural Architectures using Adversarial Attack Transferability (ECCV 2024)
[3] The Relationship Between Network Similarity and Transferability of Adversarial Attacks (2025)

[W2] Limited technical depth and scope. The proposed dataset and task splitting methods are straightforward and mainly procedural rather than methodological innovations. The experiments rely on small-scale synthetic datasets and LoRA-tuned LLMs, which limit the generalizability of the conclusions to large-scale or real-world model training. In addition, the study uses a single similarity metric (CKA) and simple regression fitting, resulting in limited technical rigor and diversity in evaluation.


Decision: Reject