Independence Tests for Language Models
Authors: Sally Zhu
Ahmed M Ahmed
Rohith Kuditipudi
Percy Liang
Venue: ICML 2025 spotlightposter (2025)
Keywords: language models
finetuning
fingerprinting
Primary area: deep_learning->large_language_models
TL;DR: We propose statistical tests to determine if two open-weight language models are independently trained from each other or not, i.e. one is finetuned.

Abstract: Motivated by liability and intellectual property concerns over open-weight models we consider the following problem: given the weights of two models, can we test whether they were trained independently---i.e., from independent random initializations? We consider two settings: *constrained* and *unconstrained*. In the constrained setting, we make assumptions about model architecture and training and propose statistical tests that yield exact p-values with respect to the null hypothesis that the models are trained from independent random initializations. We compute the p-values by simulating exchangeable copies of each model under our assumptions and comparing various similarity measures between the original two models versus these copies. We report p-values on pairs of 21 open-weight models (210 total pairs) and find we correctly identify all pairs of non-independent models. In the unconstrained setting we make none of the prior assumptions and allow for adversarial evasion attacks that do not change model output. We thus propose a new test which matches hidden activations between two models, which is robust to these transformations and to changes in model architecture and can also identify specific non-independent components of models. Though we no longer obtain exact p-values from this test, empirically we find it reliably distinguishes non-independent models like a p-value. Notably, we can use the test to identify specific parts of one model that are derived from another (e.g., how Llama 3.1-8B was pruned to initialize Llama 3.2-3B, or shared layers between Mistral-7B and StripedHyena-7B), and it is even robust to retraining individual layers of either model from scratch.

Independence Tests for Language Models
Sally Zhu Ahmed Ahmed"
Rohith Kuditipudi
Percy Liang'
Abstract
1. Introduction
Motivated by liability and intellectual property concerns over open-weight models we consider the following problem: given the weights of two models, can we test whether they were trained independenty--iie. from independent random initializations? We consider two settings: con- struined and unconstrained. In the constrained setting, we make assumptions about model archi- tecture and training and propose statistical tests that yield exact p-values with respect to the null hypothesis that the models are trained from inde- pendent random initializations. We compute the p-values by simulating exchangeable copies of each model under our assumptions and compar- ing various similarity measures between the orig- inal two models versus these copies. We report p-values on pairs of 21 open-weight models (210 total pairs) and find we correctly identify all pairs of non-independent models. In the unconstrained setting make none of the prior assumptions and allow for adversarial evasion attacks that do not change model output. We thus propose a new test which matches hidden activations between two models, which is robust to these transforma- tions and changes in model architecture and can also identify specific non- -ondindependent com- no longer obtain
Consider the ways which two models could be related: one model may be finetune of the other; one could be spliced and pruned from certain parts of the other; both models could be separately fine-tuned from a common an- cestor; finally, they could be independently trained from each other. We consider the problem of determining whether two models are independently trained versus not from their weights, which we formalize as a hypothesis testing prob- lem in which the null hypothesis that the weights of the two models are independent. We concretely treat only the weight initialization random and thus consider two mod- els with different random initial seeds as independent, even if both models were trained on the same data, or one model was distilled from the outputs of the other.
A solution to this independence testing problem would help auditors track provenance of open- ppen-weight models. This pertinent because while open-weight models enable broader access and customization, they also pose potential risks for misuse as they cannot be easily monitored or moderated (Kapoor et al., 2025). Model developers would also gain an enhanced ability to protect their intellectual property (IP) (Mensch, 2024; Peng et al., 2023) and enforce custom model licenses (Dubey al., 2024; DeepSeek-A Al et al., 2024).
We consider two settings of the independence testing prob- lem. In the constrained setting, we make assumptions on training and initialization (essentially, that the training al- gorithm is equivariant to permuting the hidden units of the random initialization) that enable us to obtain provably valid p-values. The main idea is that under these assumptions we can cheaply simulate many exchangeable copies of each model's weights and compare the value of some test statistic (e.g., cosine similarity of model weights) on each of these copies with the original model pair. The assumptions gener- ally hold in practice but preclude robustness to ad adversarial evasion attacks and architectural changes.
ponents of models. Though exact p-values from this test, empirically we find it reliably distinguishes non- coniindeeeneet models like a p-value. Notably, we can use the test to identify specific parts of one model that are de- rived from another g., how Llama 3.B was pruned initialize Llama 3.2-3B, or shared layers between Mistral-7B and StripedHyenn-785) and it is even robust to retraining individual layers of either model from scratch.
For the constrained setting, we evaluate various test statis- tics 21 models of the Llama 2 architecture (Touvron et al., 2023), including 12 fine-tunes of Llama 2 and nine indepen- dently trained models, obtaining extremely small p-values
'Equal contribution Department of Computer Sci- ence, Stanford University. Stanford, US. Correspondence to: Sally Zhu "salzhu@stanfoon @tanfford.edu>. Ahmed Ahmed <ahmedah cs. sta.ffooor....
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We share code at https: / http//bttthhoooo/oo ahmedal4 00250//moee -ordddee-rrinnngg

---

Independence Tests for Language Models
Liama O
Liama
Liama
Vicuna 1.5
Liama (8B)
(38)
Mistral M
Stripedhyena
O
O
O
O*O
Independent
Not Independent
Figure 1. Given the weights of two models, what relationships can we derive?
for all 69 non-independent model pairs. Notably, tests retain low p-values over different fine-tuning methods (e.g., different optimizers) and on models fine-tuned for many tokens from the base model such as Llemma (Azerbayev et al., 2024), which was fine-tuned on an additional 750B tokens from Llama 2 (i.e.. 37.5% of the Llama 2 training budget). We also confirm that the leaked Miqu-70B model from Mistral is derived from Llama 2-70B
The work most closely related ours is due to Zeng et al. (2024), who considered our constrained setting; they de- velop various tests to determine whether a model as whole is independent of another by computing the cosine similarity of the products of certain weight matrices in both models. They show that their tests are robust simple adversar- ial transformations of model weights that preserve model output; however, detail in Appendix G.1 other transfor- mations to perturb dependent models that evades detection by their tests. Additionally, unlike Zeng et al. (2024), in the constrained setting we obtain exact p-values from our tests. Jin et (2024) propose crafting specific queries that are likely to produce different responses among indepen- dently trained models; their method does not require access to weights but also does not produce exact p p-values.
For the unconstrained setting we develop a test robust to simple modifications to model weights and architecture, such as permuting hidden units, that can violate the assump- tions of the constrained setting an adversary applies them after fine-tuning. Though we are not able to obtain prov- ably exact p-values in the unconstrained setting, derive test whose output empirically behaves like p-value and reliably distinguishes non-independent models from inde- pendent models. In particular, we first align the hidden units of two models-wel which may each have different activation types and hidden dimensions-ane then compute some mea- sure of similarity between the aligned models. Because of the alignment step, the test is robust changes in model architecture and various adversarial evasion attacks (includ- ing those that break prior work). Moreover, it can localize the dependence: we can identify specific components or weights that are not independent between two models, even when they have different architectures.
2. Methods
2.1. Problem formulation
Let f E θ and an input We represent model training or fine-tuning process as learning algorithm A θ that takes as input set of initial parameters corresponding to either a random initial- ization or, in the case of fine-tuning, base model parameters. Specifically, A includes the choice of training data, ordering of minibatches, and other design decisions and even the randomness used during training Suining-everything other than the initial model weights.
+ y denote a model mapping parameters
to an output f(X;0) 0) e y,
We evaluate our unconstrained setting test on 141 indepen- dent model pairs and find that its output empirically behaves like a p-value in the sense that it is close uniformly dis- tributed in [0, 1] over these pairs. In contrast, it is almost zero for all dependent pairs we test (including those for which we simulate a somewhat strong adversary by retrain- ing entire layers from scratch). We also employ our test to identify pruned model pairs, which occur when one reduces the layer dimensions by retaining only select activations and identi-
Given two models 0,, O2 P for some joint distribution P E P(O1 x θ2), our goal is to test the null hypothesis
Ho : 0 O2.
(1)
where 1. denotes independence of two random variables. One example of a case where 0 and 02 might not be in- dependent if 6g is fine-tuned from 8,, i.e., O, = O (meaning the two models share the same architecture) and 0₂ = A(O2) for some learning algorithm A. We treat
weights from pre- pre-trained model; for example, fied the precise layers of Ilama 3.1 8B from which each Llama 3.2 3B and Llama 3.2 IB layer was derived.
2

---

Independence Tests for Language Models
learning algorithms deterministic functions. Thus, for
Algorithm 1: Test for computing p-values (PERMTEST)
I (f9) and 0₂ A2(82). then 09 1 0g (i.e. two models with independent random initializations) implies our null hypothesis.
Input: Model weights 81, 02
Parameters :test statistic d; discrete transformation
class II: permutation count T
Deep learning models are often nested in nature. For exam- ple, Transformer models include self-attention layers and MLP layers as submodels. We formalize the notion of submodel via the following definition.
Output: p-value ý E (0, 1]
tiiies- - 0;
for t E 1..... T do
T ~ Uni;([I);
Definition 1. A model f Oxx + contains submodel g O' x x² + 3" if there exists projection operator
4
Φi + (0), 8 8 8 + l(o: = (01, 02)}:
s
6 E ~ (oi.((()....mteesss) // break ties
proj: Q @' such that for all 0 θ we have
. p - 1 (1 + E 1f 0(01,02));
f(x;; 0) = (g (ro))))))))))
return p
for some functions fin X - X' and fout: 3" + y (which may depend on 0).
Definition (TT-equivariance). Let C 0, IT, and 6 E e. A learning algorithm A is II-equivariant if and only π (A(09)) = A()())))
Many of our experiments will involve Transformer models specifically containing MLP layers with Gated Linear Unit (GLU) activations, which are widely used among language models. It thus will be useful to define this type of MLP presently through the following example.
The main idea underlying PERMTEST is that so long as 0, = A(O9) and 0 ~ P for some Il-equivariant learning algorithm A and IT-invariant distribution P, we can simulate T exchangeable (but not independent) copies (Tx(0))) of 0. by sampling T i.d. Uni This allows to effi- ciently compute exact p-value without actually repeating the training process of 02. In effect, Definitions 2 and 3 im- ply that ı commutes with A-i.e., . (A(0))) = A(T (6P).. Under exchangeability, the p-value output by PERMIEST will be uniformly distributed over {(i + 1) + 1))T
Example 1: (GLU MLP) Let G,U E R and D E Rdxh. Let σ : R- + R be an element-wise activation function. For x E Rd and = (G,U,D, E Ohip let fmip(a; 0) = (Gr( @ (U=)). Likewise, for X E Raxd let fmip(X;;) E Rexd denote the result of broadcasting fiip over the rows of X. O
In addition to the basic independence testing problem above, we also consider the problem of localized testing: testing whether various pairs of submodels among two overall mod- els are independent or not. A prototypical example of localized testing problem is identifying which layers of larger model (e.g., Llama 3.I-BB) were used to initialize a smaller model (e.g., Llama 3.2-3B) (in this case, we treat the layers as different submodels).
Standard initialization schemes for feedforward networks are symmetric over the hidden units of the network, and so one example of a class of transformations with respect to which any such initialization is invariant is the set of permu- tations over the hidden units of the network. Moreover, the gradient of the model's output with respect to the hidden units is permutation equivariant: thus, any learning algo- rithm whose update rule is itself a permutation equivariant function of gradients (e.g., SGD, Adam, etc.) satisfies Defi- nition 3 with respect to these transformations. A (contrived) example of a learning algorithm that is not permutation equivariant is one that uses different learning rates for each hidden unit depending on the index of the hidden unit.
2.2. Constrained Setting
1.2.1. TESTING FRAMEWORK
Algorithm (PERMTEST) encapsulates our framework for computing p-values against the null hypothesis in the con- strained setting, wherein we simulate T exchangeable copies of the first model 0, by applying transformations its weights. The exchangeability of these copies holds under some assumptions on the learning algorithm and random initialization that produced the original model. We capture these assumptions in the following definitions; together, they define the constrained setting.
Example 2 (Permuting hidden units): Let (G, D) Ô mlp parameterize a GLU MLP, where recall fmip(2:; bz; 0) == D(o (z)) @ (Ux)) for some element- wise activation func- tion R - IR. Abusing notation, let IT be the set of h x h permutation matrices such that for π E 11 we define π (0) (mG, TÜ, D T) Observe fmip (z; 8) = faip (z: π (f)) and (V 0)) = V [x; (0)) for all inputs x. O
Definition 2 (II-invariance), Let II C θ - e. A distribu- tion P E P(O) is II-ivariant if for f ~ P and any л E II, the parameters 0 and π (0) are identically distributed.
The assumptions we make in the constrained setting suffice for PERMTEST to produce a valid p-value, as we show in
3

---

Independence Tests for Language Models
the following theorem, whose proof uses symmetry of the initialization and training process (full proof in Appendix A).' Importantly the result of the theorem holds (under the null hypothesis) without any assumptions on 82; so, a model developer of 0, testing other models with our methods can have confidence in the validity of our test without trusting the provider of 0z. Of course, if 02 does not satisfy the equivariance assumption on training (as in the unconstrained setting), then PERMTEST is unlikely to produce a low p-value even in cases where 0, and O2 are not independent (e.g., if an adversary finetunes Og from O2 but then afterwards randomly permutes its hidden units).
where SPEARMAN is the Spearman rank correlation (Algo- rithm 3). Equation (2) is applicable to any model architec- ture & for which we can define a suitable matrix valued function M of model parameters. For example, M could extract a weight matrix or activation matrix (based on some set of inputs) from a layer of the model, where each row corresponds to a hidden unit of the model. We use MATCH to align the rows of the two extracted matrices and compute the Spearman correlation of this alignment with the identity map between rows. We describe matching in Algorithm 2, wherein cossim denotes cosine similarity function and LAP denotes the algorithm of Ramshaw & Tarjan (2012) we use to solve the matching problem.
+ IR be a test statistic and II C - O be finite. Let A @ + θ be II-equivariant and let P E P(@) be I--ivariant. For Og P. let 01 A(09). Let 0g E @ be independent of Then p PERMTEST(O;, 02) is uniformly distributed on [+1 1T
Theorem 1. Let Φ O x
The idea is that for two dependent models, each row of M (01) should be similar to its counterpart in M thus, the alignment found by SPEARMAN will be close to the iden- tity map. Meanwhile, so long as M is a IT-equivariant map (Definition 4), then ΦM (01, 022 under the null yields valid p-values (see Theorem 2 and proof in Appendix A); so we can use the more computationally--eine convert statistics to p- p-vuuues instead of running PERMTEST.
1441)1=
We also generalize Theorem 1 to apply to randomized learning algorithms that satisfy a notion of equivariance in distribution (including dropout) in Appendix B. However, throughout the main text we will continue to treat learning algorithms as deterministic for the sake of simplicity.
Algorithm 3 to
Definition 4. (equivariant map) A matrix-valuee function M : θ + R=xm is Il-equivariant with respect class of transformations II θ - θ if there exists a bijection between II and the set of n x permutation matrices such that M (=)) = (0) for all A E O and x E II.
2.2.2. TEST STATISTICS
We have shown PERMTEST produces valid p-value regard- less of the test statistic e we use. The sole objective then in designing a test statistic is to achieve high statistical power: we would like X = PERMTEST((,, 02) to be small when 01 and 0g are not independent. The statistics in this section apply to any model pair sharing the same architecture.
Theorem 2. Let M : Q - I= be II equivariant map and let P E P(()) be II-invariant. Let 01, 02 E O be independent random variables, with 02 = A(09) for B ~ Pt. Then (0,(0,,0, uniformly distributed on 1).
Prior work (Xu et al., 2024) proposed testing whether two models are independent or not based on the & distance between their weights, summed over layers. Specifically, for
Algorithm 2: Cosine similarity matching (MATCH)
Input: Matrices W1, W2 with h rows Output: Permutation for E 1,..., h do
[h] - [h]
a model with L. layers parameterized by θ = θ x... x OL,
with 01 = (((()) ) and 02 = (o!)) J let dee (01, 02) = -E
((0)). We can obtain p-vauues from Φe, by using within PERMTEST. However, a major limitation is that in order to obtain a p-value less than 1/(T++1 1) we must recompute oi, at least T times; then the statistical power of our test using Φe, is therefore bottlenecked by computation.
2
for E 1, I...., h do
Ct +- wwsssim(ss))).. (W2))) < LAP(C):
return
Taking various such functions M yields different test statis- tics. We focus our experiments on Transformer models con- sisting of a series of L Transformer blocks that each contain GLU MLP submodel, and we take M to be either the up projection weights or the hidden-layer activations of one of these MLP submodels. In particular, let (7(1) (0) E R denote the first layer up projection weights of the ML. in the b-th block, where h is the hidden dimension and d is the input dimension, and let ((((()) E (NN denote the (flattened) hidden activations that obtain from passing N length s input sequences X E N Nxexd to the same MLP
To address this limitation, we propose a family of test statis- ties whose distribution under the null is identical for any model pair. The test statistics all share the following general form based on Algorithm (MATCH): for m, n E N and M: θ Raxm, let
@m (0,, 02) = SPEARMAN(MATCH M (01), (02)). [L...., n)),
1 AA result of the test yielding exact valuee, can directly control for the false positive rate via the significance threshold.
4

---

Independence Tests for Language Models
module (the test is valid for any X; we will specify later how we choose X in our experiments). The two main test statistics we employ in our experiments are @rrce and @rrn.
these test statistics, and the independence of the p-values from these test statistics across blocks follows directly from the independence of the statisties themselves.
Both ( and (+) are equivariant with respect to permut- ing the hidden units of the corresponding MLP, so we can directly interpret the outputs Φ17()) and (ggre) as p-values. Moreover, we can separately permute the hidden units of the MLP in the E-th block without changing the inputs or outputs of the other blocks. Thus, we show in Theo- rem 3 (proof in Appendix A), we can aggregate the p-values from & and (1(() across blocks using Fisher's 's method (Mosteller & Fisher, 1948)) to obtain a more powerful test in Algorithm 4 (FISHER).
2.3. Unconstrained Setting
For the unconstrained setting, our goal is design ro- bust test that applies to models of different architectures and is robust to output-preserving transformations of model weights. Recall our tests for the constrained setting satisfy neither of these desiderata: these tests assume both models have the same number hidden units, and it is easy to fool them without changing the output of a model by permuting the order of the hidden units in the model.
Our robust test reposes on the design of ΦM in equation (2), The goal is to identify two matrix valued functions of model parameters M, M' θ - Raxm that jointly satisfy the following condition: any output-pesserving transformation of model parameters must transform both M and M' the same way. Then, whereas previously we would correlate MATCH( M (01). M (62)) with the identity permutation, instead define
Algorithm 3: Deriving p- p-values from Spearman corre-
lation (SPEARMAN)
Input: Permutations Ii, T2 [h] + [h]
Output: p-value ý [0
6 (mn [1 x))
1
2-2
1 2 2 t);
return p
MM SPEARMAN
(ATC (3)
Algorithm 4: Aggregating p-values (FISHER)
The above goal is aspirational in the sense that for any nontrivial deep learning model we are not able to fully enu- merate the set of transformations of model parameters to which model output is invariant; nonetheless, it will serve as a useful guiding principle for designing our robust test under the framework of equation (3). We organize the description of our full robust test-which is generally applicable to variety of model architectures into two parts: first, in Section 2.3.1 we instantiate equation (3) to obtain a test for GLU MIP models. Then, in Section 2.3.2 we use our GLU MLP test as a primitive for designing a test that applies to general deep learning models (including those which do not contain any GLU MI.P submodels).
Input: p-values (0()
Output: p-value p E (0,
{
1 log p())
p 1- F(X21 < -2E);
return p
Theorem 3. Consider block indices i, E [L] with + for models with L blocks. Suppose for & E {i,j} that
1. M : θ + RAX is equivariant with respect to I1,),
Lee, for any O E O and (1) E (T()) we have
= =()
1. TESTING GLU MODELS
2. A is a P(0) is II -invariant distribution.
(T(0 -quiivarian learning algorithm and P E
Recalling our definition of a GLU MLP model in Example 1,
for k E [1,2) let 02 (G2, Up, D&) E O mip and with in-
puts E RR let (0p) U X E [m A2) XN be the output the up projection operation and let H pone (0s) = Gg.X E gmmax(hg.hs x be the output of the gate projection operation (with appropriate zero-padding when h o h2). Due the element- product operation, we conjecture that in general it is not possible to permute the rows of Gk: while preserving the output of 0 without permuting the rows Un in the same way, and so we use @M.M with M = Hpate and M' = Hup for our GLU MLP test. Henceforth, we will shorthand this test as ΦMATCH-
Let 0,,02 E O. 0f 1 for 0, A(O9) with 0f P. then
MATCH(M((())) (0)) 1 MATCH( (O)
Recall dyen and 2H are functions of MATCH(M (0, (B2)) respectively for M( = r
and (= = ( both of which satisfy the assumptions of
the theorem. Thus, the result of the theorem applies to both
5

---

Independence Tests for Language Models
As with the constrained setting, we focus much of our ex- periments on Transformer models, which recall consist of series of L Transformer blocks that each contain GLU MLP submodel. Adopting the notational conventions of Section 2.2.2, we can apply our GLU MIP test to the L-th block by taking M Hgat and M' Hup' where like before (in the case of @HCO)) we obtain the activation inputs for each block by computing a forward pass through the full model over set of length sequences of input tokens.
Algorithm 5: Generalized robust test
Input: Model parameters 01, 02 E θ Parameters distribution P over Rd Output: p E [0, 1]
for k (1,2) do
0, +-
2
|| projart))) - Smip (z: 0x) [²/
arg ming R
return P t MMATCH 02)
We can aggregate the results of these tests over blocks using FISHER, like we do for @o(n and @iiie in the constrained setting, Alternatively, we can apply the test to all possible O(L²) pairs of blocks between two Transformer models if we suspect that certain blocks from one model served as the initializations for different blocks in the other model. Specifically, we can test the i-th block of 0, and the j-th block of 02 using
P with which we learn the GLU MLP are hyperparameters of the test. See Section 3.2 for details.
3. Experimental Results
3.1. Constrained setting
(b.5) MATCH = IPEARMAN(MATCH H (01), Fr (02)), MATCH( H (01), H)) (02))).
We first validate validate the effectiveness of our tests in the constrained setting on spen-weight language models 21 models trained with the Llama-7B architecture with public documentation on ground truth model independence. These models all contain L = 32 GLU MLPs, each part of its own Transformer block. We run experiments with three different tests. Each test comprises two elements: test statistic along with a method for computing p-values from the statistic. For the first test, we use Φi, and compute p- values via PERMTEST with T = 100. For the other two tests, we compute p-values by directly aggregating the outputs of (respectively) éyro and @ over E [L] using FISHER. We obtain the inputs to the GLU MLP in the l-th required to compute ftee) by sampling sequences of tokens uniformly at random from the models' vocabulary and computing forward pass through the full model while storing the MLP hidden layer activations. The equivariant transformation class II is the set of permutations over both the hidden units of each MIP (see Example 2) and the embedding dimension of the model (i.e., the inputs passed to the both the MLP and self-attention layers in each block); we defer the precise definition of II in this case to Appendix D.
This test is relevant for pruned models, where only select blocks (layers) of 0₂ may be used to initialize the smaller Oi or, if an adversary takes only certain layers, or even only certain activations, of pre-trained model and injects other layers.
2.3.2. BEYOND GLU MODELS
Thus far we have focused on models f X Q - y containing a GLU MLP submodel. In particular, recalling Definition 1. we have assumed for some projmip θ O that
(z; = (fsp (fi mip (f)))
(4)
Now, our goal is to test more general types of models. In particular, we generalize to an arbitrary alternative submodel san: Rd x Oalt + d with projan > Oai such that f(z; 0) = (San (in (z); (0)).
(5)
In order to test whether two models 0g, 02 E θ of the more general form in equation (5) are independent, we will first construct proxy models of the form in equation (4) and then apply our previous test ΦMATCH to these proxy models. We construct these proxy models by leveraging the fact that fan shares the same input and output space with fmip- Specifically, for k E (1,2) we first learn parameters 0₂ E 0 that fm (0,) approximates fal(- :projar (0x)). We then return dM (0, (0,, Ō2). We capture this two-stage process in Algorithm 5.
1.1.1. I. BASELINE STATISTICS
We employ two test statistics from prior work as baselines: Jensen-Shannon divergence between next token output dis- tributions (ØISD, (Lin, 2006)), and Φi (Xu et al., 2024)) with PERMTEST (details in Section 3.2.2). We computed ΦISD using input sequences sampled from 110 (Merity et al., 2017; Xu et al., 2024) (consistent with prior work). Since the Lensen-Shannor divergence is (by defini- tion) invariant to any transformation of weights that does not affect model output, we cannot compute meaningful p-values using PERMTEST; instead, in our experiments we report the raw value of the test statistic itself.
Perhaps surprisingly, we show that Algorithm is effective in practice at distinguishing independent versus non indepen- dent models. The hidden dimension h and input distribution
6

---

Independence Tests for Language Models
3.1.2. LLAMA FAMILY EXPERIMENTAL RESULTS
PERNTEST effectively standardizes the raw distances. We further evaluated the efficacy of our tests through ablations by training two models with the same OLMo-7B architec- ture on the same dataset that only differ on the choice of random initialization of randomness, and report results in Appendix E.I. We also verify that Miqu-70B is not indepen- dent from Llama 2-70B (Mensch, 2024) and report further details in Appendix E.2.
The 21 models we evaluated include 6 base models (trained from scratch), we have six disjoint sets of the models based on -1ama-2- 2b- of industry labs and non-profits (Azerbayev et al., 2024; Sudalairaj et al., 2024; Liu et al., 2024; Li et al.. 2023). We consider any pair of models in the same tree as dependent and all other pairs as independent. We include examples of further tune-tunes (e.g., 1 11mma_77) of fine-tunes (e.g.. Codel. amaa- 7b-hf) among the models we test. We will mostly refer to models using by their Huggingface identi- fiers, without the organization names for clarity.
stemming from diverse mix
3.2. Unconstrained setting
For the unconstrained setting, we first assess the previous 21 models of the Llama-7B architecture. We compute ÓMATCH with the gate and up-projection matrices M gm and M' = Hip of each MLP in block & E [L], and aggregate them with FISHER. We obtain the activations in the MLPs by using input sequences sampled from WikiText-109 and computing forward pass through the full model, with re- sults on all model pairs in Appendix F.
We evaluated four test statistics: dyte (cosine similarity of weights), (He (cosine similarity of hidden activations), di, (&2 distance), and ΦISD (Jensen-Shannon Divergence). As we describe in Section 2.2.2, for dyte and @HO) we report aggregated p-values over all blocks using FISHER. We report results for subset of these pairs involving base model L1 rest and the full experimental setup details to Appendix E.
in Table 1 while deferring the
We find that the distribution of ΦMATCH on independent model pairs is close to uniform (Figure 2), whereas across all non-independent model pairs the statistic is at most &. Unlike the constrained setting, where the p-values are valid by construction, the output of the robust test does not enjoy such theoretical guarantees; however, Figure 2 suggests that even in the unconstrained setting our statistic ΦMATCH
Llana -D. 0g ==
p-values
Indep.?
dysp (log)
+x(()
(g(C))
LLama-TT-ll
-11.10
0.88
0.50
0.25
vi conn--7--11..
-10.40
0.63
0.16
0.64
Aniows
-10.69
0.75
0.36
0.88
opea- 11nna-7 7a
0.26
0.36
0.71
Vizuns-73-v1.3
CodeLLamee-t--- Ib-1st
-10.62
-10.87
0.01
0.01
2
behaves like a p-value.
11erna-7b
-10.24
02ca-2-7b
-10.34
Table 1. We report various constrained setting test statistics with 0 as amaam------ and 0g ranging over the listed models. The "independent" column the ground truth. Here, = 2.2e-308 (numerical underflow for a bit float). We find proposed tests Qyve) and e distinguish independent versus non- independent model pairs with high statistical power.
Consistent with prior work (Xu et al., 2024), we find that disD does not reliably distinguish independent versus depen- dent model pairs. For example, Codell ama- -f ex- hibits a larger divergence with 1aam--2-7b-ba than the independently-traine models 11 ama- -ooah f and Amber.
(a) Plot of x 10,1) vs. the (b) Plot of x E [0, 1) vs. the fraction of OMaCH (across all fraction of ΦMAтCи (O ag- MLP blocks) of independent gregated with FISHER) of inde- pendent model pairs less than x.
(i)
model pairs less than x.
Figure 2. We plot the fraction of ÓMATCH less than x E [0, 1). aggregated with FISHER for independent model pairs. Both plots roughly follow the line y = x. i.e. a uniform distribution in [0, 1) under the null, meaning ΦиATC empirically acts as p-value.
All other test statistics reliably distinguish independent ver- sus dependent pairs; in particular, the p-values we obtain using the other test statistics are negligible for all depen- dent pairs (for Φeg, because we run PERMTEST with T = 99 for computational reasons, we cannot obtain a p-value less than 0.01. A.0i.Ny in contrast to our findings, prior work (Xu et al., 2024) argued that the &g distance between model parameters is not reliable indicator of independence, in the sense that the l distance between dependent pairs is sometimes larger than that of independent pairs (similar to the case of ΦISD); the key difference is that Xu et al. (2024) report the raw lg distance whereas we obtain p- values from the raw distances using PERMTEST. We hypothesize that
We also validated our tests on the Mistral architecture- we compared the weights of the hy- brid StripedHyena Mooaaa-Nuusuaa 7B (Poli et al., 2023) with Mistral-7B-vD) 1 and find non-independent parame- ters via Φun We compute 6 on all parameters, which allows us to identify non-independence between specific parameters of the models - such as the self-attention matrices - rather than as models as a whole, and report values of Φvς among certain parameters in Table 6 in
7

---

Independence Tests for Language Models
Appendix F.I. From the small p-values, we infer that the embedding layer and some self-attention matrices were likely shared between the two models.
were likely used in initializing Llama 1mma-3333 3.2aaa3.2333 2-B and
1. amaam... 2-11, as Meta reported that the first two mod- els were pruned from the third (MetaAI, 2024). We match MATCH with block from 0₃ and j from 02. such that (t.5)
(i.5)
3.2.1. SIMULATING STRONG-ISH ADVERSARIES
MATCH is less than 1e-4. We report the matched layers between the L.1 ama-3. 1 and allma-33 2 models in Fig- ure 3 and in Appendix F.3.
A significant difficulty in evaluating the robustness of our test ΦMATCH to adversarial transformations is that we cannot exhaustively enumerate all such transformations. Recalling that @MATCH specifically considers the MLP layers contained within two models, we attempt to fool it by randomly reini- tializing and retraining these MLP layers individually, thus simulating somewhat strong adversary.
Figure 3. We evaluate (t.))
MATCH between all pairs of GLU MLPs of
We reinitialize the first GLU MLP module of a model 0 with an MLP with double the width, and using Algorithm 5 (generalized robust test), we train &2 with random Gaus- sians as the training distribution P. We retrain each of the 32 MLPs (keeping other layers fixed) of rrcuua--bbvz (a finetune of Llama- 2- -)) for 10k gradient steps (until the loss curve plateus). (Additional hyperparameters and a learning curve are in Appendix F.2.) For all 32 runs, we compute ΦMATCH for the retrained model with the origi- nal ..1am-2-77-be and find ΦMATCH remains very small between the non-independent models even after an MLP has been retrained. For example, retraining the first MLP module, MATCH on the first MLP was less than E = 2.2c- 308, indicating that the two models are not independent. We find the same is true for the other MLP layers as well (f)
Llama 3.1-8B and Llama 3.2-3B. Arrows indicate if (4.7) @MITCH < le-4 and suggest which Transformer blocks of Llama 3.1-8B were kept in the pruning process to initialize Llama 3.2-3B.
We also identify which hidden units were most likely shared between the blocks when MLP dimension is re- duced (from 14336 to 8192) during pruning, from the permutation x returned from the up projection matching, MATCH (6) (1 uppp g2. We plot the activation matching for L1 allma-13 3 1- and 11ama-3.22 in Appendix F.3.
4. Related & Future Work
A related line of work known model fingerprinting (Xu et al., 2024; Zhang et al., 2025; Jin et al., 2024; Yang & Wu, 2024) plants a secret signal in the weights of a model so that anyone who knows the key can detect the fingerprint from query access to the model (or fine-tunes of the model). For example, Xu et al. (2024) propose fingerprinting a model by fine-tuning on secret random string; fingerprint detection then resolves to prompting a putative fingerprinted model with prefix of the string. Unlike Xu et al. (2024), we do not intervene on the training process of the models we test; however, we do require access to model weights.
(i.e. MATCH when evaluated on retrained layer f), with full results in Table 7 of Appendix F.2.
3.2.2. GENERALIZING TO DIFFERENT ARCHITECTURES
As we describe in Section 2.3.2, can also apply our test to model architectures which do not contain GLU MLP submodels. For example, the GPT-2 architecture uses a standard 2-layer MLP rather than a GLU MLP. We apply our test (Algorithm 5) to GPT2 PMC and gpt where the former is a finetune of the latter (Radford et al., 2019). We use 30k training steps with an isotropic Gaussian input distribution learn the GLU MLP parameters with which we replace the original MLP submodels in each model. The test yields a value of 3.034e-61, thus distinguishing the two models as dependent. We show additional results on independent and non-independent models (of Llama and GPT architectures) in Appendix F.4.
Finally, a separate line of work on text watermarking aims to attribute -eeee--ge text by planting a watermark when sampling text from the model (Christ et al., 2024; Kirchenbauer et al., 2023; Kuditipudi et al., 2024; Aaronson & Kirchner, 2023). Because it intervenes on sampling, text watermarking is inapplicable to open-weight models, the focus of both model fingerprinting and our setting. Recent work demonstrates that models can directly learn to generate watermarked text but also finds the learned watermark is not robust to further fine-tuning (Gu al., 2024).
3.3. Fine-grained forensics and Localized testing
Finally, we use ΦMAACCH on models pairs with different di- mensions, specifically on pruned model pairs, when model dimensions are reduced by preserving only select weights.
Future work can consider differentiating between fine-tunes of the same base model to reconstruct a complete "family tree" of model lineage is possible (c.g. infer Llemma is a direct fine-tune of CodeLlama) (Yax et al., 2025), and whether robustness against adversarial attacks is solvable with exact guarantees warrants further exploration.
In particular, we were able to identify the specific Transformer blocks of Llama-2 3.1--..11 1-BB whose weights
8

---

Independence Tests for Language Models
Acknowledgments
Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang,
X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhu, Y. X., Zhang, Y. Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma. Y. Liu, Y., Guo, Y. Wu, Y., Ou, Y., Zhu, Y., Wang. Y., Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y. Ma, Y., Yan, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F,, Ren, 7. 7., Ren, 7.,, Sha, 7., Fu, Z., Xu, 7., Huang, 7.,, Zhang, Z., Xie, Z.. Zhang, Z., Hao, Z., Gou, Z., Ma, Z.,
We gratefully acknowledge the support of this work by an NSF Frontier Award (NSF Grant no. 1805310) and Omidyar. Sally Zhu was supported by Stanford CURIS Fellowship. Ahmed Ahmed is grateful to be supported by NSF Graduate Research Fellowship and Knight-Hennessyy Fellowship.
Impact statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, some of which we discuss in the introduction and none which we feel must be specifically highlighted here.
Yan, Z., Shao, Z., Xu, Z., Wu, Z., Zhang. Z., Li, Z.,, Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song. Z., Gao, Z.,
and Pan, Z. DeepSeek-V3. Technical Report, 2024. URL https:/ /ttp///aaxvvvv/aa//eee/eeeeeaaaa wttpo/rgggaaaaaaoeee httpaaass222211110
References
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshom, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chemn B., Caucheteux, C.,, Nayak, C., Bi, C., Marra, C., McConnelll, C., Keller, C., Touret, C., Wu, C.,, Wong. C.,, Ferrer, C. C., Nikolaidis, C.,, Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia Olamo, D., Perino, D., Hupkes, D., Lakomkin, E.,, AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, A., Kloumann, I., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J.. Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia J., Al- wala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala- Yeary, L.,, van der Maaten, L., Chen, L., Tan, L.., Jenkins, I.., Martin, I.., Madaan, I., Malo, L.., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M., Pa- supuleti, M.. Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M.. Pavlova, M.. Kambadur, M.. Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji. N., Duchenne, O., Çelebi. O., Alrassy, P., Zhang, P., Li. P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapa- thy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunoy, S., Nie, S., Narang, S., Raparthy, S., Shen,
Aaronson, S. and Kirchner, H. Watermarking GPT Outputs, 2023.
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q.. Deng, J., Biderman, S., and Welleck, S. Llemma: An Open Language Model for Mathematics. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview wwt. forum? 1==
Christ, M., Gunn, S., and Zamir, O. Undetectable Water- marks for Language Models. In Agrawal, S. and Roth, A. (eds.), Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Ma- chine Learning Research, pp- 1 1125-1139 PMLR, Jun- 03 Jul 2024. URL https: http://pooceeen...e/e////o mlr. html.
press / /v2 4 http//hrriit/
DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang. D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H. Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L.., Liang, J., Guo, J., Ni, J., Li, J.. Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K. Wang, L., Zhang, L., Xu, L., Xia, L., Zhao, L., Wang, L., Zhang, L., Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P. Zhang, P., Wang, Q.. Zhu, Q., Chen, Q. Du. Q. Chen, R. J. Jin R. L., Ge, R., Zhang, R., Pan, R., Wang, R., Xu, R., Zhang. R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang. T., Yun, T., Pei, T., Sun, T., Xiao, W. L., Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao, W. Yu, W.. Zhang, W., Li, X. Q., Jin, X., Wang, X., Bi, X., Liu, X., Wang, X., Shen, X.,

---

Independence Tests for Language Models
S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky. S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T. Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang. X., Tan, X. E., Xie, X., Jia, X.. Wang, X., Gold- schlag. Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang. Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, 7., Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoia,, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg. A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisemman, A., Yazdan, A., James, B., Maurer, B., Leon- hardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montallo, B., Parker, C., Burton, C., Mejia, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Ozgenel, F., Caggioni, F., Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lak- shminarayanan, G., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Molybog, I., Tufamov, I,, Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang, J.. Chan. J., Zhen, J., Reizenstein, J. Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J. Wang, J., Wu, K., U, K. H., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakho- tia, K., Huang, K., Chen, L., Garg, L., A, L.,, Silva, L.., Bell, L., Zhang. L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsim- poukelli, M. Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Keneally, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vy- atskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Sing- hal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P.,
Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina,
P., Ratanchandani, P., Yuvraj, P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng. S., Lin, S., Zha, S. C.. Shankar, S., Zhang. S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chin- tala, S., Max, S., Chen, S., Kehoe, S., Satterfield. S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subrama- nian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta,, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang. Hao, Y., Qian, Y., He, Y., Rait, Z., De Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The Llama Herd of Models, 2024. URL https: / arxiv. wrg.orgg/aaoooo /agg////////// 22007 21783.
Groeneveld, D., Beltagy, L., Walsh, E., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M., Pyatkin V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Sol- daini, L., Smith, N., and Hajishirzi, H. OLMo: Acceler- ating the science of language models. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653/w1/2024..cll wwgph//llllll..g..e2///////
841. URL acl- pol-long.841/:
Gu, C., Li, X. L.,, Liang, P., and Hashimoto, T. On the Learnability of Watermarks for Language Models. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: opwnrevien.eeeeeeeooeeeeeeroeewe .nt.iiewweee////eeeee forum? (ddummid=90 1ete21d=900000 9k0kkkk1k...
Jin, H., Zhang, C., Shi, S., Lou, W., and Hou, Y. T. ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models. In 2024 IEEE Conference on Communications and Network Secu- rity (CNS), pp. 1-9, 2024. doi: 10.1109/CNS6248..2000 10735575.
10

---

Independence Tests for Language Models
Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ra- maswami, A., Cihon, P., Hopkins, A., Bankston, K., Bi- derman, S., Bogen, M., Chowdhury, R., Engler, A., Hen- derson, P., Jernite, Y., Lazar, S., Maffulli, S., Nelson, A., Pineau, J., Skowron, A., Song, D., Storchan, V., Zhang, D., Ho, D. E., Liang, P., and Narayanan, A. Position: On the Societal Impact of Open Foundation Models, Pro- ceedings of the 41 International Conference on Machine Learning, ICML'22, JMLR.org, 2025.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. In International Conference Learning Representations, 2017. URL https:// openreview net mett/oorunnlddd31112aaa
MetaAI. Llama 3.2:
Revolutionizing edge Al and vision with open, customizable models, 2024. URL https: / //ai meta com/blog.
Mosteller, F. and Fisher, A. Questions and An- swers. The American Statistician, 2(5):30-31, 1948. ISSN 00031305. URL hitp: wwwp. jstor.org/ stable/2681 650.
Kirchenbauer, J., Geiping. J., Wen, Y., Katz, J., Miers, I., and Goldstein, T. A Watermark for Large Language Models. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, (eds.), Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. wwt.p...0000..00 17061-17084. PMLR, 23-29 Jul 2023. URL httcctps:/pprcceed... mmrr. www.m....sss.....mm.mmmmmmm.m... v202/ki htt.e/iiicchhhbeue2eer2. htmmm html.
Peng, S., Chen, Y., Xu, J., et al. Property Protection of DNN Models. Web, 26:1877-1911, July 2023. doi: 10.1007/ s1 11280-022-01113333 URL https: /ttp///ooiooo org/orr
Intellectual
World Wide
Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, Robust Distortion Watermarks for Language Mod- els. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https: http///eeeeereeeeeeeeeeee net
Poli, M., Wang, J., Massaroli, S., Quesnelle, J., Carlow, R., Nguyen, E., and Thomas, A. StripedHyena: Mov- ing Beyond Transformers with Hybrid Signal Process- ing Models, 12 2023.
com/
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine. S. (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 51991-52008. Curran Associates, Inc., 2023. URL https: http::::///cccceenggg..ooo neutpidpi. cc httpppeefflleepppperr///////////
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multi- task Learners. 2019. URL https: wptp:////idd.o.eiii.... openai.
models wtw.arrruuuuuuerriieeeeeee learners.pdf.
Ramshaw, L. and Tarjan, R. E. On Minimum- Cost Assignments in Unbalanced Bipartite Graphs.
https: / /ttss//appp //ppi...p//////// semanticscholar hts///oopuIID::6641100
pdf.
Lin, J. Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theor, 1551 September 2006. ISSN 0018-9448. doi: 10.1109/18.61155. URL https: //doi, .rr 10 1109 09 61115.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin-
son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muen- nighoff, N., Naik, A., Nam, C., Peters, M., Ravichan- der, A., Richardson, K., Shen, Z., Strubell, E.. Sub- ramani, N., Tafjord, O., Walsh, E., Zettlemoyer, L., Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Ku, L.-W., Martins, A., and Srikumar, (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.18663/w1/2224.aaoooogg 840. URL https: //aclanthology
Liu, Z.. Qiao, A., Neiswanger, W., Wang. H., Tan, B., Tao, T., J., Wang, Y., Sun, S., Pangarkar, O., Fan, R., Gu, Y., Miller, V., Zhuang, Y., He. G., Li, H., Koto, F., Tang, L., Ranjan, N., Shen, 7., Iriondo, R., Mu, C., Hu, Z-, Schulze, M., Nakov, P., Baldwin, T., and Xing, E. P. LLM360: Towards Fully Transparent Open- Source LLMs. In First Open-Source
Conference on Language Modeling, 2024. URL https: //openreviewt wtt.ieewweeeeeeeeeeeeeeeeeeeeeee / forum? d=:ddh JOQZFF
Mensch, A. Mistral CEO confirms Miqu model leak, August 2024. URL st
Accessed: 2024-
08-15.
hwwp:hhhhllll..o.///.oooo org/2024 ..e...c.... a -lng.. 840/.

---

Independence Tests for Language Models
Sudalairaj, S., Bhandwaldar, A., Pareja, A., Xu, K., Cox, D. D., and Srivastava, A. LAB: Large-Scale Alignment for ChatBots, 2024. URL
International Conference Learning Representations,
2025. URL https: htt.:::s//opeeeiiee.... net forum?
Sd=SnDmPKOJOO
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Bahaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I, Nie, Y. Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open Foundation and Fine- Tuned Chat Models, 2023. URL https:// hetps://arxiv.. org/ /ts//ab//tt/ 2307. 09288.
Xu, J., Wang. F., Ma, M. Koh, P. W., Xiao, C., and Chen, M. Instructional Fingerprinting of Large Lan- guage Models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3277-3306, Mexico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 180. URL naacl-long, 180/.
Yang, Z. and Wu, H. Fingerprint for Large Language Mod- els, 2024. URL / abs /2407. 01235.
Yax, N., Oudeyer, P.-Y., and Palminteri, S. PhyloLM: Infer- ring the Phylogeny of Large Language Models and Pre- dicting their Performances Benchmarks. In The Thir- teenth International Conference on Learning Represen- tations, 2025. URL https: /ttps////peeeeiieeeeee .ntt/
Zeng, B., Wang, I.., Hu, Y., Xu, Y., Zhou, C., Wang, X., Yu, Y., and Lin, Z. HuRef:: HUman- -aanable Fingerprint for Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttp:://opeereeeeeeeet id-R12gnEZsOH.
net/ forum?
Zhang, J., Liu, D., Qian, C., Zhang, L., Liu, Y., Qiao, Y., and Shao, J. REEF: Representation Encoding Finger- prints for Large Language Models. In The Thirteenth
12

---

Independence Tests for Language Models
A. Proofs of Main Theorems
Proof of Theorem 1. From our assumptions on and P and the fact that {Te){=1 are independently drawn, it follows that the collection (0,)) comprises T exchangeable copies of 0,. The independence of 02 and 0₂ thus implies {(m,(1)) 02))E= comprises T exchangeable copies of (01,0)), and so the claim follows by symmetry - o 02) is identically distributed as (0)))(()),)))1)))
(( 02) will have uniform rank among the other values. Ties (Ør s(0,, 02)) randomly contribute to p, so symmetry still holds and the p-values will be uniformly distributed under the null.
Proof of Theorem 2. As M is II-equivariant map, if 01 1 02 then letting π = LAP(C") in MATCH is equivalent in distribution to sampling π ~ TT). Then the output of MATCH is identical in distribution for any pair of independent models, and can be converted to a p-value using SPEARMAN and the distribution for the Spearman correlation coefficient (I-distribution with 2 degrees of freedom).
Proof of Theorem 3. Let 04 A(=1") om (89)) for Tı, 2 lid. Unif(IT). Then 0f is an independent copy of 0n since taking
the composition Ti o 22
(6) (f) (0i) yields an independent copy of 0, for any π. n2 E II. From 0, 1 02. it follows for E E (i.j} that MATCH( M (f) ( (8))) is identically distributed to (ATCH(M(()())) (O), (0)). The result then follows from the fact MATCH equivariant with respect to permuting the rows of its arguments: particular, for any и E II we have MATCH(TW1, W2) = *MATCH(W1, W2).
B. Randomized Learning Algorithms
One notable (non-contrived) category of deep learning algorithms that not permutation equivariant are those with random dropout masks to hidden units during training. In particular, once we fix specific setting mask values to specify a deterministic learning algorithm, this algorithm will not be permutation equivariant unless the individual dropout masks are all permutation invariant (which is highly unlikely). We provide generalized statement of Theorem for randomized algorithms.
Definition 5. Let II C ④ + O. Let π E II and 6 E θ, with 0 ~ A(0)), (0) and O' ~ A(T(()))- A randomized learning algorithm A: > PO)) is I-equivariant if and only if 0 _ 0'.
Theorem 4. Let é x o + be a test statistic and c e + be finite. Let A: e - P(O) be n-eqquvariant and let E P())) be IT-invariant. Let 01, 0g E θ be independent random variables, with 0, ~ A(oP) for o ~ Pa. Then p = PERMTEST(AL, 02) is uniformly distributed on { V=1
Proof. The proof is identical to that of Theorem 1.
C. Transformer Architecture and Notation
We consider models with the Llama Transformers architecture and define the notation henceforth, although this can casily be extended to other Transformer architectures.
Following the definition of fmip in Example 1, can define abstraction of the full Llama language model architecture consisting of L Transformer blocks sandwiched between an input and output layer. For the sequel, we will abuse notation in applying fuip to tensors by broadcasting along the last axis. We use d, n E N to respectively denote the model dimension and sequence length, where θLM = O x θ O x Ocnt with Opteck denoting the parameter space of each Transforme block and Om, Ocu denoting the
parameter spaces the input and output layers. We decompose Obtck = Oum X m and use frss eam x Raxd + gaxd to denote all remaining parts of the Transformer besides the MLP. The inputs to fest are the input and output of the MLP, and the output of fist is fod directly to the MLP of the next layer. In particular, free takes the input and output the MLP of layer i, and first performs the residual connection following the MI. of layer then the self-attention and normalization components of layer + 1, and returns the input to the
MLP of layer i 1. We use fi Om X - axd and fou: (L)) x = + respectively denote the input and output layers, i.e. the elements before the first MLP and after the last MLP, Putting everything together gives the following definition of the model; we introduce the notation x in the definition as a matter of convenience track intermediate activations.
Definition 6. (GLU Transformer model) Let (0m, (9 1 Oon) E Orm and X E X, with g(3) fim(X:0)= = [om( 0mw))
(0(9) ((4) Then
(0w)) for x = (X; (i.) and
x( frest (x((()) (x
(6)
For a Llama model, table 2 describes the shapes of the model weight matrices for i = 1,..., L, for V (vocab sizc), demb (the hidden dimension), and dmip (MLP hidden dimension). Following Definition 6, we have Om = (F)), o = (0ft)) ((0) where
13

---

Independence Tests for Language Models
Parameter name
Notation
embedding
E E R Kilems
input layernorm
Tinput, E RIxdom
attention query matrix attention key matrix attention value matrix attention output matrix post-attention layernorm
Wo, E IRdeus denb
WK, E R X dech
WV, Re X damb
Wo, E Rd X deub
ppsst-amn, 1 R.Xdem
MLP gate projection
Gi 1Rd Xheee
MLP up projection
U, E RR X daste
MLP down projection
Dj RRess X chup
final layernorm
Tinal E Rixdeen O E Rd XV
linear output
Table 2. Llama model architecture and dimensions.
(Gi: U2, D:), and lou = (Yiimll, L). We now describe forward pass of the We define softmax function on vector (VI,..., Un). softmax(v).
(4) 0 (Yimput,i, WQ WK.,, Wv,i, Wo,,, mpont-at)) o
model.
e
softmax(v): k== e%
On batched imput NNxxxs Xxxm where each X(h) [wn| /uil... [[m]] E R nm with column vectors w;, we define the softmax
softmax (x [softmax (ummmax(u)) )I... softmax );
softmax(X) (X) = jsoftmax( (I [s.ooooaax (x ())
For a forward pass of the model fix(X;0), 0), consider an input sequence of tokens X E (0, (0,1) NxV as one-hot vectors where n sequence
length. Then
We feed the input through:
1. (fin) Embedding layer:
x fin(X;0m)) XE I Nidha
2. (fans fmips fpoa) For each Transformer block = 0, l..., through funo, Smip. and fpea:
(a) Input layernorm:
x
( X
LNN
Tipput
Var(X
+
(with variance over the last axis) for some offset (typically le-6).
(b) Causal multi-head self-attention: Split x(() on the first axis into nheads X(()
JN. LNy. abeuds On each head ((1) 3
(()
(t)
WT
(X WT LN WR.)
SA. self-attn( (allf-attn(((00 ) = softmax
x
W W., Wo..
damb
IN
and concatenate SA.j along the first axis again x=()
(c) Dropout and residual connection: x(()
x(() Dropout (X)))
(d) Post-attention layernorm:
x DRi
(+)
x
X
Var(x(fl) +
(with variance over the last axis) for some offset Then have
fum (x! 1), ((N)
x-(1)
14

---

Independence Tests for Language Models
Parameter name
0
Temb (0)
Tmlp (0)
embedding
E
ErTemb
input layernorm attention query matrix attention key matrix attention value matrix attention output matrix
Vinput, Wa
Tinpus, Temb
Tinpor,
W WQ,
Wa,
WK,
W Tenh
WK,
Wy.
WembTemb
Wv.
Wo,+
Temb Wo,i
Wo,i
post-attention layemorm
posttattm.
Ppost-atta, <Temmb
7post-aitn.
MLP gate projection
G2
GyTemb
Tmlp, G2
MLP up projection
U
UgTemb
Kmpp, ;U;
MLP down projection
D.
Temb Di
IT
Dem
final layernorm
Yinal
Temb
Yinal
linear output
O
Temi O
0
Table Transformations Temb and Tmlp applied to Llama -rchittccci model.
(c) Next, we feed through faip, the multi-layer perceptron:
fmip fmip(XLN L821 mip)) = X = (ox(x (X)
for some activation a (c.g., SiLU). Finally, feed through fpe.. dropout and the residual connection:
Spost (1i))
x ((-1)
x-RR Dropout Mx))))
3. (fou) Final layernorm on the output x (N+)) from the final Transformer block:
x(1)
X
Yiinal
Var(xx! +
(with variance the last axis) for offset Then, linear output embedding and softmax mapping output probabilities:
fou(( = softmax (X O²),
(X(2O²²)
which defines the entire forward pass fLM(X;,).
D. Model Transformation Class
We describe two sets cquivariant transformations II on a Transformer model as described in Appendix C. (Abusing notation), the first set, Lemb. consists of clements Temb where Temb E R dan Xdems is permutation matrix. The second set, Immp. consists of elements Tmip where kmip E Puip hulp is permutation matrix.
1. (0): Applying embedding permutation STemb (permuting rows or columns).
X by left right multiplying all relevant matrices by Sented
2. Tmip (0): Applying MLP permutations Tmip.1 E aad to MLP layers.
These permutations are applied such that the outputs of the original model 0 and the permuted model II(0) remain aligned. We describe the details in Table 3.
E. Additional Constrained Setting Experimental Results
We report p-vulues from the statistics ot, syin, and eyte all 210 model pairs (from 21 Llama 2-architecture models) in Figures 4, 5, and where the model names colored by base model (ground truth). For all statistics, the p-values independent model pairs uniformly distributed, while they are all significant at 0.01 (smaller for &etn and @ for fine-tuned model pairs.
15

---

Independence Tests for Language Models
0.20
2:41
0:03
2:07
D:21
0:31
0:31
2:1
E:1
C.1
341
0.01
241
C40
C43
DAR
CAI
0.12
0.75
0:14
2.4
601
S-0t
0:01
S:0
0:04
0:04
S-1
6:63
6:01
0.01
9:21
0
0.09
0.44
0.12
237
233
0001
C40
0.01
0.29
039
333
0:00
001
CEC
0:03
0.15
039
114
D0c
0:00
D43
0
0.03
666
6:0
0.01
CG
0.00
0.52
stoemble
0.23
0.2L
0L03
0.22
2.29
Houshesee
0:21
DAT
0:99
2.54
24
DAM
DUY
0:0
3:2h
17
225
CET
DAW
821
21
openi www.rraiaoooooeee.. AI
0.03
8:76
0:51
0.7
5.25
57
235
CS4
DX7
D:5
6:71
0.45
0:33
15
553
RE1
C
CM
0:00
0:66
0:08
0.00
0.71
3:23
0.29
2:23
EM
C32
CAT
0.24
0.22
0.22
3.29
2:33
8:2
iestiiini Cobatanaa 2013
resettine GAbLERD to
Aanaa
dew wewetieerreeeeoooo Kndaes Samta
Figure 4. Results of p-vuuu from di, on all model pairs.
16

---

Independence Tests for Language Models
6:49
6-31
0.313
0.-9
0.791
$:397
8.433
0143
0.203
2:41
3:314
5:024
0:151
5.599
0.311
5.247
6013
DL/18
6:142
640
0.737
0-739
6:381
0.22
0.102
8:035
8:579
933
0:03
1.70
0.144
0.533
0.007
Salatry
8.377
0:338
3.31
8770
0363
635
0267
Chacl Bemens, mutratreet cometmath
6.30
872
494
6.134
0.202
procennnus Barner TD-L coscmidl
0.99
0.00
2:3
0.43
0.24
0:113
0.613
Houuhhesse Noun dereeeaaaaaaaaaa 2016-
0.44
0:03
0.000
8.479
9.31
8.737
0.25
0.643
0.711
6:57
0.115
2:33
0.57X
0.597
3.577
0:15
8:42
0:81
0:233
0.713
0.199
0.001
8:286
03322
0,22
0403
$22
6:322
6:67
6:115
0:22
0.215
200
3.2
5:309
0:362
0:55
0.21
0.903
0.227
6:14
3.366
5517
0.22
0,25
JUST www.RREEE Play
1:11
03/1
0122
0.27
0/19
0.50
5:322
Integrant Ie-intrccc
Besageast Mas.groiiggeene Baok
det Staidiio
Figure 5. Results of p-values from epce on all model pairs, where 2.2e-308.
17

---

Independence Tests for Language Models
8:33
6:01
6:223
CSSC
0.00
2.342
3.231
0::47
0:332
0:132
C1044
0.311
0.219
1.31
0.604
6:44
6:714
6:17
0.572
3.33
C-43B
6-122
6:141
6:447
0.103
0-911
0.500
6.193
4.422
6-422
6:77
0.8m
www.arra Salett
6:03
0.33
0:243
0443
0.403
0.700
8:008
0.044
6:388
0.99
M0G
0.315
0.0
srundock Barno atoemblee
0112
0.73
0.332
0.20
6677
6.72
6/917
0.311
9137
6:1
6:127
6:118
0.027
0.00
01118
2.11
1.77
0:141
0.50
0011
1:14
opener www.oooeeieoeeee......o REP8L2
#377
3.79
1:3
0.447
6:157
0:231
6:511
0.407
0.51
2:14
3.17
0.79
0:357
0.161
0.91
6.97
277
135
0.17
EXE7
6:43
0OT1
3.403
0.55
515
500
132
E31
0179
0.14
0357
0:21
testiinni Www.ernaiaaeeeaaamoo Cabekanna 2014
ieeee daoe to
you Micomiiiiiaai
Keeeiiiiie Andinn Aamaaao
Figure 6. Results of p- values from dy on all model pairs, where 2.2e-308.
18

---

Independence Tests for Language Models
train tokens
euc
2 a
Qtg
(MATCH
@ssD (log)
100M
0.641
0.119
0.07
0.809
-11.81
IB
0.789
0.483
0.06
0.443
-11.05
10B
0.707
0.277
0.93
0.343
-11.28
18B
0.819
0.141
0.64
0.027
-11.03
Table 4. Results for eytte @yte, and Φмиtch evaluated training checkpoints between two independently-rraiee OLMo models.
0. = L. -2-mmma-----hf,0000
600 E
miqu-1-70b-pytoor
1lama-3.1-22 3.1--0B
0.571
0.539
Table Results of dyce (aggregated with FISHER) with 0 L-ama-2- -20-h and 02 ranging over the listed models.
E.1. Identically distributed, Independent models
We further evaluated the efficacy of our tests through ablations by training two models with the same architecture on the same dataset that only differ the choice of random initialization of randomness. Specifically, similar (trained using the same learning algorithm) but independent (randomly initialized) models, as non-independent.
that test does not incorrectly detect two
To verify this, we randomly initialized model with OL (7B) architecture (Groeneveld et al., 2024) trained it on the Dolma v1 7 dataset ((Soldaini ct al., 2024)). We trained a second model with independently chosen initialization and data ordering.
We keep checkpoints for both seeds after 100M, IB. 10B, and 18B train tokens and evaluate the statistics dyrt,, @yte and CMATCH on the two models each training checkpoint, reported in Table 4. We highlight that the p-values broadly distributed, validating tests support independence even on similarly-trained but independent models.
E.2. Tests for Larger Models
Next, we evaluated our tests on larger models. We ran @ett on four 70B parameter models with the Llama 2-70B architecture shown in Table 5, and in particular, verify that Miqu-70B is not independent from Llama 2-70B.
F. Additional Unconstrained Setting Experimental Results
We report values of ΦыллTи all model pairs Figure The statistic low (<. 10 10-300) for all non-independent model pairs, and uniformly distributed independent model pairs, empirically acting as p-value.
F.I Striped Hyena Experiments
We report (yte) specific parameters from Stri pedHyena -aniiyeenn--uu---- Noo-s 7 and Mistral - shown in Table 6. We no longer only evaluate dyce) MLP up projection matrices, that investigate similarity in other parameters well. These p-values longer satisfy the independence requirement Theorem 2, so we not aggregate them with FISHER.
Parameter name embedding
Notation
E w
1.6le-16
attention query matrix
6.17e-190
w
attention key matrix
K
1.47e-7
w(l)
attention value matrix attention query matrix
1.56c-114
W
6.17e-190
Q
w
attention output matrix
0.010
MLP gate projection
G()
0.517
MLP up projection
(()
0.716
MLP down projection
D(I)
6.03e-80
Table dyte on parameters from StrripedHyenna-ooooe Nouaa-Nousss-aa 7B and Mistral-TPe 1B-1--7BBv0 1, some with low p-values.
19

---

Independence Tests for Language Models
B:49
6.544
0:01
0.001
0:411
0.993
3.199
0.795
3-141
0.75
0:103
0.717
3799
3372
0:44
0:00
0.204
3.770
6913
8885
0.71
0.343
0.19
1.0
01144
5.47
3310
0.75
0.077
626
Alfeeif Suliry
0.227
0.00
0:644
0.116
Doothoonil Semens, Th murrstrect comemmath
0.477
6.8
0.39
6
prommnn Barno 1D- sounmii
5:411
0.155
6.112
0:222
6.211
0.991
Houshtaaa ti Noci moao--arta-aaaa 2-16
0:19
334
6:03
0:147
0.011
0.612
3.700
9:01
3.55
8220
6502
0.71
0:164
0:73
0.311
3.954
0.3
0.555
3322
0.41
0343
0.207
0.004
3.57
6:33
7777
3553
8:300
C447
0:45
3.47
0:01
0.706
0441
0:334
357
3:334
6:51
6442
0:607
0:03
0.004
0014
300
6:330
679
6254
6:15
3:
0.594
3.519
0.77
305
3:333
0:356
0:41
3/99
OCT
0.304
3271
Figure 7, Results of values of sMATCH on all model pairs, where 2.2c-308.
20

---

Independence Tests for Language Models
MLP
Loss
logio MATCH
MLP
Loss
log1o (MMATCH)
MLP
Loss
logio (MATCH)
0.0048
479
12
0.0060
-342
23
0.0043
-593
0.012
-485
13
0.0058
-330
24
0.0047
542
0.0026
614
14
0.0066
-323
25
0.0050
497
0.0034
-880
15
0.0063
-414
26
0.0051
-534
0.0030
523
0.0061 16
-394
27
0.0052
482
0.0035
513
17
0.0063
-445
28
0.0061
-477
0.0041
533
18
0.0055
-515
29
0.0065
433
0.0042
464
19
0.0045
-571
30
0.0098
361
0.0050
439
20
0.0045
-512
31
2.313
26.4
10
0.0050
377
21
0.0047
595
32
0.0114
- 174
11
0.0060
-365
22
0.0043
-555
Table 7. ΦмхICH on individual blocks between L. ama-2-7b-hh and vicuna-7h-vv. after retraining MLP layers.
F.2. MLP Retraining Experiments
We retrain each of the 32 MLP layers by fceding in random inputs through the original MLP (gate, up, and down projection matrices.) We train for 10000 gradient steps using MSE loss and Adam Optimizer with learning rate of 0.001 and batch size of 5000. A sample learning curve Figure 8.
Learning retraining first MLP of Vicuna-1.5-7b
2.0
ISS loss
1.5
10-
0.0
2000
4000
6000
8000
10000
train steps
Figure 8. Learning curve for ML. retraining.
The ML.P retraining results for all ML.P layers of vi cuna-7b-vl 5, compared with 1,1 1l-amm------h- are in Table showing the statistic robust to retraining of layers.
F.3. Localized Testing
As described in 4.4.2, we run ΦMΛTCи all pairs of Transformer blocks between two models (of different architecture), as long they share the GLU structure. addition the Llama results, report results of matched blocks the Bheared-LLaMe and Nvidia-Minitron models, which are both pruned from Llama models.
particular, we were able to identify specific Transformer blocks of AsB = 1.1 ama 3. 1-88 whose weights were likely used initializing Asp = 1.1 a11am33.. 2-38 and 01 = 1.1 ama-3 1.aa-.22-1 12-18, B, as Meta reported that the 1.1 ama- 3. 2- 2-38 and 1,1 ama-3. 3. 2- models were pruned from am..m.
3.11. 1 -18 ((MetAAl, 2024)). We use ÓихтCи on all pairs of MLP blocks, where (dean hoan- No.a
(4096, 14336, 32). heg Neg,) ) (3072, 8192, 28), and (de, be,,- No. (2048, 8192, 16). We match blocks when the (5.5) statistic MATCH from block of model and block of model is less than 1e-4, reported in Tables 8 and 9 (with the same for the other matchings in this section).
21

---

Independence Tests for Language Models
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
j MATCH (OSB. 8BB)
5
6
10
11
12
13
14
15
i
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
j MATCH ( 03B) < le - 4
16
17
18
19
20
21
22
23
24
25
26
27
28
Table 83g = Lmama- 1- blocks matched with ABB Llama 3.2-3B blocks using OOMRTCH
i j MATCn 018) < <e
1
2
3
4
5
6
7
9
10
11
12
13
14
15
16
1
2
5
6
9
17
18
20
21
22
23
25
26
27
28
30
31
32
j (OSB, 01B) < le 4
(tJ)
10
11
15
16
Table 9. AAB = Llama-3 3.- 3.1-8B blocks matched with 0LB = Llama-3. 2BB blocks using dMATCC
Next, we have Sheared-LLaMa 2.7B, with 32 Transformer blocks, hidden dimension 2560 and MLP dimension 6912. All 32 blocks align with the 32 blocks of Llama 7B, although both hidden and MLP dimensions have been reduced through pruning.
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
j: o
MATCH (0),02) (0,CH((1
(t.)
le 90
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
j (MTTCH (0), le 90
(id)
17
18
19
20
21
22
23
24
25
27
28
29
30
31
32
Table 10. 0 Sheared-LLare
1. 3B blocks matched with 0
Llama-2- blocks using ΦMATCH
Next, have Sheared-LLaMe 1.3B, with 24 Transformer blocks, hidden dimension 2048 and MLP dimension 5504.
1
2
3
4
5
6
7
8
9
10
11
12
j MMTCC (0,,0) < 1e
2
3
6
7
8
10
12
16
13
14
15
17
18
19
20
21
22
23
24
j MATCH MATCH((1,, (2) le -
(6.))
17
18
19
20
21
22
25
27
28
29
31
32
Table 11. 0: Sheared-LLaMa 1. 3B blocks matched with 02 = Llama 1llmm-22-78 blocks using ΦMAICH
Finally, compare Llama 3.1 8B with avidia/Li aatiaaa/Lmaaa..... 1.1...1111 www.iiiittoon-----ottteeeee
pruned model by reducing from 32
16 Transformer blocks and able identify the likely shared blocks.
2
3
5
8
10
11
12
13
14
15
16
j (1.)) (0,, 02) 1e 90
2
3
4
5
6
7
8
9
10
11
12
13
14
15
32
Table 12. 0, = nvidia/Ilame 1-Minitron- -Bocoonnn-B------ ΦMAACИ
4B-44---- h Base blocks matched with 02 Llama-2-78 blocks using
E4. MLP Distillation Experiments
As we mentioned in section 3.2.2, we present further results on experiments where we distill a model without GLU MLP then test the efficacy of our approach. These models do not use GLU MLPS (instead, a different fecd-forward network) and GLU MLP
distilled as first FFN using Algorithm 5. The cases two non-independent models still have very small values of MATCH
(1)
22

---

Independence Tests for Language Models
81
0 GPT2_PMC
Independent?
(1)
MAICH
gpt2
X
3.034e-61
gpt2
artgpt2tox
X
1.049c-75
gpt2
dist 1igtt2 LLama-3.2-1 3-
X
1.079c-63
X
2.011e-70
openai-gpt
gpt2
0.359
openai-gpt
distilgptz Llama-3.0 3.2-112.1 2-1B
0,770
gpt
0,481
Table MATCH on models distilled with a GLU MLP.
Parameter name
0
Rot(O) = 0'
embedding
E
E Remh
attention query matrix Tinput, Was
input layernorm
Knppur
R₁ WQ.: diag(Ymmpu. i) Remb diag(
1
Vigue
attention key matrix attention value matrix attention output matrix Woi
WK.
R3 WK,s diag(Timpu.
Vimput, i) Remb diag(
Wv.
Wv,, diag(Tipput
Tippot a) Remb diag( RT Wo, ppsst-atm,
Tarpat
post-attention layernrr
ppost-am.. G
MLP gate projection
G, diag((ypsstanne poss-atto, (t) Remb diag(
Pont-arn,
MLP up projection
U
qUU, diag( (post-attt.)) à) Remb diag( poot-ataa.
MIP down projection
D;
Romb D3
0i
final layernorm
Yinal
Vinal
linear output
O diag (Yinna)) Remb diag
Table 14. Output preeutt-reeeeiingg rotation applied to a Llama architectre model.
G. Output-Preserving Transformations
An adversary could apply particular rotation scheme multiplying weight matrices by orthogonal rotation matrix that will also preserve outputs. We describe
transformation which breaks the invariants proposed by (Zeng et al.. 2024) by manipulating layernomss. While this list may not exhaustive, the following six transformations (with the first two described previously) "camouflage" the language model while preserving outputs:
T1. Permuting the rows of the embedding matrix (and subsequent matrices due to residual connections) by permutation Eomb R danc denh
T2. Permuting the MLP matrices (NV different permutations for each Transformer block) by permutations f....., Ex E (R K
T3. Rotating the embedding matrix (and subsequent matrices due residual connections) by orthogonal rotation matrix Remb R danb &
T4. Rotating the query and key attention matrices (N different rotations for each Transformer block) by orthogonal rotation matrices Ri...., Rx
R R dam
T5. Replacing all layernorms (input, post-attention, final) with vectors R' Xdes with non-zero elements
T6. Scaling the MLP matrices by constant non-zero factor
Consider model 0 of Llama architecture (Appendix C). Consider orthogonal matrices Rumb, Ri... R32 as described, well as elements. Finally, consider apply the rotation with these parameters to 0, to get new "rotated" model, (.) We generalize the set of transformations above as applying Rot(f) to model 6".
layernorms Thmpue, ..... T1 Tpod-aitn.11 I . Tpod--ntt.33 32 IR dunb with
constants
CI...., C32. C52. which we use transform the layernorms.
We transform all the original matrices of as in Table 14 (for = 1, 1,..., 32). Note that the transformations Tl and T2 are elements of Temb and Tmmp and the remaining transformations T3 to T6 are described in Table 14. Importantly. T5 is the transformation that (Zeng al., 2024)'s invariants robust to; unconstrained setting test ΦMATCи is robust all transformations, which show Table 15.
23

---

Independence Tests for Language Models
G.I. Breaking HuREF Invariants
Only transformations T3 and T5 are required to break the invariants from (Zeng et al., 2024). Their first invariant is M, E(Wq.) Ww,,) EE" at layer i. and for M' with embedding matrix rotation Henh where the layernorms T are replaced with T we have the invariant is
Ma E'(W..) ((WWa) (Wkka") E
M3 (ERans)
diag(
diagi Cimpon .) nZ.
R& R.Wx, (Tipua Pimpon, s Remndlagg (1 Vingar
(I E)
Vinuurs
E digg iimpur,
diag(( WW WK, diaddggggpppa... Remodlag
R E,
Vinput,
general M. * M2 unless the layernorm weights are equal constants. The other two invariants also do not hold due to changing the layernorms. (Note that our notation for Transformers is different than theirs.) Assuming in their invariant M, that Wi and Wz are the gate and down projection matrices of an MLP (this is not stated explicitly in the paper but can be inferred from experiments), the remaining invariants do not hold either.
Empirically, compute the invariants between Llama2-7h and independently trained models and hetween Llama2-7b and rotated finetuned models (including .lama2-7b) in Table 15. can see there little distinction between independent vs. non-independent model pairs.
Independent?
Ma
M
Me
ΦMATCH
Pp())
ØISD
vicuna- /b-na-
X
1.0
0.9883
0.9922
<
E
<
-10,874
Hous--ermes-LLame
X
1.0
1.0
1.0
<
<
-12.101
11ama-7h-hf
0.0884
0.0250
0.0400
0.049
0.595
0.253
-11.102
AmberChat
/
0.1289
-0.0093
0.0198
0.941
0.460
0.279
-10,281
Open1 Lama-vl
/
0.1084
0.0076
0.0057
0.286
0.357
0.703
-8.381
Rotated -1llamm--7----
0.0767
0.0908
0.1011
<
0.517
0.323
-00
Rotated calcuunn--bvl..
0.1553
0.0933
0.0977
<
0.688
0.857
-10.874
Rotated Heras--ereee---- 1eama-1a22--- 1lama-2-78
X
0.0332
0.0718
0.1060
< 0.772
0,240
-12.101
Table 15. Results for the three invariants Ma Mo, Me from (Zeng et al., 2024) between ama 2 and independent and non-
independent models.
G.2. Invariance of Outputs under Rotation
These transformations are particularly important because they preserve outputs we show in Theorem ??, and hence generally can go undetected, though ΩылTCh robust them.
Thcorem 5. For any input sequence X E {0, 1)a the outputs of models 0 and Rot(O) 0' are aligned, iee. fıM(X;0) = O
Proof. First, that an element-wiss wise product of one-dimensional vectors is equivalent multiplying by the diagonal matrix of the second vector, i.c for u, y E Rl
V. 7 vdiag(y))
We use this in our layernorm calculations.
I the output from the unrotated embedding layer be fi,X, = EX (for X (0, 1)" Then the output from the rotated embedding layer is fm,X,, E²) Remb)) Remh. Now consider Transformer block with input y and the rotated Transformer block with input y/eam. y passed into input layernorm, which returns
z LN.(v) Varly) +E
diag (iaprr))
Var(y) +
The rotated input layernorm on y' returns
Re
= =L=N((()))
2 Yinparr /Var(yRema) +
Ampurr
)ar(y')++ +
1
Remb dingdiag( (Vipu..s) = diag(
Remadiag((
VVaryy) +
Tinpur.
24

---

Independence Tests for Language Models
which follows from Ramb being orthogonal. Then we have the output from the unrotated self-attention is and the output from the rotated self atlf-attention with input z' is
=WQslaWkk))
softmax
WWV,
dhey
x' ( Rems diag(
Yapa. ))a
(,, Vingur, Rens
1)²)²
Naper.
softmax
dhey
z' (Wvvadia((Twngg) Tinuur, Rena
(R Wo,:)"
Vispur.
2' z'diagí
R diagg W RT (z'diag( diagd (inuu. 3) R?!
Tippe.
= =softmax
dhey
z' d'diag(
Remb diag (Tinpu, WT, W.
Vinpul.
diag(
R diag( (aaiiig(((/n/u Yirgus. W WK. WK,, (riggrrnuu,, x) Remadiag(
softmax
"ipprr
2 W. Wo, Remb
Vaey
zW Q WT k
softmax
zW Remb
V daay
= 1 Remb = w'
Then y and y' respectively from before the layemorm added residual connections + w and v' o/ w² VR passed into the post-attention layernorm, which returns
LN.(v) Var(v) +
/pstt-atm..
diag diag(Yppoca (Tpost
Var(w) +
Similar to the input layernorm, the rotated post-attention layernorm on of returns
BRamb
Apost-atinn
Var Remb) + Apost-utm,
Var(t") +e
1
Remndiaag Remndiaag/7pearam. (pe))
u diag(
Remadiag(Tponn atpa))
Var((u) +e
Ppost
Then the output from the unrotated MLP layer is
t = (o(uG)) @ (uu?)))
and the output from the rotated MLP on u' is
t' [G Ppost attn.)
(G.diag(
Remadiag
1)ah @ (z' (c.Uddiag (ailgg//////c (ppost ( Remediag( %/ostamm.
))
R Dz)²
Tpeatootn.
C3
(u diag(
Reme diag (T atn.. diaag
diag Tppmet daaaa GT
1
Tppottomin,
diag (ma 1 D. Rasb
(c.u diag(
Rewas (1 )diag(
7/post atto.
Yoes atmn,
C,
= [cCo(uG)) o (ut?)) DJ Remb = &.emb.
Then the output from the self-attention is added residual connection, and the final output from the unrotated Transformer block + v, and the output from the rotated Transformer block s' t' + o' Remb-
Suppose a is the output after all Transformer layers and the output after all Transformer layers in 0', Then the outputs after the final layernorms are
Var(a) +e diag(?mml)
25

---

Independence Tests for Language Models
= diag(
diag
and the logits from the linear output layer are
60th b diag( Remadiag( ddigg( Yima) Re diag(
1 )0"
Vinal
Vimal
=0'
which are the same for both models.
We attempted undo such transformation that adversary may apply by solving the least squares problem: We solve for rotation that minimizes |AX where X is weight matrix of the model and Y is the corresponding weight matrix of second model. Although this will provide a potential rotation to undo this transformation, we find that this solution will also find matrix A that aligns two independent model pairs as well. This makes undo-ing the rotation this way unreliable. The same holds for X and Y that are activations over multiple inputs.
26


Reviews:
[{"review_id": "eGlSeZaZ9M", "reviewer": "Reviewer_5ZNo", "summary": "This paper proposes a statistical test for determining whether the initializations of two language models (really, \"deep networks containing GLU MLPs\", or even really slightly weaker than that) are independent or not, when treating the algorithms themselves (and any data they use, etc) as fixed. Exactly valid tests are developed under equivariance assumptions for the training algorithms; when this is not true, the paper proposes heuristic tests which seem to behave reasonably under the null.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive feedback! We will add the Hemerik and Goeman reference mentioned, thank you. \n\nAbout some of the weaknesses discussed: \n\nFrom our empirical results, the power of our constrained and unconstrained test is as low as 2.2e-308 — in the cases of rejecting the null hypothesis, the p-value derived is less than 2.2e-308. (See Figure 5, 6, 7 in the Appendix.) In particular, in the best case where $\\theta_1 = \\theta_2$, then the power will be exp(- # hidden units). \n\n\nThe test could have trivial power in the case where the learning algorithms are constant functions, then the p-values will be non-significant for any initializations even if they are non-independent. However, this would never occur in practice for any non-trivially trained language model, and our tests have strong power for the wide array of language models we evaluate. We agree that the \"in general\" does a lot of work, so we only make our claims here via experimental results. We are happy to amend our writing to better calibrate to your feedback\n\n\"For the retraining and distilling test\": The unconstrained test is robust to permutations --- permuting the hidden units before and after the MLP (or if we permute the entire model) and distilling the permuted model would not change the efficacy of the test, as the gate and up projection matrices would need to share that original permutation. \n\nWe will also apply the edits from \"Other Comments or Suggestions.\" We are happy to answer more questions and appreciate the in-depth response!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proof techniques pass a \"smell check\" for me, but I did not carefully verify them.", "experimental_designs_or_analyses": "The setups seem reasonable but I did not carefully examine the details.", "relation_to_broader_scientific_literature": "The proposed test differs from previous approaches for the same problem in a useful way, and seems to work better.", "essential_references_not_discussed": "Theorem 1 is very closely related to e.g. Theorem 2 of [Hemerik and Goeman](https://arxiv.org/abs/1411.7565), who also cite earlier sources for roughly the same result. This is not a big deal, since that theorem in itself is simple and not a major contribution of the paper, but as that paper goes in more depth about related properties, it would be good to point readers to. (They assume a group structure on their transformations, implying invariance, while you directly assume invariance; I think this is the only difference.)", "other_strengths_and_weaknesses": "The proposed test is clever, relevant for an interesting problem, and appears to work in practice. I think the paper is worth publishing at ICML.\n\nThe proper interpretation of the null hypothesis, however, is subtle. The paper includes some discussion of these issues, and I don't think any of it is incorrect or even misleading, but thinking about independence where the data is fixed is somewhat unnatural, and I can easily see practitioners misinterpreting the outputs of the test.\n\nAs you point out, in the constrained setting you do have a valid test, and hence the only consideration is the power. You don't, though, give any formal discussion to the power of your tests. It seems like it might be possible to, for example, at least say something about how using Fisher's procedure across blocks relates in power to the permutation test? I'm not sure what \"consistency\" or similar properties would mean here since there's not iid data...but I believe there are probably some situations where despite the null hypothesis not holding, the test based on (2) has only trivial power. (I thought about it for a few minutes and couldn't come up with one in the constrained setting, but also couldn't convince myself that it's impossible; I think it likely is. In the unconstrained setting, doing so is trivial.)\n\n> Due to the element-wise product operation, we conjecture that in general it is not possible to permute the rows of $G_i$ while preserving the output of $\\theta_i$ without permuting the rows $U_i$ in the same way\n\n\"In general\" is doing a lot of work here; it is easy to construct silly examples where this is not the case (e.g. take the activation $\\sigma$ to always map to zero). This is not a big deal; for \"reasonable models,\" this should be true. But it highlights the general issue in this paper that while some attempt is made at formality, there are many parts which are difficult to really formalize. I think that's probably inherent to the problem setting, but it does highlight how basically everything in the \"unconstrained\" setting is generally \"reasonable\" but does not have any strict definitions the way the \"constrained\" setting does (even if those definitions themselves require some thought to understand).\n\nFor the retraining and distilling test: it seems that this scheme could be tricked by first permuting the hidden units before and after the MLP, then retraining the MLP layer from scratch there, right?", "other_comments_or_suggestions": "- A few times the paper refers to \"the set of permutations over the hidden units of the network\"; this isn't really right (or what you do), since it wouldn't make sense to swap hidden units across layers/different modules.\n- It would be good to add a sentence about the LAP algorithm of Ramshaw and Tarjan, even just saying that it is an algorithm for weighted matchings in bipartite graphs. You could easily save the space by not writing Algorithm 2 out in algorithm form and instead just put it in an equation display, since the algorithm form adds basically no information for this one.\n- (extremely minor) \"Our robust test reposes on the design\" – this uncommon usage of \"reposes\" seems to be used mostly in theology and is probably unfamiliar to most ICLR readers.\"Relies on\" would be far more typical.\n- Your bib file is rather sloppy; you should e.g. remove most of the urls, especially the ones from Semantic Scholar.", "questions_for_authors": "I don't think I have any specific questions, although I'd be interested to hear if you're able to say a little more about the power of the test in the constrained setting, or other points I raised above.\n\nUpdate after rebuttal\n------------------------\n\nI accidentally posted the below comment in a way that you weren't able to see. I think it would be worth considering the question below for the next revision of your paper (whether camera-ready or a resubmit).\n\n> Thanks for your reply; I remain happy with the paper.\n> \n> I realized in thinking slightly more about the setting today that: is there a particular reason you chose the statistic to compute the Spearman correlation between the best permutation and $[n]$? In particular, that induces a strong \"locality\" on any swaps: swapping hidden units 1 and 2 would \"cost\" much less to the statistic than swapping hidden units 1 and 1000. That doesn't seem to make sense to me, though, since there really isn't any locality in this sense to the network. In a related point, comparing whether the *best* match lines up is distinct from asking how much \"better\" the best match is than the identity. When permuting, I wonder whether it might not make more sense to ask about the ratio of the matching objective, or similar. This would probably be harder to avoid permuting with the approximate p-value from the Spearman correlation, though.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z3UGZcZXeM", "reviewer": "Reviewer_H5os", "summary": "The paper addresses the large model independence test: given the weights of two language models, can we determine if they were trained independently or if one model’s weights are derived from the other? Leveraging permutation invariance and equivariance in MLP neurons, it provides exact p-values. Extensive evaluations on open-weight models show that the test performs effectively.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and feedback! We will add the references mentioned. \n\n$\\textbf{Experimental Designs Or Analyses Comment 1}$ \n\nWe report experiments on Llama 70B, the hybrid StripedHyena and Mistral model, and (distilled) GPT 2 models in Table 5, 9, and 15  and find our statistics work for these different architectures as well. In Reviewer RDaz's comment, we also experiment with smaller Phi models and find the test holds high power for those models too. We believe these models likely encompass a broad range of training regimes. \n\nIn Table 7 of the Appendix, we also run our tests on adversarially-transformed models and show how our unconstrained test is robust to the transformation (whereas prior work is not). If there are other model architectures or families that would be beneficial, we would be happy to add those experiments!\n\n$\\textbf{Experimental Designs Or Analyses Comment 2}$ \n\nWe include more discussion and experiments in Appendix F.1 (HuREF Invariants) but are happy to move discussion to the main text. Specifically in Table 7, we demonstrate how our adversarial transformation can be used to break the HuREF invariants — each of the transformed Llama-2-7b-hf, vicuna-7b-v1.5, and Nous-Hermes-llama-2-7b models have low $M_a$, $M_b$, and $M_c$ values when compared with Llama-2-7b-hf, whereas our unconstrained statistic $\\phi_\\text{MATCH}$ gives a value of 2.2e-308. \n\nWe also mention in the paragraph at Line 96 of the introduction that our tests yield p-values whereas Zeng et al.’s do not. Please let us know if there is more we can provide about comparison with Zeng et al. (2024).\n\n$\\textbf{Weakness 1}$ \n\nWe agree. But in those cases, then causal relationships can be determined by first using our test, then using metadata, such as the dates of model releases. \n\n$\\textbf{Weakness 2}$ \n\nWe report our times on a Nvidia RTX A6000. For $\\phi_H$, the bottleneck is computing the forward pass to obtain the intermediate activations — and on two 7B models, the test on all 32 Transformer blocks combined takes on average less than 2 minutes. $\\phi_\\text{MATCH}$ requires the forward pass and aligning the activations; on dependent models, this will also take around 2 minutes total, whereas it may take 5-10 minutes per Transformer block for independent models. We believe this is reasonable computationally. We will include these details. \n\n$\\textbf{Weakness 3}$ \n\nThe hidden dimension $h$ is determined by the model weights, i.e. for Llama 2-7B the hidden dimension is 11008. This dimension varies for the models we test (28672 for Llama 2-70B, 8192 for Llama 3.2-3B) but do not affect the strength of our tests. \n\nIn our paper, we only report results for using WikiText as the input distribution, but in fact we are able to achieve similar performance using random tokens from the vocabulary as the input distribution. We will include further experiments and ablations on this distribution. \n\n$\\textbf{Question 1}$ \n\nWe choose the gate and up projections of an MLP because they are two matrices that are combined via a direct product and activation function — which makes significant transformations to the weights difficult. However, the results in Figure 2 from the GLU MLP remain a conjecture. We reason that the gate and up projections are trained to be very aligned during training with a very large loss landscape for two independent models. \n\n$\\textbf{Question 2}$ \n\nEmpirically, we find our results hold with a threshold of even 2.2e-308, and e-61 for the generalized unconstrained test where we distill the MLP. In practice, a third party or model provider may choose $\\alpha =$ 1e-5 for example, or their own confidence level. \n\n$\\textbf{Question 3}$ \n\nWe have tested non-bijective cases such as between Llama 3.1-8B and Llama 3.2-3B, where the dimension is reduced from 11008 to 8192 (more than 30%). We test $\\phi_\\text{MATCH}$ on many pruned models including the Nvidia Minitron models and Sheared Llama models in Appendix I.2, where the bijective assumption does not hold and find that the test is still strong. \n\nAlso, if the reviewer is asking about many-to-many sub-models, we also run experiments, such as with the hybrid StripedHyena model, where only some layers are taken from the Mistral 7B model (i.e. embedding) but not others (MLP projections). We are happy to run more experiments or provide more clarification. \n\nWe will also add more explanation with regards to equivariance and invariance. We appreciate the in-depth review and are happy to answer more questions!", "claims_and_evidence": "Overall, the claims made in the paper are consistently supported. The strong statements are backed by theoretical proof or experiments. I did not find any major claim that is unsupported.", "methods_and_evaluation_criteria": "1. The paper employs cosine similarity as a core metric for comparing model parameters. However, its reliance on linear relationships may overlook nonlinear dependencies between models. This limitation could lead to false negatives, particularly in the unconstrained setting where architectural differences and adversarial modifications are more prevalent, and the paper does not explore alternative metrics to address this gap.  \n2. Additionally, while the unconstrained setting handles size inconsistencies via zero-padding, this approach may introduce bias or reduce sensitivity when the dimensional disparity is significant. The effectiveness of zero-padding is demonstrated empirically (e.g., Llama-3.1-8B vs. Llama-3.2-3B), but the paper lacks analysis of its impact under extreme size mismatches or discussion of alternative alignment strategies, potentially compromising robustness in broader scenarios. \n3. In the unconstrained setting, the paper employs a matching approach via the MATCH algorithm (Algorithm 2) to compare $\\theta_1$ and $\\theta_2$ despite potential dimensional inconsistencies, using zero-padding to align matrix sizes. While this method effectively identifies dependencies in experiments, its reliance on a strict one-to-one correspondence may be inadequate when the true relationship between models is one-to-many or many-to-many, such as in scenarios involving pruning, expansion, or complex retraining. Such relationships, which do not align with the bijective assumption of MATCH, could lead to false negatives, especially when dimensional disparities are significant, as zero-padding might obscure nuanced dependencies. The paper does not explore these possibilities or test the method’s robustness against non-bijective dependencies, limiting its applicability to more intricate model relationships.", "theoretical_claims": "I checked the proofs for Theorems 1-3 in the constrained setting. The logic is correct. \n\nOne issue is that the authors treat learning algorithms as deterministic functions, ignoring the possible randomness of dropouts, etc. This assumption limits the practical applicability of Theorem 1. Although partly discussed in Appendix A, the main text should provide a clear discussion of this limitation.", "experimental_designs_or_analyses": "1. The evaluation primarily focuses on the Llama-7B architecture, which raises concerns about the generalizability of the proposed methods. Expanding the experiments to include a broader range of training regimes or adversarial scenarios would enhance confidence in their robustness.  \n\n2. While baselines such as $\\phi_{JSD}$ and $\\phi_{l_2}$ are included, the comparison to Zeng et al. (2024) is insufficiently addressed in the main text. A more thorough discussion of how the proposed methods outperform or complement prior work would strengthen the significance of the claims.", "relation_to_broader_scientific_literature": "This contribution bridges classical statistical tools with modern deep learning to apply in a novel real-world problem, language model independence tests, which adapt permutation methods to a new domain while leveraging neural network symmetries.", "essential_references_not_discussed": "Some related works about permutation tests may help unfamiliar readers to better understand Algorithm 1, e.g., [1, 2].\n\nReferences:\n1. Phillip Good. Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses. \n2. E.L. Lehmann, Joseph P. Romano. Testing Statistical Hypotheses.", "other_strengths_and_weaknesses": "Strengths:\n1. The problem of testing model independence has significant implications.  \n2. The methods are computationally efficient (e.g., avoiding full retraining by using permutations or proxy models), making them feasible for large-scale models.  \n3. The formal definitions (e.g., $\\Pi$-invariance, $\\Pi$-equivariance) and theorems provide a solid theoretical backbone.  \n\nWeaknesses:\n1. From a practical application perspective, I think determining the direction of causal relationships between large models may be more meaningful than merely testing for the existence of a dependency. However, the latter remains a highly important and intriguing problem.\n2. The running time of the test is also a very important practical consideration. Authors need to explicitly report needed times.  \n3. In the unconstrained setting, Algorithm 5’s proxy GLU MLP construction depends on the choice of hidden dimension $h$ and input distribution $P$. The paper offers little insight into their impact and how they were selected. The sensitivity of the method to these hyperparameters is also unclear.", "other_comments_or_suggestions": "1. Adding the mathematical definition of \"exchangeable\" is better. \n2. The assumptions of $\\Pi$-equivariance and $\\Pi$-invariance are mathematically clear but may be difficult to readers unfamiliar with deep learning symmetries. More intuitive explanations or examples could improve accessibility.", "questions_for_authors": "1. The motivation for choosing gate and up projections in $\\phi_{MATCH}$ (Section 2.3.1) is described as a conjecture rather than a derived principle, which may leave readers questioning its robustness. Could you provide more justification?\n2. In real-world applications, how to set significance level $\\alpha$? \n3. When the true relationship between sub-models is one-to-many or many-to-many, which do not align with the bijective assumption of MATCH, is there any better solution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8uQ4WCDG1P", "reviewer": "Reviewer_L4qp", "summary": "This paper introduces a rigorous statistical framework for testing whether two language models were trained independently. Concretely, the authors propose hypothesis tests in both constrained and unconstrained settings. The constrained setting assumes known model architecture and training conditions, allowing for exact p-value computation through simulations of exchangeable copies of each model. The unconstrained setting removes these assumptions, making the test robust to adversarial modifications that preserve model outputs but alter internal weight structures. The proposed methods are validated on many open-weight models.\n\n##  update after rebuttal\n After reviewing the rebuttal addressed to me and those for other reviewers, I am willing to maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive feedback!\n\n$\\textbf{Weakness 1}$ \n\nThank you for bringing up this concern. Standard machine learning algorithms such as SGD  which are the most common algorithms for training language models follow the equivariance condition, as the gradients are permutation-equivariant, explained briefly in Example 2. We also emphasize that only one of the models needs to satisfy these assumptions, so a trusted model’s developer (who used SGD for example) can run this test without assumptions on the training strategy for another model. \n\n$\\textbf{Weakness 2}$ \n\nYes, we agree, and we do not claim our test is in fact robust to all adversarial attacks. We have at least empirically validated that our test is robust to a superset of attacks over prior work (Zeng et al. 2024).\n\n$\\textbf{Weakness 3}$ \n\nWe agree that the final model weights are heavily influenced by the training data. However, in our definition of a learning algorithm (Section 2.1) we write “$A$ includes the choice of training data…” Given the training data, minibatch ordering, and other parameters, then in fact $A$ is deterministic and it is possible to fully reproduce the learning algorithm given the initial weights. \n\nThis is an abstraction to simplify our framework and may be an unconventional way to describe learning algorithms, so we will add more clarification. We also acknowledge there are other sources of randomness which deterministic functions with a fixed seed do not capture such as dropout; thus, in Appendix A, we state and prove a more general version of Theorem 1 for randomized learning algorithms, for which our conclusions still hold.\n\nThank you, and we are happy to answer more questions!", "claims_and_evidence": "Yes, the claims made in the submission appear to be supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method makes sense for the independence testing of two language models.", "theoretical_claims": "Yes, I check some proofs, including Theorems 1, 2 and 3.", "experimental_designs_or_analyses": "Yes. The experimental designs and analyses appear to be sound.", "relation_to_broader_scientific_literature": "This paper aims to address a fundamental question in model provenance and intellectual property protection.", "essential_references_not_discussed": "No, the paper includes essential references.", "other_strengths_and_weaknesses": "Strengths \n- This paper is well written and its structure is clear.\n- The paper frames model independence as a hypothesis testing problem.\n- In constrained setting: this paper uses exchangeable model copies under specific assumptions to compute exact p-values.\n\nWeaknesses:\n\n- The assumption for permutation-equivariant training may not always hold in real-world applications.\n- While the unconstrained test is empirically robust, theoretical guarantees are lacking.\n- More importantly, the assumption that the learning algorithms are deterministic functions is seriously inconsistent with the facts. Obviously, the outputs of the learning algorithms are primarily influenced by the training data and thus are random.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzW8OF38qw", "reviewer": "Reviewer_RDaz", "summary": "The paper investigates a method to determine whether two models’ weights were trained independently (i.e., from different random initializations) or if one model was derived from the other through fine-tuning, pruning, or partial reuse. This is framed as a hypothesis test for independence between two sets of model weights.\n\nThe study considers two settings:\n- Constrained setting: Both models have the same architecture. The authors assume the training process is equivariant to permutations of the hidden units, allowing them to compute exact p-values under the null hypothesis of independent training. They validate this method on 21 open-weight models and correctly identify all non-independent pairs.\n- Unconstrained setting: Models can have different architectures. The authors develop a robust test based on aligning hidden activations, which remains effective despite architectural changes or adversarial modifications. Though this test does not produce exact p-values, it empirically behaves like one and can even pinpoint specific model components that are shared or derived.\n\nOverall, the authors claim that the proposed methods reliably distinguish independent models from non-independent ones, even in cases where dependencies are obscured by architectural modifications or selective weight reuse.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and positive feedback! \n\nWe find the tests also work on smaller-scale models such as the Phi-3 family. Both $\\phi_\\text{CSH}$ and $\\phi_\\text{MATCH}$ return a statistic approximately 1e-308 (aggregated with Fisher’s method) on the fine-tuned model pair microsoft/Phi-3.5-mini-instruct and numind/NuExtract-v1.5 (3.8B parameters). We are also happy to add more experiments on other model families and will update our experiment section in our paper\n\nThank you!", "claims_and_evidence": "The paper provides strong empirical evidence for the effectiveness of its proposed methods, particularly in detecting non-independent model pairs and identifying shared components.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No issues as far as I know.", "experimental_designs_or_analyses": "Overall, the experiments conducted by the author is sound, tests were conducted across a wide range of models (mostly those related to Llama). For the unconstrained case, it would have been interesting to also explore other model families such as the Microsoft's Phi family.", "relation_to_broader_scientific_literature": "The paper contributes to the broader literature on intellectual property protection and model fingerprinting by demonstrating that model weights themselves can serve as a fingerprint for tracing model lineage. Unlike prior work that relies on embedding traceable signals in model outputs or specific responses, this study shows that statistical tests on model weights can effectively determine whether a model was trained independently or derived from another. This insight enhances provenance tracking and provides a new tool for enforcing licensing restrictions and protecting intellectual property in machine learning.", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AeMDoBmaAp", "reviewer": "Reviewer_wPPE", "summary": "This paper introduces a method to assess whether two large language models (LLMs) are independent or if their training procedures exhibit dependencies. The core idea is based on the principle that if two LLM weights are independent, the distribution of differences between arbitrary permutations of their weights should be uniform. Conversely, if the models are dependent, the differences between their original weights will be significantly smaller than those between permuted versions of their weights. Motivated by this observation, the authors propose a method to compute p-values from the distribution of weight differences, which quantify the probability of the two models being independent. The paper presents both the conceptual framework and algorithmic implementation of their approach. Empirical results on some cases demonstrates its effectiveness in detecting dependencies between LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and address some of their concerns below. We will add the references suggested. \n\n$\\textbf{Claims and Evidence Comment 1}$\n\nFor the constrained test, we in fact guarantee that the test leads to exact p-values under the null hypothesis when two models are independent, i.e. two independent models will not lead to a low p-value. \n\nBut with regards to a Type II error, we agree that our tests do not inherently guarantee that two non-independent models will always lead to a low p-value. However, empirically we find this is true in all our experiments. Specifically, Figures 5 and 6 in Appendix E show that on all (69) dependent Llama 7B pairs, the tests $\\phi_{U}$ and $\\phi_H$ yield p-values less than 2.2e-308 (which is our Type-II error rate, i.e. the maximum p-value we observe in the case that the null hypothesis (independence) does not hold). The same holds for $\\phi_\\text{MATCH}$ (Figure 7). We believe that this is sufficiently addressed through evidence from the Llama 7B and 70B results. We note that with regards to a Type I error, that our theorem does guarantee a uniform distribution for two independent models under the equivariance assumption, which we discuss more below.\n\n$\\textbf{Claims and Evidence Comment 2}$\n\nStandard machine learning algorithms such as SGD, which are the most common algorithms for training language models follow the equivariance condition, as the gradients are permutation-equivariant, explained briefly in Example 2. We also emphasize that only one of the models needs to satisfy these assumptions, so a trusted model’s developer (who used SGD for example) can run this test without assumptions on the training strategy for another model. \n\n$\\textbf{Methods and Evaluations Criteria}$\n\nThe independence is statistical independence of two random variables, $\\theta_1$ and $\\theta_2$ (equivalently, zero mutual information), i.e. $X \\perp Y$ if and only if $P(X|Y) = P(X)$. We will clarify this in a footnote in updated writing.\n\nIn this case, if $\\theta_1^0$ and $\\theta_2^0$ are independent random variables, then by the post-processing inequality, since $A$ is a (deterministic) function, then $A(\\theta_1^0)$ and $A(\\theta_2^0)$ are also independent. We will add these details and this explanation. \n\nThe Type-I error rate is the $\\alpha$ threshold determined by the test user, since our tests give p-values. If the user chooses a threshold of $\\alpha = 0.0001$ for example, then the Type-I error rate is 0.0001. This result is a consequence of our Theorem 1 with the equivariance condition. Further, we also plotted the null distribution for the 141 independent model pairs and found that the values are uniformly distributed, with exact values shown in Figure 6, for example. \n\nIn our experiments (Figures 5, 6, 7 in the Appendix) (using ground truth from Hugging Face), choosing $\\alpha =$ 1e-307 would yield a Type-I error rate of 1e-307 and a Type-II error rate of 0. \n\n$\\textbf{Experimental Designs Or Analyses}$\n\nOur experiments are already conducted on neural networks; $\\phi_U$, $\\phi_H$, and $\\phi_\\text{MATCH}$ are all defined over the individual MLP component of the Transformer models, which are in principle simple neural networks. For example, $\\phi_U$ uses the weights of the up-projection matrix, i.e. one layer of a neural network. \n\nBy varying model size and the dimensions of the weight / activation matrices from 1B to 7B to 70B models, we test the effectiveness and robustness of the test; we also discuss Type-I and Type-II errors above. \n\nOur tests $\\phi_U$, $\\phi_H$, and $\\phi_\\text{MATCH}$ do not use a $T$ value, as they follow the general form provided in Equation (2). In principle, $T$ is actually the total number of permutations, which is why we get a very low p-value like e-308. \n\n$\\textbf{Theoretical Claims comment and Weakness 1}$\n\nWe agree that our proof holds for a broad class of machine learning models and neural networks. We intentionally chose this to highlight the generality of our method, but we focused our empirical experiments on language models due to their widespread re-use and the subsequent intellectual property concerns. Specifically, language model capabilities and pretraining costs are growing, which makes models more at risk of being stolen. Furthermore, many parties will fine tune an open-source model for a downstream task rather than pretraining their own model. We briefly discussed this in the second paragraph of the introduction but can further expand on the motivation. But, we note that the unconstrained test is geared towards language models (GLU MLPs). \n\nWe acknowledge your concern about model-specific considerations and would appreciate it if you could point out any aspects that would significantly benefit from incorporating more language model specific characteristics, and if there was a specific issue you found with the proof. \n\nWe will also work on the clarity of writing and appreciate the in-depth feedback.\n\nThank you!", "claims_and_evidence": "The submission makes claims in Theorem 3 that require further clarification and evidence. Specifically:\n- The theorem does not guarantee that the parameters (thetas) are independent when two matches are independent, which raises concerns about the potential for a high Type-II error in the proposed test. This limitation is not sufficiently addressed, and the evidence supporting the test's robustness in such scenarios is unclear.\n- The equivariant-type condition, which is central to the theorem, may not hold in practice. The submission does not provide adequate justification or empirical validation for this condition, casting doubt on the generality and applicability of the claimed results.", "methods_and_evaluation_criteria": "I have some concerns about the proposed methods and evaluation criteria for the problem at hand. \n\nFirst, the definition of independence in Section 2.1, presented in Equation 1, is not well-defined. The notation $\\theta_1 \\perp \\theta_2$ is used without a clear explanation of it. For instance, does it imply statistical independence, zero mutual information, or some other form of independence? This lack of clarity makes it difficult to assess the validity of the proposed test. Furthermore, the authors suggest that if (non-independent initializations), then (non-independent final weights). However, they also imply that if or even $A_1 = A_2$ (i.e., non-independent or identical training procedures), then $A_1(\\theta_1^0) = A_2(\\theta_2^0)$ (i.e., independent final weights), without providing sufficient justification. \n\nSecond, the evaluation criteria do not seem adequate for the problem. For example, the authors do not report results for Type-I error rates or statistical power, which are critical for assessing the reliability and effectiveness of the proposed test. Including these metrics would provide a more comprehensive evaluation of the method's performance.", "theoretical_claims": "Yes, I have reviewed the proofs provided in Appendix B of the paper. I have the following concerns regarding their correctness. \n\n1. The definition of independence in Appendix B is not well-defined. The lack of a precise and rigorous formulation makes it difficult to assess the validity of the theoretical claims and proofs that rely on this definition.\n\n2.  The proof appears to be overly general and does not leverage any specific characteristics of language models. While this generality might seem advantageous, it raises questions about whether the proof is sufficiently tailored to the problem at hand. The absence of model-specific considerations limits the depth of the theoretical insights and their relevance to the application domain.", "experimental_designs_or_analyses": "I have concerns about the soundness and validity of the experimental design and analyses in the paper:\n\n- Given that the proposed method is agnostic to neural networks, it would be beneficial to include experiments with simple neural network architectures to study Type-I and Type-II errors. This would help validate the method's effectiveness and robustness in a controlled setting, which is currently missing from the experimental design.\n\n- $T$ used in the experiments appears to be too small, which may limit the reliability of the results.", "relation_to_broader_scientific_literature": "This paper's contributions relate to the broader literature on independence testing, particularly for large language models (LLMs). It may be helpful for applications like LLM-based ensemble methods and model voting.", "essential_references_not_discussed": "This paper employs a permutation-based technique to test independence. However, recent relevant works, such as [1-2], are omitted from the discussion.\n\nReference\n\n- [1] Berrett, Thomas B., Ioannis Kontoyiannis, and Richard J. Samworth. \"Optimal rates for independence testing via U-statistic permutation tests.\" The Annals of Statistics 49.5 (2021): 2457-2490.\n\n- [2] Kim, Ilmun, Sivaraman Balakrishnan, and Larry Wasserman. \"Minimax optimality of permutation tests.\" The Annals of Statistics 50.1 (2022): 225-251.", "other_strengths_and_weaknesses": "- Strengths: The paper is mathematically thorough and bases its thesis in well formulated assumptions. The idea is interesting and the method relatively simple and inexpensive, which may be attractive for implementation.\n\n- Weaknesses: \n  - The significance of independence testing for language models is not sufficiently motivated or clarified. The paper would benefit from a more detailed and board discussion of why this problem is important in the context of language models and their applications.\n  - The clarity of writing could be improved. The presentation of ideas is often unclear, which hinders the reader's ability to fully grasp it.", "other_comments_or_suggestions": "Typos:\n\n- Section 3.1, the first sentence: \"We first validate validate the effectiveness\"", "questions_for_authors": "Please refer to **Claims and Evidence**, **Methods and Evaluation Criteria**, and **Weaknesses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles the question of whether two given models were trained independently (precisely, deep networks containing GLU MLPs). A statistical test with an exact p-value is developed under equivariance assumptions for the training algorithms. More precisely, the test leverages permutation invariance and equivariance in MLP units. A separate heuristic-based test was proposed if the assumptions are not met. 

The paper addresses a timely problem. All the reviewers noted that this work proposes interesting approaches that are theoretically grounded. The contributed theoretical results (e.g., testing under equivariance, exact p-value computation) are solid (H5os, wPPE). The authors submitted a strong rebuttal that addresses most of the concerns raised, mostly about the permutation-equivariant assumptions, the assumption that the learning algorithms are deterministic, and other clarification questions  (e.g., runtime). 

Much of the AC-reviewers discussion was on two topics: 1. potentially uncontrolled type-I error, 2. no guarantee that asymptotically type-II error goes to zero. These are valid concerns. The first concern was resolved by a consensus that the test (at least in the constrained setting) yields an exact p-value and so type-I error is guaranteed to be in control. The AC resolved the second concern by pointing out that the “asymptote” is not a well-defined concept here, unlike other more common testing scenarios where asymptote is tied to observing a larger sample. Here, sample means the two given models. Formally quantifying the type-II error will require the knowledge of the distribution under the alternative, which is non-trivial. The AC thinks that even without this analysis, as one of the few works that tackles this direction, the rest of the theoretical contributions are already sufficient.

AC recommendation: accept.


To the authors:

* Please include an impact statement in the final version. This is a must. Please see https://icml.cc/Conferences/2025/CallForPapers. This work is theoretical in nature. The AC does not expect this work to have direct ethical impacts or direct societal implications.
Decision: Accept (spotlight poster)