Quasi-Equivariant Metanetworks
Authors: Viet-Hoang Tran
An Nguyen The
Benoît Guérand
Thieu Vo
Tan Minh Nguyen
Venue: ICLR 2026 Poster (2026)
Keywords: metanetwork
functional equivalence
Primary area: learning theory
TL;DR: We introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity

Abstract: Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.

Published as a conference paper at ICLR 2026
QUASI-EQUIVARIANT METANETWORKS
Viet-Hoang Tran"
An Nguyen'
Department of Mathematics National University of Singapore hoang. t travvitt nus edu
FPT Software AI Center Vietnam
annt 683 fpt com
Benoft Guérand
Thieu N. Vo
Department of Mathematies National University of Singapore benoit. guerandeu. nus edu
Department of Mathematics National University of Singapore thi Chieuvo@nus edu. sg
Tan M. Nguyen
Department of Mathematics National University of Singapore tanmn@ nus edu. sg
ABSTRACT
Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parame- ters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, moti- vating the development of equivariant metanetworks, which incorporate equivari- ance principles to respect architectural symmetries. Existing approaches, how- ever, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariiane, which allows metanetworks move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation. show that -qqivvriatt metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theo- retical understanding of weight-space learning and provide a principled founda- tion for the design of more expressive and functionally robust metanetworks.
INTRODUCTION
Modern problem-solving increasingly relies on neural networks, which encode vast amounts of information within their trainable parameters during learning, ranging the application from computer vision (Huang et al., 2020; Krizhevsky et al., 2012; He et al., 2015), natural language processing (Vaswani al., 2017: Rumelhart al., 1986; Hochreiter Schmidhuber, 1997: DeepSeck-AL al., 2025), and nature science (Raissi et al., 2019; Jumper et al., 2021). While these parameters capture rich knowledge, accessing and interpreting it remains challenge.
'Equal Contribution
Please correspond to: hoang.tran edu and tanmn @ www.ed...g

---

Published as a conference paper at ICLR 2026
[0x]
oquivariance
9F16)
02-
[0=]
90
of ((()))
F(0))
quasiequiivaiaane
0s [0s]
F(e
F
04 [0x]
e
Figure 1: (Left) Illustration of the partition of parameter space into functional equivalence classes. (Right)
Illustration of the quasi-cquivariance property and its distinction from strict equivariance.
Metanetworks. Metanetworks were introduced to analyze and process other neural networks by treating their weights, gradients, and sparsity patterns structured inputs. Early work focused on evaluating their generalization and revealing properties of neural network behavior (Baker et al., 2018; Bilertsen et al.: Unterthiner et al., 2020; Schürholt al., 2021; 2022a:b). Common strategies include flattening parameters or extracting statistics before feeding them into multi-layer percep- trons (MLPs) (Unterthiner al., 2020: Dupont al., 2022; Luigi al.). Beyond these foundations, metanetworks have been applied to extracting structure from implicit representations (Müller et al., 2023; Stanley, 2007; Mildenhall et al., 2021), developing learnable optimizers (Bengio et al., 2013; 3; Runarsson & Jonsson, 2000; Andrychowicz et al., 2016; Metz et al., 2022), performing model edit- ing (Sinitsin et al., 2020; Cao et al., 2021; Mitchell et al., 2022), evaluating policies (Harb et al., 2020), and enabling Bayesian inference (Sokota al., 2021). Nevertheless, designing metanetworks remains challenging due to the complexity and high dimensionality of the underlying structures.
Functional Equivalence. A major challenge in designing metanetworks lies how to capture functional equivalence- the fact that multiple distinct parameter configurations can realize the same input-output function (Allen-Zhr et al., 2019; Belkin et al., 2019; Du et al., 2019; Frankle & Carbin, 2019; Novak et al., 2018). This problem was first posed by Hecht-Nielsen (Hecht-Nielsen, 1990). A key observation is that swapping unchanged, provided their outgoing connections are permuted accordingly (Allen-Zhu et al., 2019; Du al., 2019; Frankle & Carbin, 2018; Belkin al., 2019; Neyshabur al., 2018). For the same class of MLPs, Fefferman & Markel (1993) established stronger result: the input- outuut mapping of an MLP with tanh activations uniquely determines both its architecture and its weights, up to permutations and sign flips. Subsequent work extended these identifiability results to broader MLP settings (Albertini Sontag, 1993b;a; Mai Lampert, 2020; Chen al., 1993; Kurkova Kainen, 1994) and, in parallel, to convolutional neural networks (CNNs) (Brea et al., 2019; Novak et al., 2018; Mai & Lampert, 2020; Tran et al., 2024a; Vo et al., 2025).
hidden units in MLP leaves its input-output mapping
Equivariant Metanetworks. Building the insight from permutation invariance in neural net- works, researchers have developed permutation-equuiaaiaa metanetworks (Navon et al., 2023; Zhou et al., 2024a; Kofinas et al., 2024; Zhou et al., 2024b), which naturally account for neuron reorder- ing within hidden layers. More recent architectures extend beyond permutation equivariance by incorporating additional symmetries such as scaling and sign changes (Kalogeropoulos al., 2024; Tran et al., 2024a; Vo et al., 2025). Furthermore, recent works (Tran et al., 2025b; Knyazev et al., 2025; Tran al., 2025t) have characterized the maximal symmetry groups of Multihead Attention and Mixture-of-Expert establishing necessary and sufficient conditions for functional equivalence, offering new insights into the structural properties of Transformer and Mixture-of-Expeee weights.
Despite advances in metanetwork design, most approaches enforce strict equivariance at the level of individual weights. However, the true goal is not preserve the weights themselves, but to capture the functions they implement-thar is, to respect the functional equivalence classes defined by the parameters. In this context, relaxed or approximate equivariance has emerged in deep learning to handle imperfect symmetries in real-world data. Early approaches include weight-relaxed convolu- tions (Wang et al., 2022), soft constraints via multitask losses (Elhag et al., 2024; Pertigkiozoglou et al., 2024), G-biases for group convolutions (Wu et al., 2025), and extensions to E(3) (qquirrauva graph networks (Hofgard et al., 2024). On the theoretical side, Kaba & Ravanbakhsh (2023) and Huang et al. (2022) formalized relaxed equivariance and analyzed its bias varinna trade-offs. To- gether, these insights motivate a class of metanetwork architectures that relax strict weight-level equivariance, enabling more flexible representations of functional symmetries.
2

---

Published as a conference paper at ICLR 2026
Contributions. Building this motivation, we introduce framework for quasi- equivariuv metanetworks-a a novel paradigm that relaxes strict equivariance to balance symmetry preservation with representational flexibility. The paper is organized as follows:
1. In Section 2, examine the parameter space of a parameterized function, characterize its associated symmetry group, and introduce the formal notion of maximality within symme- try groups, establishing direct connection Functional Equivalence.
2. In Section 3, examine the sufficiency of strict equivariance for metanetwork design. Building on this, we introduce quasi-equivvriance, which enables metanetworks to over- come the limitations of strict equivariance while still maintaining functional identity.
3. In Section 4, we present general framework for quasi-cquivariant metanctworks and demonstrate its application to feedforward neural networks and multihead attention.
4. In Section 5, we integrate the framework into existing metanetworks. Experiments on multiple metanetwork benchmarks show that this layer enhances performance considerably while incurring only a slight increase in the number of parameters.
Supplementary materials, including a comprehensive notation table, theoretical derivations, detailed proofs, and experimental setups, are provided in the Appendix.
PRELIMINARIES ON EQUIVARIANT METANETWORKS 2
In this section, present the details of the parameter space of parameterized function, its asso- ciated symmetry group, and introduce the formal notion of maximality in symmetry groups, which connects directly the concept of Functional Equivalence (FE).
2.1 PARAMETER SPACE OF A PARAMETERIZED FUNCTION AND ITS SYMMETRY GROUP
Parameter space. Let f(:; 0) be function parameterized by E θ = Rdi The set θ is called the parameter space (or weight space) of f. Assume a group G acts on Ô. For each O, we define the set of parameter vectors that yield functionally equivalent models:
[0] = (ō E e [(=:0) = f(:5)) ce
The parameter space serves merely as a proxy for the function class, and the mapping 0 - f(= is non-injective, distinct parameter configurations can yield identical behaviors. This phenomenon is illustrated in Figure 1. FE thus focuses on characterizing the sets [0]. Explicitly enumerating all such sets is impractical. A more systematic approach to view these equivalence classes as orbits under group action on O, naturally leading to the notion of the symmetry group of f.
Symmetry group. Consider group acting on the space O For O, the G-orbit of is defined as GO = {ge E G) S O We now introduce the following definition.
Definition 2.1 (Symmetry Group). A group G is called symmetry group of the function f if GO C [0] for all E O Equivalently, for every g E G and 0 θ one has f(=: f(=
The phrase "a symmetry group" acknowledges that multiple such groups may exist. In particu- lar, every subgroup of a symmetry group is itself a symmetry group. Our goal is to represent the equivalence classes [8] using G-orbits. To build intuition, we present two following observations.
First observation. Consider the function f(:: a, b): - R, defined by x - abx, parameterized by 0 (a, b) E 2 Ô. It is straightforward and only ab = ab. This naturally suggests the following group action: let R2 denote the multiplicative group of nonzero real numbers. Define the action of c R. on (a, e- b) + (ac, c c-) It is straightforward to verify that Rx is a symmetry group of f. However, does not fully capture the equivalence classes. Indeed, for (a, E g2a with ab of O, one has
that (a, b) and (a, b) yield the same function
R2 by
[(a, ((a, b) E 2 ab ab) ((ac,c- C R R. R( (a,
In contrast, for (a. E 2 with ab = O. one obtains ((a, 6)] = R x U R L R (0. Thus, while Rx almost completely describes the functional partition, it fails on the degenerate subset
3

---

Published as a conference paper at ICLR 2026
of the parameter space where ab = 0. It is difficult to identify larger natural group that extends the action to cover these exceptional cases.
Second observation. Classical group theory ensures that any partition of a set can be realized as the orbit decomposition of some group action. Accordingly, there always exists group G and an action of G on @ such that the G-orbits match the functional partition. However, constructing such group typically requires explicit mappings, which are often intractable and impractical. In the context of parameterized models, where θ finite-dimensional real space, it is natural to focus on group actions arising from standard operators such as addition, multiplication, or permutation.
These observations present trade-off: the tractability of the group and its action versus the descrip- tive capacity of the functional partition, motivating the notion of maximality of symmetry groups
Maximal symmetry group. The above observations lead to the following intuitive and informal description of a maximal symmetry group:
Under generic parameters, the symmetry group G captures all functional equivalences, up to a sufficiently small exceptional set.
In other words, let E denote sufficiently small subset of O, and consider the restricted domain c. The group action of on θ restricts naturally to è \ E. Then, for all 0, 0 E θ \ E such that f(;;0) 0) = f(= there exists g G with 0 = gO. Hence, although there may exist parameters in O for which G does not capture FE, this exceptional set negligible, and G may still be regarded as characterizing FE of O. The subset is typically taken to coincide with the zero set of finitely many nonzero polynomials, that is, a proper real algebraic variety, consistent with prior work on FE in neural architectures (Hecht-Nielsen, Nielsen, 1990; Fefferman & Markel, 1993; Mai & Lampert, 2020).
Definition 2.2 (Maximal symmetry group). A symmetry group G said be maximal there exists proper real algebraic variety ε - θ such that, for all 0, 0 θ \e, whenever f(: () = f(; f), there exists g E with 0 = =g0.
Remark 2.3. In the above observation on f(; a, b). let e = {(a,b) E R2 ab = 0}. Then e is proper real algebraic variety, and the group Hx serves as a maximal symmetry group of f.
In the next section, we demonstrate that this notion of maximality coincides with prior analyses of FE in feedforward and convolutional neural networks, as well as in multihead attention.
2.2 ON THE ROLE OF EQUIVARIANCE IN METANETWORKS.
A metanenwork is a map F: - X' that takes as input the parameters of a model. Depending on the application, F may return another element of θ (as in network editing tasks) vector in Rd for some integer (as in prediction tasks). The fundamental objective is to determine whether the parameters of model contain sufficient information reveal properties of the function realized by the model itself. Since F' receives 0 as input, it is natural to require that F' depend only on the underlying function represented by 0, rather than on the particular parameterization. Equivalently, the input of F should be the equivalence class [0], as all elements of [0] define the same function. It would be undesirable for F(O) and F(0) to produce incompatible outcomes whenever [0] (0).
A principled approach to this requirement is to impose equivariance or invariance with respect to symmetry group G. In particular, suppose F: - θ is G-equivariant. By definition, this means
F(gf)= = ( for all E G, A NEO. O.
(3)
Consequently, the equivalence classes are preserved in the sense that [F(g0)] [g) = [F(0)]
thereby ensuring consistency across parameterizations that correspond to the same function. If G a maximal symmetry group of the underlying model, such equivariance is sufficient to guarantee that F operates solely the functional content of 0. This observation underscores the importance of characterizing the maximal symmetry group group-equine waap---uuivlllly...... of understanding FE-as as a prerequisite for the systematic study of equivariant metanetworks.
3 Is STRICT EQUIVARIANCE NECESSARY FOR METANETWORK?
As discussed in Section 2.2, equivariance provides a principled mechanism for preserving the func- tional behavior of input networks. Nevertheless, equivariance should be regarded a sufficient
4

---

Published as a conference paper at ICLR 2026
condition for such preservation, rather than necessary one. This naturally leads to the question:
Is strict equivariance necessary for metanetworks?
We now introduce a broader notion, namely muuuii-euvvriancee Throughout the remainder of the paper, let denote the maximal symmetry group, and let F denote metanetwork map.
Quasi-equivariance. We first address equivariance, deferring the discussion of invariance to a later stage. The requirement of functionality preservation for map F:
O can be stated as follows:
for all f E [0], one requires that F(0) [F()].. By the maximality of G, the condition @ E [Ø] implies that there exists g E such that = g0. The same holds for F (0) and F(O). Consequently, the above requirement can be reformulated, motivating the following definition.
Definition 3.1 (Quasi-quiivariance) A map F: θ - θ is said to be g E G and
if, for all
O, there exists g' = g' (g. 0) such that F(g0) = (()).
The notation of - g 0) emphasizes that g' may depend both g and 0. Figure illustrates the quasi-equivariance property. By definition, every G-equivariant map is also G-quasi-equivarias Moreover, G-quasi--quivvariacee ensures functionality preservation. Given the maximality of G (Definition 2.2), it provides necessary and sufficient condition for map to preserve functionality. Indeed, for 0, 0 O such that [0] = 0, one has f = g0 for some E G. Thus, F(O) F(g = g' (FF(0) for some g' E G. Therefore, [F) l F(0)] [F(O)]-
Remark 3.2. A natural question is how to construct a map F' that satisfies the quuii--quivaiant property. By Definition 3.1, one natural attempt
first choose an arbitrary group-va valued function a: G x θ - G' and then solve for map F θ - θ satisfying F(gf) agg 0) F(0). However, for general choice of ax such an F does not exist. Appendix A. provides the necessary conditions on under which at least one corresponding F can exist. Although this approach is theoretically motivated, it is not practical for constructing metanetworks. Therefore, in Section 4, we will present more effective and implementable design for F.
Invariance. For invariance, introducing quasi-version is unnecessary. Indeed, to ensure that map F: θ - preserves funetionality, it suffices require F()) F(0), which is equivalent to F(gØ)= = F(0). Hence, strict invariance is necessary.
Properties. In practice, equivariant and invariant metanetworks are constructed by stacking equiv- ariant and invariant layers on top of one another, in the same manner as deep models are typically built. This construction relies on standard closure properties: the composition of two equivariant maps is equivariant, and the composition of equivariant map with an invariant map is invariant. For equivaiqqiv,,,, analogous properties hold, as stated in the following result.
Proposition 3.3 (Composition). Let yp sb: - θ maps. Then:
1. If both yp and vb are G-quasi- qatuss--quivvriant, then o yp E-quasi-equivarian 2. If ip is G-quasi-equivariant and v/ is G-invariant, then up o up is G-invariant.
The assumption that y,, vh: θ - O is made for notational simplicity. In fact, the results remain valid if θ is replaced by any domain on which the notions of E-quasi-equiivrrinne and G-invariance are well-defined, The proof of Proposition 3.3 is provided in Appendix A.2.
Remark 3.4 (Comparison with relaxed notions of equivariance in the literature). In Kaba & Ravan- bakhsh (2023). the notion of relaxed equivariance is introduced. Given a group G' acting on X' and V, map p: X' - said satisfy relaxed equivariance if, for all g E G and X. there exists g' E gG.-with G- the stabilizer subgroup of r-such that (pgæ)) = g (z().. This definition is natu- rally subsumed under the broader notion of quasi-equivariance given in Definition 3.1. Other works, such as Wang et al. (2024), employ related but distinct perspective, where relaxed equivariance is interpreted as an approximation to strict equivariance, namely v(gx) RJ gg(x).
4 QUASI-EQUIVARIANT METANETWORKS
In this section, we establish a general framework for quasi-equivariant metanetworks and subse- quently apply it to feedforward neural networks and multihead attention.
5

---

Published as a conference paper at ICLR 2026
4.1 A GENERAL FRAMEWORK FOR THE DESIGN OF
METANETWORKS
Given the notation f. 0, 6 and G from the previous section, we focus on constructing G-quasi- equivariant networks. The invariant case immediate, since it can be obtained by stacking an invariant layer on top of an equivariant backbone, as observed in Proposition 3.3.
A Gu-uusi-euivvarian layer is defined as follows. Let a:: θ G be a map into the group, and let
3: θ = ④ be an equivariant map. Define
F - O,
= [10)=a(0))80)
(4)
By construction, F is G-quaaii-qqiivarinnt In this framework, the design of 3 follows directly from prior work equivariant metanetworks. The central task is therefore construct that outputs group elements of G. This extension is motivated by the observation that strict equivariance necessary for metanetworks, and that enforcing it often yields sparse models due the strong constraints imposed on the network weights. By introducing a, we aim to relax these constraints, thereby improving both the expressivity and performance of metanetworks.
We now examine several representative cases of G, assuming a is continuous. In machine learning, continuity-and in practice differeeniabblity-is essential for gradient-based optimization via back- propagation. The parameterized maps considered here will primarily be feedforward networks, convolutional neural networks, and multihead attention modules, as these are the predominant ar- chitectures in existing datasets of pretrained weights.
Remark 4.1. In the instances considered in the next part, the group G, although a group, can also be embedded into R" for some n. this setting, the group-valued map a : θ G may be regarded as continuous. We then recall the classical fact that the continuous image of connected space is connected. Since θ I connected, the image a(@) must also connected. Consequently, G' is discrete, must be constant. This observation will be useful in the next part: when G contains a discrete component (such as permutations), any continuous a cannot meaningfully vary over O. Hence, the discrete part of G' can be ignored in the construction, and the focus is placed on the continuous component of G.
4.2 THE CASE OF FEEDFORWARD AND CONVOLUTIONAL NEURAL NETWORKS
We primarily focus on the feedforward neural network. The convolutional counterpart can be treated in an analogous manner without loss of generality.
Parameter space. Consider feedforward neural network f with layers, having n, neurons in the hhh layer and activation a. Here, no and nt are the input and output dimensions. The map f is parameterized by = (W by 1 where W. E 2 and b, E R": is expressed as:
fx;; 0) = ft. 0 ft-100000 ................................ f((x),
(5)
where fo: R t such that x W₁ . by. The parameter space of f is:
θ Xni-1 RT))
nm
R"²) (R Xno RR
Maximal symmetry group. We define group action Q by monomial matrices. Let be positive integer. A monomial matrix of size n x n is a matrix in which each row and each column contains exactly one nonzero entry. Denote g220 as the sets of monomial matrices of size n with all non-zero entries positive. Now, define the group G = 9 X x G200 Denote = (gL-1..... where g: E Gn,, for elements of G. By convention, denote gL = In,. and go = Inos which are identity matrices. The group action of G on defined by
g0 = (W., (W where W; = 9s W; 9 and b, = 9: b,-
is straightforward to check that forms a symmetry group for f. One expects G to be maximal, however, for the general setting of f. it is open question whether G is maximal. Prior studies only proved G to be maximal when restricted to a restricted setting. For instance, if n > > n2 2 ni
no 1, then G is maximal (Mai & Lampert, 2020; Grigsby et al., 2023).
Design of the map a. First, we decompose the group G into factors Gn, for i E [L - 1], and construct maps Q Gn; for each i. More generally, the goal is define map θ - 9n. for

---

Published as a conference paper at ICLR 2026
arbitrary positive integer n. To this end, we further analyze the structure of Gn by decomposing as follows. Define Pn as the set of monomial matrices whose nonzero entries are all equal to diagonal matrices with
1, that is. the set of permutation matrices. Consider also the set of positive diagonal entries, which is isomorphic to R where Ry0 denotes the multiplicative group of positive real numbers. Every monomial matrix in 9n. can be expressed uniquely as the product of such a diagonal matrix and permutation matrix, that is,
Go = (DP : DEDDRRR and P E P...
Formally, 9,, is isomorphic to the semidirect product 9n, R2 x n. (see Dummit & Foote (2004)). Thus, constructing a: θ - Gn reduces to specifying two maps: one into Pr, and the other into RR
P,,
The case of the group Pn. Since Pn is discrete, any continuous map a:: θ Pn must be constant.
The case of the group R8 The group RR05 can be further decomposed into its coordinate factors, so that the construction of a: θ I2% reduces to specifying n. independent maps a,= O - -g0 for j E [n]. To do this, the main idea is to construct the map from 0 E θ to a vector of size n, after which we take the sin of entries, scale by a small >> 0 and add a unit vector 1 The detailed implementation of a in practice is described in Appendix B.1.
Remark 4.2 (Extension to CNN). For CNNS, the approach follows the same principle but adapted for convolutional filters. Each bias vector 6; retains the same dimensions as in MLPs, while the con- volution filter W; E R x (for ID convolution) or W; E - xexw (for 2D convolution) contains additional spatial dimensions. To handle this, we apply the group action to both the filter's channel dimensions and spatial dimensions. Specifically, the filter W; is treated as having dimen- sions n, X 7 (cw), where e represents the number of input channels (or more generally, any extra spatial dimensions like spatial channels). This allows us to apply the quasi-equivariant layer to the filter a manner similar to the MI.P case, where we perform the scaling operation across the output channels and input channels.
Remark 4.3 (On activation functions beyond ReLU). In the literature, FE has also been studied for feedforward neural networks with other activation functions σ. For instance, in the case of the tanh activation, maximal symmetry group can be determined (see Chen et al. (1993); Fefferman & Markel (1993)). However, for most widely used activation functions, the maximal symmetry group is discrete, making the construction of trivial. For this reason, our analysis focuses on ReLU.
4.3 THE CASE OF MULTIHEAD ATTENTION
Parameter space. Let d denote the token dimension, L. the sequence length, and h the number of heads, where all are positive integers. Define the space of token sequences as S L For a fixed head dimension d,, let W.?, W.K. W.,, WOO E RRdxd, for each i E [M]. and set 0 (W. W.K W.Y, W = Given an input sequence x = (#1...., LL) E R C S the Multihead Attention (MHA) mechanism with heads defined by
h
MHA (x: 0)
Σ softmax (xW, (xWY) (WP)T.
Here, the softmax operator is applied row-wise the similarity matrix (xW2) (XWR) R producing the attention for x. Each row of this matrix forms probability distribution that deter- mines the relative influence of all input tokens on a given output token. Typically, the head dimension is set to dn. d/h. The parameter space of the MHA map is then defined as e = (Rdx).) 4h
We denote by Gd.(dh) the general linear group of degree dn, i.e., the set of all invertible da x dr real matrices.
Maximal symmetry group. Define the following group G = Sn x (GL(d)) x Gd()))) This group is exactly the direct product of the permutation group S₁ with h copies of GL (d,). Each element E G can be written as y = (a, (U,, vi){-)) where a E Sn and U, Vi2 E (dx).). The group G acts naturally the parameter space O as follows:
g0 = (wog (00)
U.U. WK 0(1)
U.1 W. V,, Woo V=-1) im1
(10)
7

---

Published as a conference paper at ICLR 2026
Scale networkk
Leauned scoles
Strtiticaal
3.0
1.18
5:00
2
6.00
Network weights
Meen Virianee
3
Quantiles
wil
Equivariant
Metanetwork
Extracted weights
Illustration of the design of the quasi- equivariant layer. Statistical features are extracted from network weights and biases, then passed through a Scale network to learn the group action. This corresponds to the MLP case, where scaling vector is learned for each layer's weights and biases. The learned scales are applied to the outputs of the equivariant layer, enhancing expressiveness while adding only minimal parameters.
Figure 2:
It is evident that G serves as a symmetry group of the MHA map. The reasoning is follows: the general linear action cancels within the matrix multiplications, while the permutation action induced by commutes with addition. Furthermore, G is maximal, as formalized in the following result.
Theorem 4.4 (See Tran et al. (2025b)). Consider two MHA maps with h heads, parameterized by 0 = (W., W.K. WY, WO Jimi and 0 = (W,, W.K, WV WP) in θ, respectively. Assume that All matrices W.O, W.K, W., W and W.O, W.K, WY, W., for all feasible i, are of rank di-
2. From 0, the matrices (W, ( Fiz1 are pairwise distinct. The same condition holds for 0.
If the two MHA maps are identical, there exists g G such that 0 = go.
Remark 4.5. Note that the conditions on 0 and in Theorem 4.4 can both be expressed as the vanishing of finitely many nonzero polynomials. This corresponds precisely the real algebraic variety= e introduced in Definition 2.2 of maximal symmetry groups
Design of the map a. In analogy with the feedforward case, restrict the construction of a: Q G' to the design of a map e - GL(n) for a general positive integer n. The idea is as follows: first, reshape 0 E θ into an sine function, scale the result by small c > O, and finally add the identity matrix I,, i.e. 0 - sin()(0)) c In. By continuity, and since the range of the sine function [-1, 1], there exists sufficiently small > O such that the resulting matrix is invertible. The detailed implementation of practice is described in Appendix B.I.
matrix via a feedforward network 7. Next, apply the entrywise
Remark 4.6. An alternative approach to constructing map θ GL(n) is to the matrix exponential exp: x GL(n). However, both in theory and in practice, this approach tends to be slow and numerically unstable. Our experimental trials confirmed these issues, and we therefore did not pursue this direction further.
5 EXPERIMENTS
In this section, integrate the quasi-equivariant layer with existing metanetworks: Monomial-NFN for MIP/CNNS and Transformer-NFN for Transformers. The overall layer is illustrated in Figure 2. We provide detailed implementation of the network for each case (MLP/CNN or Transformers) in Appendix B. We evaluate these models on three tasks: predicting CNN generalization from weights, classifying image INRs, and predicting Transformer generalization from parameters. For each task, we compare against the original baseline and an expanded version with more parameters, ensuring fair comparison with our method. Our aim is twofold: the quasi-equivariant layer improves performance efficiently with minimal parameter increase, and it preserves performance under group action transformations. Results are averaged over five runs; hyperparameters and parameter counts are detailed in Appendix B.
5.1 PREDICTING CNN GENERALIZATION
Experiment Setup. We aim to predict pretrained CNN generalization using only their weights, without test data. Experiments use the Small CNN Zoo dataset (Unterthiner et al., 2020), containing
8

---

Published as a conference paper at ICLR 2026
Table 1: Performance prediction of CNNs on the ReLU subset Small CNN with varying scale aug- mentations. The metric used is Kendall's 7. Uncertainties indicate the standard error across 5 runs.
Augment settings
No augment
2/'1, 10*)
24(1, 10a)
2411,
STATNet (Unterthiner et al., 2020)
0.915 + 0.002
0.894 +: 0,0001
0.853
0.007
0.523 = 0.02
0.516 = 0.001
NP (Zhou al., 2024a)
0.003
0.002
0.898
0.884
0.002
HNP (Zhou al., 20244))
0.926 0.003
0.913 0.001
0.903 0.003
0.801 4: 0.003
0.601 0.02
Graph-NN (Kolinas et al., 2024)
0.897 + 0.002
0.892 0.003
0.885 0.002
0.858 :: 0.003
0.851 = 0.002
Monomial -NN (Trun et al., 2024a)
0.922 :: 0.001
0.920 0.001
0.919 0.001
0.920 4: 0.002
0.920 +: 0.001
Monomial-NFF NFN large (68. 65% params ++)
0.923 + 0. 00.0
0.920 + .000
0.920 +: 0.002
0.919 :: 0.001
0.920 = 0.001
Monomial NFN Quasi (ours) (3. 89% params +++
0.926 1: 0. 0.002
0.924 1: 0.002
0.924 0. 002
0.923 1: 0.001
0.924 1: 0.002
Table 2: Classification train and test accuracies (%) for implicit neural representations of MNIST, FashionM- NIST, and CIFAR-I0. Uncertainties indicate standard error over 5 runs.
MNIST
CIFAR-10
FashionMNIST
MLP
10.62
0.71
9.95 0.36
68
31.74
58.21 1
HNP (Zhou al., 2024a)
06.02 0.51
31.61 0.22
0.44
Monomial-NeN et 2024a)
1:
31.23 0.33
61.15 0.55
Momomial-NEN luned 3% params+++)
34.26
61.44
Monomial-NEN Quasi (ours) 3% params 4-4)
70. 0.34
35
0.56
62.
0.27
CNNs trained with varying hyperparameters and activations. Following Tran et al. (2024a), we analyze the ReLU subset, where models follow the group action M Robustness to group-action transformations is evaluated by augmenting the dataset with variants from diagonal matrices D
2(11 10'] for E 21, 3,4) 4] and random permutation matrices Pn. Prediction is measured with Kendall's rank correlation (Kendall, 1938), which quantifies agreement between predicted and true accuracy rankings. Our approach extends Monomial-NET (Tran et al., 2024a), denoted Monomial- NFN Quasi, and is compared with STATNN (Unterthiner et al., 2020), NP, HNP (Zhou al., 2024a), and Graph-NN (Kofinas et al., 2024). For Mononial-NFN, we test both the original and an enlarged, carefully tuned variant for fair comparison.
Table reports model performance. Scaling Monomial-NFN (+68.659 65% parameters) gives minor gains, whereas Monomial-NIFN Quasi shows notable improvement with only +3.89% param- eters. On the original subset, where Monomial-NFN lags behind HNP, the Quasi layer matches its performance. This holds under augmentation, showing that small parameter increases via the Quasi-equivariant layer can enhance expressiveness and flexibility considerably.
Results.
5.2 CLASSIPYING IMPLICIT NEURAL REPRESENTATIONS OF IMAGES
Experiment Setup. This experiment focuses classifying the source class of pretrained Implicit Neural Representation (INR) weights. Following the setup in (Tran et al., 2024a), we use three INR weight datasets introduced in (Zhou et al., 2024a), each corresponding
different image dataset: CIFAR-10 (Krizhevsky & Hinton, 2009), FashionMNIST (Xiao et al., 2017), and MNIST (LeCun & Cortes, 2005). In these datasets, each INR is trained to represent single image, encoding image structure by mapping pixel coordinates (x. y) to pixel color values. CIFAR- 10 images are encoded as 3-channel RGB outputs, while MNIST and FashionMNIST are represented with a single grayscale channel. Since excessively increasing parameters in Monomial-NFN leads to overfitting in this setting, we introduce a variant, MoNomial-NFN tuned, which is carefully adjusted to match the parameter count of Monomial-NFN Quasi. This ensures a fair comparison between the two models.
Results. Table 2 shows the performance of all models on INR classification. The tuned Monomial- NFN, which adds 3% more parameters, yields only minor improvement. In contrast, adding the proposed quasi layer with the same parameter increase allows Monomial-NEY, Quasi to outperform NP on the MNIST task and achieve the best results across all three datasets. The performance gap between Monomial-NFF Quasi and Monomial-NFF NFN is about 1 1% on CIFAR- 10 and FashionMNIST,, and 1.78% MNIST. These results demonstrate the consistency of the proposed method.

---

Published as a conference paper at ICLR 2026
Table Performance measured by Kendall's r of all models on MNIST- and AGNews-Transtoree Transformers datasets. Uncertainties indicate standard error over 5 runs.
Accuracy thresbold
No threshold
20%
40%6
6076
80%
MNIIT-Trongömmeen
MLP
0.866 0.002
0.873 :: 0.001
0.874 + 0.003
0.874 + 0.006
0.873 + 0.007
STATNN (Unterthiner al., 2020)
0.001
0.872 0.001
+
0.800
0.956 0.001
XGBoost (Chen Guestrin, 2016)
0.860 + 0.002
0.839 0.004
0.800 0.003
0.846 + 0.001
0.884 + 0.001
LightGBM (Ke et al., 2017)
0.858 0.002
0.835 d: 0.001
0.647 4: 0.001
0.822 :: 0.001
0.830 :: 0.001
Random Forest (Breiman, 2001)
0.772 0.002
0.758 +: 0.004
0.709 +: 0.001
0.752 + 0.001
0.759 + 0.001
Transformer- NEN (Tran et 2025b)
0.905 0.002
0.899 0.001
0.895
0.002
0.002
Trunsformer-Ne NEN large (57. 66% params ++)
0:907 0.001
0.904
0.002
0.597 1: 0.002
0.897 1: 0.002
0,890 0.001
Transformer-e NEN Quasi (Ours) (4.54% params ++)
0.911 = 0.001
0.905 1: 0. 001
0.898 1: 0.002
0.897 : 0.001
0. 0.892 2: 0.001
AGNews- AGNewo-Tramfformer
MLP
0.879 0.006
0.875 0.001
0.841 1: 0.012
0.842 d: 0.001
0.562 d: 0.006
STATNN (Unterthiner al., 2020)
0.002
0.839
0.813
0.001
XCBoost (Chen& Guestrin, 2016)
0,858 0.001
0.852 0.002
0.672 1: 0.002
0.871 0.001
0.872 0.001
LightGBM (Ke et al., 2017)
0.835 4: 0.001
0.845 à: 0.001
0.637 4: 0,001
0.835 1: 0.001
0.820 0.001
Random Forest (Breimam, 2001)
0.774 0.003
0.801 0.001
0,797 1: 0.061
0.798 :: 0. 0.002
0.773 0.001
Transformer-
et 2025b)
0.910
0.908 0.001
0.897
1:
Transforme--NE large (59. 38% params ++)
0.913 0.000
0.910 0.002
0.898 + 0.002
0.898 + 0.001
0.893 0.002
Camssoomar-NE Quasi (Ours) (5.27% params +++
0.914 :: 0.001
0.913 1 0. 002
0.901 1: 0.001
0.903 :: 0.002
0. 0.896 0.001
5.3 PREDICTING TRANSFORMERS GENERALIZATION
Experiment Setup. In this task, we predict the accuracies of pretrained Transformer checkpoints, aiming to test whether metanetworks capture structural patterns in Transformer weights. Follow- ing (Tran et al., 2025b), we integrate our quasi-layer into Transformer-NFY and conduct evaluations on two datasets: TranSeeran,s,ommerss, built from models trained for MNIST image classification, and Transffrrrr,,,,,,rsssssss derived from models trained for AGNews text classification. Perfor- mance is assessed using Kendall's т. To probe model performance under varying difficulty levels, evaluate not only on the full dataset but also on four subsets defined by minimum accuracy thresholds of 20%, 40%, 60%, and 80%. Since most pretrained models in these datasets achieve high accuracy, maintaining strong Kendall's T becomes more difficult as the threshold increases.
and AGNews- wwwws--ranffomerr In both benchmarks, scaling up -NN to a larger version (with up to 59.38% more parameters) can improve Kendall's r, but adding the quasi-equivariant layer achieves even greater gains with far fewer extra parameters (only up to 5.27%). )). The improvement holds across all accuracy thresholds, demonstrating both the efficiency and effectiveness of our approach.
Results. Table 3 presents the results on
CONCLUSIONS, LIMITATIONS, AND FUTURE DIRECTIONS
This paper introduces quasi-equivariant metanetworks, a framework that relaxes strict equivariance balance symmetry preservation with representational flexibility. By analyzing parameter spaces, their symmetry groups, and maximality, we establish theoretical connection to functional equiva- lence and formalize equivriquiva as a principled extension of strict equivariance. We demon- strate applicability of the framework to feedforward networks and multihead attention, and validate its effectiveness across multiple metanetwork benchmarks, achieving substantial performance gains with minimal additional parameters. A current limitation of our work is that quasi-equivariance has so far been applied primarily to metanetworks with linear architectures. Extending this framework to more complex structures, such as graph-based metanetworks, remains unexplored due to the di- versity and rarity of such architectures. Moreover, the quasi-equivariant design could be beneficial in various other fields, such as computational chemistry, physics, and materials science, where sym- metries may only approximately hold and greater modeling flexibility is required. We view these as promising directions for future research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISGG-TCC-2023-01--5000 supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00) and the NUS Artificial Intelli- gence Institute-Seed Funding (A-8003062-000-00)
This research project is
(FY2023) (A-8002040-00-00) A A-8002039-00-000
Ethics Statement. Considering the scope and focus of this work, we do not anticipate any adverse socictal ethical consequences.
Reproducibility Statement. The source code for all experiments included in paper's sup- plementary materials. Detailed descriptions of our experimental setup can be found in Section 5 and Appendix B. All datasets used are publicly accessible via an anonymous link provided in the README of the supplementary materials.
LLM Usage. In this work, we use large language models (LLMs) solely as a tool to assist and refine the presentation of our ideas. The LLM was employed only improve clarity, grammar, and overall readability, without influencing the scientific content, methodology, or experimental results. All technical contributions, analyses, and conclusions in the paper entirely the authors' original work.
REFERENCES
Francesca Albertini and Eduardo D. Sontag. Identifiability of discrete- time neural networks. In Proceedings of the European Control Conference, pp. 460-465. Springer Berlin, 1993a.
Francesca Albertini and Eduardo D Sontag. For neural networks, function determines form. Neural networks, 6(7):975-2)) -900, 1993b.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International conference on machine learning, pp. 242-252 PMLR, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https:ttp/// openrevieveee net forum? fow.m?id-Huge.. 3N1vG
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849 15854, 2019. doi: 10.1073/pnas. 11073//paas.9003701..... 1903070116. URL https:
/ www. pnw.ppnass.......... wwtp////oooooooooooooooooooooooe /ww//b/i/abb//// abs 10 1073/ /ppass/as...// 1903070114
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pp. 287. Routledge, 2013.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss land- scape. arXiv preprint arXiv:1907.029// 2019.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp. 6491-6506. Association for Computational Linguistics, 2021.
An Mei Chen, Haw minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network emor surfaces. Neural Computation, 5(6)9910-927, 1993.

---

Published as a conference paper at ICLR 2026
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 11 of KDD '16, pp. 785-794, ACM, August 2016. doi: 10.144529399722293955.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen. Guowei Li, H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang. Haowei Zhang. Honghui
Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun. Tian Pei. Tianyu Sun. W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An. Wen Liu, Wenfeng Liang, Wenjun Gao. Wengin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li. Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei. Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong. Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025. URL https: / /ttp/aaavvvv/..///////...//..av. org/ abs 2412 19437.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675- 1685. PMLR, 2019.
David Steven Dummit and Richard M Foote. Abstract algebra, volume 3. Wiley Hoboken, 2004.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5694-5725. PMLR, 2022. URL https:/ / httpp//ocediingg iig...inngg........ ml ptr..pprss///.... v1 62/ dupont html.
Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: Dissecting the weight space of neural networks. In ECAI 2020 24th European Conference on Artificial Intelligence, volume 325 of Frontiers in Artificial Intelligence and Ap- plications, pp. 1119-1126. IOS Press. URL https: //oii/iio//////// orgg / 0. 3233/FATA200200
Ahmed A Elhag, T Konstantin Rusch, Francesco Di Giovanni, and Michael Bronstein. Relaxed equivariance via multitask learning. arXiv preprint 1xXx:24100117776 2024.
Charles Feffermman and Scott Markel. Recovering a feed-forward net from its output. In Advances in Neural Information Processing Systems 6, [7th NIPS Conference. Denver, Colorado, USA, 1993], pp. 335-342. Morgan Kaufmann, 1993. URL http: / paperr....... nips. cc/paper/ 748 wwwooecooeee..... ring -ww--ffffddd----ooooooo---- forward-net---- from it output.
12

---

Published as a conference paper at ICLR 2026
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th Interaational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2018. URL https:/ /wwpn//evewwweeeeeeeeeeee/ net wtt./oormmm/m /twpp////w/ppppp//////eereroeoo.
=J1=b3RcF7.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019. OpenReview.net.
Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In International Conference on Machine Learning, pp. 11734-11760. PMLR, 2023.
Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint ar arXiv:2002.1/833 2020.
Kaiming He, X. Zhang. Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 770-778, 2015. URL https: /aapppi.... semant
206594692.
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Ad- vanced Neural Computers, pp. 129-135. Elsevier, 1990.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, 1997. doi: 10. 19.11///NEC...999.9....... neco. 1997. 9. 135..
URL httsst/::// :/tpss://ciiii org org/10.1162/
Elyssa Hofgard, Rui Wang, Robin Walters, and Tess Smidt. Relaxed equivariant graph neural net- works. In /CML 2024 Workshop on Geometry-grounde Representation Learning and Generative Modeling. 2024. URL https: / openreview net/foouumiddeeeeeeeeeee
Wenbing Huang, Jiaqi Han, Yu Rong. Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanies networks with constraints. In International Conference Learning Represen- lutions, 2022. URL https::/p/// /pwprevieeeeeeeeeeooeeeeeee net/ did-
Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan Nguyen, Doris Tsao, and Anima Anand- kumar. Neural networks with recurrent generative feedback. Advances in Neural Information Processing Systems, 33:535-545, 2020.
John M. Jumper, Richard O. Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russell Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, R. D. Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David L. Silver. Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, July 2021. doi: 10.108884411866-21--088888 URL ht https: : / /wensee....// wrg/103 212 983 826 945.
Sékou-Oumal Kaba and Siamak Ravanbakhsh. Symmetry breaking and equivariant neural networks. arXiv preprint arXiv:2312.0901 2023.
Ioannis Kalogeropoulos, Giorgos Bouritsas, and Yannis Panagakis. Scale equivari- ant graph metanetworks. In Advances in Newral Information Processing Systems 38, 2024. URL http:// wwtp:sssssss...p.i..eeeeacc http://iisssseeaeeeeeee //hash// Ceetract-Cooffeenneee.i html.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: highly efficient gradient boosting decision tree. Advances neural information processing systems, 30, 2017.
M. G. Kendall. A NEW MEASURE OF RANK CORRELATION. Biometrika, 30(1-2):81-93, 06 1938. ISSN 0006-3444. doi: 10.1099/bimee/33.1122..11 URL https: /doi. org/ 2//g//0. 10. 1093/br omet//me//a
13

---

Published as a conference paper at ICLR 2026
Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie, Eugene Belilovsky, and Simon macoste-Julien. Accelerating training with neuron interaction and nowcasting networks. In The Thirteenth Inter- national Conference on Learning Representations, 2025. forum? d=cUFT il 6hEG.
nett/
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant represen- tations of neural networks. In The Twelfih International Conference on Learning Representations. 2024. URL https: / woppreeeeeeieeeee/e net/ / forum? dd=ommmii000FFf oF sMyDBB..
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009, URL https: / /tww://ww..cc//////////// cs. cs. wwwp:eddddddkkeeeeeeeeee
features- cettuures-2009-110 2009-TR.pdE
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems 25, pp. 1106-1114, 2012. URL https: http:///pppoeeeeee.geeeeeee cc. https//eeerriss.ccp.pp/////222 http//ppp/////aaaaeeeee c399862d3b (e1obb6664436666 hat.aa/ee600..1111116646665----0 -bbsaaaaaaa..obbbaaaaaat........ html.
Vera Kurkova and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543-558, 1994.
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. In Proceedings of the IEEE, 2005. URL https: semant i icscholar www.aar.rrrooooooDDDD. 60282629.
Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In The Eleventh International Conference on Learning Representations, ICLR 2023. URL https: // //wppeeeeieww...////////////...e net westt//oommiid-oooo 3uadi.
Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu net- works. In 8th International Conference on Learning Representations (ICLR), 2020.
Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09700 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In The Tenth International Conference on Learning Representations, 2022. URL
Andreas Müller, Carlo Curino, and Raghu Ramakrishnan. Mothernet: foundational hypernetwork for tabular classification. arXiv preprint arXiv:23/2.08594 2023.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. In International Conference Machine Learning, pp. 25790-25816 PMLR, 2023.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of wer-paraametiizaiion in generalization of neural networks. arXiv preprint arXiv: 18056 2018
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: An empirical study. In 6th International Con- ference on Learning Representations (ICLR), Vancouver, BC, Canada, 2018 OpenReview.net.
Stefanos Pertigkiozoglou, Evangelos Chatzipantazis, Shubhendu Trivedi, and Kostas Dani- ilidis. Improving equivariant model training via constraint relaxation. In Advances in Neu- ral Information Processing Systems, volume 37, pp. 83497-83520. Curran Associates, Inc., fwles/paper//000/ pappr/202//
2024. URL https:://pccceeedaeaaeeee
nwwriiisdddeii.inis.seeeeeeeee.e ec wttpscee_eeapeeeeeeeeeeee fi ftte:22002essAAAaaaa2aa233133322 can
14

---

Published as a conference paper at ICLR 2026
Duy-Tung Pham, Viet-Hoang Tran, Thieu Vo, and Tan Minh Nguyen.
tree-sliced wasserstein distance. In The Fourteenth International Conference on Learning Representations, 2026. URL https::///... httpss///openeeew..cooo ntt.///wwweeeeeeeeeeeeeeeeeeeeee
M. Raissi, Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equa- tions. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991. doi: https://ooi.
org/10..1016/..jp.2011 10.04. 10.045. URL https: /www wwwp:::sieieieeieiee.e......eeee com/ coom/science/
David E. Rumelhart, Geoffrey E. Hinton, and J. Williams. Learning internal representations by error propagation. 1986. URL nttpss://ap wtt.aapp...aiiiiiieeemmmm semanticscholaa wtw.aa.oroo/ooooootta
62245742.
Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Net- works. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pp. 59-63. IEEE, 2000.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self- Self-super representation learn- ing on neural network weights for model characteristic prediction. Advances in Neural Informa- tion Processing Systems, 34:16481-16493, 2021.
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nicto, and Damian Borth. Hyper- representations generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 35:27906-27920, 2022a.
Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems, 35:38134-38844 2022b.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object rep- resentations for manipulation. 2022 International Conference on Robotics and Automation
(ICRA), pp. 6394-6400. IEEE, 2022.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. Ed- itable neural networks. In 8th International Conference on Learning Representations, 2020. URL https: /ttpp:://cpenreviw net / forum Fid=HJedXaBt....
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. December 6-12. 2020, virtual, 2020. URL https::/ppooceeddee hew.rreeeeednnngn.eei...se..... wttpccppppppp//././eeeeeoe haap///h/2hhah/////// 3300118811112211183333333333
Abst html.
Samuel Sokota, Hengyuan Hu, David Wu, Zico Kolter, Jakob Nicolaus Foerster, and Noam Brown. A fine-tuning approach belief state modeling. International Conference on Learning Representations, 2021.
Kenneth O Stanley.
Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:131-162, 2007.
Hoang Tran, Thieu Vo, Tho Huu, An Nguyen The, and Tan Nguyen. Monomial matrix group equivariant neural functional networks. In Advances in Neural In- formation Processing Systems, volume 37, pp. 48628 48665. Curran Associates, Inc., 2024a. URL https: http:p/:///pppeeeegsss.sooo
wtweediiinnnnnniiiiie neurips oc httpprrrpppccppppeeeeeeeeeeeeeee wwwes//ppee/eee/ee wowppe-ennoeeeeneee..o .ddf.
fi htw.::7:cc88811111eeeee611113333
15

---

Published as a conference paper at ICLR 2026
Hoang V Tran, Khoi NM Nguyen, Trang Pham, Thanh T Chu, Tam Le, and Tan M Nguyen. Distance- based tree-sliced wasserstein distance. arXiv preprint arXiv: 2025a.
Hoang V. Tran, Thicu Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. In The Thirteenth International Conference on Learning Representations, ICLR, 2025b.
Hoang- Viet Tran, Thicu N Vo, Tho Tran Huu, and Tan Minh Nguyen. A clifford algebraic approach to (u)-equivariant high-order graph neural networks. arXiv preprint arXiv:2410.0469. 2024b.
Thanh Tran, Viet-Hoang Tran, Thanh Chu, Trang Pham, Laurent El Ghaoui, Tam Le, and Tan M Nguyen. Tree-sliced wasserstein distance with nonlinear projection. arXiv preprint 1xXiv:2505.00966 2025c.
Viet-Hoang Tran, Trang Pham, Tho Tran, Minh Khoi Nguyen Nhat, Thanh Chu, Tam Le, and Tan M Nguyen. Tree-sliced wasserstein distance: geometric perspective. arXiv preprint arXiv:2406. 13725, 2024c.
Viet-Hoang Tran, Thanh T Chu, Khoi NM Nguyen, Trang Pham, Tam Le, and Tan M Nguyen. Spherical tree-sliced wasserstein distance. arXiv preprint arXiv:2503. /249, 2025d.
Viet-Hoang Tran, Thanh Tran, Thanh Chu, Tam Le, and Tan Minh Nguyen. Tree-sliced entropy par- tial transport. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025e.
Viet-Hoang Tran, Van Hoan Trinh, Khanh Vinh Bui, and Tan M Nguyen. On linear mode connec- tivity of mixture-of-exppee architectures. arXiv preprint arXiv:2509. 11348, 2025f.
Viet-Hoang Tran, Thanh Tran, Thanh Chu, Duy-Tung Pham, Trung- Khang Tran, Tam Le, and Tan Minh Nguyen. Revisiting tree-sliced wasserstein distance through the lens of the fer- mat-weber problem. In The Fourteenth International Conference Learning Representationss 2026a. URL https:t/p//.. wpwpe/vveweeeeeee////e/.. net/ forum? d=kDqG03v055.
Viet-Hoang Tran, Thanh Tran, Thanh Chu, Duy-Tune Pham, Trung- Khang Tran, Tam Le, and Tan Minh Nguyen. Tree-sliced sobolev IPM. In The Fourteenth International Conference Learning Representations, 2026b. URL. https:/ /oppnrevieeeeeeeee / ew. net/ forum?id= HHNOSxaLkF.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predict- ing neural network accuracy from weights. arXiv preprini arXiv:2002. /448, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998-6008, 2017. URL https:ttp.... http://////oed//oeeeeoeeoo igs. neurips. oc
html.
Thieu Vo, Hoang V. Tran, Tho Tran Huu, An Nguyen The, Thanh Tran, Minh-Khoi Nguyen-Nhat, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant polynomial functional networks. In Forty- second International Conference on Machine Learning, 2025. URL https: /openreview. fer/ff TDgEpgg2.
Robin Walters, Jinxi Li, and Rose Yu. Trajectory prediction using equivariant continuous convolu- tion. arXiv preprint arXiv:2010.113:2 2020.
Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly sym- metric dynamics. In International Conference on Machine Learning, pp. 2022.
PMLR,
Shih-Hsin Wang, Yung-Chang Hsu, Justin Baker, Andrea L. Bertozzi, Jack Xin, and Bao Wang. Re- thinking the benefits of steerable features in 3d equivariant graph neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https: :pp:// / //operreview.
net. nermf d=mGG.TAyR@w..
16

---

Published as a conference paper at ICLR 2026
Zhiqiang Wu, Yingjie Liu, Licheng Sun, Jian Yang, Hanlin Dong, Shing-Ho Lin, Xuan Tang, Jinpeng Mi, Bo Jin, and Xian Wei. Relaxed rotational equivariance via g-biases vision. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 8541-8549, 2025.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.0777 2017. URL abs/ /bb//708. 07747.
org/
Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang. Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals, Advances in Neural Infor- mation Processing Systems, 36, 2024a.
Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burn,, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems, 36, 2024b.
17

---

Published as a conference paper at ICLR 2026
TABLE OF NOTATION
Parameter space the network
W,
Weight of feed forward neural network in layer Bias of feed forward neural network in layer i Monomial matrix of size n
b;
Gn
h
Number of head of Attention module Hidden dimension of model
da
Hidden dimension of head in the model
Da
Dimension of key/query vector in Attention module Weight of query matrix of head
w&
WK
Weight of key matrix of head Weight of value matrix of head Weight of out projection matrix of head i
W.
Wo
Symmetric group of the weight space
o
Relu activation
Sa
Head permutation group action in Attention module Equivariant layer
EO
T
Invariant layer
Rd
d-dimensional Euclidean space standard dot product disjoint union
(5))
g
element of group
GL(d,)
General linear group of invertible da d, matrices over IR
a0
Quasi-equivariant map
80
Equivariant map
18

---

Published as a conference paper at ICLR 2026
Supplement to "Quasi-Equivariant Metanetworks"
Table of Contents
A Quaii-Equivariant Metanetworks
19
On the Well-Definedness of the Muasi-Equivariance Property
19
A.2 Proof of Proposition 3.3
21
Additional Details of Experiments
22
B.1 Details Group Action Learning
22
B.2 Predicting CNN Generalization
23
B.3 Classifying implicit neural representations of images
24
B.4
Predicting Transformers Generalization
25
B.5 Ablation on the MLP network for Group Action Learning
26
B.6
Experiments on Augmented AGNews- Transformers dataset
27
B.7
Analysis the learned scaling
27
B.8
Sensitivity of
28
B.9 Weight space style editing
29
A QUASI-EQUIVARIANT METANETWORKS
Invariance and Equivariance in Machine Learning. Equivariant neural networks (Cohen & Welling, 2016) incorporate task-specific symmetries directly into their architectures, leading to im- proved generalization and greater sample efficiency. They have achieved strong empirical success across diverse domains, including trajectory prediction (Walters et al., 2020), robotics (Simeonov et al., 2022), graph representation learning (Satorras et al., 2021; Tran et al., 2024b), and Optimal Transport- based methods (Pham et al., 2026; Tran al., 2026a;b; 2024c; 2025e;d;arc). Leveraging equivariance consistently enhances performance, data efficiency, and robustness under distribution shifts.
In the main paper, we introduced the concept of quasi- euuivaiiuuva and dis- cussed its key insights. Here, establish formal theoretical guarantees and provide rigorous proofs for the results related to this notion.
A.1 ON THE WELL-DEE BELL-DEFINEDNESS OF THE QUASI- EQUAVA-QQIVARAANCE PROPERTY
We now examine the well-definedness of the quasi-equivariance property, stated in Remark 3.2. Although the construction of quasi-equivariant maps discussed in this section is not used in our implementation due its inefficiency, we include the following analysis for completeness and for potential future work, where one may wish to construct quasi-equivariant maps using this approach.
Setup. Let group
on set O. Let (possibly the same) group Hl act on a X. map F: θ - X' is called a-quasi- a-quasi--quuivaiann if there exists
a a: Q
(11)
such that for all g E and E O,
F(g0) = a(g, 0) F(0).
(12)
When H G and the group action of H on X is the given group action of G, this reduces to original formulation on quasi-equivariance.
Well-definedness across different representatives. The relation 9101 g202 gives two represen- tations of the same point in e. Thus Equation (13) must produce the same value of F. A necessary and sufficient condition
presented as follows.
19

---

Published as a conference paper at ICLR 2026
Proposition A.1 (Normalized 1-cocycle condition). The following statements equivalent:
1. For every choice of a section S C θ meeting each G-orbit exactly once, and for every seed map S - X' satisfying the stabilizer constraint (see Proposition A.2), there exists unique map F: θ obeying the quasi-equivarianee relation
F(g0) alg, 0) F(0), for all G,0 E O
(13)
such that F is independent of the choice of representative of point in θ
2. The function a : G x O - H satisfies, for all 91, 92 E G and 8 E O, a(e, CH.
(14)
o(g192, 0) = agg), 920) alg2
(15)
Proof. The proof proceeds as follows.
Necessity. Fix 91, 02, 0. Using Equation (13) twice,
F( a(91, 920) (9200) al91, 920) alg,, 0) F(0).
(16)
On the other hand, applying Equation (13) once with 9192 gives
F(g1920) = (g1 0) F(0): so Equation (15) of 1-cocycle follows. Setting y in Equation (13) yields Equation (14) of Normalization.
(17)
Sufficiency. Given any representative g0o of point in the orbit of lo. define F(g0o) by Equa- tion (13). If 9:00 9200, then 92 91 E Ga, and the cocycle identity implies the two definitions coincide provided the stabilizer constraint Proposition A.2 holds.
Stabilizers and orbit descent. Denote Go {h G: h0 0). The values of Go control whether F is well-defined from orbit data alone. Proposition A.2 (Stabilizer constraints). Assume Equations (14) and (15). Then for each O and E Gg. one has
F(0)= F(h0) (((1,)))((()
(18)
Hence F(O) must lie in the fixed-point sei
Fixx(a(Go.0)) = fi E X a(h,0) r x for all h E Go).
(19)
In particular:
If F required to be definable independently of any additional constraints on values (i.e., to descend G/G without restricting the image), then necessary and sufficient condition is
a(h, 0) cH for all h Ge, 0 e θ.
(20)
More generally, F(0) is allowed to live in the moving fixed-point set above: then triviality stabilizers is not required, but the image of F is constrained.
Gauge/coboundary equivalence. Two quasi-equivvaiant structures related by a change of variables in X' are equivalent.
Proposition A.3 (Gauge transform and trivial class). Let B: O - H. Define
a (g, 0) 3(g0)a(g, 0) 8(0) F= = 3(0) F()).
(21)
Then F satisfies Equation (13) with a if and only if Fd satisfies Equation (13) with a In particular, if or is coboundary, i.e.
alg, 0) = 3(0 3(0)-1,
(22)
then with F'(0) = 3(0) -1 F(() one has the strict equivariance:
F(gf)= (0)
if H acts trivially,
(23)
g- F'((()
if ß is valued in G and H = G.
(24)
Thus, normalized cocycles modulo coboundaries classify muasi-equivariann structures up to gauge (a H22 of the transformation groupoid G x O with coefficients in H).
20

---

Published as a conference paper at ICLR 2026
Regularity (topological/mmoohh settings). If I,,, X' are topologiaal (or smooth) spaces and the group actions are continuous (or smooth), then to ensure F is continuous (or smooth) whenever the seed s is, one additionally asks a(:,)) to be continuous (or smooth) and the group actions to be continuous (or smooth). The results above remain valid verbatim.
Conclusion. We have the following observations.
1. Strict equivariance. af 0) = g (and H = G with the given action), then Equation (15) is automatic and we recover the standard G-equivariance F(gØ) = 9 F(O).
2. Homomorphic twist. If a(g, 0) = (g)g) for homomorphism v G - H. then Equa- tion (15) holds. To descend to orbits without image constraints one needs (b(h) eg for all h E Gg and all 0 (i.e., 1 trivial on all stabilizers).
3. Coboundary (gauge) case. If (g. 0) = 3(g0) one can gauge to a strictly equiv- ariant F' as in Proposition A.3.
To make sense of the equivariuie relation F(gØ) = agg. 0) F(0) independently of how a point of O is represented, the essential structural requirement is that be normalized 1-cocycle on the transformation groupoid G x e with values in H (Equations (14) and 15). Descent to the orbit space without restricting the image of F additionally demands triviality on stabilizers. Up to gauge, quasi-equivariant structures are classified by the corresponding first cohomology set; coboundaries are precisely those that can be turned into strict equivariance by a change of variables.
Remark A.4. The three conclusions regarding the conditions on serve as necessary requirements for the existence of a map F satisfying
F(gØ) = (g)
(25)
A.2 PROOF OF PROPOSITION 3.3
In this section, we provide the proof for Proposition 3.3.
Proof of Proposition 3.3. We provide proof of each part of the proposition.
Part 1. Assume i and vi are G-quasi-equivarriat.
wh E G. V0 E O, 3hg E G (o(hf) h14(())
(26)
Then
4y G,, 4O E O, 3hg G (hh10) = (1)
(27)
Taking 9 = ((0),
vh E G, 40 E θ 3hg E G: vv((h)) = ((h)p(0)) h2)()(())
(28)
Hence
4h G, 40 E 0, 3hg E G: (hh)) h2 ()),
(29)
so th o yo is G-quasi-equiivariant.
Part 2. Assume is G-quasi-equiivarian and & is G-invariant:
Vh G, va o, 3hg E G: v(h0) = h)))(0,
(30)
and
AkE 04 : (k() (00).
(31)
Taking A = hi and 0 0- (()))
wh E G, 40 E : v(v(h()) (h))p()) = ())()))
(32)
Therefore
G. 0 ( = (* o ()).
(33)
so vp o is G-invariant.
The proof is complete.
21

---

Published as a conference paper at ICLR 2026
B ADDITIONAL DETAILS OF EXPERIMENTS
B.1 DETAILS ON GROUP ACTION LEARNING
We provide more details on building the process to learn group actions for two cases: MLP/CNN
and Transformers.
Our quasi-equivariant layer is designed in two stages: feature selection and group action learning. The goal is learn a group action (a) from network weights and biases, and then apply it to the outputs of existing equivariant layers.
Feature Selection: A straightforward approach is flatten and concatenate weights and biases, but this introduces excessive parameters and becomes infeasible for large networks. To address this, inspired by STATNET (Unterthiner et al., 2020), we instead compute sta- tistical features: mean, variance, and five quantiles (0, 0.25, 0.5, 0.75, 1) of weights and biases. These features are concatenated into a compact representation that scales consis- tently with network size while retaining essential information.
We adopt MLP to model the group action, tailored to the underlying MIP Transformer weight space. encourage stability, the action is learned around the identity. Inspired by Fourier analysis, where sine functions form basic signal components, we introduce a structured noise mechanism: à = sin Wsatte + dscate) e + {1., 1,); where Waaae and dwale are parameters of MLP, and is small learnable factor. This generates mild oscillations centered at unity, preserving the base equivariant behavior while providing flexibility for improved learning.
Group Action Learning:
Implementation of Auaas--qquivarintt Layer for MLP/CNN weights. In MLPs and CNNs, goal is to learn positive scale vector for each layer. The group action applied to the output dimensions of the weights and biases by scaling all neurons in the output dimension, while cor- respondingly scaling the input dimension of the subsequent layer with the reciprocal value of the scale.
To construct this, we first extract the weights and biases from the input network. For each layer, we compute seven statistical features: the mean, variance, and five quantiles (0, 0.25, 0.5, 0.75, I). Features from weights and biases are concatenated, yielding a 14-dimensional representation per layer. Aggregating across all layers produces a feature vector of size 14 L, where the total number of layers in the network.
The group action for each layer is parameterized as positive scale vector of size Nout, where Nout denotes the layer's output dimension. We learn this vector using Gated-MI.P with hidden dimension 32, assigning a separate network to each layer. To ensure stability and positivity, we incorporate the structured noise mechanism described above. The resulting scale is then applied to the final equivariant layer the metanetwork (specifically, Monomial-NFN),
Implementation of Quasi-Equivariant Layer for Transformer weights. Transformers, our objective is to learn two invertible matrices, M and N, for each layer, representing the GL. group action applied to W W., W,, and W., Specifically, M is applied to the query and key matrices, yielding W, W,M" and W, W- This ensures that the attention score remains unchanged, since
(W.M")((..M-)))) W.WTT
Similarly, N is applied to the value and output projection matrices, transforming them into W.N and N W..
To construct these transformations, we extract all weights and biases from the input network: Wg. Ws, W., Wo, WA. ba, WB. bg- For each layer, we compute seven statistical features the mean, variance, and five quantiles (0, 0.25, 0.5, 0.75, 1). Concatenating features from all weights and biases gives a 56-dimensional representation per layer. Aggregating across L layers results in feature vector of size 56 L.
22

---

Published as a conference paper at ICLR 2026
The group action for each layer is parameterized two invertible matrices of shape (D, D), where D denotes the hidden dimension in attention, and each attention head has its own pair (M, To learn these matrices,
employ MLP with hidden dimension 32, assigning The MLP maps the feature vector to n4 D.D, which is then reshaped into (nn, D, D). To guarantee stability and numerical soundness, procedure produces matrices that are invertible almost everywhere, allowing stable training, The learned transformations are finally applied the last equivariant layer of the metanetwork (specifi- cally, Transformer-NFN),
network per layer.
apply the structured noise mechanism described above. This
On the design of quusi-equivariant layer. the MLP/CNN case, the map a consists of two parts: constant map for the group Pn and a map for the group R To learn the latter map, we construct a network that can translate from the weight space to a diagonal matrix in R%o. which can also be represented as vector in R" with all positive entries. We compute statistical features of the weights, with shape D. to serve as input to the network. Therefore, the introduced network is basically MLP (Waatee; bscate } that maps from D n.
The formula sin seeett I -C 1, naturally relaxes the strict equivariance typically imposed in metanetworks. By introducing sine function, we allow the transformation gently oscillate around the identity, creating a smooth, controlled variation. This approach is inspired by Fourier analysis, where sine waves naturally introduce periodic fluctuations, offering flexibility without disrupting the overall structure. The small learnable parameter simply scales these oscillations, determining how much relaxation apply. This design provides a natural way balance stability and flexibility, enabling the model to remain expressive while avoiding the constraints of strict equivariance.
B.2 PREDICTING CNN GENERALIZATION
Dataset. The Small CNN Zoo provides a ReLU subset containing 6,050 samples for training and 1,513 for testing. To enlarge the dataset, apply data augmentation with a factor of 2, generating one additional variant for every original instance. This procedure yields an augmented ReLU subset with 12, 100 training examples and 3,026 test examples.
Baselines.
We evaluate our model against five established baselines:
STATNN (Unterthiner et al., 2020): extracts statistical features from network weights and biases.
Graph-NN (Kofinas al., 2024): models network parameters as graphs and applies Graph
Neural Networks for processing.
NP and HNP (Zhou et al., 2024a): integrate neuron permutation symmetries into neural functional networks.
Monomial-NFN (Tran et al., 2024a): generalizes the action on weights from permutation matrices to monomial matrices, incorporating scaling and sign-flip symmetries.
Model Configurations. Our Monomial-NFN Quasi extends Monomial-NFY which follows the architecture of Zhou al. (2024a). The base model consists three equivariant Monomial-NFN layers with 16, 16, and 5 channels, each followed by ReLU activation. In our variant, the fi- nal Monomial-NFN layer is replaced with a Monomial-NFN Quasi layer, where a learned scale applied to the output of the standard Monomial-NFN layer.
The resulting weight-space features are then fed into an invariant Monomial-NFN layer with Monomial-NFN pooling, which contains learnable parameters. This pooling layer normalizes the weights along the hidden dimension and computes averages across rows (first layer), columns (last layer), or both (intermediate layers). The pooled output is flattened and projected to 200 followed by an MLP with two hidden layers and ReLU activations. Finally, the output is linearly mapped to scalar and passed through sigmoid function.
23

---

Published as a conference paper at ICLR 2026
Table 4: Number of parameters of all models for prediciting generalization task.
Model
STATNN
NP
HNP
Monumial NFN
Monumial NFN large
Monumial NFN Quasi
Number of parameters
1.6M
2.05M
2.8M
0,25M
0.43
0.26M
Table 5: Hyperparameters for Monomial-NEN and Monomial-NFN an on predicting generalization task.
Mosomial-NFN dim
MLP dim
Optimizer
Learning
size
Epoch
Monomial-NFN
16.3]
200
Binary cross-cntropy
Adam
0.001
8
50
Monomial.NFN large
[20,20.5]
Rinary crosseenropy
Adam
0.001
50
Mosomial-NFF Quasi 16.5 16.5 (quasi layer)
200
Binary cross-entropy
Adam
0.001
50
We train the model with Binary Cross Entropy (BCE) loss for 50 epochs, applying early stopping based on a validation threshold т. On an A100 GPU, the full training process requires approximately 35 minutes.
Training.
Other Baselines. For Graph-NN, we adopt the official implementation provided in (Kofinas et al., 2024) Monomial-NFF follow the setups in (Zhou et al., 2024a) (https://github..c http:://giihub..oo//A/ Aolm/ YangZhou/nfi) and (Tran et al., 2024a). For these models, we use three equivariant layers with channel sizes of 16, processed through average pooling, and subsequently passed
The implementations NP, HNP, and
16, and 5. The features extracted into three MLP layers, each with a hidden size of 200. We provide the number of parameters and hyperparameters in Table and Table
B.3 CLASSIFYING IMPLICIT NEURAL REPRESENTATIONS OF IMAGES
Dataset. We employ the original INRs dataset, which includes three subsets: CIFAR-10 10, MNIST, and Fashion-MNIST Each image in these datasets is represented using single SIREN model, as described in Zhou et al. (2024a). Following the setup in Tran et al. (2024a), no data augmentation is applied. The numbers of training, validation, and test samples for each dataset summarized in Table 6.
Baselines. Follow (Zhou al., 2024a; Tran al., 2024a), we compare our method against MLP, NP, HNP, and Monomial-NFN, For baselines, we follow the architecture in Zhou et al. (2024a), with a reduced hidden dimension (512 - 256) to mitigate overfitting. All baseline models and our base model use hidden dimension of 256. The hyperparameters of Monomial-NFN and the tuned version is given Table
Model Configurations. In these experiments, architecture consists of two Monomial-NFN layers with sine activations, followed by one Monomial -NNN Quasi layer with absolute activation. The hidden dimensions for Monomial-NEN Quasi layers vary by dataset and are listed in Table 8. We also provide the parameters count for all models in Table 9
The subsequent design follows the NP and HNP models of Zhou al. (2024a). Specifically, we apply Gaussian Fourier transformation to encode the input with sine and cosine components, ex- panding from one dimension to 256 dimensions. When using NP as the base layer, the features are further processed by IOSinusoidalEncodigg mum frequency of 10 and six frequency bands. The encoded features are then passed through three NP or HNP layers with ReLU activations, followed by average pooling. The pooled output is flat- tened and fed into an MLP with two hidden layers of 1000 units each, also with ReLU activations. Finally, the output is linearly projected scalar.
positional encoding tailored for NP-with maxi-
For the MNIST dataset, we insert a Channel Dropout layer (after each HNP ReLU activation) and Dropout layer (after each MLP ReLU activation), both with dropout rate of O.1. Training Binary Cross Entropy (BCE) loss for 200,000 steps, which takes approximately hour 50 minutes on an A 100 GPU.
24

---

Published as a conference paper at ICLR 2026
Table 6: Dataset size for Classifying INRs task.
Train
Validation
Test
CIFAR-10 10 MNIST
45000
5000
10000
45000
5000
10000
Fashion-MNIST
45000
5000
20000
Table 7: Hyperparameters of Monomial NFN and Monomial-NFF tuned (in parentheses) for each dataset in Classify INRs task.
MNIST
Rashion-MNIST
CIFAR-10
Monomial-NHN hidden dim
64 (128)
64 (128)
16 (64)
Base model
HNP
NP
HNP
Base model hidden dim
256
256
256
MLP hidden neurons
1000
500
1000
Dropout value
0.1
Leamnnng rate
0.000075
0.0001
0.0001
Batch size
32
32
Number training steps
200000
200000
200000
Loss function
Binary cross-entropy Binary cross-entropy
Binary cross-entropy
Table 8: Hyperparameters of Monomial-NFN Quasi for each dataset in Classify INRs task.
MNIST
Fashion-MNIST
CIFAR-10
Scale network hidden dim
32
32
Scale network weight initialization
Xavier
Xavier
Xavier
Scale network initialization
0.01
0.01
0.01
Monomial-NFi NFN hidden dim
64
64
16
Base model
HNP
NP
НNN
Base model hidden dim
256
256
256
MLP hidden neurons
1000
500
1000
Dropout value
0.1
0
0
Learning rate
0.000075
0.0001 32
0.0001 32
Batch
32
Number training steps Loss function
200000
200000
200000
Binary cross-entropy
Binary cross-entropy
Binary cross-entropy
Table 9: Number of parameters of all models for classifying INRs task.
CIFAR-10
MNIST
Fashion -NNIST
MLP NP HNP
2M
2M
2M
16M
15M
15M
42M
22M
22M
Monomial-NNi
16M
22M
20M
Monomial-NFP tuned
16.3M
22.3M
20.7M
Monomial-NEN Quasi (ours)
16.3M
22.2M
20.5M
B.4 PREDICTING TRANSFORMERS GENERALIZATION
Dataset. We use the Small Transformer Zoo dataset (Tran et al., 2025b), which includes two sub- sets: MNIST-Trrnnsoomerr and Transforrr.....mm.r...... These datasets are generated by training Transformer models on MNIST image classification and AGNews text classification tasks, while varying key hyperparameters such training fraction, dropout rate, learning rate, and weight ini- tialization. In total, the zoo contains 62,756 models for Following the experimental setup (Tran al., 2025b), we use the check- points at epoch 75, corresponding to 15,689 models in MNIST--rrasformer and 15,949 models in AGNews-Transfomeer The ratio of train, validation and test set is 0.7, 0.15, and 0. 15, respectively.
and 63,796 models for
Baselines.
The baselines implemented following (Tran et al., 2025b). More specifically:
MLP. For the MLP baseline, each network component is treated independently. The corre- sponding weights flattened and passed through dedicated ML.ss single hidden layer
25

---

Published as a conference paper at ICLR 2026
Table 10: Number of parameters for all models
Model
Trass-Transsormrrs
AGNews- TaakloswTTansrmmmrss
Transformer
1.812M
1.80M
Transformer-Nii large
2.857M
2.857M
Transformer- NHH quasi
1.894M
1.877M
MLP
0.933M
0.896M
STATNN
0,203M
0.168M
Table 11: Performance measured by Kendall's 7 of all models on L.GNess-ranssoore dataset. Uncertain- ties indicate standard runs.
Accuracy threshold 10%
Quasi dim
No threshold
20%
60%
80%
Transformer NFN
0.910 1 0.001
0.908 1: 0.001
0.897
0.001
0.896
0.001
0.890 0.001
Transfomer--FFt Quasi
0.911 0.002
0.909
0.001
0.897
0.897
0.891
0.001
Transformer NFN Quasi
16
0.913 + 0.002
0.911 + 0.001
0.901 + 0.001
0.902 + 0.002
0.894 0.001
Transformer-Ne Quasi
32
0.914 + 0.001
0.913 1: 0.002
0.901
0.001
0.903
0. 0.002
0.896 0.001
Transfommer NFN Quasi
64
0.913 1: 0.001
0.911 1: 0.001
0.899 0.002
0.903 1: 0.002
0.894 0.002
with 50 units used for both the transformer block and the embedding, while the classifier is modeled by a two-layer MLP with 50 units per layer. The outputs from these modules are concatenated and further processed by final MLP that produces the prediction.
STATNN (Unterthiner et al., 2020). To adapt STATNN to transformer architectures, we first compute statistical summaries from the weights of the query, key, value, and output projections, together with the weights and biases of the two feedforward layers. These features are concatenated and given input to a one-layer MLP with 256 hidden units. For the classifier, we retain the original STATNN feature extraction scheme, followed by another MLP with 256 hidden units. The embedding is handled separately with one-layer MLP of 64 hidden units. The outputs from all three components are merged and passed through final single-layer MI.P to obtain the prediction.
XGBoost (Chen & Guestrin, 2016). LightGBM (Ke et al., 2017), and Random Forest (Breiman, 2001). For tree-based approaches, we directly flatten all component weights and use them input features. Across all three models, set the hyperparameters to maximum tree depth of 10, a minimum child weight of 50, and at most 256 leaves.
Model Configurations. Our approach built on the Transformer-NEY backbone, with the Transformer-Ne Quasi model consisting of three modules that process the weights of a transformer network. The embedding and classifier components are modeled as MLPs with ReLU activations, each applied independently to their respective inputs. The transformer block is handled separately using an invariant architecture: several equivariant Transformer-NFN Quasi layers with ReLU acti- vations are applied to the two MLP submodules of the block, and their outputs are passed through an invariant polynomial Transformer-Nii layer. The resulting vectors from all components are concatenated and processed by final MLP with Sigmoid activation to produce the prediction.
In our experiments, the embedding module is implemented as a one-layer MLP with 10 hidden units, while the classifier is two-layer MLP, each with 10 hidden units. For the transformer block, we use an equivariant Transformer-Nei NFN Quasi layer with 10 hidden channels, followed by an invariant Transformer- NEN layer and MLP that generates 10-dimensional output vector. These outputs are concatenated and fed into a classification head to yield the final prediction.
B.5 ABLATION ON THE MLP NETWORK FOR GROUP ACTION LEARNING
Experiment Setup. To assess the robustness of the scale network, we conduct experiments by systematically varying the hidden dimension of the MLPs. Our evaluation carried out using the Transformer-Ner NFN Quasi architecture tigate the capacity of the model to learn the two invertible matrices M and under different hidden dimensions, specifically 8, 16, 32, and 64.
the
dataset. In particular, we inves-
26

---

Published as a conference paper at ICLR 2026
Table 11 reports the performance of Transformer-NFR Quasi under different hidden di- mensions of the scale network. Incorporating the quasi-equivarian layer consistently improves per- formance, with gains increasing the hidden dimension grows, and reaching the highest score 32 dimensions. Based on this observation, we select a hidden dimension of 32 for the final model.
Results.
B.6 EXPERIMENTS ON AUGMENTED AGNEWS- SGNEWS-TRANSFONE TRANSFORMERS DATASET
Experiment Setup. We evaluate the robustness of Transformer-No Quasi under strong group actions in weight space, testing whether the quasi layer compromises architectural symmetry. Fol- lowing the setup in (Tran et al., 2025b), we conduct experiments on the AGNews- TeNNews-ransffrmers dataset augmented with the group action Gu- Both training and test sets are 2-fold augmented: the original weights are retained, and additional weights are generated by applying permutations and scaling transformations Transformer modules. The elements of M and N are uniformly sampled from --1, 1, [-10, 10). and [ 100, 100].
Results. Table 12 summarizes the results. Transformer-NFi shows stable Kendall's т values across all ranges of scaling, with a consistent score of 0.914. While our model does not gain from increasingly large augmentation scales, its performance remains steady, highlighting the balanced trade-off of Transformer-NFI Quasi: it enhances the expressiveness of the base model while pre- serving its inherent symmetry.
Table 12: Performance measured by Kendall's r of all models on augmented dataset using the group action Gu. Uncertainties indicate standard error over runs.
Original
[-1,1]
[-10, 10]
--00, 100
XGBoost
0.859 + 0.001
0.799 = 0.003
0.800 + 0.001
0.802 + 0.003
LightGBM
0.835 + 0.001
0.785 = 0.003
0.784 + 0.003
0.786 +: 0.004
Random Forest
0.774 1 0.003
0.714 : 0.001
0.715 1: 0.002
0.716 1: 0.002
MLP
0.879 + 0.006
0.830 = 0.002
0.833 + 0.002
0.833 + 0.005
STATNN
0.841 + 0.002
0.793 0.003
0.791 + 0.003
0.771 + 0.013
Transformer-No NFN
0.910 1 0.001
0.912 0.001
0.912 1: 0.002
0.913 0.001
Transformer-Ne Quasi
0.914 + 0.001
0.914 = 0.002
0.914
0.914
B.7 ANALYSIS ON THE LEARNED SCALING
Experiment Setup. To analyze the sensitivity of and the learned scaling (sin(>(0)) e + 1,,)
MLP/CNN case, we conduct experiments on the task of Predicting CNN Generalization. We introduce learned scale layers (with corresponding e) on top of Monomial-NEN equivariant layers and explore two cases: one with fixed and one with learnable c. The results are presented in Table 13. The table reports the following values:
Initial e:
The initial value
Learable/Fixed: How changes during training
Final e:
The final value of the first layer after training
Learned scale: The first layer learned scaling (sin(Wscntelt bucate) - + 1n)
Kendall's r:
Evaluation metric (Higher better)
When is fixed at a small value, the learned scaling slightly deviates from the identity, resulting in marginal performance gains. However, when e learnable, the learned scaling can deviate further from the identity, leading to broader scaling range that enhances performance. Additionally, when e is learnable, the final e values tend to converge, regardless of their initial values. For larger values of instability may arise due to significant deviations in the early steps. Consequently, in our study, we choose a learnable e with an initial value of 0.01 to ensure training stability.
Results.
27

---

Published as a conference paper at ICLR 2026
Table 13: Analysis on the learned scaling with varying
Initial
Learnable/Fixed
Final
Learned scale
Kendall's
0 (Baseline)
0.922
0.001
Fixed
0.001
1.000 + 0.001
0.922
0.01
Fixed
0.01
0.999 0.007
0.923
0.1
Fixed
0.1
1.003 = 0.079
0,923
0.2
Fixed
0.2
0.933 = 0.135
0.923
0.4
Fixed
0.4
0.972 = 0.378
0.924
0.001
Learnable
0.966
0.796 = 0.668
0.925
0.01
Learnable
0.970
0.941 0.710
0.926
0.1
Learnable
0.971
0.918 0.758
0.926
0.2
Learnable
0.946
0.743 0.626
0.925
0.4
Learnable
0.988
0.762 = 0.624
0.925
B.8 SENSITIVITY OF
Experiment Setup. We present an ablation study on the sensitivity of e by conducting the "Pre- diet Transformer Generalization" experiment on the MNIST-Transfone analyze two cases: when e is learnable versus fixed, and report the performance alongside statistics of learned noises (#)) 6) for the matrices M and N, which represent the two GT. group ac- tions applied to the weights of the MHA layer. For each noise, we compute the mean and standard deviation the diagonal and off-diagonal elements. The results are summarized Table 14. The table reports the following values:
dataset. Specifically, we
Initial The initial value of for both M and N.
Leamable/Fixed: Whether is changed during training.
Final <M, <x: The final value of after training.
Diagonal M, Mean and standard deviation of the learned noises sin()(0)) c for M and N, computed for diagonal elements.
Off-diagonal M, N: Mean and standard deviation of the learned noises sin(?(()) ) for M and N. computed for off- dfagoia elements.
Kendall's Evaluation metric (higher better).
Table 14: Ablation study on with metrics for M and N.
Initial
Type
Final eM
Final EN
Diagonal M
Off-diagonal M
Diagonal N
Off-diagonal N
Kendall's s
Buseline
0,905
0.001
0.001
0.001
0.000
0.062
0.000 0.000
0.063
0.242
0.01
Fixed
0.01
0.01
0.001
0.002
0.062
0.242
-0.001 + 0.005
0.062
0.242
0.909
0.1
Fixed
0.1
0.1
0.005
0.020
0.062
0.245
-0,088 + 0.065
0.061
0.244
0,907
0.2
Fixed
0.2
0.040 0.010
0.061 0.249
-0.002 + 0.144
0.059
0.285
0,910
0.4
Fixed
0.4
0.021
0.060 0.265
0.055 0.221
0.073 0.323
0,909
0.001
Learnable
0.001
-0.187
0.000
0.000
0.063 0.242
0.068 4: 0.117
0.059 0.260
0,908
0.01
Learnable
0.010
0.182
0.001 0.002
0.062 0.241
0.056 1: 0.084
0.061 0.244
0,911
0.1
Leamahle
0.234
0.2
Learnabee
0.200
0.290
0.010 0.040
0.061 0.249
0.011 0.218
0.062 0.316
0,909
0.4
Learnalee
0.400
0.298
0.021 0.079
0.060 0.265
0.011 1 0.179
0.065 1: 0.298
0,909
Results. When the GL group is learned through our quasi-equivariant layers, the off-diagonal elements provide additional flexibility, leading to noticeable improvement in performance even with a fixed c. With a learnable e, the range of values for N expands and become more stable, resulting in better predictive performance. Overall, this study suggests that increasing the relaxation through e can enhance performance. For our experiments, we select a learnable with an initial value of 0.01 ensure the training stability.
28

---

Published as a conference paper at ICLR 2026
B.9 WEIGHT SPACE STYLE EDITING
Experiment Setup. In this experiment, we focus on modifying the image content encoded in each SIREN model (Sitzmann al., 2020) by adjusting its weights. Specifically, we leverage pretrained models from (Zhou et al., 2024a), which encode images from the CIFAR-10 and MNIST datasets. The experimental setup follows the approach described in (Zhou et al.. 2024a: Tran al., 2024a). Our main goal is to explore two tasks that involve altering the embedded information within the SIREN model: first, enhancing the contrast of CIFAR-I0 images, and second, applying dilation to MNIST images.
To evaluate the effectiveness of these modifications, we compute the mean squared error (MSE) loss between the images encoded by the modified SIREN network and the corresponding ground truth images, which have undergone contrast enhancement CIFAR- and dilation for MNIST. The baseline for comparison is the Monomial highly optimized version. this case, further increasing the parameters does not lead to a noticeable improvement in performance, serving useful benchmark for our experiments. The results from these tasks are summarized Table 15.
model (Tran et al., 2024a), which is already
Table 15: Weight space style editing (INRs editing)
Model
Contrast (CIFAR-10)
Dilate (MNIST)
MLP
0.031 0.001
0.306 0.001
HNP (Zhou et al., 2024a)
0.021 0.001
0.071 1 0.001
NP (Zhou et al., 2024a)
0.020 1 0.002
0.068 1 0.002
Monomial-NEN (Tran al., 2024a)
0.020 0.001
0.069
0.002
Monomial-NEy Quasi (1 params ++)
0.019 0.001
0.066 0.001
Results. In this task, Monomial-NFN model initially underperforms compared to the NP model. However, by incorporating newly introduced Quasi layer, which adds only 1% more param- eters, we are able to significantly improve the performance of the Monomial-NFN model. This enhancement allows surpass the performance of the NP baseline in both tasks, demonstrating the effectiveness of our approach with minimal increase in model complexity.
29


Reviews:
[{"review_id": "tRQ79Fex80", "reviewer": "Reviewer_ya8b", "summary": "This work proposes a novel theoretical framework for relaxing the rigidity of strict equivariant meta-networks while still preserving functional identity. The authors formalize this relaxed notion of equivariance as quasi-equivariance and propose a general construction where a learned group action $\\alpha(\\theta)$ modulates an already equivariant backbone. After describing how they can implement such a network in the case of MLPs, CNNs, and Transformers, they apply it in generalization prediction tasks and the INR image classification task. Across all architectures, the proposed relaxation of the strict equivariance seems to provide small improvements in the overall performance without a significant increase in the number of parameters.", "questions": "- If the loss used to train the quasi-equivariant networks is invariant to the transformation introduced by $\\alpha(\\theta)$, since both the original and transformed parameters map to the same function, does this imply that the loss gradient with respect to $\\alpha(\\theta)$ is zero? If so, does this phenomenon occur in the current experimental setup?\n- How do other approximate or relaxed equivariant methods compare to the proposed quasi-equivariant networks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The framework of quasi-equivariant networks and their connection to the maximal symmetry group and the preservation of the functional identity are presented with significant mathematical rigor. As such, it provides a solid theoretical foundation and a promising starting point for the community to further investigate the proposed framework and its properties.\n- The proposed framework and the decomposition of quasi-equivariant networks into $F(\\theta)=\\alpha(\\theta)\\beta(\\theta)$ is quite general, allowing the approach to be implemented for a variety of tasks and network architectures. The authors partially demonstrate this adaptability by applying it to the most commonly used architectures.\n- The experimental evaluation shows that the proposed network provides consistent improvement over the baseline equivariant network without a significant increase in the computational complexity of the method.", "weaknesses": "- The authors claim that introducing the $\\alpha(\\theta)$ improves the expressivity of the networks, but they do not provide sufficient  evidence to support such a claim. Specifically, if the task or loss we are interested in is invariant to the group transformation introduced by $\\alpha$, does the preservation of the functional identity imply that the proposed quasi-equivariant architecture will have the same expressivity as the original one?\n- The experimental section lacks ablation studies comparing the proposed approach with other methods that also relax exact equivariance. Thus, it is difficult to conclude whether the improvements in performance come from the specific framework of the quasi-equivariant network or are more generally caused by the relaxation of the equivariant constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IL7ODuJH3q", "reviewer": "Reviewer_7DXF", "summary": "The paper concerns the efficient use of equivariance in the design of metanetworks, which are architectures intended to process the weights of another, predefined model. As different parameter values may identify the same functional mapping (the input-output relation of the model), it is a crucial advantage to identify and process functionally-equivalent sets of parameters.\nThis work in particular argues that exact equivariance is a too strict assumption to design effective metanetworks, while the weaker notion of quasi-equivariance is sufficient. It then introduces architectures that are able to implement this quasi-equivariance for feed-forward NNs and MHA, and demonstrates convincing results on a number of tasks.", "questions": "Apart from the points discussed above, there are the following minor points: \n- Section 2.1: $f(\\cdot; \\theta)$ should be defined with a domain $D$, so that (1) has a precise meaning. Moreover, is $f$ at least continuous?\n\n- Section 2.2: \"...to produce incompatible outcomes\". What does incompatible mean? So fa we are requiring $F(\\theta)=F(\\bar\\theta)$ if the two parameters are in the same orbit (depend only on the underlying function represented by $\\theta$).\n\n- \"Moreover, $G$-quasi-equivariance ensures functionality preservation\": A one-line equation explaining this key point would be helpful.\n\n- Section 4.1: The notion of continuity of a group-valued function should not be assumed to be common knowledge. Moreover, in Remark 4.1 the set $\\Theta$ has connected components. But so far we always had $\\Theta=R^{d}$. Is this no more the case? Moreover, for $\\Theta=R^d$ this seems to imply that $\\alpha$ is a constant function (see below for the matrix permutation group).\n\n- Theorem 4.3: How reasonable are conditions 1/2 in practical, learned MHAs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The topic is of current interest, and the paper does a convincing job in guiding the reader through computational and theoretical motivations for the work.\n- The discussion around quasi-equivariance is interesting and convincing, and the notion is a novel contribution that has potentially wide application in the design of meta-networks. \n- The discussion of the limitations (current limited applicability beyond linear architecture) is convincing.", "weaknesses": "**Definition and use of notion of quasi-maximality:**\n- Definition 2.2: The sentence \"Under generic parameters\" is unclear. The entire informal definition seems to serve little purpose, as the formal one is given 10 lines after. I suggest Definition 2.2 to be simply stated in the text, without a \"definition\" environment.\nMoreover, 1) the reason to consider epsilon to be a real algebraic variety is unclear here (instead of another set with other structure), and should be at least motivated if not clearly defined; 2) The notion of maximality seems to depend on $\\varepsilon$, but it would be more clear if first it is stated without it (e.g., defining maximality and then $\\varepsilon$-maximality).\n- Section 2.2: \"If G is a maximal symmetry group of the underlying model, such equivariance is sufficient to guarantee that F operates solely on the functional content of θ.\". This again is not explicitly demostrated or discussed, and it is especially unclear why considering an algebraic variety is a sensible choice. This is partially motivated by the important example of MHA, which however comes only in Remark 4.4, several pages later.\n\n**Well posedness of the notion of quasi-equivariance:**\n- It is mentioned that \"Further discussion and insights on this perspective are provided in Appendix A.1\". \nSince the conclusions of A.1 are quite involved, a short summary of these insights should be given in the main test. Positive and negative examples would be particularly helpful. \n- Explanations would be appreciated also considering that this may guide practitioners in choosing or not this solution depending on the group of interest. Related to this, G is f-dependent. How does this relate to the results of Section A.1?\n- The entire Section A.1 relies on heavy terminology and notions that are not tied to the rest of the paper and are not defined elsewhere. There should be either a concise introduction of the main tools, or at least references to the literature.\n\n**Design of the quasi-equivariant map:**\n\nThe construction of the map $\\alpha$, especially for MLPs (and specifically the part mapping to the diagonal matrices) is quite involved, and it is completely unclear why such an elaborate construction is required. Appendix B.1 does not help in understanding this. There should be some effort put into explaining why this construction is required in this crucial step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVQrwOyiod", "reviewer": "Reviewer_9pd6", "summary": "This work posits the question whether strict equivariance is necessary for metanetworks, and introduces the notion of quasi-equivariance, allowing metanetworks to move beyond strict equivariance. The authors demonstrate the effectiveness of their method in a suite of experiments that includes INR classification, and predicting CNN/Transformer generalization.", "questions": "1. The experiments are only focusing on invariant tasks. Why don't the authors perform experiments on equivariant tasks (e.g. INR editing)?\n\n2. The performance reported for baselines is often different from the original works (e.g. the works of Zhou et al., Kofinas et al.). In the case of INR classification, specifically, the performance gap is quite large. Why is that?\n\n3. The performance gains of the proposed method compared to the baselines are often marginal. Are there any scenarios/experiments where the proposed method can be expected to boost the baseline greatly? If so, can that be shown through an experiment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The manuscript is well-written, and the math formalism is mostly clear.\n\nThe research question posed, i.e. whether strict equivariance is necessary for metanetworks is a very topical and significant question.\n\nThe proposed method is simple and has a small computational overhead.", "weaknesses": "The method details (namely the design of the quasi-equivariant layer) are not described in detail and are, thus, unclear.\n\nA large part of the manuscript is used to re-iterate existing group theory and the math behind parameter space, which obfuscates the novelty of the proposed work.\n\nThe datasets used are somewhat saturated, and the performance gains are often marginal.\n\nThere are no ablation studies to examine the significance of various hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NkvYhbXTX", "reviewer": "Reviewer_J9ur", "summary": "This paper introduces the concept of quasi-equivariance specifically for metanetworks, a novel framework designed to overcome the limitations of strict equivariance in metanetworks. Metanetworks are models that process the parameters of other neural networks, and a key challenge is functional equivalence—where different parameter sets can yield the same input-output function. While strictly equivariant metanetworks respect the symmetries underlying functional equivalence (e.g., neuron permutations), they can be overly restrictive and lack expressivity.\n\nThis work proposes quasi-equivariance as a principled relaxation. A map $F$ is quasi-equivariant with respect to a symmetry group $G$ if for any transformation $g \\in G$ applied to its input $\\theta$, its output is transformed by some other group element $g' \\in G$, i.e., $F(g\\theta) = g'F(\\theta)$. This ensures the output remains in the same functional equivalence class, thus preserving functional identity, while allowing for greater flexibility than the strict condition $F(g\\theta) = gF(\\theta)$.\n\nThe paper provides a theoretical foundation for this concept, connecting it to the notion of a maximal symmetry group, and proposes a constructive method for building quasi-equivariant layers. This is achieved by augmenting a standard equivariant map $\\beta(\\theta)$ with a learned, parameter-dependent group action $\\alpha(\\theta)$, yielding a layer of the form $F(\\theta) = \\alpha(\\theta)\\beta(\\theta)$. The framework's effectiveness is demonstrated by applying it to feedforward, convolutional, and transformer networks. Empirical results across three distinct metanetwork tasks—predicting model generalization, classifying implicit neural representations, and predicting transformer performance—show that quasi-equivariant layers significantly improve the performance of state-of-the-art metanetworks with only a minimal increase in parameter count.", "questions": "The weaknesses above are important to address in the main paper, even given the space constraints. The following questions can be addressed in an appendix and referred to in the main paper. \n\nHave you performed any analysis of the learned group elements from the map $\\alpha(\\theta)$ in the experiments? For instance, how far from the identity do the learned scaling factors or matrices tend to be, especially in the case of the experiments presented in section 5.1?\n\nThe construction of the map to $GL(n)$ via $\\sin(\\gamma(\\theta)) \\cdot \\epsilon+ I_n$ is an interesting and practical choice. How sensitive is the model's performance to the hyperparameter $\\epsilon$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: This paper introduces the novel concept of \"quasi-equivariance\" for metanetworks, which is a significant and original extension to the existing paradigm of strict equivariance. This relaxation addresses a critical limitation of previous approaches, offering a new direction for designing more flexible and expressive models.\n\nQuality: This paper provides a strong theoretical foundation for quasi-equivariance, connecting it to functional equivalence and maximal symmetry groups. The empirical evaluations are thorough, comparing the proposed method against several established baselines across diverse tasks, datasets, and architectures that the metanetwork processes, consistently demonstrating its effectiveness.\n\nClarity: This paper is overall well-written and well-structured, with clear explanations of complex concepts such as functional equivalence, symmetry groups, and the mathematical definitions of equivariance, supported by figures and tables of results.\n\nSignificance: The introduction of quasi-equivariance has significant implications for the design of metanetworks, enabling better trade-offs between symmetry preservation and representational flexibility. The empirical results demonstrate substantial performance improvements with minimal additional cost.", "weaknesses": "Scope Limitations: \n* Your proposed methods framework are only demonstrated on MLP metanetworks that operate on the parameters of MLPs, CNNs, and Transformers, whose symmetry groups have a relatively simple direct-product structure. You briefly mention that extending this framework to more complex architectures like graph-based metanetworks are only briefly mentioned, but expanding upon what the challenges are and how they might be tackled in a few sentences could provide better context for the audience.\n* You provide clear evidence that the proposed quasi-equivariant framework is useful for metanetworks, but are there other fields of machine learning that could benefit from it? Adding a sentence with some ideas would help broaden the audience of this paper.\n\nThe implementation for MLPs and CNNs in Section 4.2 is currently only provided for MLPs, but the experiments focus on CNNs. Can you add a few sentences on how the provided MLP implementation translates to the CNN case?\n\nAlthough $\\text{GL}(d_h)$ is defined in the notation table, it would be worth also defining around line 344.\n\nThe text is table 3 is quite small and difficult to read. Please make it at least the same size as the text in the other tables.\n\nFormatting: On line 066, use \\citet instead of \\citep to get \"...class of MLPs, Fefferman & Markel (1993) established...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)