Scale Equivariant Graph Metanetworks
Authors: Ioannis Kalogeropoulos
Giorgos Bouritsas
Yannis Panagakis
Venue: NeurIPS 2024 oral (2024)
Keywords: graph neural networks
weight space networks
implicit neural representations
symmetries
Primary area: graph_neural_networks
TL;DR: We introduce a graph metanetwork framework that allows scaling and permutation equivariant  neural network processing.

Abstract: This paper pertains to an emerging machine learning paradigm: learning higher- order functions, i.e. functions whose inputs are functions themselves, particularly when these inputs are Neural Networks (NNs). With the growing interest in architectures that process NNs, a recurring design principle has permeated the field: adhering to the permutation symmetries arising from the connectionist structure of
NNs. However, are these the sole symmetries present in NN parameterizations? Zooming into most practical activation functions (e.g. sine, ReLU, tanh) answers this question negatively and gives rise to intriguing new symmetries, which we collectively refer to as scaling symmetries, that is, non-zero scalar multiplications and divisions of weights and biases. In this work, we propose Scale Equivariant Graph MetaNetworks - ScaleGMNs, a framework that adapts the Graph Metanetwork (message-passing) paradigm by incorporating scaling symmetries and thus rendering neuron and edge representations equivariant to valid scalings. We introduce novel building blocks, of independent technical interest, that allow for equivariance or invariance with respect to individual scalar multipliers or their product and use them in all components of ScaleGMN. Furthermore, we prove that, under certain expressivity conditions, ScaleGMN can simulate the forward and backward pass of any input feedforward neural network. Experimental results demonstrate that our method advances the state-of-the-art performance for several datasets and activation functions, highlighting the power of scaling symmetries as an inductive bias for NN processing. The source code is publicly available at https://github.com/jkalogero/scalegmn.

Scale Equivariant Graph Metanetworks
Toannis Kalogeropoulos!.2
Giorgos Bouritsas 1.2+
Yannis Panagakis 1.2
National and Kapodistrian University of Athens
"Archimedes/Athena RC, Greece
Abstract
This paper pertains to an emerging machine learning paradigm: learning higher- order functions, i.e. functions whose inputs are functions themselves, purticularly when these inputs are Neural Networks (NNs). With the growing interest ar- chitectures that process NNs, recurring design principle has permeated the field: adhering to the permutation symmetries arising from the connectionist structure these the sole symmetries present in NN parameterizations?
NNs. However, Zooming into most practical activation functions (e.g. sine, ReLU, tanh) answers this question negatively and gives rise intriguing collectively refer to as scaling symmetries, that is, non-zero scalar multiplications and divisions of weights and biases. In this work, we propose Scale Equivariant Graph MetaNetworks ScaleGMNs, a framework that adapts the Graph Metanet- work imessage-passingg paradigm by incorporating scaling symmetries and thus rendering neuron and edge representations equivariant to valid scalings. We in- troduce novel building blocks, of independent technical interest, that allow for equivariance or invariance with respect to individual scalar multipliers or their product and use them in all components of ScaleGMN. Furthermore, we prove that, under certain expressivity conditions, ScaleGMN can simulate the forward and backward pass of any input feedforward neural network. Experimental results demonstrate that our method advances the state-of-the-art performance for several datasets and activation functions, highlighting the power of scaling symmetries an inductive bias for NN processing. The source code is publicly available at http://gahhb.coom//jooonn/cgggg
symmetries, which we
Introduction
Neural networks are becoming the workhorse of problem-solving across various domains. To solve task, they acquire rich information which is stored in their Icarnable parameters throughout training. Nonetheless, this information is often opaque and difficult to interpret. This begs the question: How can we efficiently process and extract insights from the information stored in the parameters of trained neural networks, and how to do in a data-driven manner? In other words, can devise architectures that learn to process other neural architectures?
The need to address this question arises in diverse scenarios: NN post-pree post-processing. e.g. analy- sisfinterpretation (i.e. inferring NN properties, such as generalisation and robustness [74 19)), as well editing (e.g. model pruning 22 merging 775 adaptation new data)
NN synthesis (e.g. for
optimisation [12 or more generally parameter prediction/geneen /gnenratinnn 31
67 32
58) ). Furthermore,
with the advent of Implicit Neural Representations] [13 52 57 70
533 trained NN parameters
are increasingly used to represent datapoint signals, such as images or 3D shapes, replacing raw representations, i.e., pixel grids or point clouds (18). Consequently, many tasks involving such data,
"Equal contribution. Corrresooneecee ioakalogero@dii.ot di.uo..gr Sauuritas@aathnarrc
INR: A coordinate-proceesing NN "overfitted" given signal, e.g. image, 3D shape, physical quantity.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

across various domains such as computer vision 47 and physies 68 which are currently tackled using fomainn-specine architectures (e.g. CNNs for grids, PointNets [3] or GNNs [60] for point clouds and meshes), could potentially be solved by NNs that process the parameters of other NNs.
The idea of processing/prediccing NN parameters per se is not new the deep learning community, c.g. it has been investigated in hypernetworks [27 or even earlier (65]. However recent advancements, driven by the need for INR processing, leverage a crucial observation: NNs have symmetries.. In particular, several works have identified that the function represented by an NN remains intact when applying certain transformations its parameters [28 with the most well-known transformations being permutations. That is, hidden neurons can be arbitrarily permuted within the same layer, along with their incoming and outgoing weights. Therefore, all the tasks mentioned in the previous paragraph can be considered part of equivariant machine learning [TO], as they involve developing models that are invariant or equivariant to the aforementioned transformations of neural networks. Recently Navon et al. [54] and Zhou et al. [86] acknowledged the importance of permutation symmetries and first proposed equivariant architectures for Feedforward NN (FFNN) processing. demonstrating significant performance improvements against non-equivariant baselines. These works, as well as their improvements [85] and extensions to process more intricate and varying architectures 33 88 are collectively known weight space networks metanefworks, with graph metanetworks being a notable subcase (graph neural networks with message- sesssageppssing).
Nevertheless, it is known that permutations are not the only applicable symmetries. In particular, theoretical results in various works 6T mainly unified in (5 show that FFNNs exhibit additional symmetries, which we collectively here refer to scaling symmetriex: multiplying the incoming weights and the bias of a hidden neuron with non-zero scalar a (with certain properties) while dividing its outgoing weights with another scalar b (often b a), preserves the NN function. Intuitively, permutation symmetries arise from the graph/connectionis! structure of the NN whereas different scaling symmetries originate from its activation functions i.e. when it holds that a (az)) (z).. However, equivariance w.r.f scaling symmetries has not received much attention so far (perhaps with the exception of sign symmetries a E [1,, 4 40 ) and it remains effectively unexplored in the context of metanetworks.
To address this gap, we introduce in this paper graph metanetwork framework, dubbed as Scale Equivariant Graph MetaNetworks- ScaleGMN, which guarantees equivariance to permutations and desired scaling symmetries, and can process FFNNs of arbitrary graph structure with variety of activation functions. At the heart of our method lie novel building blocks with scale invari- ance/equivariance properties w.r.t. arbitrary families of scaling parameters. We prove that ScaleGMN can simulate the forward and backward pass of an FFNN for arbitrary inputs, enabling it to reconstruct the function represented by any input FFNN and its gradients.
Our contributions can be summarised as follows:
We extend the scope of metanetwork design from permutation to scaling symmetries.
We design invarian/equivaaiint networks scalar multiplication of individual multipliers or combinations thereof, originating from arbitrary scaling groups
We propose scale equivariant message passing, using the above as building blocks, unifying permutation and scale equivariant processing of FFNNs. Additionally, the expressive power of our method is analysed w.r.t. its ability to simulate input FFNNS.
Our method is evaluated on 3 activations: ReLU (positive scale), tanh (sign) and sine (sign we first characterise this using technique from (25) on several datasets and three tasks (INR classification/editnng & generalisation prediction), demonstrating superior performance against common metanetwork baselines.
2
Related work
Neural network symmetries. Long preceding the advent of metanetworks, numerous studies delved into the inherent symmetries of neural networks. Hecht-Nielsen [28] studied FFNNs and discovered the existence of permutation symmetries. Chen et al. [T] showed that, in FFNNS with tanh activations, the only function-preserving smooth transformations are permutations and sign flips (multiplication of incoming and outgoing weights with a sign value)- this claim was strengthened in (11; this was the first identification of scaling symmetries. Follow-up works extended these observations to

---

other activations, such as sigmoid and RBF (37 and ReLU [56 51 61 64 26 and architectures, such as RNNs [3, characterising other scaling symmetries and providing conditions under which permutations and scalings are the only available
preserving symmetries or the parameters are identifiable given the input-output mapping. Recently, Godfrey et al. [25] provided a technique to characterise such symmetries for arbitrary activations that respect certain conditions, unifying many previous results. Additionally, symmetries have been found in other layers, e.g. batch norm [6 16) (scale invariance) or softmax [36] (translation invariance). Prior to metanetworks, symmetries were mainly studied to obtain a deeper understanding of NNs or in the context of optimisation//eamingg
dynamics [56]
36 5 83 [4] and/or model merging/ensembling (20 69 59 55
Metanetworks. The first solutions proposed for NN processing and learning did not account for NN symmetries. Unterthiner et al. [74] and Eilertsen et al. [1)]. initially employed standard NNs on vectorised (flattened) CNN weights or some statisties thereof, to predict properties of trained neural networks such as generalization or estimation of a hyperparameter resp.). Similar methods have been proposed to process continuous data represented as INRs. In 78 high-order spatial derivatives are used (suitable only for INRs), in [47 the architecture operates stacked parameter vectors (but is constrained by the assumption that all INRs are trained from the same initialization), while in [[8] and 77 the authors learn low-dimensional INR embeddings (which are further used for downstream tasks) by jointly fitting them with meta-learning. Finally, Schürholt et al. [66] learn representations of NN weights with supervuue learning and in [67] this is extended to generative models.
In contrast to the above methods, our work follows a recent stream of research focusing on equivariant metanetworks. Navon et al. (54) and Zhou et al. [5 first characterised all linear equivariant layers to permutation symmetries of MLPs and combined them with non-lineasisies, while in the latter this was extended to CNNs. These approaches derive intricate weight-sharing patterns but are non- locaf and cannot process varying architectures. In follow-up works, Zhou et al. [86] constructed [8 the above characterisation was generalised to arbitrary input architectures and layers (e.g. RNNs and transformers) introducing an algorithm for automatic linear layer (with weight-sharing construction. A different route was taken in the very recent works of Kofinas et al. [33] and Lim et al. [4], where input NNs are treated as a special class of graphs and naturally processed with Graph Neural Networks (GNNs), with appropriate symmetry breaking wherever necessary. This perspective has been adopted several times in the deep learning literature and in few examples, GNNs were applied on the graph representation for, e.g. neural architecture search (82 31), albeit without mapping parameters to the graph edges. In parallel our work, (73 extended the framework of [5 to incorporate scaling and sign-flipping symmetries. They construct non-local equivariant layers, resulting in an architecture with fewer trainable parameters. Nevertheless, their method suffers from limited expressive power, while their experimental results showcase limited advancements. In contrast to our method, this approach limits the choice of activation functions to those equivariant to the studied symmetries. Finally, their framework cannot be extended to other activation functions, as the parameter sharing must be re-designed from scratch, while it is not suitable for processing diverse architectures.
equivariant attention layers and in
9
23
45
77
[0)
Scale Equivariance Equivariance to vector scaling (or more generally to matrix multiplication with diagonal matrices) remains to date underexplored in the machine learning community. Sign symmetries received attention in the work of Lim et al. [42), where an invariant network (SignNet) was designed, mainly to process eigenvectors, with the theoretical analysis revealing universality under certain conditions. This was extended in (39], where the importance of sign equivariance on several tasks was demonstrated and sign equivariant network was proposed. In our work, draw inspiration from these architectures and extend their formulations to arbitrary scaling symmetries.
3 Notation and Preliminaries
Notation. We denote vectors, matrices, and tensors with bold-face letters, e.g., x, X, X, respectively and sets with calligraphic letters X'. A normal font notation will be used for miscellaneous purposes (mostly indices and functions). Datapoint (input) functions/signals will be denoted with u, while higher-order functions (functions of functions) will be denoted with fraktur font 8.
Functions of Neural Networks. Consider functions of the form uG X + &. Each function is parameterised (1) by computational graph G, which determines all the mathematical operations
They can be perceived the metanetwork analogue of Invariant Graph Nets [50 for conventional graphs
3

---

that should be performed to evaluate the function u at a datapoint x É X. When ur is neural network, G is determined by the NN architecture. Additionally, u is parameterised by (2), by tuple of numerical parameters 9, on which the aforementioned mathematical operations are applied (along with the input x) these are the learnable parameters, which can be arranged into a vector. We are interested in learning unknown higher-order functions $ : X + y of the form 3 (uG,a). In our case, the goal is to learn 3 by accessing solely the parameters (G, 0) of each u, i.e. via functions f: G x O + y of the form f(G, 0), where & is space of auchiteturre//oopptatinnn graphs and 2 space of parameters We typically interested in learning functionals () C Rd) or operators
O' = g x 2 or y = )) To approximate the desired higher-orde order function, we assume access to dataset of parameters sampled i.i.d. from an unknown distribution p on G x O. For example, in supervised setup, we aim to optimise the following: argmin -) ((ua.o).f(GG f(G, 8)), where ((:,)) is a loss function and F is an NN processing hypothesis class (e.g. metanetworks).
Feedforward Neural Networks (FFNNs), In this paper, we focus our analysis on Feedforward Neural Networks (FFNNs), i.e. linear layers interleaved with non-linearities. Consider NNs of the form UG & + Rd of the following form:
X = x, X de
+ be), uc,o(x) = _X
where L: the number of layers, W E Rdexde 1; the weights of the NN, b, E Ie the biases of the NN, dg = dm- dp = doat. Oe E IR activation functions applied element- -wise. Here, the learnable parameters are 0 = (WI...., 1...... WL, bL........b. bL) and the computational graph encodes the connections between vertices, but also the rype of activations used in each layer.
Neural Network symmetries. One of the major difficulties with working with function parameters directly is that the same function can be represented with more than one parameter, i.e. there exists transformations that when applied to any parameter (G, 0), keep the represented function intact. Formally, an NN symmetry is induced by a Vi of transformations 1b 9 x O + O such that (x) = ( (x), Vx E X, V(G 0) E G x O. If for a pair parameters 0), (G',0') Feb such that (G, 0) = (b''', 0¹), we will call the two parameters equivalent and write (G,0) ~ (G', . To appropriately represent a functional 3, hypothesis (metanetwork) f should be invariant to transformations in v: f (w (G, 0)) = f(G, 0). For operators, f should be equivariant to transformations: f (v (G, 0)) = 3 (f, 0)), such that identical functions map to identical functions.
Permutation symmetries (connectionist structure). For fixed computational graph G, perhaps the most well-known symmetry of FFNNs are those induced by hidden neuron permutations (28). As far metane/works are concerned it is to date the only NN symmetry that has been accounted for see Section 2 This symmetry implies that permuting hidden neurons (along with their biases and incoming and outgoing weights) within each layer preserves the NN function (regardless of the activation function). This reads:
W/ P,W,P -1 b/2 = Pyb; - (We, b - = 0 (We, br)
where E E (L....,, Po P and P.- dd,, that they are different for each layer, with the input and output neurons held fixed.
xd, are arbitrary permutation matrices. Observe
Graph Metanetworks (GMNs). A recently introduced weight space architecture advantage of the permutation symmetries and treats FFNNS (among others, e.g. CNNs) as graphs, processing them with conventional GNNs. In particular, let G = (V,E) be the computational graph, i E V an arbitrary vertex in the graph (neuron) and (1.j) E an arbitrary edge from vertex j to vertex zf Additionally, let Xy E R x be the vertex features and X E [ the edge features (i.e. biases and weights resp. in FFNN). The general form of a T iteration (layer) GMN reads: (Init)
B3 44 takes
hi- (1) = V (xv (i)), [1, = INIT (XE (i..))
mf-(i) = + JEN( MSG (h+ (1), 10) (
(0)
(Msg)
h (i) UPD (1) (i), hl (i.j) UPD (i), (t), (()
(Upd)
hg = READ (nT(())
(Readout)
e Note the difference with architectures that access u via input-output pairs (Xi, uG,o (xx))
We use this convention to align with the indexing of the weights W (i,

---

where hú, h'g are vertex and edge representations at iteration t and b is the overall graph (NN) representation. INIT, MSG, UPD are general function upproximators (e.g. MLPs), while READ is permutation invariant aggregator (e.g. DeepSets [81). The above equations have appeared with several variations in the literature, e.g. in some cases the edge representations are not updated or the readout might involve edge representations as well. Another frequent strategy is to use positional encodings pv, P to break undesired symmetries. In FFNNs, Eq. (2) reveals that input and output vertices are not permutable, while vertices cannot be permuted across layers. Therefore, vertices (or edges) that are permutable share the same positional encoding (see Appendix [A. for more details). Remark: Although, typically, the neighbourhood (t)) contains both incoming and outgoing edges, in Section 5 we will illustrate our method using only incoming edges: forward neighbourhood New (i) D e V layer (t) layer (j) 1} and backward where layer (i) gives the layer neuron i belongs. Backward neighhourhoods (i) are defined defined similarly. In Appendix A.2 we show more elaborate bidirectional version of our method, with both neighbourhoods considered.
4 Scaling symmetries in Feedforward Neural Networks
Scaling symmetries (activation functions). Intuitively, permutation symmetries stem from the graph structure of neural networks, or put differently, from the fact that hidden neurons do not possess any inherent ordering. Apart from the affine layers Wy that give rise to the graph structure, it is frequently the case that activation functions de have inherent symmetries that are bestowed to the NN.
Let us dive into certain illustrative examples: for the ReLU activation (x) = max(2;, 0) it holds that a (az)) = max(ax (x., 0) = a max(x, 0), Va >> 0. For the tanh and sine activations (xx = tanh(z), (æ) = sin(x) respectively, holds that (ax) (x)), Va E (-1, In a slightly more complex example, polynomial activations (xx) ott we have a (axx) a (x). i.e. the multiplier differs between input and output. In general, we will be talking about scaling symmetries whenever there exist pairs (a, b) for which it holds that a ((a)) = (zr(z). To see how such properties affect NN symmetries, let us focus on FFNNs (see Appendix AA.3 for CNNs): for neuron i (we omit layer bias and
subseripts) we have a (aW(i,)) + ab(i)) = bo (Wi,,:)x++ ((():. i.e. multiplying all incoming weights with constant results in scaling its output with a corresponding constant b. Generalising this to linear transformations, we may ask the following: which are the pairs of matrices (A, B) for which we have a (AWx- Ab) = Bø [Wx + b)? Godfrey et al. [25] provide an answer for any activation that respects certain conditions. We restate here their most important results:
Proposition 4.1 (Lemma 3.1. and Theorem 1.14 from 25 Consider an activation function R-- R. Under mild conditions the following hold:
For any d E N+, there exists a (non-empty) group group of invertible matrices defined as: Ta,d = (A E Raxd : invertible 3 E Rdad invertible, such that: a ((Ax) Bx(x))
(intertwiner group), and mapping function Φo, such that B = (a,)(AA). (A).
* Every A E Lo,d is of the form PQ. where P: permutation matrix and Q = diag (gt,.... qu) diagonal, with 9; E D. = fa E R (0} a(az) = (z)) : the 1-dimensional group. and (o,((A) = Pdiag (9)) ) )o.1
This is a powerful result that completely answers the question above for most practical activation functions. Importantly, not only does it recover permutation symmetries, but also reveals symmetries to diagonal matrix groups, which can be identified by solely examining Ør,1, i.e. the dimeennnsi case and the set D2 (easily proved to be group) we have already discussed in our examples above.
Using this statement, Godfrey et al. (25 characterised various activation functions (or recovered existing results), e.g. ReLU: Lo,a contains generalised permutation matrices with positive entries of the form PQ. Q diag(g) .o.............................. ... ga). q > 0 and (PQ) = PQ (56). Additionally, here we characterise the intertwiner group of sine (used in the popular SIREN architecture [70] for INRs). Not surprisingly, it has the same intertwiner group with tanh I 21 (we also recover this here using Proposition 1.I.. Formally, (proof in Appendix A.7.II.
Corollary 4.2. Hyperbolic tangent a (Ix) tanh(r) and sine activation o(z) sin(wa)) satisfy the conditions of Proposition 4.1 when (for the latter) w + kT, k E L. Additionally, Lo,d contains signed permutation matrices of the form PQ, with Q diag(qi.... ...g(g.....9..... .a)), 9: = +1 and do, (PQ) PQ
SSee Appendix A.7.1 A.7.I for the precise statement and more details about do.d.
5

---

It is straightforward to see that the symmetries of Proposition 4.1 induce equivalent parameterisations
for FNNNs. In particular, it follows directly from Proposition 3.4. in [5], that for activation functions
og satisfying the conditions of Proposition 4.1 and when do,e (Q) Q we have that:
w - P.Q. Q P b/- = P.Q.b = (We. b 0' = 0 (We, bE)
6-11
(3)
where again E L). Po = Qo = P = QQ==1
Scale Equivariant Graph MetaNetworks
As previously mentioned, the metanetworks operating weight spaces that have been proposed so far, either do not take any symmetries into account or are invariant/cuivaaiant to permutations alone as dictated by Eq. (2). In the following section, introduce architecture invariant/equivarian to permutations and scalings, adhering to Eq. 3 An important motivation for this is that in various setups, these are the only fuunction-preservinn symmetries, i.e. for fixed graph uG.0 uG,00 7
(0, 0') satisfy Eq. (3) e.g. see (21) for the conditions for tanh and 66 26 for ReLU.
Main idea. Our framework is similar in spirit to most works on equivariant and invariant NNs [00 In particular, we build equivariant GMNs that will preserve both symmetries at vertex- and edge-level. i.e. veriex representations will have the same symmetries with the biases and edge represenlations with the weights. To see this, suppose two parameter vectors are equivalent according to Eq. (3). Then, the hidden neurons representations- the discussion on input/output neurons is postponed until Appendix(A.I.A should respect the following (the GMN iteration t is omitted to simplify notation):
L (i) qe (Iee (1)) by (Te (i)), t= layer (i) E L 13
h (i, qe (Te (1)) hg (Te ( (i)) (Te-1 ()), l layer (i) E (,,...., L 1}, (5)
where ne : Ve Ve permutes the vertices of layer (denoted with V\)) according to P, and qe: : V/- + R {0} scales the vertex representations of layer according to Qe- We will refer to the latter as forward scaling in Eq. (4) and bidirectonal scaling Eq. (5). To approximate operators (equivariance), we compose multiple equivariant GMN layers/iterations and in the end, project vertex/edge representations to the original NN weight space, while to approximate functionals (invariance), we compose a final invariant one in the end summarising the input to scalar/vector.
To ease exposition, we will first discuss our approach w.r.t. vertex representations. Assume that vertex representation symmetries are preserved by the initialisation of the MPNN Eq. ([mil) and so are edge representation symmetries for all MPNN layers. Therefore, we can only focus on the message passing and vertex update steps Eq. (Msg) and Eq. (Upd). Additionally, let us first focus on hidden neurons and assume only forward neighbourhoods. The following challenges arise:
Challenge 1- Scale Invariance / Equivariance. First off, the message and the update function MSGy, UPDy should be equivariani to scaling in this case the forward scaling using the multiplier of the central vertex qe()). Additionally, the readout READ, apart from being permutation invariant should also be invariant to the different scalar multipliers of each vertex. . Dealing with this requires devising functions of the following form:
Fi . X,) 9:99 (x1, x,.. Vgg D fl....,n.
(6)
where D; a 1-dimensional scaling group as defined in Proposition 4.1 Common examples are those discussed in Section e.g. D; {1. -1. } or D: = R+ The first case, i.e. sign symmetries, has been
discussed in recent work 43 40 Here we generalise their architecture into arbitrary scaling groups
In specific, Scale Equivariant networks follow the methodology of 44), i.e. they are compositions of multiple linear transformations multiplied elementwise with the output of Scale Invariant functions:
Scalelnv (X) = p² (K)..... ,,..,,,,),,,
(Scale Inv. Net)
ScaleEq = PK ....o
o ft, (x( = (Tfx)..... ",,,..,,,,,,,,, @ Scalelnv (X),
(Scale Equiv. Net)
where p² I[ X, RE:. dt universal approximators (e.g. MLPs), I X R linear transforms (for each of the ls invariant/equivariant layers resp in practice, we observed experimentally that single layer K 1 was sufficient), and &; are explained in detail in the following paragraph.
Central to our method is defining a way to achieve invariance. One option is canonicalisation, i.e. by defining function canon X' + X' that maps all equivalent vectors x to representative (obviously

---

non-equivalent vectors have different representatives); Iff x = E X, then x = cx) ca)on(y). In certain cases, these are easy to define and differentiable almost everywhere, for example for positive scaling: canon(x) = Txi T For sign symmetries, this is not straightforward: in
1-dimension one can use Ir|, but for arbitrary dimensions, a more complex procedure required, as
recently discussed in 49 48 Since the group is small - two elements one can use symmetrisation instead 779, as done by Lim al. [M: symm(x) >x MLP(y), i.. for sign: symm(x)
( x). Therefore, we define: Scalelnv R (X) p (x1, ......... Rn), with x; = cc,oon
x or
symm (x)) Importantly, it is known that both cases allow for universality. see g 29 and
79
62
respectively.
Challenge 2 Rescaling: Different multipliers. Scale equivariance alone is not sufficient, since the input vectors of the message function MSGy are scaled by different multipliers- central vertex: (e),), neighbour: - (i), edge: (e(1)g__1(3) (i), while its output should be scaled differently as well-
(e()). We refer to this problem as rescaling. Dealing with Challenge 2 requires functions of the form:
( 9nx,) = g(x), x..)
II
qis vq E D,.
We call these functions rescale equivariant. Note, that this is unusual symmetry in equivariant NN design. Our approach is based on the observation that any n-other monomial containing variables Collecting all these monomials into single representation
from all vectors x; is rescae--quivariann is precisely the outer product X,, = X @ ... 2 Xx, where
= III (i)... Therefore,
the general form of our proposed Rescale Equivariant Network is follows:
ReScaleEq(xI.... .x)) = (celeEE((ee(X,.)).
(ReScale Equiv. Net)
In practice, given that the size of X., grows polynomially with n, we resort to a more computationally friendly subease, i.e. hadamard products, i.e. ReScaleEq(xx... ..,....,)))...).))... 0 T,x, r, X3 - Re Contrary to the original formulation, the latter is linear (lack of multiplication with an invariant layer).
Scale Equivariant Message Passing. We are now ready to define our pessage-passingg scheme. Starting with the message function, we require each message vector mi to have the same symmetries the central vertex i. Given the scaling symmetries of the neighbour and the edge, for forward neighbourhoods, this reads: MSGv (gxx, Y, 4n.9 ee) = q. MSGv (x, y,e. In this case, we opt to eliminate qy by multiplication as follows:
(8) where [ denotes concatenation, ReScaleEq(quy 4s ee) qu ReScaeeEqqy c). In our experi- ments, we used only y and c, since we did not observe significant performance gains by including the central vertex x. Now, the update function is straightforward to implement since it receives vectors with the same symmetry, i.e. it should hold that: UPDy () = Qn UPDy (x, m) which is straightforward to implement with a scale equivariant network, after concatenating x and m:
MSGv (x, e) = ScaleEq ([x, ReScaleEq ey)))),
UPDy (x, m) ScaleEq m)).
Finally, to summarise our graph into a single scalar/vector we require scale and permutation- invariant readout. The former once more achieved using cunonicalised/symmerrred versions of the vertex representations of hidden neurons, while the latter using a DeepSets architecture as usual:
READy (X) := DeepSets (KI,..., Xn), X; canon; (x:) or x; = symm, (x;) (10)
show how to handle symmetries in the rest of the architecture components (i.e. initialisation, positional encodings, edge updates and i/o vertices) and provide an extension of our method to bidirectional message-passing (Appendix A.2 which includes backward neighbourhoods.
In Appendix A.1
Expressive power. Throughout this section, we discussed only scaling symmetries and not permu- tation symmetries. However, it is straightforward to conclude that ScaleGMNN is also permutation equivariann/iivariantt since it subcase of GMNs; if one uses universal approximators in their message/update functions (MIP), our corresponding functions will be expressible by this architecture, e
which was proved to be permutation equivariant in 44. Although this implies that GMN can express ScaleGMNN, this is expected since PQ symmetries are more restrictive than just P. Note that these symmetries are always present in FFNNs, and thus it is desired to explicitly model them, to introduce more powerful inductive bias. Formally, on symmetry preservation (proved in Appendix A.7.2)
Fwd: y ạy
Tall
(q 0). Reverse: |||
TyT
x
7

---

Table 1: INR classification on MNIST, F-MNIST, CIFAR-10 and Aug. CIFAR-10 10. We train all methods on 3 seeds and report the mean and std. (*) denotes the baselines trained by us and we report the rest as the corresponding papers. Colours denote First, Second and Third.
Method
MNIST
F-MNIST
CIFAR-10
Augmented CIFAR-10
MLP
17.55 + 0.01
19.91 + 0.47
11.38 + 0.34
16.90 + 0.25
Inr2
47
23.69 + 0.10
22.33 + 0.41
DWS [54]
85.71 + 0.57
67.06 0.29
34.45 + 0.42
41.27 + 0.026
NFNNP 185
78.50 = 0.23*
68.19 0.28*
33.41 + 0.01
46.60 0.07
NFN
85
79.11 + 0.84*
68.94 - 0.64*
28.64 = 0.07
44.10 0.47
NG-GNN
33
91.40 0.60
68.00 = 0.20
36.04 +0.44*
45.70
ScaleGMN (Ours)
96.57 + 0.10
80.46 = 0.32
36.43 + 0,41
56.62 = 0.24
ScaleGMN-B (Ours)
96.59 0.24
80.78 0.16
38.82 0.10
56.95 0.57
Table 2: Generalization pred.: Kendall-T on subsets of SmalICNN Zoo w/ ReLU/Tanh activations.
CIFAR-10-GS ReLU
SVHN-GS ReLU
CIFAR-10-GS Tanh
SVHN-GS Tanh
CIFAR-
Method
both act.
StaNNN
0,9140 0.001
0.8463
0.004
0.9140
0.000
0.8440
0.001
0.915 0.002
NFNxp
0.9190
0.010
0.8586
0.003
0.9251
0.001
0.8580
0.004
0.922 + 0.001
NENNNP
85
0,9270
0.001
0.8636
0.002
0.9339
0.000
0.8586
0.004
0.934 + 0.001
NG-GNN
33
0.9010
0.060
0.8549
0.002
0.001 0.9340
0.8620
0.003
0.931 + 0.002
ScaleGMN (Ours)
0.9276
0.002
0.8689
0.003
0.94181:
0.005
0.8736
0.003
0.941 + 0.006
ScaleGMN-B (Ours)
0.9282 0.003
0.8651 0.001
0.9425 0.004
0.8655 0.004
0.941 0.000
Proposition 5.1. ScaleGMN is permutation scale equivariant. Additionally, ScaleGMN is permu- tation & scale invariant when using readout with the same symmetries.
Finally, analysed the ability of ScaleGMN to evaluate the input FFNN and its gradients, i.c. simulate the forward and the backward pass of an impul FFNN. To see why this is desired inductive bias, recall that functional/operator can often be written via an integral on the entire domain) or of the input function's 's derivatives (e.g. via a differential equation). By simulating the FFNN, one can reconstruct function evaluations and gradients, which an additional module can later combine. Formally (proof and precise statement in Appendix Jop..ddixA.7.22.
function of input-output pairs (e.g.
Theorem 5.2. Consider an FFNN per Eq. I with activation functions respecting the conditions of Proposition (4.1 Assume a Bidirectional ScaleGMN with sufficiently expressive message and vertex update functions. Then, ScaleGMN can simulate both the forward and the backward pass of the FFNN for arbitrary impuls, when ScaleGMN's iterations (depth) are L and 2L respectively.
Experiments
Datasets. We evaluate ScaleGMN on datasets containing NNs with three popular activation functions:
sine, ReLU and tanh. The former is prevalent in INRs, which in turn are the most appropriate testbed for GMNs. We experiment with the tasks of (1) INR classification (invariant task), i.c. classifying functions (signals) represented as INRs and (2) INR editing (cquivariant), i.c. transforming those functions. Additionally, ReLU and tanh are common in neural classifiersrregressorss. A popular GMN benchmark for those is (3) generalisation prediction (invariant), i.e. predicting a classifier's test accuracy. Here, classifiers instead of FFNNS are typically CNNs (for computer vision tasks) and to this end, we extend our method to the latter in Appendix A.3] We use existing datasets that have been constructed by the authors of relevant methods, are publicly released and have been repeatedly used in the literature (8 datasets in total, 4 for each task). Finally, we follow established protocols: we perform hyperparameter search and use the best-achieved validation metric throughout training to select our final model. We report the test metric on the iteration where the best validation is achieved.

---

Table 3: Dilating MNIST INRs. MSE between the reconstructed and ground-truth image.
Table 4: Ablation: Permutation equivariant mod- els + scaling augmentations.
CIFAR- 10-GS
Method
F-MNIST
ReI.U
Method
MSE in 10"
DWS
67.06 0.29
MLP
5.35 + 0.00
DWS
S
>aug.
71.42 0.45
DWS
[54
2.58 0.00
NFNNr
83
68.19 1 0.28
0.9190 0.010
NFN NP
85
2.55 + 0.00
NFN
85
+aug.
70.34 0.13
0.8474 + 0.01
NFN
88
2.65 0.01
NFN
85
08.94
0.9270 0.001
NG-GNN-O 33
2.38 + 0.02
NEN
85
+
aug.
7.447
0.8906 0.01
NG-GNN
33
68.0 0.20
0.0010+0.000
NG-GNN-64
33
2.06 + 0.01
NG-GNN
33
aug.
72.01 = 1.4
0.8855 0.02
ScaleGMN (Ours)
2.56 = 0.03
ScaleGMN (Ours)
80.46 0.32
0.9276
0.002
ScaleGMN-B (Ours)
1.89 + 0.00
ScaleGMN-B (Ours)
80.78 + 0.16
0.9282 0.003
is non-local metanetwork that uses all linear permutation equivariant/ivvariant layers interleaved with non-linearities. NFNNNP [85] mathematically equivalent to DWS, while NFNNp [85 makes stronger symmetry assumptions in favour of parameter efficiency. The two variants are also designed to process CNNs contrary to DWS. NG-GNN [33 converts each input NN to graph (similar to work) and employs GNN (in specific PNA (5). Importantly, the last three methods transform the input parameters with random Fourier features while all methods perform input normalisations to improve performance and facilitate training. These tricks are in general not equivariant to scaling and were unnecessary in the ScaleGMN experiments (more details in Appendix A.4). On INR classification include naive MLP on the flattened parameter vector and Inr2Vec 47]. a task-specific non-equivariant method. For generalization prediction, we also compare StatNN 774, which predicts NN accuracy based on statistical features of its weights/biases.
Baselines. DWS [54]
INR classification. We design a ScaleGMN with permutation & sign equivariant components (and invariant readout). Additionally, we experiment with the bidirectional version denoted as ScaleGMN- B. We use the following datasets of increasing difficulty: MNIST INR, F-MNIST INR (grayscale images) and CIFARIO-INR, Aug. CIFARIO-INE
INRs representing images from the MNIST (38), MashionMNIST (76) and CIFAR (35] datasets resp. and an augmented version of CIFARIO-INE containing 20 different INRs for each image, trained from different initialisations (often called views). The reported metric is test accuracy. Note that [85] 86) use only the augmented dataset and hence rerun all baselines for the original version. All input NNs correspond to functions WG.e R² + I i.e. pixel coordinate to GS/RGB value. Further implementation details can be found in Appendix 1 As shown in Table D. ScaleGMN consistently outperforms all baselines in all datasets considered, with performance improvements compared to the state-of-the-ar ranging from approx. 3% (CIFAR-10) 13% (F-MNIST). While previous methods often resort additional engineering strategies, such as probe features and advanced architectures (33 or extra training samples [85] boost performance, in our case this was possible using vanilla ScaleGMNNs For example, in MNIST and F-MNIST, NG-GNN achieves 94.7 + 0.3 and 74.2 = 0.4 with 64 probe features which is apprpx. 2% and 6% below our best performance. The corresponding results for NG-T 33 (transformer) were 97.3 + 0.2 and 74.8 = 0.9, still on par or approx. 6% below ScaleGMN. Note that all the above are orthogonal to our work and can be used to further improve performance.
Predicting CNN Generalization from weights. As in prior works. we consider datasets of image classifiers and measure predictive performance using Kendall's + (30] We select the two datasets used in 855, namely CIFAR-10-GS. and SVHN-GS originally from Small CNN 7oo (74). These contain CNNs with ReLU or tanh activations, which exhibit scale and sign symmetries respectively. To assess the performance of our method (1) on each activation function individually and (2) on a dataset with heterogeneous activation functions we discern distinct paths for this experiment. In the first case, we split each dataset into two subsets each containing the same activation and evaluate all baselines. As shown in Ta once again ScaleGMNN outperforms all the baselines in all the examined datasets. This highlights the ability of our method to be used across different activation functions and architectures. Performance improvements here are less prominent due to the hardness of the task, a phenomenon also observed in the comparisons between prior works. Note that in this case, additional symmetries arise by the softmax classifier (36), which are currently not accounted for by none of the methods. We additionally evaluate method on heterogeneous activation functions.

---

In principle, our method does not impose limitations regarding the homogeneity of the activation functions of the imput NNs- all one needs to have a different caronicclisatoon//mmeetttisiii module for each activation. Experiments on CIFAR-IO-GS show that ScaleGMN yields superior performance compared to the baselines, significantly exceeding the performance of the next-best model. Further implementation details can be found in Appendix A.4.2
INR editing. Here, our goal is transform the weights of the input NN, to modify the underlying signal to a new one. In this case, our method should be equivariant to the permutation and scaling symmetries, such that every pair of equivalent input NNs is mapped to a pair of equivalent output NNs. Hence, we employ ScaleGMN similar to the above experiments but omit the invariant readout layer. Following [33,. we evaluate our method on the MNIST dataset and train our model to dilate the encoded MNIST digits. Further implementation details can be found in Appendix (A.endi AA.4.3 As shown in Table 3 our bidirectional ScaleGMN--B B achieves an MSE test loss (10-2) equal to 1.891, surpassing all permutation equivariant baselines. Notably, our method also outperforms the 3 baseline that uses 64 probe features. Additionally, our forward variant, ScaleGMN, performs par with the previous permutation equivariant baselines with an MSE loss of 2.56. Note that the performance gap between the forward and the bidirectional model is probably expected for equivariant tasks: here we are required to compute representations for every node of the graph, yet in the forward variant, the earlier the layer of the node, the smaller the amount of information it receives. This observation corroborates the design choices of the baselines, which utilize either bidirectional mechanisms (NG-GNN (3 or non-local operations (NFN (55).
NG-GNN
A Ablation study: Scaling data augmentations. We baseline our method with permutation equivariant methods trained with scaling augmentations: For every training datapoint, at each training iteration, we sample diagonal scaling matrix for every hidden layer of the input NN and multiply it with the weights/bias matrices as per Eq. (3) (omitting the permutation matrices). We sample the elements of the matrices independently as follows: Sign symmetrry Bernoulli distribution with probability 0.5. Positive scaling: Exponential distribution where the coefficient À is a hyperparameter that we tune on the validation set. Observe here that designing augmentations in the latter case is a particularly challenging task since we have to sample from a continuous and unbounded distribution. Our choice of the exponential was done by consulting the norm plots where in some cases the histogram resembles an exponential Fig. I Nevertheless, regardless of the distribution choice we cannot guarantee that the augmentations will be sufficient to achieve (approximate) scaling equivariance, due to the lack of upper bound. We evaluate on the F-MNIST dataset for the sign symmetry and on the CIFAR- 10- 10-GS-ReLU for the positive scale. As shown in Table 4 regarding the former, augmenting the training set leads consistently to better results when compared to the original baselines. None of these methods however achieved results on par with ScaleGMN and ScaleGMN-B. On the other hand, we were unable to even surpass the original baselines regarding the latter task. This indicates that designing an effective positive scaling augmentation might be non-trivial task.
Limitations. A limitation of our work is that it is currently designed for FFNNs and CNNs and does not cover other layers that either modify the computational graph (c.g. skip connections) introduce additional symmetries (e.g. softmax and normalisation layers). In both cases, in future work, we plan to characterise scaling symmetries (certain steps were made in (25) and modify ScaleGMN for general computational graphs as in [44] 88). Additionally, a complete characterisation of the functions that can be expressed by our scale/rescale equivariant building blocks is an open question (except for sign (4)). Finally, an important theoretical matter is a complete characterisation of the expressive power of ScaleGMN, similar to all equivariant metanetworks.
7 Conclusion
In this work, we propose ScaleGMN metanetwork framework that introduces the field of NN processing stronger inductive bias: accounting for function function-preservve preserving scaling symmetries that arise from activation functions. ScaleGMN can be applied to NNs with various activation functions by modifying any graph metanetwork, is proven to enjoy desirable theoretical guarantees and empirically demonstrates the significance of scaling by improving the state-of-the-arr in several datasets. With our work, we aspire to introduce a new research direction, i.e. incorporating into metanetworks various NN symmetries beyond permutations, aiming to improve their generalisation capabilities and broaden their applicability in various NN processing domains.
10

---

Acknowledgements
IK. GB and YP were partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the European Union under the NexIGenerationEU Program. This work was partially supported by computational resources grant from The Cyprus Institute (HIC system "Cyclone") as well from an AWS credits grant provided by GRNET National Infrastructures for Research and Technology.
References
[1] Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[2] Francesca Albertini and Eduardo D Sontag. Identifiability of discrete-time neural networks. In Proc. European Control Conference, pages 460-465. Springer Berlin, 1993.
[3] Francesca Albertini and Eduardo D Sontag. For neural networks, function determines form. Neural networks, 6(7):955-990, 1993.
[4] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2): 251-276, 1998.
[5] Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierre-Marc Jodoin. Neural teleportation. Mathematics, 11(2):480, 2023.
[6] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. NIPS Workshop on Optimization for Machine Learning (OP12015), 2015.
[7] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Richard Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv: 2302.03130, 2023.
[8] Benjamin Bloem-Reddy Yee Whye, et al. Probabilistic symmetries and invariant neural networks. Journal of Machine Learning Research, 21(90):1-611 2020.
[9] Léon Bottou and Patrick Gallinari. A framework for the cooperation of learning algorithms. Advances in neural information processing systems, 3, 1990.
[10] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
[11] An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910-927, 1993.
[12] Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Hcaton, Jialin Liu, Zhangyang Wang. and Wotao Yin. Learning to optimize: A primer and benchmark. Journal of Machine Learning Research, 23(189): 2022.
[13] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9939-59443 2019.
[14] Minhyung Cho and Jachyung Lee. Riemannian approach to batch normalization. Advances in Neural Information Processing Systems, 30, 2017.
(15) Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13277 2020.
[16] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMIR, 2017.
11

---

[17] Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homoge- neous models: Layers are automatically balanced. Advances in neural information processing systems, 31, 2018.
[18]
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosen- baum. From data to functa: Your data point is function and you can treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 5694-5725. PMLR, 17-23 Jul 2022. URL hntps://pprcceeet
mlr popres/vv62/duue pttp.eeee/.122/uuuooo222222 httml
[19] Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Clas- sifying the classifier: Dissecting the weight space of neural networks. ECAI 2020 24th Europeun Conference Artificial Intelligence, Frontiers in Artificial Intelligence and Applica- tions, pages 1119-1126. IOS Press, 2020.
[20] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[21] Charles Fefferman and Scott Markel, Recovering feed-forward net from its output. Advances in neural information processing systems, 6, 1993.
[22] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL
https://opearrevev. http:nttoooommmddd:d1bbbbbb
[23] Cedric Gegout, Bernard Girau, and Fabrice Rossi. mathematical model for feed-forward neural networks: theoretical description and parallel applications. PhD thesis, Laboratoire de l'informatique du parallélisme, 1995.
[24] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference
Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1263-1272. PMLR, 2017. URL http: http:///proceedings. mlr wwwps/ww0/ htt.//////////////////
html
[25] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:1 1893-11905, 2022.
[26] Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In International Conference on Machine Learning, pages 1 11734-11760, PMLR, 2023.
[27] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference Learning Representations, 2017. URL https::/ppenrrvvier..t wttp://wwww....ettooo.//d//////e
[28] Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-133 135. Elsevier, 1990.
[29] Sékou-Oume Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravan- bakhsh. Equivariance with learned canonicalization functions. In International Conference Machine Learning, pages 15546-15545 PMLR, 2023.
[30] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938.
[31] Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems, 02224333-24443 2021.
12

---

[32] Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters of diverse imagenet models? In International Conference on Machine Learning. pages 17243-17259 PMLR, 2023.
[33] Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL htttps::/openreeeee...tto http:anneeetooommmmmdoo0000--30
[34] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadeneshelii Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes, Journal of Machine Learning Research, 24(89):1-97, 2023,
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny (200),. 2009.
[36] Daniel Kunin, Javier Sagastuy- Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021.
[37] Véra Kürková and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543 1994.
[38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2778-2324, 1998.
[39] Derek Lim, Joshua Robinson, Stefanic Jegelka, and Haggai Maron. Expressive sign equivariant networks for spectral geometric learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 URL http::/ nips. http//ppe__fffeeee www.faaaapppppp/222a2.. /apprr/022//aah/ Cor.teeooooeeeeccc.coo html
Neural Informa-
16. 2023, 2023.
[40] Derek Lim, Joshua Robinson, Stefanie Jegelka, and Haggai Maron. Expressive sign equivariant networks for spectral geometric learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttps://openrevieeeee http:///peaaeeieeeeeee////o///a/ aeee/fforummd--llWdyyaaCCooo
[41] Derek Lim, Joshua David Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Fleventh International Conference Learning Representations, 2023.
[42] Derek Lim, Joshua David Robinson, Lingxiao Zhao. Tess E. Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpt:t//ppeeeeeeeee net/ pdf?id=-- UigMorzi1
[43]
Derek Lim, Joshua David Robinson, Lingxiao Zhao, Tess E. Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:// net/ pdf Q
[44] Derek Lim, Haggai Maron, Mare T. Law, Jonathan Lorraine, and James Lucas. Graph metanet- works for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations (ICLR). 2024.
[45] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL httpttt::///peeee///// net net//forummLd--S1eeHtooGGG
13

---

[46] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 2229, 2021.
[47] Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. BtRl.httpps//openreeee
net/ pdf w Frid=00OIW-3aad
[48] George Ma, Yifei Wang, Derek Lim, Stefanie Jegelka, and Yisen Wang. A canonization perspective on invariant and equivariant learning. arXiv preprint arXiv:2405. 2024.
[49] George Ma, Yifei Wang, and Yisen Wang. Laplacian canonization: A minimalist approach to sign and basis invariant spectral embedding. Advances in Neural Information Processing Systems, 36, 2024.
[50] Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: net/ forum?id-Syx72jc99m)
[51] Qi Meng, Shuxin Zheng, Huishuai Zhang, Wei Chen, Qiwei Ye, Zhi-Ming Ma, Nenghai Yu, and Tie-Yan Liu. G-SGD: optimizing relu neural networks in its positively scale-invariant space. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
[52] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 4460-4470, 2019.
[53] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes neural radiance fields for view synthesis. Commun. ACM, (5(1):99-106, 2022.
[54] Aviy Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pages 25790-25811 PMLR, 2023.
[55] Aviy Navon, Aviy Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024. URL Inttps://openrooe
net/forrum/iddnnPnnnie
[56] Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path- notb-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
[57] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[58] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints, arXiv preprint arXiv:2209.12892 2022.
[59] Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminheidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20237-20246. 2023.
[60] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. Learning mesh-based based simulation with graph networks. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net,200 2021.
14

---

[61] Mary Phuong and Christoph H. Lampert. Functional vs. parametric equivalence of re(lu) networks. In International Conference on Learning Representations, 2020.
[62] Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25-29,
2022. OvenRevi,,.net, 2022.
[63] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[64] David Rolnick and Konrad Kording. severse-engineerne deep relu networks. In International conference on machine learning, pages 8178-8187. PMLR, 2020.
[65] Jürgen Schmidhuber. Learning to control fast- weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-13)) 139, 1992.
[66] Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, 34:16481-16492 2021.
[67] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth, Hyper- representations as generative models: Sampling unseen neural network weights. Advances in 2022.
Neural Information Processing Systems,
[68] Louis Serrano, Lise Le Boudec, Armand Kassai Koupai, Thomas X Wang, Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari. Operator learning with neural fields: Tackling pdes on general geometries. Advances in Neural Information Processing Systems, 36, 2024.
[69] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-2205 2020.
[70] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7447, 2020.
[71] Pierre Stock, Benjamin Graham, Rémi Gribonval, and Hervé Jégou. Equi-normalization of neural networks. In 7th International Conference Learning Representationss ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[72] Veronika Thost and Jie Chen. Directed acyclic graph neural networks. In International Conference on Learning Representations, 2021. URL /tRl:attss://peeeeevii wttp:evwwwwwweeeftteeeeee net/forum? 1d=JbuYF437WBB
[73] Hoang V Tran, Thieu N Vo, Tho H Tran, An T Nguyen, and Tan Minh Nguyen. Monomial matrix group equivariant neural functional networks. arXiv preprint arXiv:2409. 1697, 2024.
[74] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint 1rXVv22002.11448 2020.
[75] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning. pages 23965-23999 PMLR, 2022.
[76] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for henchmarking machine learning algorithms. arXiv preprint arXiv: 1708.077477 2017.
[77] Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEFF/CVE International Confer- ence Computer Vision, pages 1284-1293, 2019.
15

---

[78] Dejia Xu, Peihao Wang, Yifan Jiang, Zhiwen Fan, and Zhangyang Wang, Sig- nal processing for implicit neural representations. In Advances Neural Infor- mation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, IA, USA, November 28 Decem- URL. httt:://ppp:prrr/ nips ctaps.cc/ppp wwcppaper__eeeeeee files/ httppppppp////a/a//a/ee 37cc550013ddeeeeteeeebbbbbaadaa Abstract----ttt-ttoeo Costrrctt-Confeeee Conf Conference.e html
ber 9, 2022, 2022.
[79] Dmitry Yarotsky. Universal approxxmatios of invariant maps by neural networks. Constructive Approximation, 55())4407-474, 2022.
[80] Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10881- 10891. PMLR, 13-18 Jul 2020. URL https::/ html
mlr. press/v119/you205
[81] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[82] Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Gameet,, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. Advances in neural information processing systems, 32, 2019.
[83] Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated optimization. Advances in neural information processing systems, 35: 15:16169-,,690, 2022.
[84] Bo Zhao, Robert M. Gower, Robin Walters, and Rose Yu. Improving convergence and gener- alization using parameter symmetries. In The Twelfth International Conference on Learning Representations, 2024. URL https:://///peereeiie met//forumnddd-1OO0phh111
[85] Allan Zhou, Kaien Yang. Kaylee Burns, Adriano Cardace, Yiding Jiang. Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems, 36, 2023.
[86] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, Zico Kolter, and Chelsea Finn. Neural functional transformers. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. httthttss://
wwt. netrforum?id-
JahyIaOazI
[87] Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. abs/2402.05232,, 2024. URL httph:t/t///ccct //do: 0rg//0.48850 485500//r/// arxiv. 2402.05232
CoRR.
[88] Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. arXiv preprint MxXv:24422.02332 2024.
16

---

A Appendix supplemental material
A.1 Additional practical considerations of ScaleGMN
In the following section, we describe the additional practical considcrations that were omitted from Section [ For the purposes of this section, we introduce the functions layer (i) and pos (i) which return the layer of a vertex i and its position within this layer respectively.
A.1.1 Initialisation
As previously discussed, all vertex/edge representations throughout the network should respect the symmetries given by Eqs. (4) (5). This is straightforward to achieve at initialisation, by using the possible to use scale equivariant layer here as
NN biases and weights (similarly to B31.. well, e.g. a linear layer, which is what was performed in our empirical evaluation. For notation convenience, denote the biases and weights of Eq. I with b, W, such that b(i) = Dlaye((f) (pos (i)),
W(i,) = W Wyyer(t) (pos (i) pos (j)). Now we can define the initialisation:
INITV (xv (1)) (i), xv(i) [b (t), if layer (i) E (1.....,A) L},
(11)
1, if layer (i)
E (xg (i.j)) = - (i,)) xE(i,)) = w (i,5), if layer (i) E [I..... L}.
(12)
Note that the above different from the initialisation/graph construction used 44 For example, in this work, an extra vertex is created for each bias, which is connected to its corresponding neuron vertex with an edge initialised with the bias value. However, this strategy would complicate scaling symmetry preservation and was thus not used in our work.
A.1.2 Positional Encodings
As discussed in 3 not all vertices/edges of an FFNN are permutable. In particular, vertices corresponding the input and output neurons are fixed, while vertices corresponding to hidden neurons can be permuted only within the same layer. Similarly, edges originating from an input neuron permutable only with edges originating from the same neuron, and the same holds for edges terminating at an output neuron, while edges of hidden layers can be only permuted within the same layer.
To adhere to the above, we need devise a symmeery--reakinn mechanism, to ensure that the GMN is not equivariant to undesired symmetries (e.g. permuting output neurons). This is the role of positional encodings, which are learnable vectors (in the literature fixed vectors have been also used), shared across all input graphs, that take part in sassage-passing In particular, for vertex positional encodings the following holds:
(i) = pv(,), if layer (i) layer (i) (0,
(13)
and for edge positional encodings:
j = jr, layer (j) E L), layer (j) = layer (j') * O, layer (i) layer (t') # L, i layer (i) L),
p=(i.j)= = if
(14)
while otherwise, the positional encodings are free take different values. Symmetry- usually happens at initialisation (which is then inherited by the subsequent message-passing layers) and
Eq. (Ini becomes:
(1) INITV (xv (i), (i)) (1) INIT z (x (i,j),PP(( (i))),))
(15)
or alternatively, it can be performed at all layers, where Eqs. Msg and (pdd
become:
mf)(1) + jEA'()
MSG ( ())
()) byg (1.1). Pv (1), Pr ( (E).)(()))))
(16)
(i) = UPD (ht mf(()))) (1).
(17)
17

---

Challenge 3 - Incorporating positional encodings into scale equivariant layers. Although the above formulation allows for the preservation valid permutation symmetries, special treat- ment is needed to preserve valid scaling symmetries as well. Since the positional encodings are fixed across datapoints, they do not have the same scaling symmetries as vertex/edge rep- resentations. Therefore, message-passing components do not need to be equivariant to scal- ing of the positional encodings. In particular, the following should hold: INITV (gx, p) = gINITE (c. (). MSGv (9=X, qyY, 9:9y c. Pas Py.Py.PP) M.MSGV (x. y. ex, Py: Pe) and UPDy (9,x, q. p) U2UPDv (x, p).
qINITy (x.p). p). INIT (qc.p)
Therefore, Eqs. 15 166) 17 introduce a new challenge (apart from those discussed in Section 5p:
being scale equivariant w.r.t. one argument and not equivariant. w.r.t another one. Interestingly, our ScaleNet formalism can straightforwardly deal with this case, by defining the canonicalisation function for the non-symmetric argument to be the identity (since in the absence of symmetry equivalent points must be equal). To discern this from the classical scale invariant/equivarian layers, we will refer to it augmented scale invarian/equivvriann layer and it is formalised follows:
AugScalelnv (X. = p² (xI...... Xn. P), X; = cx,)n(x;) or x; symm(x;) (18)
AugScaleEq = f o o...of. ....off, fl, (*()) (Ifx,,..... x,,)) @ AugScalelnv (X,
(19)
Concluding, we obtain the new Iearnable functions, by simply replacing scale equivariant with augmented scale equivariant layers:
INITV (x, p) = AugScaleEq (x, p) INIT (e. p) = AugScaleEq (e.p): p),
(20)
MSGv (x, y, Pz, Py, P.) AugScaleEq ([x, ReScaleEq (y, e)], [Pa, Pa Pel),
(21)
UPDy m, Pa) AugScaleEq ([x, m) Pw)
(22)
A.1.3 Edge Updates.
To respect the symmetries given by Eq. ( for edges (i.j). with layer (i) E L 13 the edge update function should have the following property: UPD Ky8,4.4 e)
9 'UPD (x, Observe, that this is once again reminiscent of the rescaling problems dis- cussed in Section []] but with the difference that qy appears both as a multiplier and as a divisor. The most straightforward solution would be to elementwise invert y and then proceed as in the message function of Eq. (8) i.c.:
UPD (x.y, c) ScaleEq ([c, ReScaleEq (x, 1 2 y)l)
(23)
In experiments though, opt for simpler solution due to the numerical instabilities (high- magnitude gradients) produced by the element-wise division and choose an invariant layer for the arguments y. Formally:
UPD (x,y, e) AugScaleEq (e, Scalelnv (x,y)).
(24)
When we include positional encodings into the edge updates, Eq. Upd) (RHS) becomes:
(i,j) = UPD B (1 (1), (1) h "
pv (v) (r)))
(25)
and Eq. 24 is further modified into:
UPD y, Pa, Py, P.) AugScaleEq (e, [Scalelnv (x,y), Pai Py: pol)
(26)
A.1.4 Updates on input and output vertices
So far, we have only discussed how vertex/edge representations corresponding to hidden neurons are updated, since Eqs. (4) (5) do not hold for input and output neurons. particular, there exist no valid scaling or permutation symmetries for these neurons, while for the corresponding edges, permutations and scalings are unilateral see Eq. (3). Therefore, we require the following:
(1) hy (1), = layer (t) E {0, L), (No equivariance) h'g (i. ae (m (i)) be (me (1) .)... {= layer (i) = 1,
(27)
(28)
(i, = hg (1, ( (i)) -e-1 ( (i).
8 = layer (i) = L.
(29)
18

---

This implies that the conditions for the learnable functions should change for i/o neurons. In particular, the following should be non-scale-equivariant: INITv. MSGy. UPDv;o V. UPDy,L. The same should hold for MSGv,L, but with the additional requirement of invariance to rescaling: MSGv,L Yy 9y e, Px, Py, Pe) L y, Pv. Py, Pe)
In traditional GNNs, where only permutation symmetries are taken into account, sammetry-breakinnn can be efficiently handled by positional encodings as discussed in Appendix A.1.2 However, this is not the case here, since positional encodings only break permutation symmetries, but not scaling symmetries. Instead, we resort to a simpler solution (albeit incurring an increased number of different initialisation, message and update functions for input/output neurons. v,, MSGiv,, V, INIT UPDy.1 are general function approximators (MIPs), while M y, Px, Py,Py, Pe) MIP (x, ReScaleEq (y,e), e), Pa, Py: Pe).
parameters):
As for the i/o edge representations, the following should hold at initialisation: qy LNITTE, p), and for UPD (y- Qa Pa. Pys Pa) = In UPD I y, Pa, Py, Pe), Jy UPDEL y, Pr. Py: Pe). Therefore, edge ini-
INIT (g) = Ia E, p), INT,, e. the edge updates: UPDE,L(x, 4y., fy e, Pz, Py-Pe) tialisation should be equivariant to the edge scale and therefore can remain is Eq. (20). The same holds for the edge updates Eq. (26). which allows us to retain the same architecture in our experiments, although the interested reader may observe that one can use more powerful architectures since there is no need for rescaling here.
A.1.5 Readout
Regarding the readout, devise permutation and scale invariant aggregator follows:
READy (X)
MLP ([DeepSese (x....., Xn) Xo: XL.) cxnon(xx) x, symm (x;).
(30)
This means that we aggregate scale canoricaiseed/ymmeerried representations of hidden neurons with permutation invariant aggregator (Deepsets [81)), concatenate the result with all i/o neuron representations (that cannot be permuted or scaled) and transform the result with a universal function approximator. A simpler alternative is to only concatenate the output neuron representations (or also the input for the bidirectional version) since after sufficient sessage--assiigg iterations they are expected to have collected information from the entire graph. We include and evaluate both choices in our hyperparameter grid.
A.2 Bidirectional ScaleGMN
Undoubtedly, using only forward neighbourhoods, although convenient and aligned with the com- putational graph of the input datapoint, may restrict the expressive power of ScaleGMN. This is self-explanatory since vertices receive information only from parts of the graph that belong to earlier layers. This might be detrimental, especially for equivariant tasks (operators). Additionally, it is contrary to the first works on weight space networks [64) where vertices/edges collect information from the entire graph, while backward neighbourhoods are crucial for GMNs to express NP-NEN [6) as shown in Proposition 3 in Lim et al. [4].
However, making bidirectional message passing scale equivariant is challenging since back- ward messages have different symmetries than forward ones. In particular, preserving the same assumptions for the symmetries of vertex/edge representations, the input triplet mes- sage function (i), h i). with j E Ni((1) and = layer (i) is equivalent to qr(i) (i), 9241 (J), q (j)hr(j (r,((,))) i)q; (i) (we have omitted permutations here for simplic- ity). Therefore, doing the same rescaling as before inappropriate since it does preserve the desired scaling coefficient (i).
There several options may follow circumvent this discrepancy. However, most involve an elementwise division of a certain representation, which we experimentally observed to face numerical
"This is redundant in the forward ScaleGMN since input neurons do not have any forward neighbours. In the backward case Appendix A.2 layer similar to MSGv.L.
19

---

instabilities. The simplest one is to add backward edges and initialise them with inverted features:
I x (XE (1.1)) = Fxl 2 xx(i.t), x=(i
W (i.t), if - layer (i) < layer (i) L. (31)
In this case, we have that E (1)+ ( (i:8) - q0(1) ()INITE (x (i, i)), and thus (i are initialised with the same symmetries as (i,j)- Now can proceed as before with the message and update functions. In particular, we introduce a backward message function to discern the two message directions (alternatively this can be done with edge positional encodings to reduce the number of parameters):
mmv =
BW (1) '1, hE ".,1) (v()), PE(J.)) jEN
MSGV,Bw(X, y. Pa: AugScaleEq ((x, ReScaleEq c), PP, Pel).
It is easy to see that:
MSGy, 4uy, Oy ex Pz, Py, Pu, ga BS (,, Pa. Py, Pe).
Now to incorporate backward messages into the update function we simply concatenate the outputs of the two message functions and apply a scale equivariant layer usual:
h (1) UPD (b'(())
(1), Bw (i), pv (i)):
(32)
UPDv (x, mew, mgw Pa) AugScaleEq ([x, mew, mgw] Pw)
Once, again the desired symmetries are preserved:
UPDv(gaX x, qa Maw Oa Maw, Px) = UPDy (x, mew. maw, Pa)-
Bidirectional ScaleGMN for sign symmetries. Importantly, the above edge weight inversion of Eq. (31) is not necessary when dealing with sign symmetries. In particular, for any E {-1,1} it holds that = g. Therefore, one can proceed with the above formulation but initialise the edge weights as before -Eq. (12). We found this crucial in our experimental evaluation since weight inversion led to an unusual distribution of input features: since typically the weight distribution was nearly Gaussian, the distribution of the inverses was close to reciprocal normal distribution, bimodal distribution with undefined mean and higher-order moments. This frequently led to numerical instabilities as well (albeit less so compared to computing reciprocals of neuron/edge representations), which explains why it was easier train bidirectional models for sign symmetric NNs compared to positive-scale symmetric ones.
A.3 Extension to CNNs
A similar methodology to Section 5 can be applied Convolutional Neural Networks with only a few changes. As observed by prior works on metanetworks (Zhou et al. [85]. Kofinas et al. B3]), the permutation symmetries in a CNN arise similarly to MLPs; permuting the filters (output channels) of hidden convolutional layer, while also permuting the input channels of each filter in the subsequent layer does not alter the underlying function of the network. This is easy to since convolution is nothing else but a linear layer with weight sharing. Weight sharing constrains the valid permutations far pixels/input coordinates concerned (e.g. one cannot permute the representations corresponding to different pixels), but allows for permutations of input/output channels, except for the input/output layers as always.
Similar rules apply to scaling: one may scale the output channels hidden convolutional layer, while also scaling the input channels of each filter in the subsequent layer per Proposition 4.1 Note here that, again, all weights corresponding to the same input channel can only be scaled with the same multiplier for all pixels/input coordinates, due to weight sharing. Below, describe implementation for all CNN layers, which closely follows [33].
Convolution. Being the main building block of CNNs, convolutional layers consist of kernels and biases. For a hidden layer i we can write the former as W, E RRd xde-xd_1xk x and the latter as b, E Rd following the FFNN notation. Aligning with the procedure of [33], we construct one vertex for
20

---

each i/o channel and the imput graph has the following node and edge features: Xy (i) - b(i) E R. (i) ji,,) E Rwaeehh respectively, where Wmax. /max are the maximum width and height of the kernels across layers (zero-padding is used to allow for fixed-size edge representations across all vertices in the graph). Again here, the positional encodings are responsible for the permutation symmetry breaking mechanisms within the CNN. As with FFNNS, the CNN input and output neurons not permutable, while the hidden neurons are permutable only within the layer they belong. Given that filter positions are not permutable and are associated with a single neuron, they are represented as vectors the edge features.
Average Pooling. Average pooling commonly placed after the last convolutional layer (typical in modern CNNs instead of flattening + linear layer to allow for variable-size images) and is responsible for pooling the output feature map in a single feature vector. First off, since it is linear operation, is amenable to scaling symmetries. It is easy to see this by perceiving average pooling as a KrK convolution with no learnable parameters (equal to K. where K the dimension of the image, and a single output channel. Hence, no additional modifications are required, similarly to 33
Linear layer. Commonly used as the last layer of CNNs extract a vector representation of an image, linear layers require a different approach within CNNs compared to the one in MLPs, as in the former case the edge features are vectors, while the latter scalars. Aligning with the method proposed by [33], the simplest solution is to zero-pad the linear layers to the maximum kernel size and consequently flatten them to meet the dimension of the rest of the edge features. By handling linear layers this way, no additional measures are needed to ensure permutation and scale equivariance.
A.4
Implementation Details
A.4.1 INR Classification
Datasets:
We evaluated our method on three INR datasets. Provided as open-source by Navon et al. (54, the datasets MNIST and FashionMNIST contain single INR for each image of the datasets MNIST 338 and FaSHionMNIST 766 respectively. The selection of these datasets was encouraged by the fact that they were also selected by prior works, establishing them useful first benchmark on INR metanetworks. As a third INR dataset, we use CIFAR-10, publicly available by Zhou et al. [85), which contains one INR per image from CIFARIO (35. Regarding the last dataset, the authors also provide an augmented training dataset, consisting of 20 additional copies of each INR network, but with different initializations. Training on such bigger dataset allows them to achieve better results, probably because allows them to counteract overfitting. To demonstrate the capabilities of our method train ScaleGMN both on the "original" dataset as well as the augmented one, achieving better results in both cases. In the latter case we carefully follow the training and evaluation procedure used in [85), i.e. we train for 20000 steps and evaluate every 3000 steps. In all the above datasets, we use the same splits as in prior works, we train for 150 epochs and report the test accuracy based on the validation split, following the same procedure Navon et al.
Hyperparameters: The GNN used for ScaleGMN is a traditional Message Passing Neural Network 124], with carefully designed update and message functions that ensure sign equivariance. We opti- mise the following hyperparameters: batch size in (64, 128, 256]. hidden dimension for node/edge features in (64, 128, 256]. We also search learning rates in (0.001, 0.0005, 0.0001}. weight decay in f0.01. 0.001}. 3. dropout in {0, 0.2) and number of GNN layers in {2, 3,4 4, 5). Moreover, we experiment with using only vertex positional encodings or also employing edge positional encodings. We apply layer normalization within each MLP and use skip connections between each GNN layer. The last two steps were proven valuable to stabilise training. Finally, for each MLP within the architecture we use SiL.U activation function, one hidden layer and no activation function after the last layer. All the experiments were conducted on NVIDIA GeForce RTX 4090.
As mentioned above, the core of all building blocks, i.c. Scale Invariant Net is designed to be invariant to sign flips. In our experiments, we explore two means of designing sign equivariance, either by elementwise absolute value |x| (canonicalization only in 1-dimension), or symmetrisation, symm(x) = MLP(x) + MLP(-x), Although the former leads to reduced expressivity, in some cases we observed better performance. A possible explanation could point to the extra training parameters that are added because of the additional MLP.
Results: As shown in table Table D our method is able to push the sfate-of-the--at all the datasets considered. Notably, many additional techniques that were employed by previous works in order to
21

---

achieve good results were not required in our method. Specifically, probe features when inserted into the node features as in 33 were proven to boost the performance much higher, and could potentially
enhance the performance of ScaleGMN too. In the reported results we compare with variants of [33
that do not use probe features and rely solely on the parameters of the input network, although our
method outperforms even the probe features variants. Moreover, [85] and [33] also apply random Fourier features during the initialization of the weight and bias features, which in our experiments
showcased better performance when used in traditional MPNN framework, but were not employed for our method. In our framework, we also do not include any normalization of the input as conducted by previous works, where the normalization is applied either parameter level (85). [54] or layer level [33], computing the mean and standard deviation. Although this step typically leads to better performance, it could not be applied in our scale and permutation equivariant framework. Nevertheless, even without all the above techniques, ScaleGMN achieves remarkable results. Finally, opting for Transformer, as in [R), instead of a GNN layer is also an orthogonal to our work addition that could enhance the performance of ScaleGMN.
A.4.2 Predicting CNN Generalization from weights
For evaluating our method on the CNN Generalization task, we select CIFAR-IO-GS and SVHN-GS datasets from 774,, following the evaluation of prior works (Zhou et [ Zhou et al. [86], Lim et al. [44] Kofinas et al. B3). The above datasets contain CNNs trained with ReLU and Tanh activation functions. In order to assess our method separately for each of these activation functions, as they introduce different symmetries, we distinguish two different types of experiments. In the first case we deviate from the path of previous evaluations and split each of these datasets into two subsets. The first one contains CNNs trained with ReLU activation function and the second networks with Tanh. In order to compare with previous methods, we train them on each subset and report Kendall's s correlation r metric [30) as originally proposed by Zhou et al. [85].
Heterogeneous activation functions: In the second case, we evaluate ScaleGMN on the whole CIFAR- and SVHN-GS datasets. We are able to do so, as our method does not impose any limitations regarding the homogeneity of the activation functions of the input neural networks. The sole necessary modification lies in the construction of the invariant layer. Specifically, following the notation of Eq. Scale Equiv. Net), each equivariant layer f* is now equipped with two invariant nets, Scalelnv rela, and Scalelnvianh- to be applied on the datapoints with ReI.U and Tanh activation functions respectively. The rest of the architecture remains the same.
Hyperparameters: For the ReLU datasets we implement scale equivariant update and message functions, while for the Tanh sign equivariant. In all of our experiments we use MSE training loss. Re- garding the hyperparameters we search: batch size in {32, 64, 128]. hidden dimension for node/edge features in {64, 128, 256]. We also search learning rates in (0.001, 0.0005, 0.0001]. weight decay in (0.01, 0.001). ). dropout (0, 0.2) and number of GNN layers in (2,3,4,5) Again, we experiment with using only node positional encodings or also employing edge positional encodings and apply layer normalization within each MIP. For the Tanh datasets, we also experiment with the applying canonicalization symmetrization of the sign symmetry. Finally, for the experiments with heterogeneous activation functions, we apply the same hyperparameter search as above.
A.4.3 INR editing
To evaluate the performance of ScaleGMN on equivariant tasks we opted for the task of INR editing. Aligning with the setup of (33), we utilize the MNIST INR dataset provided by Navon et al. [54] and evaluate our method on dilating the encoded MNIST digits. Specifically, for each image i of
the dataset, where [N], we compute the ground truth dilated image using the OpenCV image processing library and denote it as fú, where f. R22 + R. Let 0 denote the INR parameters of the i-th image and fSIREN the encoded function. Then, fSIREN (z, y: 6,) is the output of the INR at the (I, y) coordinates, when parameterized by 0.. The updated INR weights are computed as: 0- 0, + 7. ScaleGMM(O..,. where y a learned scalar initialized to 0.01. Finally, we minimize the mean squared error on the function space, that is:
1 N d
L =
Σ
ISIREN (z, (2, 0.) - fa v)l
(33)
N.
22

---

Simply put, we compare the reconstructed new INR with the dilated ground truth image. Again, we do not apply any normalizations nor feed the model samples of the encoded input image (probe features).
Hyperparameters: The GNN used for ScaleGMN is traditional Message Passing Neural Network [24], with carefully designed update and message functions that ensure sign equivariance. In contrast to the above experiments we do not add a final readout layer, as we do not compute any final graph embedding. We optimise the following hyperparameters:: batch size in {64, 128, 256}. hidden dimension for node/edge features in 64, 128, 256 and number of GNN layers in {8. 9, 10. 11}. We also search learning rates in {0.001, 0.0005, 0.0001}, }. weight decay in {0.01, 0.001) } and dropout in {0, 0.1, 0.2). Moreover, we experiment with using only vertex positional encodings or also employing edge positional encodings. We apply layer normalization within each MLP and use skip connections between each GNN layer. The last two steps were proven valuable to stabilise training. Finally, for each MLP within the architecture we use SiLU activation function, one hidden layer and no activation function after the last layer. The learned scalar y, also used in previous work Zhou al. [85,, was always initialized at the same value 0.001 and consistently led to better results. All the experiments were conducted on NVIDIA GeForce RTX 4090.
As shown in Table 3 the bidirectional variant achieves better results than all the previous works on metanetworks, without incorporating any of the additional techniques described above. Notably, ScaleGMN-B -B surpasses the performance of the graph-based permutation equivariant baseline NG-GNN equipped with 64 probe features [33] Interestingly, our method demonstrated much better results as we added more GNN layers. This experimental observation possibly corroborates with the findings of Theorem (5.2 Regarding our forward variant, we can see that although it is able to perform on par with the previous baselines, it struggles to match the results achieved by the bidirectional variant. This behaviour, observed exclusively in this task, is theoretically anticipated for equivariant tasks. Specifically, in the forward case, neurons of layer é only receive information from layers e" < f, which hinders the computation of meaningful embeddings for nodes of the first layers. Taking into account that the INR models are rather shallow (only 2 hidden layers of 32 neurons each), then big proportion of the whole set of nodes is being updated with less meaningfull information.
Results:
A.4.4 Extra symmetries of the sine activation function
Regarding the sine activation function, can casily show the presence of an additional symmetry due to the periodicity of harmonic functions. To see this, recall from Eq. the formula for a feedforward L-layer SIREN:
Xg x, (i) = sin (wo (i ()x=)xx- + be(i)).
We will prove that the following guarantees function preservation:
W Q We, b'g Q.b. + OgT = (W,, bl) (We, be)
(34)
with Qe defined as diagonal sign matrices, O 2M, + inv- Qt), where M.
[0, q 0
diag (oog(mm())).... (m((4)....(((())) mi) Z and we defined inv-sign(g)
for compactness
1,, q < 0.
(we have omitted permutation matrices here for brevity). Observe here that the matrix transformations are "single-sided", contrary to Eq. (3). We show by induction that, when Eq. (34) holds, then x² = X
Xx-1- In order to have Base case: X = Xg by defintion. Induction step: Assume X
x xx, Vx E X. it suffices that 3me(1) E Z. ge()) E 11,
wo ( (1) = (wo (i, + be (i)) qe()) + (2mB + inv-sign(ae(())) (i.) -: (i) b' (f) = ge(i)b, (3) + (2mx(1) + inv-sign(ar(t)))
This can be straightforwardly extended to include permutations.
To account for those symmetries, in this paper, we opted for simple solution, based on the observation
E ( - (Tm/2 /2). then m must be O and q must be 1 in all cases: For gr(i) 1, since 0 < [b) bz(1) < n, it must be that
that they cannot exist if the biases are bounded. particular, if (i). be(i)
S [2me(i)] < <> 0 < mmm < 1/2 = m = 0. For qi))) = - 11, since # < b/(()bbeii) S m,
it must be that -1 < (1) + 1 < + -1 < me(i) S 0 = mi(i) 0. Further, since the
23

---

Algorithm 1: Bias shift (elimination of symmetries due to periodicity)
Input: b: bias by(1), w: weight row We(i,:)
2: if h < ( then
3: bj
b; WI t w: g1 c -1// Sn[wow][xeL1 b1) sin(wow" NowTTxe xx-1
4: else
5: b1 f- b; W +- w; q1 f-1
6: end if// Now bi 0.
7: if b1 > 2n then
8:
b2 bj mod 2n; w2 91 f-
// b1 = 2 + b2 sin(wa X + b2) = sin(wow xx-1 + b1)
9: else
10: b2 t- b,: W2 + Wi: t-1
11: end if// Now 0 S b2 2TT
12: if π < b2 I 2n then
b3 b2 m. W3 t W2: q3 1// sin(wow__e_
xe-1 + 6z)
sin(wowg xc-1 b2)
14: else
b2: W3 W2i03 +
16: end if// Now 0 < by S T.
i by // then
18: by + by; W4 t W3: f-1 1
Return: (ba, Wa: 94)
20: else
b by n; W4 < W3i Q4 f 1// sin(wow,] xe-1 + b4) sin(wowg xx-1 + bz)
22:
Return: (ba, W4.94)
23: end if// Now -n/2 4y S 7/2..
24: Output: New bias (i) t II by and new weight row We(i,) + II * ws.
// sin(I] g X - (b4 = II 4.94 sin x,-1 +bg) = ... sin w T Xx- +b)
equality holds only when both biases are equal to r/2, then the case qe(i) = 1, m 0 reduces to (i) = 1, m
Overall, by constraining the biases in the (-T/2, x/2 interval, these symmetries are eliminated. In all datasets considered, this constraint already satisfied. For completeness, provide Algorithm I that illustrates a method to shift biases in the desired interval.
A.5
Additional ablation studies
A.5.1 Distribution of Scaling coefficients
Empirically, accounting for given symmetry might not prove fruitful if the given dataset is by construction canonicalised, c.g. if all parameters were already positive or negative in the presence of sign symmetries. To ensure the occurrence of the studied symmetries within the used datasets, we explore their statistics. Regarding the positive-scaling symmetry, the distribution of norms of weights and biases within each network layer in the CIFAR-10 Concerning sign symmetries, we similarly plot the distribution of the sign values for the MNIST-INR dataset in Fig. 2 and the CIFAR-IO-GS-taa
dataset is shown in Fig. [
dataset in Fig. 3 Observe that in the former case, the distribution has relatively high variance, showcasing that the weights/biases are not by training Even more prominently, in the latter case, the probability of each sign value is close to 1/2 which corroborates our claims on the need for sign equivariance//ivariacee
A.6 Additional Considerations
In the following section, explore key questions raised by the reviewers during the evaluation process. These questions touch upon interesting aspects of our methodology that merit in-depth exploration.
Complexity and performance (runtime) degradation between ScaleGMN and ScaleGMN-B In the bidirectional case, add backward edges to graph and introduce backward message

---

giite layer
Misess layer 2
Masea leger
ghts
Meses
140
ghts
130
Figure I: Norm distribution on the weights and biases in the CIFAR--O-GS--RLL dataset.
waight byyer
ases layer
135
2M000
6.
layer
3800
2003
0800
3009
2800
Figure 2: Sign distribution of the weights and biases in the MNIST INR dataset.
25

---

100m
weggla laye
braace lggo
0:0
-120
-00
C30
4.00
-30
0.00
-120
4000
15000
Figure 3: Sign distribution of the weights and biases in the CLFAR-10-GS-ER dataset.
function discern the two message directions. Subsequently, concatenate the outputs of the two message functions and apply the UPD function. Consequently, the additional complexity of ScaleGMN-B B is introduced solely by the extra message function, with complexity of OEE). Given that ScaleGMN has the complexity of a standard GNN model O(V + E), the final complexity of ScaleGMN-B is O(V 2E).
Trainable parameters: GMN vs ScaleGMN(-B)? Going from a GMN model to ScaleGMN requires adapting the MSG and UPD functions to be scale equivariant, leading to more learnable parameters, as opposed to using plain MLPs. Another design choice of ScaleGMN that introduces more parameters using different MLPs for the I/O nodes (A.1.4).
Are there any limitations on the choice of activations the ScaleGMN network? Importantly, our method does not impose any limitations on the choice of the activation functions. We are able to select any activation function, because these are only applied within the MLPs of the invariant modules. As discussed in Section 5] the MLPs (equipped with non-linearities) are only applied after ammonicalizaioo/symmeriaiaaii
function. In case one chose to place activations in different computational part of ScaleGMN, this would indeed limit their options so not to compromise scale cquivariance. However, this is not the case in our method.
Canon/symm mappings for all activation functions. In general, we cannot guarantee that canon- intiratioo/smmmttiizzaaooo mappings are easy construct for any activation function, since it is currently unknown if we can provide general characterisation of the possible symmetries that may arise. Our results extend to all positively homogeneous activations, o (Ax) (z)) À > and all odd ones, of -r) - (z). We refer the reviewer to Table of Godfrey et al. 25), where we can see that LeakyReLU also falls the first category. Regarding polynomial activations, which demonstrate non-zero scaling symmetries, one option would be: (1) norm division to canonicalise scale, and (2) sign symm/canon as discussed this paper. The above, cover a fairly broad spectrum of common activation functions.
26

---

A.7
Deferred Proofs
A.7.1
Characterisation of symmetries
First we restate Proposition 4.1 and provided the detailed conditions for the activation function a.
Proposition A.1 (Proposition restated). Consider an activation funnctoio o IR R.
(Lemma 3.1. (25).) y If the matrix (La) is invertible, then, for any d E (1,2.... J. there exists
a (non-emptyy group of matrices, called the interbwiner group of da defined as follows:
la, = {A E E Rddd invertible JB E Raxd invertible, such that: o (Ax) Bxx())
and group homomorphism Do, Todd + GLa(R), such that B Φo,a(A) o (A)o
(Theorem E.14. (25))) Additionally, if is non-constant, non-linear and differentiable on dense open set with finite complement, then every A E lo is of the form PQ. where P: permutation matrix and Q diag( diag(q.....gg (91....9.) diagonal matrix, with Φo,a(A) Pdiag (fo.1(9)).... (9,1
Note that La is the identity matrix GL,(R) is the general linear group. The above two state- ments are proven in [25]. The exact form of ( is not precisely given, but is easy to derive: Do,d is group homomorphism and therefore (PQ) (P,a(P) Pdiag
Φ (qa)( where we also used Lemma E.10. from 25 which states that é = P for permutation matrices.
Proposition A.2 (restated). Hyperbolic tangent activation function o(z) tanh(x) and SIREN activation function, i.e. (xx) sin(wx), satisfy the conditions of Proposition [4. when (for the latter) * kT, k E Z. Additionally, Iod contains signed permutation matrices of the form PQ. with Q diag .d, 9: +1 and a =QQ.
Proof. Example 1: (z)) = sin(wx), with
constant:
This is the most common activation function for INRs (70). We can easily find the conditions under which o (Ia) is invertible invoking Lemma E.7. from Godfrey et al. [25]. In particular, the latter states that if or (1) * (0), (1) * -dd (0), then (Ia) is invertible.
In our case, (0) = sin(w0) = 0, ((1) = sin(w). Therefore, it should hold that w * Km, E Z, otherwise (La) is an all-zeros matrix. Additionally, o is non-constant, non-linear and differentiable everywhere. Thus, as dictated by Proposition 4.1 we can characterise the intertwiner group, by identifying the function (a)) In particular, b (a,1(a) if the following holds:
sin (wax) (sin = sin?(wax) = 62 (wz) = 1 - cwsz) = 6² sin (wx)
wa cos(waxz) = bw cos(war) + a² cos? = 1² cos (war),
where in the second line we took the derivative of each side since the equation holds for all E IR and divided by w + O. Now summing up the two equations we obtain:
1 + cos2 (war( 1) = 6² + cos² (wax)) (auax)((i (a² - 1) = 62 - o2 - 1
The above is true because a² + 1, then one would conclude that cos? (war)) constant. Thus, = 11 and b- = +1. Additionally a cannot be different than b, since in that case we would have e.g. sin (w:r) = sin(wx) + sin(war) = Therefore:
(a) = a, = (-1,1} and (PQ) PQ. PQ: : signed permutation matrices. (35)
Example (x) tan)
We can easily verify that (I2) is invertible invoking the same Lemma E.7. from Godfrey et al. 25 as before. We have, o (0) tanh(0) = 0, o(1) tanh(1) _ * 0. Additionally, is non-constant, non-linear and differentiable everywhere. Thus, again dictated by Proposition 4.1
27

---

will characterise the intertwiner group, by identifying the function (o)1 In particular, b - (0,1 the following holds:
tanh(ax) = b tanh(x) = tanh (ax) = b² tanh²(x)
tanh (ax)) = b(1 tanh (x))
where in the second line we took the derivative of each side since the equation holds for all i E IR. Now summing up the equations obtain:
+ tanh (ax) a) b (x) - b) tanh? b² +bb = b
= and b² b+ 1 0 and b² b and +1.
The above is true because b2 - a + then one would conclude that tanh 2 (ax) constant. Therefore:
00,1(0) =a, a= (-1,11 and (P.) PQ. PQ: signed permutation matrices. (36)
A.7.2 Theoretical analysis of ScaleGMN
Proposition A.3 (Proposition reasettte) ScaleGMN is permutation/scaale equivariant or permuta- tion/scale invariant when using readout with the same symmetries.
Proof. We will prove this by induction. In particular, consider two equivalent parameterisations 0', as in Eq. (3. This implies that vertex/edge features will have the same symmetries, i.e. starting with the hidden neurons:
x (t) ge (Ie (i) Xy (Te (i)), e layer (i) E L L- 1 -1
x=(i,j) ge (Tz X (T (i) mE- (7)) प. (Te-1 (f)), é layer (i) E ( L 1},
We will show that these symmetries are propagated to representations for every 2 i.e. that Eq. and Eq. (5). Let us start with the base f = O.
(1) INITV (Xy (1) (i) INITV (1e (TE (i)) xv (me (i)) pP
ge (Ie (i)) INITV (xV (IT (i)) PV (i) = qe (TT (i) h (me (i)),
h()()) INIT n (x'r ,p)(()))))
INIT & (qe (IT (i)) Xg (IT (i) Fe-1 (i)) 9-1 ( (7) (i.
de ( (i)) 0. (Te-1 ())) INIT X (Te (i) m ( (i.).)
ge (me (i) 9 (TE-1 (f)) bg (T (i), - (j))
28

---

This is due to the properties of the initialisation functions see Eq. (20). Now we proceed with the induction step. Suppose Eq. and Eq. ( hold for t 1. We will show this implies they hold for
m (1) JEN (
MSG (bv-'(), (G), h (i.j))
MSG
qe (Te (8)) hy ( (i)) 9--1 (Te-1 (i)) by (TT-1 (i)),
j (c)
(TE (i)) h (m (i) Te- (i)) g6-1 (TT- (i))
(e) (Te (i)) MSG (hi (I (1)) by (Te-1 (i)) big (Te (i) Te (i1)
qe (Te (i))
(E MSG (hi (TH (i)) by ()), h' (IT (i).
qe (m (i)) my (Te (i)).
hy (1) UPD (h/ (i), (i) = UPD (ge (Te (1)) by (me
(mee (t)), qe (Te (i)) m (Te (i))
qr. (Ty (i)) UPD (by (m (mp (m), (1)) = q (मx (i)) (i)
h) UPD (hy (
(0) (i.5)
h/ (i),
= -UD''s (a (Te (i)) (0-) (Te-1 (j)) by (Te-1 (i))
(i))
(Te (i)) h' (Te (i) ne- (i)) q (T (i)) (me
qe (Ie (1)) 9- (me-1 (1) UPD (bl ( (1)) h (Te-1 (i), (me Te-1 (i))
= qe (Te (1)) - (m (7) h'g (TT (i), ; (j)
Therefore we have shown the desideratum for hidden neurons, where again we used the properties of the message and update functions (we omitted positional encodings here for brevity). Now for the input neurons, we have:
(1) = (v),,, { = layer (i) =
(1, = de (Fe (i)) X (T (i), (), layer (i) = 1,
Further, we perform the induction again for Eq. (7 and Eq. 28 hold. Base case f = 0.
(1) INIT v.O (xy (1), Pv (oppv(i)) (i)) INIT v.0 xv((), pv(i))
I (x) hy (i).
Induction step:
(i) 0 (i).
(i) = UPD (h-'(() m)) = UPD (b)'((0, ()) = (t),
since input vertices do not have any incoming edge. Finally, for the output neurons, have:
(i) (t)..
layer (i)
x = 1 ( (i)) Xx (i, TE-1 (i)), l layer (1) T.,
Induction for Eq. 27 and Eq. (9) Base case = O.
(1) = INIT (xv (1),v()) INITV,A (x)())
INITVA (f). = h& (f),
h) INIT (X'E
ENITE, E.L (Tr1) (TC-1 (i)) xg (i, T (f)) .PE(i.)))
= =qr² (TT-- (i)) INIT F. XF (1, TE-1 (I)),p(((,))
= (TT-1 (i)) bg TE- (i)).
29

---

And the induction step:
my (i)
MSG (1)) / (j), (1.
(e
MSG (h- (0) by (Te-1 (i))
h (())
MSGV h (1). (
b' (1, (1 (1) (1))
)EN
MSG. (b+ (i). h, (1) h (1.5)
jENim(())
(i).
(i) UPD (hy (1). (i) = = (n (i), (i), mi) h²)
h () UPD (hy (hv- (1),
=UPDY (b (1), 91-1 (TT-1 (j)) hy (Te-1 (j))
h (1, M--1 (f)) 9--1 (me-1 (f))
- -1 (me- (i)) UPD (b)-(()) - (j)), h'z (1,me- (i))
- (me-1 h/s (i, me-1 (j))
Again used the properties of the message and update functions defined in Appendix A.1.4 And with that, we conclude the proof.
Theorem A.4 (restated). Consider a FFNN as per Eq. ① with activation functions respecting the Sittirettionl-SSaaeeiiee
conditions of Proposition 4.7 Assume can express the activation functions ag and their derivatives o'c- Further, assume that ScaleGMN has access to the inputs Xa and the gradients of (optional) loss function L w.r.t. to the output Vx. f. via its positional encodings. Then, ScaleGMN can simulate both a forward and a backward pass of the FFNN, by storing pre-activations, post-activations and their gradients at the vertex representations. In particular, to compute the forward pass at the FFNN layer E, 1 2 ScaleGMN layers are required, while for the corresponding backward, the requirement is L 2 2L - l.
with vertex update functions that
Proof. Consider a loss function L Res x Rela + R. which is computed on the output of the FFNN C(xL, :). The second argument is optional and is used when we have labelled data - we will omit in our derivations for brevity. Our proof strategy is based on two observations:
First, the pre-activations of each layer (denoted here as Z Wxx11 + by) have the same symmetries as the biases. This is straightforward to see by induction:
2 = Xo = Zo: X0= = =Xo
(base case 1)
2 A W 120 + Aibi AIZi, xi =0=o((A = do, (A1)x1
(base case 2)
2/ A Woor (A7-1)x(11
Arb, = AoWroo (At-))AAAA))))1111 + Arbs (induction step)
Aizis X =00 (Ae)x
Second, the gradients of w.r.t. the pre-activations of each layer (denoted here as Ve, Cxxr)
Vzr-i L(x1), where the dependence of x is implied sometimes we will omit the argument xL for brevity have the inverted + transpose symmetries of the biases. This is due to the invariance of the output to the symmetries considered:
Smell
x (o4) = Xx. (ze)
> L (xL. (x() = L (2x))
(x (z Ve, (xL (zz))
Jelyy Vzz C (xL (z()) Ve, (zz))
Ozi
AJ Vzg L (xL (z))) = T (zx)) = Volf (AT) 'Va,
30

---

and similarly Vx,C= (Ae) 'v,L.
Now recall that in the cases we consider in this work, Oo, (A) = A and A = PQ. Therefore, given that (P (P))- P and (Q) Q-1, we have:
= P.Q.ze
= P V&LC-
P. Vx,
(37)
The latter is what motivates us to enforce vertex representations to have the same symmetries with the biases: Assuming sufficient expressive power of ScaleGMN, we can anticipate that vertex representations will be able to reconstruct (1) the pre- post-activations and (2) the elementwiss-inverse of the pre- or post-activation gradients.
Now recall the formulas that we want to recover:
(i)
Σ jEN
Weli.j)xe-(() br(i)=
E JENF(
-eli))oo- be(1)
x((1) (ze (1)) de Σ We(t,))xe bb(1)
(ENE(()
V 2=((1) ( (a2(41) T Vzeni c)(0) d((z((i)
Σ jENe(1
V
Ozi
Vx (1 (QUet1)
()()==
E V jENm(()
dxi
Now let us proceed the proof. We will show that there exists parameterization of ScaleGMN, such that after f = l layers of message-passing, all vertices with layer (i) < l will have stored in part of their representation the pre-/post-activation for one or more inputs to the datapoint NN. Additionally, we will show that after f 2L l layers of message- message-passing, all vertices i with layer (i) 2 & will have stored in part of their representation the pre-ipost-activation gradients for one or more inputs to the datapoint NN. Therefore, L layers of (forward) message-passing are needed to compute the output of the datapoint NN, and 2L layers of (bidirectional) message passing to calculate the gradients of the output/loss w.r.t. the input.
For reasons that will become clearer as we proceed with the proof, we construct the below pa- rameterization of ScaleGMN, First off, recall the input vertex and edge representations are given by:
layer (i) = =0, (iayer( (pos (i)), layer (1) E [L.,
XV (i)
Xg xE(i.)) W (s) (pos (i), pos (j)),i> XE(j. i) Ø W myyert) (pos (j) pos (i)), >
Then, we use the vertex positional encodings to store the information of the imput values and the gradients w.r. the outputs:
Zo (pos (i)) Xo (pos (i)) layer (i) 0
(i
0. 0), layer (i) [L L-1]
(V (Ve (postt)) -)- (V (pos(t)) Cl) layer (i) L
Throughout the network, the vertex representations will be 5-dimensional: will use relay its neuron's 's bias across ScaleGMN's 's layers, two for the pre-and post- post-activations and two for their respective gradients. One can repeat the process for m inputs, which case the dimension will be m + 1. The edge representations will be 1-dimensional and will be simply used to relay the NN's weights throughout the network.
Now, let us define the vertex initialisation functions.
INITv: (x,P.) P..)) = MIP (x, p) (x[1), P. [1), Pa 2,
INIT V.L. (x, P..) = MLP p..) = [x[1], 0, P.. (1) P. []]
INITy (x.p) = AugSSaleEq (x. = 0x11)000.000
31

---

Regarding the first two functions, it is obvious that can be expressed by an MLP. Regarding the third one, it can be expressed by AugScaleEq as follows: AugScaleEq(xx:)) p) r @ AugScaleinv(x, p) 0,, [ o [L, 1, 1, 1,", where the the constant vector can be expressed by AugScalelnv since it is invariant to the first argument. Further, for all edge initialisation and update functions (forward and backward):
INIT (e, (c) c,
(x.,,)
which can both be expressed by AugScaleEq using the identity as linear transform and an AugScalelnv with constant output.
We will not use positional encodings in the message and update functions, therefore we use their definitions of Eq. Eq. In particular, the forward/backward message functions:
MSG V.W y, e) = ScaleEq (xx, ReScaleEq (y, e)l) = =y[3] ee[1]
MSG X, e, = Im (y(4) e(1]),
where g(.) = ScaleEq (-) an arbitrarily expressive scale equivariant function. In the first case, this (since it's unnecessary), identity transform on the output of ReScaleEq and a Scalelnv with constant output (ones). Then what remains is ReScaleEq (y, e) fyy @ e [0, 0, 1,0,,,0,,0 O, 0] y e[[] = y e([].
can be expressed as follows: zero linear transform
Finally, the vertex update functions:
UPD (x, mpw, mgw) = MIP(X, mpw, mgw)
[[]],xx2],,xx (3)) (c)( gu (mpw gu (mgw())) [4))
UPD (x, mew, mpw) MLP((x mrw, mmw)
[x[1]. x11] mpw [1), oz (x[1) + mrw (1]) x(4], x[5]
UPD (x, mpw, mgw) ScaleEq mpw, mp, mgw))
[x[1], x [1] + mpw [4], o (x[1] mfw mm)
(o'(x(2) 1 gu (mgw [1)), gu (mgw[1])]
where gu (caleEq()) an arbitrarily expressive scale equivariant function. The first two can obviously be expressed by an MLP (note also that usually do, Jr are the identity). Regarding the third, first, we assumed a common activation function across hidden layers. In case the activation is not shared, a different update function should be used per layer. This is because, as we will see below, the activation and its derivative need be learned for ScaleGMN to simulate the forward and the backward passes. Now, let's see how its element can be expressed by ScaleEq. Recall that (caleEq(a) Ta @ Scalelnv(a). Here, E 5 (7 inputs. 5 outputs). The first element (which will be the bias), the second (pre-activation) can be expressed by a Scalelnv with constant output (ones) and (1) an all-zero vector except for the /st or the /st and 6th coordinates respectively.
The third (post- prost-cttvattion,..oe fourth (pre-activation gradient) and the fifth (post-activation gradient) elements not straightforward to simulate with elementary operations. However, one observe that all are scale equivariant. In particular, the third: a (gx + qu) = qo (x + 2) (by assumption about the symmetries of the activation function). Additionally, the fourth (assuming a scale equivariant go') ): T: this is because the derivative of a scale equivariant activation function is scale (2)
gu (qw)
(q
invariant The fifth is by definition scale equivariant. Therefore, if ScaleEq is sufficiently expressive,
so as to contain all scale equivariant functions, then it can express the third to fifth elements.
It remains to establish few last statements that will be necessary in our induction. First, the edge representations are constant throughout ScaleGMN and equal to the weights of the input NN:
(i,5) = INIT E.+ (XE
= xg (i. = W (iyer(t) (pos (i),
pos (j))
(i,j) UPD (1), h (i), (i.j) -h
(i,)) b
-... (i.
(gr)) (z( - go'(gz) = go (④) - d' (qz) o' (x) for non-zero scalar multipliers.
32

---

Second, the first element of the vertex representations is constant throughout ScaleGMN and equal to the biases of the input NN:
1, layer (i) = 0, (iayer(t) (pos layer (i) E [L].
h (+]1] = INITV v.x (xv (r),P(()) xx =
bf (0[1)) = - (hv ( (i), Fw (i), mi) = -h (1[1 -.- (1)[4)
Now, our first induction hypothesis is the following: for all vertices with layer (i) and 2 é we have that (1) 12: 3] Zelpos (t)), X (pos (i))), then the same should hold for vertices with layer (i) = f + 1 when 2 E + 1.
(Base case:) If layer (i) 0:
(1)(2: : 3] UPD (bt mm :3
= ( 3] = (i)(: :]
INITv;0 (xv (i) pv(1)) [2 3]
2. (pos (1) Xy (pos (i))
Therefore, the hypothesis holds for 2 0 = layer (i) for the base case.
(Induction step:) Suppose the hypothesis holds for vertices with layer (1) = L. Now, if layer (i) = l, for 2 l we have:
mf rw (i)
MSG (n (n)''1), (1), by " ((.j))
().
jEN
Σ JENz(0) by (i)(3) h [1, (layer (j) = l - 1,t 12
Σ SENT(O Xx-1 (pos (j)) Wpos pos
pos (j))
hy (1)[2: : UPD1 (ht='(),, (2))
= [h (1) + mv.w (i), a (h (1)11] + mw (1))
= [poss (i)) Σ (eNn(() xe- (pos (j)) Wp pos pos (j)),
0g (h// (i)[1] + m w. (i))
=iz (pos (i)) oe (ze (pos (i))) [ze (pos (i)) x (pos (i))).
Similarly for the vertices with layer (i)
Our second induction hypothesis is the following: for all verfices with layer (i) and 1. 2L - 6 (p) then the same should hold for
have that hiy((i)) (i)[4 5] = [(V as (por(())) ()- (V vertices with layer (i) - 1 when 2 2L f+1.
(Base case:) If layer (i) L:
(t)[4 UPD (hy (i). (1)
-h)- 5]
(1)(44 5]
= -INITv:. (xv (i) (i)) [4: 5]
= -[Ven, C) (V (p)s(t) L) '1
Therefore, the hypothesis holds for 2 L = layer (i) for the base case.
33

---

(Induction step:) Suppose the hypothesis holds for vertices with layer (1) = E + 1. Now, if
layer (i) f. for t 2 2L L have:
m (i)
(EN MSG
(1.5)
Σ jENo(1) Im (hg (1144) (t,)))) (t- 12 2L (e+ 1))
Σ (EN Im ((V (p+())))))) 2)
We (pos (i) poss (i))).
(1)(: 5] UPD (b) (f)))(((0)) 5]
(1),
gv (m v.Bw (i)
Su (miz
o' (hl (1)
d'(ze(pos (i)) (pms(()) L' (V (Vxe(ro(())) L)
[(V ne (pon( (i)) ())) (Vx (pos(())) -)
In the last step, we assumed the following: gu E Im (x)) sufficiently expressive scale equivariant and permutation invariant function. This is in order to express the following:
(V ()) (EjeNu(s) 2e (pos()))
We(pos (j), pos (i))
with:
(( C. ge+1 (j) (e
We(pos (i), pos (1))
gr(1)
pos Veeti (pen(())) C We(pos pos, (i))
(i) x; (pos(e)) C)
Therefore all gu is the inverse of the gradient.
Σ Im (qx)) q Iu 2 9m (x)) can be expressed by Iu' Im then so
Following similar rationale we can prove the case for the vertices with layer (i)
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main contribution of this paper is the design and evaluation of an NN pro- cessing architecture (metanetwork) that accounts not only for NN permutation symmetries but also scaling. The abstract and introduction introduce the reader to scaling symmetries for certain NN architectures (FFNNs and CNNs) and explain our main technical innovations to achieve our goal. The importance of scaling symmetries as inductive bias is showcased in theoretical and empirical results, explained in the abstract and introduction. Guidelines:
Justification:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Yes, the limitations of our work (applicable input NN layers/architectures and open questions regarding expressivity) are discussed thoroughly in the last paragraph before the conclusion.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
35

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions, precise statements and detailed proofs of our theoretical results are provided and presented thoroughly in Appendix A.7.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe our architecture in detail and all the details needed to fully reproduce our results are provided in Section Appendix [nd Appendix A.4 Addition- ally, our code, including used configurations, and modified datasets will be made publicly available at later stage.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
36

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The datasets are already public (we simply split two datasets into datasets based on the contained activation functions). The code will be made publicly available at later stage after all the required steps are taken for proper documentation to make it easy to use for the interested user.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
ponnooollyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips.
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the training and testing procedure carefully described in the Ap- pendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
37

---

Justification:
Yes, each method we ran (including baselines) was tested for 3 different random seeds and the mean and standard deviation were reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Yes, provide sufficient technical details in Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///eeerreee.//////////caaa
Answer: [NA]
Justification: Yes, we do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
38

---

Answer: [NA]
Justification: We do not foresee and negative societal impact and therefore it is not discussed in the manuscript.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: We do not use existing assets. Regarding code and data, all works were properly cited.
Guidelines:
The answer NA means that the paper does not existing assets.
39

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
40

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
41


Reviews:
[{"review_id": "Y73sOp5AU6", "reviewer": "Reviewer_AXwX", "summary": "This work develops new GNN-based metanetworks that are equivariant to scaling symmetries induced by nonlinearities in input neural networks. Their ScaleGMNs extend metanetworks, which are typically only permutation equivariant (if at all equivariant), to also account for other symmetries in input neural networks' parameters. The architecture is proved to be equivariant to the desired symmetries and also expressive in that it can simulate forward and backward passes of the input. Experiments show improvements over merely-permutation-equivariant metanetworks.", "questions": "1. Do you have a way to handle translation symmetries in nonlinearities?", "limitations": "Limitations are discussed on Page 9", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Great writing. Nice introduction and related work, as well as good setup and notation in Section 3.\n2. Nice theoretical results. That ScaleGMN is can express the forward and backward pass is a good way to check that its expressive power is not overly limited when adding the additional scaling equivariances. Also, there is an interesting discussion in Appendix A.2 on equivariant for bidirectional ScaleGMNs.\n3. Large empirical improvements, especially on INR classification (without many of the unfair or expensive tricks that others use!), with ScaleGMN. I say some previous tricks are \"unfair\" because, for instance, random probe features of Kofinas et al. as used on INRs can essentially be used to see the input image pixels, and hence the prediction task is also taking in the image as well as the INR representing it. That ScaleGMN can beat the prior methods without these tricks is very impressive.\n4. Several other interesting empirical findings. These include the fact that ScaleGMN does not need random Fourier features or data augmentations, and that the bidirectional version can vary in performance (sometimes drastically as in augmented CIFAR-10).", "weaknesses": "1. It would be interesting to see how ScaleGMNs perform on different tasks, especially an equivariant task (rather than just invariant tasks) such as INR editing.\n2. ScaleInv is oddly described. The equation on Page 6 should probably have $\\tilde x_i$ or something similar as its arguments, instead of $x_i$ (because if the $\\rho^k$ are just general MLPs as you say right after the equation, then this is not scale invariant).\n3. At the end of Page 6 and beginning of Page 7, you say that sign canonicalization can only be used for dimension 1, but this is not quite true. In Ma et al. 2023 and Ma et al. 2024, algorithms are given for canonicalizing with respect to the sign group, for use on inputs to a neural network.\n\nReferences  \n* [Ma et al. 2023] Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embedding. https://arxiv.org/abs/2310.18716\n* [Ma et al. 2024] A Canonization Perspective on Invariant and Equivariant Learning.\n https://arxiv.org/abs/2405.18378", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n>*Weakness 1. It would be interesting to see how ScaleGMNs perform on different tasks, especially an equivariant task (rather than just invariant tasks) such as INR editing.*\n\n\nMetanetworks can indeed find interesting applications that require our model to be permutation and scale equivariant. Please refer to the global response for our experiments on INR editting and the corresponding results in Table 1 of the attached PDF. The bidirectional variant of our method, ScaleGMN-B, is able to surpass all the GNN-based metanetworks, even the ones using probe features. \n\n---\n\n>*Weakness 2. ScaleInv is oddly described. The equation on Page 6 should probably have or something similar as its arguments, instead of (because if the are just general MLPs as you say right after the equation, then this is not scale invariant).*\n\nThank you for spotting this. Indeed, there is a typo in the definition of ScaleInv (below L265). The arguments of the function $\\rho^k$ (universal approximators - MLPs) should have been $\\tilde{\\mathbf{x}}_i$ instead of $\\mathbf{x}_i$, where $\\tilde{\\mathbf{x}}_i$ are explained later in the text (L277) and are the outputs of a canonicalisation or a symmetrisation function, i.e. $\\tilde{\\mathbf{x}}_i = \\text{canon}(\\mathbf{x}_i)$ or $\\tilde{\\mathbf{x}}_i = \\text{symm}(\\mathbf{x}_i)$, which ensures invariance. We will fix this in an updated version of the manuscript.\n\n---\n\n>*Weakness 3. At the end of Page 6 and beginning of Page 7, you say that sign canonicalization can only be used for dimension 1, but this is not quite true. In Ma et al. 2023 and Ma et al. 2024, algorithms are given for canonicalizing with respect to the sign group, for use on inputs to a neural network.*\n\nWe are grateful to the reviewer for bringing up these two recent references that deal with sign canonicalisation - we were not aware of these works and, indeed, they are useful for our setup. Given the fact that symmetrisation introduces additional parameters (i.e. the internal MLP, see L276), we are interested in conducting experiments in the future with the proposed canonicalisation method, so as to examine if similar performance can be achieved with a reduced parameter count. Additionally, we will update our text accordingly to complement our discussion with this missing point.\n\n\n### Questions\n\n>*Question 1. Do you have a way to handle translation symmetries in nonlinearities?*\n\n\nTranslation symmetries (such as those induced  by the softmax activation)  are an important next step in this research direction. Our method currently does not handle this case. A potentially straightforward modification might be to follow the same rationale with scale equivariant networks: first, define a translation invariant module via canonicalisation and second, use it to achieve equivariace (e.g.  translate the input by the output of the invariant module). For example, for symmetries of the form $\\mathbf{x}’ =  \\mathbf{x} + a$, where $a$ is a scalar, we can canonicalise as follows  $\\tilde{\\mathbf{x}} =  \\mathbf{x} - \\frac{1}{N} \\sum_{i=1}^N x_i $. \n\nWe believe that characterizing even more nonlinearities (or families of them) and designing the respective invariant modules, is a prosperous future work towards implementing a unified framework able to handle various types of networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QdPLQcrGD5", "reviewer": "Reviewer_x2pG", "summary": "This paper addresses the emerging and fascinating field of deep-weight space learning where neural nets used to process weights and biases of another deep model. The authors have introduced new methods based on GNN architecture called ScaleGMN and ScaleGMNB for Scale Equivariant Graph MetaNetworks. The latter is a bi-directional variant. Both approaches tackle the scale symmetries presented by the input neural model's activation functions.\n\nThe authors claim the following contributions:\n- Extending the scope of metanetwork design from permutation to scaling symmetries.\n- Designing networks that are invariant or equivariant to scalar multiplication from arbitrary scaling groups.\n- Theoretical analysis of the expressive power of ScaleGMN. \n- Extensive empirical comparison with recent work on various datasets in the field of weight space learning.", "questions": "- Why DWS and INR2VEC are missing in Table 1? (CIFAR-10 | Augmented CIFAR-10 experiments).\n- In common bi-directional architectures (e.g. LSTM) we see performance (runtime) degradation compared to non-bi-directional design is that the same for ScaleGMN and ScaleGMNB? What is the computational complexity of both methods?\n- Can ScaleGMN and ScaleGMNB handle input data with heterogeneous activation functions, i.e. one network with ReLU activations and another with only tanh activations? (as long as they respect Prop. 4.1)\n- The results for the Augmented CIFAR-10 experiments are odd. ScaleGMNB performs worse than all baselines except MLP, while in other experiments, it outperforms them. Do the authors have an explanation for this observation? \n- Adding a figure that illustrates ScaleGMN and ScaleGMNB architectures and their design w.r.t permutation and scale symmetries, would be beneficial.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- With the growing number of works on Implicit Neural Representation (INR) and the increasing need to process neural networks, this paper tackles a crucial field and problem, advancing the area in a way that can benefit many practitioners.\n- The authors introduce a new structure that incorporates both permutation and scale symmetries by ensuring it is equivariant to scale and permutation.\n- The authors provide a theoretical analysis of the expressive power of the proposed approaches. Additionally, they show that ScaleGMN can simulate forward and backward passes of arbitrary inputs in the processed NN.\n- The empirical results show a significant improvement compared to recent works in the field of weight space learning.", "weaknesses": "- The writing can be significantly improved, particularly by breaking down long sentences that are hard to follow. Additionally, the writing pace is somewhat slow. While this might be beneficial for the average reader, the authors allocate too much space to exposition and problem formulation. As a result, the presentation of the proposed methods begins relatively late in the paper.\n- The experimental section focuses only on invariant tasks, i.e. INR classification and NN generalization prediction. It would be interesting to see how well ScaleGMN and ScaleGMNB deal with equivariant tasks like the ones presented in [1,2] which are considered harder for weight space architectures.\n- The processed architectures are not diverse dealing only with small-sized feed-forward and CNN architectures. It would be interesting to see more diversity in the processed architectures like deeper nets, attention-based methods, etc.\n \n-------\n[1] Equivariant Architectures for Learning in Deep Weight Spaces, Navon et al.\n\n[2] Permutation Equivariant Neural Functionals, Zhou et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses\n\n>*Weakness 1. Writing Style*\n\nWe thank the reviewer for their suggestions. Please refer to the global response about the writing pace and style.\n\n>*Weakness 2. Equivariant tasks*\n\nTo evaluate the performance of our method on tasks that require permutation and scale equivariance, we selected the task of INR editing. Please refer to our global response and to Table 1 of the attached PDF. In summary, the bidirectional variant, ScaleGMN-B, surpasses all baseline metanetworks, including GNNs using additional information (probe features)\n    \n>*Weakness 3. Processing diverse and complex architectures*\n\nPlease refer to our global response regarding the experiments on larger and more complex architectures.\n\n### Questions\n>*Question 1.  DWS and INR2VEC on CIFAR-10*\n\n\nIn Table 1 of our paper we include, together with our results, all the baselines from the literature. For the CIFAR-10 dataset we did not include any experiments with DWS and INR2VEC, as they were not included in [1], where the task was proposed. However, we have now run both experiments using the **DWS** model, which achieved $34.45 \\pm 0.42$ on CIFAR-10 and $41.27 \\pm 0.026$ on Augmented CIFAR-10. As expected, these results are on par with the rest of the permutation equivariant models. \n\nRegarding INR2VEC [2], we have not reimplemented it and tested on the CIFAR-10 dataset - we only used results present in the literature. However, INR2VEC demonstrated significantly worse performance than DWS [3] and NFN [1] on the task of INR classification, as can be found in the literature, and therefore testing it on extra data will probably offer limited added value, as it. Nevertheless, we plan to also include this baseline in an updated version.\n\n\n>*Question 2. Performance (runtime) degradation between ScaleGMN and ScaleGMNB and complexity*\n\n\nThe reviewer here makes a correct comment regarding the complexity of the forward (ScaleGMN) and the bidirectional variant (ScaleGMN-B) of our method. In the latter case, we add *backward edges* to our graph and introduce a *backward message function* to discern the two message directions. Subsequently, we  concatenate the outputs of the two message functions and apply the UPD function. Consequently, the additional complexity of ScaleGMN-B is introduced solely by the extra message function, with a complexity of $O(E)$. Given that ScaleGMN has the complexity of a standard GNN model $O(V+E)$, the final complexity of ScaleGMN-B is $O(V+2E)$. \n\nTo measure the differences regarding the runtime performance, we conduct a controlled experiment on the F-MNIST dataset. We observe that ScaleGMN-B only needs $6.83$ more seconds per epoch, when compared to ScaleGMN. Regarding the inference time this difference is $0,0132$ seconds per datapoint. Please refer to our response to reviewer ugM4 as well as to Table 2 in the PDF within our global response for the complete results.\n\n\n>*Question 3. Heterogeneous activation functions.*\n\n\nThis is an interesting question and of importance to ensure the generality of our method. *In principle, our method does not impose any limitations regarding the homogeneity of the activation functions of the input neural networks*. To see this, observe that the only part of the metanetwork that gets affected by the symmetry induced by the activation function, is the *symmetrisation/canonicalisation* function. In other words, all the modules of the metanetwork can be reused for any input NN with arbitrary activations, as long as the datapoints are symmetrised/canonicalised accordingly. \n\n**Experiments**. Experimentally, we opted to split the datasets into two subsets (one with ReLU Nets and one with tanh Nets). This choice was made solely to evaluate our method separately for each type of symmetry and assess the different invariant methods that we employed for each case. Following, the reviewer's suggestion, we extend our evaluation to heterogeneous activation functions (a dataset containing both ReLU and tanh Nets). We conducted epxeriments on the CIFAR-10-GS dataset and report the results on Table 4 in the attached PDF of our global response. The baselines are reported as in [4]. **Interestingly, we observe that ScaleGMN demonstrates superior performance compared to the previous baselines, significantly exceeding the performance of the next best model.** We thank the reviewer for suggesting this experiment - we will include this in the updated version of the manuscript.\n    \n\n>*Question 4. The results of ScaleGMN-B on Augmented CIFAR-10.*\n\n\nThis was indeed a confusing result, which was merely due to suboptimal hyperparameter search. Unfortunately, due to the large size of this dataset ($20$ times larger than \"CIFAR-10\") and limited computational resources, the hyperparameter search for the ScaleGMN-B on the Augmented CIFAR-10 dataset had not finished by the time of the submission. Hence, the reported result does not reflect the real performance of the model. We completed the hyperparameter search post-submission, and achieved accuracy equal to **$56.95 \\pm 0.57$** - this result follows the same pattern with the rest of the datasets.\n\n\n>*Question 5. Figure of ScaleGMN and ScaleGMNB.*\n\nWe thank the reviewer for suggesting to include a figure depicting our architectures. We will consider designing one and including it in an updated version of the manuscript.\n\n---\n[1] Zhou, Allan, et al. \"Permutation equivariant neural functionals.\" Advances in neural information processing systems 36 (2024).\n\n[2] De Luigi, Luca, et al. \"Deep Learning on Implicit Neural Representations of Shapes.\" The Eleventh International Conference on Learning Representations.\n\n[3] Navon, Aviv, et al. \"Equivariant architectures for learning in deep weight spaces.\" International Conference on Machine Learning. PMLR, 2023.\n\n[4] Kofinas, Miltiadis, et al. \"Graph Neural Networks for Learning Equivariant Representations of Neural Networks.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DUkRTaQP2R", "reviewer": "Reviewer_ugM4", "summary": "The paper considers the emergent and fascinating field of learning over weight spaces, that is, neural nets that process other neural networks. The processing NN is referred to as a metanetwork. Previous approaches showcased the importance of accounting for the input NN’s symmetries by designing equivariant architectures. However, they were mainly focused on permutation symmetries. This paper proposes a GNN-based metanetwork, which is permutation and scale equivariant. The paper studies the expressiveness (in terms of simulating the forward and backward pass of the input NN) of the proposed arch. The proposed method is evaluated using several INRs and classifiers datasets.", "questions": "1. Is it always feasible and relatively easy to design and implement either the canon or symm mappings for all activation functions?\n2. The bidirectional version of the method achieves on-par performance as the non-bidirectional one, except for the Augmented CIFAR-10 dataset, where the performance is much worse. Could you provide some insights regarding this result?\n3. Additionally, how does the ScaleGMNB and ScaleGMN compare in terms of runtime?\n4. How do ScaleGMN and ScaleGMNB compare to GMN in terms of the number of trainable parameters and runtime?\n5. Did you use any augmentation on the input NNs?\n6. Are there any limitations on the choice of activations of the ScaleGMN network?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper deals with an important and timely problem of learning in deep weight spaces, and presents a novel architecture by incorporating scale and permutation equivariance. \n2. The paper provides theoretical analysis and results regarding the expressive power of the proposed approach.\n3. Empirical results show significant improvement over baseline methods.", "weaknesses": "My main concern is the limited empirical evaluation and missing natural baselines. While the presented empirical results show significant improvement over baseline methods, at the current state of the learning over weight spaces literature, I would expect a more diverse, challenging, and comprehensive empirical evaluation.\n\n1. The writing and formatting require enhancement and refinement—specifically, long sentences, many slashes, many footnotes, etc. Also, in my view, the proposed method is introduced too late in the paper (page 6).\n2. A missing natural baseline is to use a permutation equivariant baseline like DWS/NFN or NG-GNN together with scaling data augmentations as in [1].\n3. The experimental section only considers invariant tasks. Additional experiments using equivariant tasks (e.g. INR editing, domain adaptation, etc.) would significantly strengthen the empirical study.\n4. Some evaluation and comparison of runtime and memory consumption w.r.t. baselines would be beneficial.\n5. Furthermore, adding experiments with larger input networks and diverse input architectures (like a varying number of layers) would again significantly strengthen the empirical study.\n6. Also, adding some ablation regarding design choices would be beneficial.\n7. Why are the results for DWS and INR2Vec missing in Table 1?\n8. Minor:\n   - Line 198: should be a^k.\n   - Line 336: “transformations We” -> “transformations. We” \n\nReferences:\n\n[1] Improved Generalization of Weight Space Networks via Augmentations, ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*For weaknesses: 1,3,4,5,7 and question 3 please refer to Comment.*\n\n### Weaknesses\n \n>*Weakness 2. Scaling data augmentations*\n\nAn effective method to get such augmentations is to sample scaling matrices for every training datapoint  - diagonal sign matrices for sign and diagonal positive scaling matrices for positive scale symmetries, respectively. Subsequently, we transform the input network parameters by applying the matrices to the hidden layers. (like in Eq.3 in our paper, but without the permutation matrices)\n\nThe way to perform this random sampling, however, is not straightforward and we have to deal the two cases separately.\n\n**Sign symmetries**: Since the diagonal sign matrices are of the form $\\mathbf{Q} = \\text{diag}(q_1 , \\dots , q_d ), q_i = ±1$, we sample each element of the matrix **independently and uniformly at random with probability 0.5**. We observe that augmenting the training set leads consistently to better results when compared to the original baselines. **None of these methods however achieved results on par with ScaleGMN and ScaleGMN-B.** (Table 3 in the PDF)\n    \n**Positive-scale symmetries**: The diagonal positive scaling matrices are of the form: $\\mathbf{Q} = \\text{diag}(q_1 , \\dots , q_d ), q_i \\in (0, +\\infty)$. Hence, we have to sample from a **continuous and unbounded distribution** making the augmentation strategy much more difficult to design. Consulting the plots in the appendix A.5, we opted for an exponential distribution and experiment with the coefficient $\\lambda$. Nevertheless, regardless of the distribution choice we cannot guarantee that the augmentations will be sufficient, due to the lack of upper bound. We observe that we were not able to surpass the original baselines, which indicates that designing an effective baseline of this type is not straightforward. (Table 3 in the PDF)\n\nWe thank the reviewer for proposing this important baseline.\n    \n### Questions\n\n>*Question 1. Canon/symm mappings for all activation functions.*\n\nIn general, we cannot guarantee that these mappings are easy to construct for *any* activation function, since it is currently unknown if we can provide a general characterisation of the possible symmetries that may arise. Our results  extend to all positively homogeneous activations, $\\sigma(\\lambda x) = \\lambda \\sigma(x), \\lambda > 0$, and all odd ones, $\\sigma(-x) = - \\sigma(x)$. We refer the reviewer to Table 1 of [1], where we can see that LeakyReLU also falls in the first category. Regarding polynomial activations, which demonstrate *non-zero scaling symmetries*, one option would be: (1) norm division to canonicalise scale, and (2) sign symm/canon as discussed in the main paper (and the rebuttal). The above, cover a fairly broad spectrum of common activation functions.\n    \n    \n>*Question 2. Forward vs Bidirectional and Augmented CIFAR-10.*\n\nIndeed, in most invariant tasks, the bidirectional variant does not provide significant advantages. This is inline with what has been observed in [2], where a forward variant is also sufficient. We speculate that this might be related to the fact that the tasks we considered may be solved by simulating the forward pass of the input NN alone. Given that the forward pass can be expressed by the forward variant alone, this might provide an explanation for its efficacy on the invariant tasks we considered. The significance of the bidirectional variant is highlighted mostly on  *equivariant* tasks, where ScaleGMN-B significantly outperforms the forward variant. Please refer to the global response, where we report our results on INR-editing and discuss the limitations of the forward variant on the equivariant tasks.\n\nPlease refer to our response to Question 4 of reviewer x2pG for ScaleGMN-B on the Augmented CIFAR-10.\n\n\n>*Question 4. Trainable parameters: GMN vs ScaleGMN(-B)?*\n\nGoing from a GMN model to ScaleGMN requires adapting the MSG and UPD functions to be scale equivariant, leading to more learnable parameters, as opposed to using plain MLPs. Another design choice of ScaleGMN that introduces more parameters is using different MLPs for the I/O nodes (Appendix A.1.4).\n\nPlease refer to Table 2, where we report the training and inference runtimes.\n\n\n>*Question 5. Did you use any augmentation on the input NNs?*\n*No, we do not use any augmentation on the input NNs.*\n\nAugmentations are only used for the dataset \"Augmented CIFAR-10\", where we follow the augmentation procedure from [3] for comparison reasons with the rest of the baselines. ScaleGMN and ScaleGMN-B rely solely on the original training dataset and on built-in equivariance/invariance.\n\n>*Question 6. Are there any limitations on the choice of activations of the ScaleGMN network?*\n\n*Importantly, our method does **not** impose any limitations on the choice of the activation functions.*\n\nWe are able to select any activation, because these are only applied within the MLPs of the invariant modules.  As discussed in Section 5 of our paper, the MLPs (equipped with non-linearities) are only applied after the canon/symm function. In case one chose to place activations in a different computational part of ScaleGMN, this would indeed limit their options so as not to compromise scale equivariance. However, this is not the case in our method. We thank the reviewer for noticing this important detail.\n     \n---\n[1] Godfrey, Charles, et al. \"On the symmetries of deep learning models and their internal representations.\" Advances in Neural Information Processing Systems 35 (2022): 11893-11905.\n\n[2] Kofinas, Miltiadis, et al. \"Graph Neural Networks for Learning Equivariant Representations of Neural Networks.\" The Twelfth International Conference on Learning Representations.\n\n[3] Zhou, Allan, et al. \"Permutation equivariant neural functionals.\" Advances in neural information processing systems 36 (2024).\n\n[4] Shamsian, Aviv, et al. \"Improved generalization of weight space networks via augmentations.\" arXiv preprint arXiv:2402.04081 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their thorough evaluation of our paper and their constructive feedback, which helped us improve our empirical evaluation to further corroborate our claims and identify potential future directions. In the following comments, we gather the strengths pointed out by the reviewers and summarise our rebuttal response and changes that will be made in an updated version of the manuscript.

- **Rev. AXwX** *found our paper well-written* (AXwX: "Great writing. Nice introduction and related work, as well as good setup and notation in Section 3."), 
- **Rev. x2pG, ugM4** *underlined the significance of the studied problem* (x2pG: "[...] this paper tackles a crucial field and problem, advancing the area in a way that can benefit many practitioners.", ugM4: "The paper deals with an important and timely problem of learning in deep weight spaces.").
- **Rev. AXwX, x2pG, ugM4** *acknowledged the novelty of our method* (AXwX: "Their ScaleGMNs extend metanetworks, which are typically only permutation equivariant (if at all equivariant), to also account for other symmetries in input neural networks' parameters.", x2pG: "[...] a new structure that incorporates both permutation and scale symmetries [...].", ugM4: "[...] presents a novel architecture by incorporating scale and permutation equivariance.").
- **Rev. AXwX, x2pG, ugM4** *acknowledged the importance of our theoretical contributions regarding the expressive power of ScaleGMN* (AXwX: "Nice theoretical results. That ScaleGMN can express the forward and backward pass is a good way to check that its expressive power is not overly limited when adding the additional scaling equivariances.", x2pG: "The authors [...] show that ScaleGMN can simulate forward and backward passes of arbitrary inputs in the processed NN.", ugM4: "The paper provides theoretical analysis and results regarding the expressive power of the proposed approach."
- **Rev. AXwX, x2pG, ugM4** *appreciated the empirical impovements reported in our experimental evaluation* (AXwX: "Large empirical improvements, especially on INR classification.", x2pG: "The empirical results show a significant improvement compared to recent works in the field of weight space learning.", ugM4: "Empirical results show significant improvement over baseline methods.").
- In addition to the above, **Rev. AXwX** *pinpoints that these improvements are achieved with built-in equivariance alone and without having to resort to additional practical tricks* (AXwX: "without many of the unfair or expensive tricks that others use! [...] That ScaleGMN can beat the prior methods without these tricks is very impressive. [...] ScaleGMN does not need random Fourier features or data augmentations")

## Rebuttal Summary

### Equivariant tasks
Reviewers AXwX, x2pG, ugM4 correctly mention the need for an equivariant task. To that end, we evaluate on **INR editing**. Following [2], we select the MNIST-INR dataset and evaluate our method. Again, no additional tricks or augmentations were used. 

**Results**. (Table 1) Bidirectional *ScaleGMN-B* achieves an MSE test loss ($10^{-2}$) equal to $1.89$, **surpassing all baselines**, outperforming even the *NG-GNN baseline with 64 probe features*. Note that the performance gap between the bidirectional and the forward model (which achieves a loss of $2.56$) is expected for equivariant tasks: in this case we are required to compute representations for every graph node, yet, in the forward variant, the earlier the layer of the node, the less information it receives. Similarly, our baselines are either bidirectional (NG-GNN [2]) or non-local (DWS [3], NFN [1]).

### Heterogeneous activation functions
Reviewer x2pG  pointed to heterogeneous activation functions. *In principle, our method does not impose any limitations regarding their homogeneity*. 

**Results**. Evaluated on CIFAR-10-GS, **ScaleGMN demonstrates superior performance** compared to the baselines, *significantly surpassing the next best model*. (Table 4)

### Scaling data augmentations
Reviewer ugM4 proposed baselining with a permutation-equivariant-only model combined with random scaling augmentations.

**Results.** (Table 3)
* **Sign symmetries**: We augment w/  sign flips **independently** (probability 0.5). This surpasses the original baselines, but **not ScaleGMN and ScaleGMN-B.**
* **Positive-scale symmetries**: We sample positive scalars  (NB: **continuous and unbounded distribution**), but observe performance deterioration compared to the original baselines.

### More complex architectures
We acknowledge that experimenting with diverse architectures would strengthen our contributions. However, there are two reasons why this was not possible in the current work. First, the characterisation of scaling symmetries holds for MLPs and can be extended to CNNs. However, extending it to other architectures requires further efforts and is therefore more appropriate to be considered in future work. 

Second, weight-space learning is currently missing curated benchmarks of complex architectures. Lim et al. [4] and Zhou et al. [5] experiment with diverse architectures, using private datasets. Small DNN Zoo [6] and ModelZoos [7] contain trained CNNs of fixed architectures. Finally, the diverse CNN Wild Park [2] was only made public a few days ago. *Keeping in mind that processing such networks is not among our main contributions*, we opted to align with the previous works on metanetworks and selected the datasets used in [1], [2].

### Exposition/writing style
We thank the reviewers x2pG and ugM4 for their suggestions. We allocated a good portion of the paper to problem formulation and background, as the topic of weight space learning is relatively new. The characterisation of scaling symmetries is also recent and has not yet gathered significant attention. Consequently, we opted for a smooth and detailed introduction before delving into our contributions to make our work self-sufficient.

Meta-review: This paper introduces Scale Equivariant Graph Meta Networks (ScaleGMNs), extending metanetwork design to include scaling symmetries. The work presents strong theoretical analysis and empirical improvements in the emerging field of deep weight space learning.
Initial concerns about experimental scope and presentation were adequately addressed in the authors' rebuttal. They provided additional experiments and committed to improving the paper's structure.
Given the novel contributions, theoretical foundation, and empirical results, I recommend accepting this paper for NeurIPS 2024. The authors should incorporate the promised improvements in the camera-ready version.
Decision: Accept (oral)