Scale Equivariant Graph Metanetworks

Authors: ioannis  kalogeropoulos
giorgos  bouritsas
yannis  panagakis
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:41:55

Abstract: This paper pertains to an emerging machine learning paradigm: learning higher- order functions, i.e. functions whose inputs are functions themselves, particularly when these inputs are Neural Networks (NNs). With the growing interest in architectures that process NNs, a recurring design principle has permeated the field: adhering to the permutation symmetries arising from the connectionist structure of
NNs. However, are these the sole symmetries present in NN parameterizations? Zooming into most practical activation functions (e.g. sine, ReLU, tanh) answers this question negatively and gives rise to intriguing new symmetries, which we collectively refer to as scaling symmetries, that is, non-zero scalar multiplications and divisions of weights and biases. In this work, we propose Scale Equivariant Graph MetaNetworks - ScaleGMNs, a framework that adapts the Graph Metanetwork (message-passing) paradigm by incorporating scaling symmetries and thus rendering neuron and edge representations equivariant to valid scalings. We introduce novel building blocks, of independent technical interest, that allow for equivariance or invariance with respect to individual scalar multipliers or their product and use them in all components of ScaleGMN. Furthermore, we prove that, under certain expressivity conditions, ScaleGMN can simulate the forward and backward pass of any input feedforward neural network. Experimental results demonstrate that our method advances the state-of-the-art performance for several datasets and activation functions, highlighting the power of scaling symmetries as an inductive bias for NN processing. The source code is publicly available at https://github.com/jkalogero/scalegmn.

Main paper: 
 introduction: 
 Neural networks are becoming the workhorse of problem-solving across various domains. To solve a task, they acquire rich information which is stored in their learnable parameters throughout training. Nonetheless, this information is often opaque and difficult to interpret. This begs the question: How can we efficiently process and extract insights from the information stored in the parameters of trained neural networks, and how to do so in a data-driven manner? In other words, can we devise architectures that learn to process other neural architectures?The need to address this question arises in diverse scenarios: NN post-processing, e.g. analysis/interpretation (i.e. inferring NN properties, such as generalisation and robustness [74, 19] ), as well as editing (e.g. model pruning [22] , merging [75] or adaptation to new data) -or NN synthesis (e.g. for optimisation [12] or more generally parameter prediction/generation [31, 67, 32, 58] ). Furthermore, with the advent of Implicit Neural Representations, 1 [13, 52, 57, 70, 53] trained NN parameters are increasingly used to represent datapoint signals, such as images or 3D shapes, replacing raw representations, i.e., pixel grids or point clouds [18] . Consequently, many tasks involving such data, across various domains such as computer vision [47] and physics [68] , which are currently tackled using domain-specific architectures (e.g. CNNs for grids, PointNets [63] or GNNs [60] for point clouds and meshes), could potentially be solved by NNs that process the parameters of other NNs.The idea of processing/predicting NN parameters per se is not new to the deep learning community, e.g. it has been investigated in hypernetworks [27] or even earlier [65] . However recent advancements, driven by the need for INR processing, leverage a crucial observation: NNs have symmetries. In particular, several works have identified that the function represented by an NN remains intact when applying certain transformations to its parameters [28, 11] with the most well-known transformations being permutations. That is, hidden neurons can be arbitrarily permuted within the same layer, along with their incoming and outgoing weights. Therefore, all the tasks mentioned in the previous paragraph can be considered part of equivariant machine learning [10] , as they involve developing models that are invariant or equivariant to the aforementioned transformations of neural networks. Recently Navon et al. [54] and Zhou et al. [86] acknowledged the importance of permutation symmetries and first proposed equivariant architectures for Feedforward NN (FFNN) processing, demonstrating significant performance improvements against non-equivariant baselines. These works, as well as their improvements [85] and extensions to process more intricate and varying architectures [33, 44, 88] , are collectively known as weight space networks or metanetworks, with graph metanetworks being a notable subcase (graph neural networks with message-passing).Nevertheless, it is known that permutations are not the only applicable symmetries. In particular, theoretical results in various works [11, 61] , mainly unified in [25] show that FFNNs exhibit additional symmetries, which we collectively here refer to as scaling symmetries: multiplying the incoming weights and the bias of a hidden neuron with a non-zero scalar a (with certain properties) while dividing its outgoing weights with another scalar b (often b = a), preserves the NN function. Intuitively, permutation symmetries arise from the graph/connectionist structure of the NN whereas different scaling symmetries originate from its activation functions σ, i.e. when it holds that σ(ax) = bσ(x). However, equivariance w.r.t. scaling symmetries has not received much attention so far (perhaps with the exception of sign symmetriesa ∈ {-1, 1} [41, 40] ), and it remains effectively unexplored in the context of metanetworks.To address this gap, we introduce in this paper a graph metanetwork framework, dubbed as Scale Equivariant Graph MetaNetworks-ScaleGMN, which guarantees equivariance to permutations and desired scaling symmetries, and can process FFNNs of arbitrary graph structure with a variety of activation functions. At the heart of our method lie novel building blocks with scale invariance/equivariance properties w.r.t. arbitrary families of scaling parameters. We prove that ScaleGMN can simulate the forward and backward pass of an FFNN for arbitrary inputs, enabling it to reconstruct the function represented by any input FFNN and its gradients.Our contributions can be summarised as follows:• We extend the scope of metanetwork design from permutation to scaling symmetries.• We design invariant/equivariant networks to scalar multiplication of individual multipliers or combinations thereof, originating from arbitrary scaling groups. • We propose scale equivariant message passing, using the above as building blocks, unifying permutation and scale equivariant processing of FFNNs. Additionally, the expressive power of our method is analysed w.r.t. its ability to simulate input FFNNs. • Our method is evaluated on 3 activations: ReLU (positive scale), tanh (sign) and sine (sign -we first characterise this using a technique from [25] ) on several datasets and three tasks (INR classification/editing & generalisation prediction), demonstrating superior performance against common metanetwork baselines.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main contribution of this paper is the design and evaluation of an NN processing architecture (metanetwork) that accounts not only for NN permutation symmetries but also scaling. The abstract and introduction introduce the reader to scaling symmetries for certain NN architectures (FFNNs and CNNs) and explain our main technical innovations to achieve our goal. The importance of scaling symmetries as an inductive bias is showcased in our theoretical and empirical results, as explained in the abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Neural network symmetries. Long preceding the advent of metanetworks, numerous studies delved into the inherent symmetries of neural networks. Hecht-Nielsen [28] studied FFNNs and discovered the existence of permutation symmetries. Chen et al. [11] showed that, in FFNNs with tanh activations, the only function-preserving smooth transformations are permutations and sign flips (multiplication of incoming and outgoing weights with a sign value) -this claim was strengthened in [21] ; this was the first identification of scaling symmetries. Follow-up works extended these observations to other activations, such as sigmoid and RBF [37] and ReLU [56, 51, 61, 64, 26] , and architectures, such as RNNs [3, 2] , characterising other scaling symmetries and providing conditions under which permutations and scalings are the only available function-preserving symmetries or the parameters are identifiable given the input-output mapping. Recently, Godfrey et al. [25] provided a technique to characterise such symmetries for arbitrary activations that respect certain conditions, unifying many previous results. Additionally, symmetries have been found in other layers, e.g. batch norm [6, 14, 16] (scale invariance) or softmax [36] (translation invariance). Prior to metanetworks, symmetries were mainly studied to obtain a deeper understanding of NNs or in the context of optimisation/learning dynamics [56, 71, 4, 17, 36, 5, 83, 84] and/or model merging/ensembling [20, 1, 69, 59, 55] .Metanetworks. The first solutions proposed for NN processing and learning did not account for NN symmetries. Unterthiner et al. [74] and Eilertsen et al. [19] , initially employed standard NNs on vectorised (flattened) CNN weights or some statistics thereof, to predict properties of trained neural networks such as generalization or estimation of a hyperparameter resp.). Similar methods have been proposed to process continuous data represented as INRs. In [78] high-order spatial derivatives are used (suitable only for INRs), in [47] the architecture operates on stacked parameter vectors (but is constrained by the assumption that all INRs are trained from the same initialization), while in [18] and [7] the authors learn low-dimensional INR embeddings (which are further used for downstream tasks) by jointly fitting them with meta-learning. Finally, Schürholt et al. [66] learn representations of NN weights with self-supervised learning and in [67] this is extended to generative models.In contrast to the above methods, our work follows a recent stream of research focusing on equivariant metanetworks. Navon et al. [54] and Zhou et al. [85] first characterised all linear equivariant layers to permutation symmetries of MLPs and combined them with non-linearities, while in the latter this was extended to CNNs. These approaches derive intricate weight-sharing patterns but are nonlocal 2 and cannot process varying architectures. In follow-up works, Zhou et al. [86] constructed equivariant attention layers and in [87] the above characterisation was generalised to arbitrary input architectures and layers (e.g. RNNs and transformers) introducing an algorithm for automatic linear layer (with weight-sharing) construction. A different route was taken in the very recent works of Kofinas et al. [33] and Lim et al. [44] , where input NNs are treated as a special class of graphs and are naturally processed with Graph Neural Networks (GNNs), with appropriate symmetry breaking wherever necessary. This perspective has been adopted several times in the deep learning literature [9, 23, 45, 77, 80] and in a few examples, GNNs were applied on the graph representation for, e.g. neural architecture search ( [82, 72, 31] ), albeit without mapping parameters to the graph edges. In parallel to our work, [73] extended the framework of [85] to incorporate scaling and sign-flipping symmetries. They construct non-local equivariant layers, resulting in an architecture with fewer trainable parameters. Nevertheless, their method suffers from limited expressive power, while their experimental results showcase limited advancements. In contrast to our method, this approach limits the choice of activation functions to those equivariant to the studied symmetries. Finally, their framework cannot be extended to other activation functions, as the parameter sharing must be re-designed from scratch, while it is not suitable for processing diverse architectures.Scale Equivariance Equivariance to vector scaling (or more generally to matrix multiplication with diagonal matrices) remains to date underexplored in the machine learning community. Sign symmetries received attention in the work of Lim et al. [42] , where an invariant network (SignNet) was designed, mainly to process eigenvectors, with the theoretical analysis revealing universality under certain conditions. This was extended in [39] , where the importance of sign equivariance on several tasks was demonstrated and a sign equivariant network was proposed. In our work, we draw inspiration from these architectures and extend their formulations to arbitrary scaling symmetries.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Yes, the limitations of our work (applicable input NN layers/architectures and open questions regarding expressivity) are discussed thoroughly in the last paragraph before the conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Yes, each method we ran (including baselines) was tested for 3 different random seeds and the mean and standard deviation were reported. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
notation and preliminaries: 
 Notation. We denote vectors, matrices, and tensors with bold-face letters, e.g., x, X, X, respectively and sets with calligraphic letters X . A normal font notation will be used for miscellaneous purposes (mostly indices and functions). Datapoint (input) functions/signals will be denoted with u, while higher-order functions (functions of functions) will be denoted with fraktur font F.Functions of Neural Networks. Consider functions of the form u G,θ : X → X . Each function is parameterised (1) by a computational graph G, which determines all the mathematical operations that should be performed to evaluate the function u at a datapoint x ∈ X . When u is a neural network, G is determined by the NN architecture. Additionally, u is parameterised by (2) , by a tuple of numerical parameters θ, on which the aforementioned mathematical operations are applied (along with the input x) -these are the learnable parameters, which can be arranged into a vector. We are interested in learning unknown higher-order functions F : X X → Y of the form F (u G,θ ). In our case, the goal is to learn F by accessing solely the parameters (G, θ) of each u, i.e. via functionsf : G × Θ → Y of the form f (G, θ), where G is a space of architectures/computational graphs and Θ a space of parameters. 3 We are typically interested in learning functionals(Y ⊆ R d ) or operators (Y = G × Θ or Y = Θ).To approximate the desired higher-order function, we assume access to a dataset of parameters sampled i.i.d. from an unknown distribution p on G × Θ. For example, in a supervised setup, we aim to optimise the following:argmin f ∈F E (G,θ)∼p L F(u G,θ ), f (G, θ) ,where L(•, •) is a loss function and F is an NN processing hypothesis class (e.g. metanetworks).Feedforward Neural Networks (FFNNs). In this paper, we focus our analysis on Feedforward Neural Networks (FFNNs), i.e. linear layers interleaved with non-linearities. Consider NNs of the form u G,θ : R din → R dout of the following form:EQUATIONwhere L: the number of layers, Neural Network symmetries. One of the major difficulties with working with function parameters directly is that the same function can be represented with more than one parameter, i.e. there exists transformations that when applied to any parameter (G, θ), keep the represented function intact. Formally, an NN symmetry is induced by a set Ψ of transformations ψ :W i ∈ R d ℓ ×d ℓ-1 :G × Θ → G × Θ, such that u G,θ (x) = u ψ(G,θ) (x), ∀x ∈ X , ∀(G, θ) ∈ G × Θ. If for a pair parameters (G, θ), (G ′ , θ ′ ), ∃ψ such that (G, θ) = ψ(G ′ , θ ′), we will call the two parameters equivalent and write (G, θ) ≃ (G ′ , θ ′ ). To appropriately represent a functional F, a hypothesis (metanetwork) f should be invariant to transformations in Ψ: f (ψ (G, θ)) = f (G, θ). For operators, f should be equivariant to transformations: f (ψ (G, θ)) = ψ (f (G, θ)), such that identical functions map to identical functions.Permutation symmetries (connectionist structure). For a fixed computational graph G, perhaps the most well-known symmetry of FFNNs are those induced by hidden neuron permutations [28] .As far as metanetworks are concerned it is to date the only NN symmetry that has been accounted for -see Section 2. This symmetry implies that permuting hidden neurons (along with their biases and incoming and outgoing weights) within each layer preserves the NN function (regardless of the activation function). This reads:EQUATIONwhere ℓ ∈ {1, . . . , L}, P 0 = P L = I and P ℓ ∈ R d ℓ ×d ℓ are arbitrary permutation matrices. Observe that they are different for each layer, with the input and output neurons held fixed.A recently introduced weight space architecture [33, 44] , takes advantage of the permutation symmetries and treats FFNNs (among others, e.g. CNNs) as graphs, processing them with conventional GNNs. In particular, let G = (V, E) be the computational graph, i ∈ V an arbitrary vertex in the graph (neuron) and (i, j) ∈ E an arbitrary edge from vertex j to vertex i. 4 Additionally, let x V ∈ R |V|×dv be the vertex features and x E ∈ R |E|×de the edge features (i.e. biases and weights resp. in a FFNN). The general form of a T iteration (layer) GMN reads:EQUATIONwhere h t V , h t E are vertex and edge representations at iteration t and h G is the overall graph (NN) representation. INIT, MSG, UPD are general function approximators (e.g. MLPs), while READ is a permutation invariant aggregator (e.g. DeepSets [81] ). The above equations have appeared with several variations in the literature, e.g. in some cases the edge representations are not updated or the readout might involve edge representations as well. Another frequent strategy is to use positional encodings p V , p E to break undesired symmetries. In FFNNs, Eq. ( 2) reveals that input and output vertices are not permutable, while vertices cannot be permuted across layers. Therefore, vertices (or edges) that are permutable share the same positional encoding (see Appendix A.1.2 for more details). Remark: Although, typically, the neighbourhood N (i) contains both incoming and outgoing edges, in Section 5 we will illustrate our method using only incoming edges: forward neighbourhood N FW (i) = {j ∈ V | layer (i)layer (j) = 1} and backward where layer (i) gives the layer neuron i belongs. Backward neighbourhoods N BW (i) are defined defined similarly. In Appendix A.2, we show a more elaborate bidirectional version of our method, with both neighbourhoods considered. 
scaling symmetries in feedforward neural networks: 
 Scaling symmetries (activation functions). Intuitively, permutation symmetries stem from the graph structure of neural networks, or put differently, from the fact that hidden neurons do not possess any inherent ordering. Apart from the affine layers W ℓ that give rise to the graph structure, it is frequently the case that activation functions σ ℓ have inherent symmetries that are bestowed to the NN.Let us dive into certain illustrative examples: for the ReLU activation σ(x) = max(x, 0) it holds that σ(ax) = max(ax, 0) = a max(x, 0), ∀a > 0. For the tanh and sine activations σ(x) = tanh(x), σ(x) = sin(x) respectively, it holds that σ(ax) = aσ(x), ∀a ∈ {-1, 1}. In a slightly more complex example, polynomial activations σ(x) = x k , we have σ(ax) = a d σ(x), i.e. the multiplier differs between input and output. In general, we will be talking about scaling symmetries whenever there exist pairs (a, b) for which it holds that σ(ax) = bσ(x). To see how such properties affect NN symmetries, let us focus on FFNNs (see Appendix A.3 for CNNs): for a neuron i (we omit layer subscripts) we have σ aW(i, :)x + ab(i) = bσ W(i, :)x + b(i) , i.e. multiplying its bias and all incoming weights with a constant a results in scaling its output with a corresponding constant b. Generalising this to linear transformations, we may ask the following: which are the pairs of matrices (A, B) for which we have σ AWx + Ab = Bσ Wx + b ? Godfrey et al. [25] provide an answer for any activation that respects certain conditions. We restate here their most important results: Proposition 4.1 (Lemma 3.1. and Theorem E.14 from [25] ). Consider an activation function σ : R → R. Under mild conditions,5 the following hold:• For any d ∈ N + , there exists a (non-empty) group of invertible matrices defined as:I σ,d = {A ∈ R d×d : invertible | ∃ B ∈ R d×d invertible, such that: σ(Ax) = Bσ(x)}(intertwiner group), and a mapping function ϕ σ,d such that B = ϕ σ,d (A).• Every A ∈ I σ,d is of the form PQ, where P: permutation matrix and Q = diag q 1 , . . . q d diagonal, withEQUATIONa)σ(x)}: the 1-dimensional group, and ϕ σ,d (A) = Pdiag ϕ σ,1 (q 1 ), . . . ϕ σ,1 (q d ) .This is a powerful result that completely answers the question above for most practical activation functions. Importantly, not only does it recover permutation symmetries, but also reveals symmetries to diagonal matrix groups, which can be identified by solely examining ϕ σ,1 , i.e. the one-dimensional case and the set D σ (easily proved to be a group) we have already discussed in our examples above.Using this statement, Godfrey et al. [25] characterised various activation functions (or recovered existing results), e.g. ReLU: I σ,d contains generalised permutation matrices with positive entries of the form PQ, Q = diag(q 1 , . . . , q d ), q i > 0 and ϕ σ,d (PQ) = PQ [56] . Additionally, here we characterise the intertwiner group of sine (used in the popular SIREN architecture [70] for INRs). Not surprisingly, it has the same intertwiner group with tanh [11, 21] (we also recover this here using Proposition 4.1). Formally, (proof in Appendix A.7.1): Corollary 4.2. Hyperbolic tangent σ(x) = tanh(x) and sine activation σ(x) = sin(ωx), satisfy the conditions of Proposition 4.1, when (for the latter) ω ̸ = kπ, k ∈ Z. Additionally, I σ,d contains signed permutation matrices of the form PQ, with Q = diag(q 1 , . . . , q d ), q i = ±1 and ϕ σ d (PQ) = PQ.It is straightforward to see that the symmetries of Proposition 4.1, induce equivalent parameterisations for FNNNs. In particular, it follows directly from Proposition 3.4. in [25] , that for activation functions σ ℓ satisfying the conditions of Proposition 4.1 and when ϕ σ,ℓ (Q) = Q, we have that:EQUATIONwhere again ℓ ∈ {1, . . . , L}, P 0 = Q 0 = P L = Q L = I. 
scale equivariant graph metanetworks: 
 As previously mentioned, the metanetworks operating on weight spaces that have been proposed so far, either do not take any symmetries into account or are invariant/equivariant to permutations alone as dictated by Eq. ( 2). In the following section, we introduce an architecture invariant/equivariant to permutations and scalings, adhering to Eq. ( 3). An important motivation for this is that in various setups, these are the only function-preserving symmetries, i.e. for a fixed graph u G,θ = u G,θ ′ ⇒ (θ, θ ′ ) satisfy Eq. ( 3) -e.g. see [21] for the conditions for tanh and [61, 26] for ReLU.Main idea. Our framework is similar in spirit to most works on equivariant and invariant NNs [10] .In particular, we build equivariant GMNs that will preserve both symmetries at vertex-and edge-level, i.e. vertex representations will have the same symmetries with the biases and edge representations with the weights. To see this, suppose two parameter vectors are equivalent according to Eq. (3). Then, the hidden neurons representations -the discussion on input/output neurons is postponed until Appendix A.1.4 -should respect the following (the GMN iteration t is omitted to simplify notation):EQUATIONEQUATIONwhere π ℓ : V ℓ ↔ V ℓ permutes the vertices of layer ℓ (denoted with V ℓ ) according to P ℓ and q ℓ : V ℓ → R \ {0} scales the vertex representations of layer ℓ according to Q ℓ . We will refer to the latter as forward scaling in Eq. ( 4) and bidirectonal scaling in Eq. ( 5). To approximate operators (equivariance), we compose multiple equivariant GMN layers/iterations and in the end, project vertex/edge representations to the original NN weight space, while to approximate functionals (invariance), we compose a final invariant one in the end summarising the input to a scalar/vector.To ease exposition, we will first discuss our approach w.r.t. vertex representations. Assume that vertex representation symmetries are preserved by the initialisation of the MPNN -Eq. (Init) -and so are edge representation symmetries for all MPNN layers. Therefore, we can only focus on the message passing and vertex update steps -Eq. (Msg) and Eq. (Upd). Additionally, let us first focus on hidden neurons and assume only forward neighbourhoods. The following challenges arise: Challenge 1 -Scale Invariance / Equivariance. First off, the message and the update function MSG V , UPD V should be equivariant to scaling -in this case to the forward scaling using the multiplier of the central vertex q ℓ (i). Additionally, the readout READ, apart from being permutation invariant should also be invariant to the different scalar multipliers of each vertex . Dealing with this requires devising functions of the following form:EQUATIONwhere D i a 1-dimensional scaling group as defined in Proposition 4.1. Common examples are those discussed in Section 4, e.g. D i = {1, -1} or D i = R + . The first case, i.e. sign symmetries, has been discussed in recent work [43, 40] . Here we generalise their architecture into arbitrary scaling groups.In specific, Scale Equivariant networks follow the methodology of [40] , i.e. they are compositions of multiple linear transformations multiplied elementwise with the output of Scale Invariant functions:ScaleInv k (X) = ρ k (x 1 , . . . , xn ) , (Scale Inv. Net) ScaleEq = f K • • • • • f 1 , f k (X) = Γ k 1 x 1 , . . . , Γ k n x n ⊙ ScaleInv k (X), (Scale Equiv. Net)where ρ k :n i=1 X i → R n i=1 d ki universal approximators (e.g. MLPs), Γ k i : X i → R d k i linear transforms (for each of the k invariant/equivariant layers resp -in practice, we observed experimentally that a single layer K = 1 was sufficient), and xi are explained in detail in the following paragraph.Central to our method is defining a way to achieve invariance. One option is canonicalisation, i.e. by defining a function canon : X → X that maps all equivalent vectors x to a representative (obviously non-equivalent vectors have different representatives): Iff x ≃ y ∈ X , then x ≃ canon(x) = canon(y). In certain cases, these are easy to define and differentiable almost everywhere, for example for positive scaling: canon(x) = x ∥x∥ . 6 For sign symmetries, this is not as straightforward: in 1-dimension one can use |x|, but for arbitrary dimensions, a more complex procedure is required, as recently discussed in [49, 48] . Since the group is small -two elements -one can use symmetrisation instead [79] , as done by Lim et al. [41] : symm(x) = y:y≃x MLP(y), i.e. for sign: symm(x) = MLP(x)+MLP(-x). Therefore, we define: ScaleInv k (X) = ρ k (x 1 , . . . , xn ), with xi = canon(x i ) or xi = symm(x i ). Importantly, it is known that both cases allow for universality, see [8, 29] and [79, 62] respectively.Challenge 2 -Rescaling: Different multipliers. Scale equivariance alone is not sufficient, since the input vectors of the message function MSG V are scaled by different multipliers -central vertex: q ℓ (i), neighbour: q ℓ-1 (j), edge: q ℓ (i)q -1 ℓ-1 (j), while its output should be scaled differently as wellq ℓ (i). We refer to this problem as rescaling. Dealing with Challenge 2 requires functions of the form:EQUATIONWe call these functions rescale equivariant. Note, that this is an unusual symmetry in equivariant NN design. Our approach is based on the observation that any n-other monomial containing variables from all vectors x i is rescale-equivariant. Collecting all these monomials into a single representation is precisely the outer productX n = x 1 ⊗ • • • ⊗ x n , where X n (j 1 , . . . , j n ) = n i=1 x i (j i ).Therefore, the general form of our proposed Rescale Equivariant Network is as follows:ReScaleEq(x 1 , . . .x n ) = ScaleEq vec(X n ) .(ReScale Equiv. Net) In practice, given that the size of X n grows polynomially with n, we resort to a more computationally friendly subcase, i.e. hadamard products, i.e. ReScaleEq(x 1 , . . .x n ) = ⊙ n i=1 Γ i x i , Γ i : X i → R d .Contrary to the original formulation, the latter is linear (lack of multiplication with an invariant layer).Scale Equivariant Message Passing. We are now ready to define our message-passing scheme. Starting with the message function, we require each message vector m V (i) to have the same symmetries as the central vertex i. Given the scaling symmetries of the neighbour and the edge, for forward neighbourhoods, this reads: MSG V q x x, q y y, q x q -1 y e = q x MSG V (x, y, e). In this case, we opt to eliminate q y by multiplication as follows:MSG V (x, y, e) = ScaleEq ([x, ReScaleEq (y, e)]) , (8) where [•, •] denotes concatenation, ReScaleEq(q y y, q x q -1 y e) = q x ReScaleEq(y, e). In our experiments, we used only y and e, since we did not observe significant performance gains by including the central vertex x. Now, the update function is straightforward to implement since it receives vectors with the same symmetry, i.e. it should hold that: UPD V (q x x, q x m) = q x UPD V (x, m) which is straightforward to implement with a scale equivariant network, after concatenating x and m:EQUATIONFinally, to summarise our graph into a single scalar/vector we require a scale and permutationinvariant readout. The former is once more achieved using canonicalised/symmetrised versions of the vertex representations of hidden neurons, while the latter using a DeepSets architecture as usual:EQUATIONIn Appendix A.1, we show how to handle symmetries in the rest of the architecture components (i.e. initialisation, positional encodings, edge updates and i/o vertices) and provide an extension of our method to bidirectional message-passing (Appendix A.2), which includes backward neighbourhoods.Expressive power. Throughout this section, we discussed only scaling symmetries and not permutation symmetries. However, it is straightforward to conclude that ScaleGMN is also permutation equivariant/invariant, since it is a subcase of GMNs; if one uses universal approximators in their message/update functions (MLP), our corresponding functions will be expressible by this architecture, which was proved to be permutation equivariant in [44] . Although this implies that GMN can express ScaleGMN, this is expected since PQ symmetries are more restrictive than just P. Note that these symmetries are always present in FFNNs, and thus it is desired to explicitly model them, to introduce a more powerful inductive bias. Formally, on symmetry preservation (proved in Appendix A.7.2): StatNN [74] 0.9140 ± 0.001 0.8463 ± 0.004 0.9140 ± 0.000 0.8440 ± 0.001 0.915 ± 0.002 NFN NP [85] 0.9190 ± 0.010 0.8586 ± 0.003 0.9251 ± 0.001 0.8580 ± 0.004 0.922 ± 0.001 NFN HNP [85] 0.9270 ± 0.001 0.8636 ± 0.002 0.9339 ± 0.000 0.8586 ± 0.004 0.934 ± 0.001 NG-GNN [33] 0.9010 ± 0.060 0.8549 ± 0.002 0.9340 ± 0.001 0.8620 ± 0.003 0.931 ± 0.002 ScaleGMN (Ours) 0.9276 ± 0.002 0.8689 ± 0.003 0.9418± 0.005 0.8736 ± 0.003 0.941 ± 0.006 ScaleGMN-B (Ours) 0.9282 ± 0.003 0.8651 ± 0.001 0.9425 ± 0.004 0.8655 ± 0.004 0.941 ± 0.000 Proposition 5.1. ScaleGMN is permutation & scale equivariant. Additionally, ScaleGMN is permutation & scale invariant when using a readout with the same symmetries.6 Fwd: x ≃ y ⇒ x = qy ⇒ y ∥y∥ = qx |q|∥x∥ = x ∥x∥ (q > 0). Reverse: x ∥x∥ = y ∥y∥ ⇒ x = ∥x∥ ∥y∥ y ⇒ x ≃ yFinally, we analysed the ability of ScaleGMN to evaluate the input FFNN and its gradients, i.e. simulate the forward and the backward pass of an input FFNN. To see why this is a desired inductive bias, recall that a functional/operator can often be written as a function of input-output pairs (e.g. via an integral on the entire domain) or of the input function's derivatives (e.g. via a differential equation). By simulating the FFNN, one can reconstruct function evaluations and gradients, which an additional module can later combine. Formally (proof and precise statement in Appendix A.7.2): Theorem 5.2. Consider an FFNN as per Eq. (1) with activation functions respecting the conditions of Proposition 4.1. Assume a Bidirectional-ScaleGMN with sufficiently expressive message and vertex update functions. Then, ScaleGMN can simulate both the forward and the backward pass of the FFNN for arbitrary inputs, when ScaleGMN's iterations (depth) are L and 2L respectively. 
experiments: 
 Datasets. We evaluate ScaleGMN on datasets containing NNs with three popular activation functions: sine, ReLU and tanh. The former is prevalent in INRs, which in turn are the most appropriate testbed for GMNs. We experiment with the tasks of (1) INR classification (invariant task), i.e. classifying functions (signals) represented as INRs and (2) INR editing (equivariant), i.e. transforming those functions. Additionally, ReLU and tanh are common in neural classifiers/regressors. A popular GMN benchmark for those is (3) generalisation prediction (invariant), i.e. predicting a classifier's test accuracy. Here, classifiers instead of FFNNs are typically CNNs (for computer vision tasks) and to this end, we extend our method to the latter in Appendix A.3. We use existing datasets that have been constructed by the authors of relevant methods, are publicly released and have been repeatedly used in the literature (8 datasets in total, 4 for each task). Finally, we follow established protocols: we perform a hyperparameter search and use the best-achieved validation metric throughout training to select our final model. We report the test metric on the iteration where the best validation is achieved. 2.58 ± 0.00 NFN NP [85] 2.55 ± 0.00 NFN HNP [85] 2.65 ± 0.01 NG-GNN-0 [33] 2.38 ± 0.02 NG-GNN-64 [33] 2.06 ± 0.01 ScaleGMN (Ours) 2.56 ± 0.03 ScaleGMN-B (Ours) 1.89 ± 0.00 [85] 68.19 ± 0.28 0.9190 ± 0.010 NFN NP [85] + aug.70.34 ± 0.13 0.8474 ± 0.01 NFN HNP [85] 68.94 ± 0.64 0.9270 ± 0.001 NFN HNP [85] + aug.70.24 ± 0.47 0.8906 ± 0.01 NG-GNN [33] 68.0 ± 0.20 0.9010 ± 0.060 NG-GNN [33] Baselines. DWS [54] is a non-local metanetwork that uses all linear permutation equivariant/invariant layers interleaved with non-linearities. NFN HNP [85] is mathematically equivalent to DWS, while NFN NP [85] makes stronger symmetry assumptions in favour of parameter efficiency. The two variants are also designed to process CNNs contrary to DWS. NG-GNN [33] converts each input NN to a graph (similar to our work) and employs a GNN (in specific PNA [15] ). Importantly, the last three methods transform the input parameters with random Fourier features while all methods perform input normalisations to improve performance and facilitate training. These tricks are in general not equivariant to scaling and were unnecessary in the ScaleGMN experiments (more details in Appendix A.4). On INR classification we include a naive MLP on the flattened parameter vector and Inr2Vec [47] , a task-specific non-equivariant method. For generalization prediction, we also compare to StatNN [74] , which predicts NN accuracy based on statistical features of its weights/biases. [38] , FashionMNIST [76] and CIFAR [35] datasets resp. and an augmented version of CIFAR10-INR containing 20 different INRs for each image, trained from different initialisations (often called views). The reported metric is test accuracy. Note that [85, 86] use only the augmented dataset and hence we rerun all baselines for the original version. All input NNs correspond to functions u G,θ : R 2 → R c , i.e. pixel coordinate to GS/RGB value. Further implementation details can be found in Appendix A.4.1. As shown in Table 1 , ScaleGMN consistently outperforms all baselines in all datasets considered, with performance improvements compared to the state-of-the-art ranging from approx. 3% (CIFAR-10) to 13% (F-MNIST). While previous methods often resort to additional engineering strategies, such as probe features and advanced architectures [33] or extra training samples [85] to boost performance, in our case this was possible using vanilla ScaleGMNs. For example, in MNIST and F-MNIST, NG-GNN achieves 94.7 ± 0.3 and 74.2 ± 0.4 with 64 probe features which is apprpx. 2% and 6% below our best performance. The corresponding results for NG-T [33] (transformer) were 97.3 ± 0.2 and 74.8 ± 0.9, still on par or approx. 6% below ScaleGMN. Note that all the above are orthogonal to our work and can be used to further improve performance.Predicting CNN Generalization from weights. As in prior works. we consider datasets of image classifiers and measure predictive performance using Kendall's τ [30] . We select the two datasets used in [85] , namely CIFAR-10-GS and SVHN-GS originally from Small CNN Zoo [74] . These contain CNNs with ReLU or tanh activations, which exhibit scale and sign symmetries respectively. To assess the performance of our method (1) on each activation function individually and (2) on a dataset with heterogeneous activation functions we discern distinct paths for this experiment. In the first case, we split each dataset into two subsets each containing the same activation and evaluate all baselines. As shown in Table 2 , once again ScaleGMN outperforms all the baselines in all the examined datasets. This highlights the ability of our method to be used across different activation functions and architectures. Performance improvements here are less prominent due to the hardness of the task, a phenomenon also observed in the comparisons between prior works. Note that in this case, additional symmetries arise by the softmax classifier [36] , which are currently not accounted for by none of the methods. We additionally evaluate our method on heterogeneous activation functions.In principle, our method does not impose limitations regarding the homogeneity of the activation functions of the input NNs -all one needs is to have a different canonicalisation/symmetrisation module for each activation. Experiments on CIFAR-10-GS show that ScaleGMN yields superior performance compared to the baselines, significantly exceeding the performance of the next-best model. Further implementation details can be found in Appendix A.4.2.Here, our goal is to transform the weights of the input NN, to modify the underlying signal to a new one. In this case, our method should be equivariant to the permutation and scaling symmetries, such that every pair of equivalent input NNs is mapped to a pair of equivalent output NNs. Hence, we employ a ScaleGMN similar to the above experiments but omit the invariant readout layer. Following [33] , we evaluate our method on the MNIST dataset and train our model to dilate the encoded MNIST digits. Further implementation details can be found in Appendix A.4.3. As shown in Table 3 , our bidirectional ScaleGMN-B achieves an MSE test loss (10 -2 ) equal to 1.891, surpassing all permutation equivariant baselines. Notably, our method also outperforms the NG-GNN [33] baseline that uses 64 probe features. Additionally, our forward variant, ScaleGMN, performs on par with the previous permutation equivariant baselines with an MSE loss of 2.56. Note that the performance gap between the forward and the bidirectional model is probably expected for equivariant tasks: here we are required to compute representations for every node of the graph, yet in the forward variant, the earlier the layer of the node, the smaller the amount of information it receives. This observation corroborates the design choices of the baselines, which utilize either bidirectional mechanisms (NG-GNN [33] ) or non-local operations (NFN [85] ).Ablation study: Scaling data augmentations. We baseline our method with permutation equivariant methods trained with scaling augmentations: For every training datapoint, at each training iteration, we sample a diagonal scaling matrix for every hidden layer of the input NN and multiply it with the weights/bias matrices as per Eq. ( 3) (omitting the permutation matrices). We sample the elements of the matrices independently as follows: Sign symmetry: Bernoulli distribution with probability 0.5. Positive scaling: Exponential distribution where the coefficient λ is a hyperparameter that we tune on the validation set. Observe here that designing augmentations in the latter case is a particularly challenging task since we have to sample from a continuous and unbounded distribution. Our choice of the exponential was done by consulting the norm plots where in some cases the histogram resembles an exponential Fig. 1 . Nevertheless, regardless of the distribution choice we cannot guarantee that the augmentations will be sufficient to achieve (approximate) scaling equivariance, due to the lack of upper bound. We evaluate on the F-MNIST dataset for the sign symmetry and on the CIFAR-10-GS-ReLU for the positive scale. As shown in Table 4 , regarding the former, augmenting the training set leads consistently to better results when compared to the original baselines. None of these methods however achieved results on par with ScaleGMN and ScaleGMN-B. On the other hand, we were unable to even surpass the original baselines regarding the latter task. This indicates that designing an effective positive scaling augmentation might be a non-trivial task.Limitations. A limitation of our work is that it is currently designed for FFNNs and CNNs and does not cover other layers that either modify the computational graph (e.g. skip connections) or introduce additional symmetries (e.g. softmax and normalisation layers). In both cases, in future work, we plan to characterise scaling symmetries (certain steps were made in [25] ) and modify ScaleGMN for general computational graphs as in [44, 88] . Additionally, a complete characterisation of the functions that can be expressed by our scale/rescale equivariant building blocks is an open question (except for sign [40] ). Finally, an important theoretical matter is a complete characterisation of the expressive power of ScaleGMN, similar to all equivariant metanetworks. 
conclusion: 
 In this work, we propose ScaleGMN a metanetwork framework that introduces to the field of NN processing a stronger inductive bias: accounting for function-preserving scaling symmetries that arise from activation functions. ScaleGMN can be applied to NNs with various activation functions by modifying any graph metanetwork, is proven to enjoy desirable theoretical guarantees and empirically demonstrates the significance of scaling by improving the state-of-the-art in several datasets. With our work, we aspire to introduce a new research direction, i.e. incorporating into metanetworks various NN symmetries beyond permutations, aiming to improve their generalisation capabilities and broaden their applicability in various NN processing domains. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. 
 
 Appendix: In the following section, we describe the additional practical considerations that were omitted from Section 5. For the purposes of this section, we introduce the functions layer (i) and pos (i) which return the layer of a vertex i and its position within this layer respectively.As previously discussed, all vertex/edge representations throughout the network should respect the symmetries given by Eqs. ( 4), (5) . This is straightforward to achieve at initialisation, by using the NN biases and weights (similarly to [33] ). It is possible to use a scale equivariant layer here as well, e.g. a linear layer, which is what was performed in our empirical evaluation. For notation convenience, denote the biases and weights of Eq. ( 1), with b, W, such that b(i) = b layer(i) (pos (i)), W(i, j) = W layer(i) (pos (i) , pos (j)). Now we can define the initialisation:EQUATIONEQUATIONNote that the above is different from the initialisation/graph construction used in [44] . For example, in this work, an extra vertex is created for each bias, which is connected to its corresponding neuron vertex with an edge initialised with the bias value. However, this strategy would complicate scaling symmetry preservation and was thus not used in our work.As discussed in Section 3, not all vertices/edges of an FFNN are permutable. In particular, vertices corresponding to the input and output neurons are fixed, while vertices corresponding to hidden neurons can be permuted only within the same layer. Similarly, edges originating from an input neuron are permutable only with edges originating from the same neuron, and the same holds for edges terminating at an output neuron, while edges of hidden layers can be only permuted within the same layer.To adhere to the above, we need to devise a symmetry-breaking mechanism, to ensure that the GMN is not equivariant to undesired symmetries (e.g. permuting output neurons). This is the role of positional encodings, which are learnable vectors (in the literature fixed vectors have been also used), shared across all input graphs, that take part in message-passing. In particular, for vertex positional encodings the following holds:EQUATIONand for edge positional encodings:EQUATIONwhile otherwise, the positional encodings are free to take different values. Symmetry-breaking usually happens at initialisation (which is then inherited by the subsequent message-passing layers) and Eq. (Init) becomes:EQUATIONor alternatively, it can be performed at all layers, where Eqs. (Msg) and (Upd) become:EQUATIONEQUATION)Challenge 3 -Incorporating positional encodings into scale equivariant layers. Although the above formulation allows for the preservation of valid permutation symmetries, special treatment is needed to preserve valid scaling symmetries as well. Since the positional encodings are fixed across datapoints, they do not have the same scaling symmetries as vertex/edge representations. Therefore, message-passing components do not need to be equivariant to scaling of the positional encodings. In particular, the following should hold: INIT V (qx, p) = qINIT V (x, p), INIT E (qe, p) = qINIT E (e, p), MSG V q x x, q y y, q x q -1 y e, p x , p y , p e = q x MSG V (x, y, e, p x , p y , p e ) and UPD V (q x x, q x m, p) = q x UPD V (x, m, p). Therefore, Eqs. ( 15), ( 16), (17) introduce a new challenge (apart from those discussed in Section 5): being scale equivariant w.r.t. one argument and not equivariant. w.r.t. another one. Interestingly, our ScaleNet formalism can straightforwardly deal with this case, by defining the canonicalisation function for the non-symmetric argument to be the identity (since in the absence of symmetry equivalent points must be equal). To discern this from the classical scale invariant/equivariant layers, we will refer to it augmented scale invariant/equivariant layer and it is formalised as follows:EQUATIONEQUATIONConcluding, we obtain the new learnable functions, by simply replacing scale equivariant with augmented scale equivariant layers:EQUATIONEQUATIONEQUATIONA.1.3 Edge Updates.To respect the symmetries given by Eq. ( 5), for edges (i, j), with layer (i) ∈ {2, . . . , L -1} the edge update function should have the following property: UPD E (q x x, q y y, q x q -1 y e) = q x q -1 y UPD E (x, y, e). Observe, that this is once again reminiscent of the rescaling problems discussed in Section 5, but with the difference that q y appears both as a multiplier and as a divisor. The most straightforward solution would be to elementwise invert y and then proceed as in the message function of Eq. ( 8), i.e. : UPD E (x, y, e) = ScaleEq ([e, ReScaleEq (x, 1 ⊘ y)]) .(EQUATIONIn our experiments though, we opt for a simpler solution due to the numerical instabilities (highmagnitude gradients) produced by the element-wise division and choose an invariant layer for the arguments x, y. Formally: UPD E (x, y, e) = AugScaleEq (e, ScaleInv (x, y)) .When we include positional encodings into the edge updates, Eq. (Upd) (RHS) becomes:EQUATIONand Eq. ( 24) is further modified into:EQUATIONSo far, we have only discussed how vertex/edge representations corresponding to hidden neurons are updated, since Eqs. ( 4), (5) do not hold for input and output neurons. In particular, there exist no valid scaling or permutation symmetries for these neurons, while for the corresponding edges, permutations and scalings are unilateral -see Eq. ( 3). Therefore, we require the following:EQUATIONEQUATION)This implies that the conditions for the learnable functions should change for i/o neurons. In particular, the following should be non-scale-equivariant: INIT V,0 , MSG V,07 , UPD V,0 , INIT V,L , UPD V,L . The same should hold for MSG V,L , but with the additional requirement of invariance to rescaling: MSG V,L (x, q y y, q -1 y e, p x , p y , p e ) = MSG V,L (x, y, e, p x , p y , p e ) In traditional GNNs, where only permutation symmetries are taken into account, symmetry-breaking can be efficiently handled by positional encodings as discussed in Appendix A.1.2. However, this is not the case here, since positional encodings only break permutation symmetries, but not scaling symmetries. Instead, we resort to a simpler solution (albeit incurring an increased number of parameters): we use different initialisation, message and update functions for input/output neurons. INIT V,0 , MSG V,0 , UPD V,0 , INIT V,L , UPD V,L are general function approximators (MLPs), while MSG V,L (x, y, e, p x , p y , p e ) = MLP (x, ReScaleEq (y, e) , p x , p y , p e ).As for the i/o edge representations, the following should hold at initialisation: INIT E,1 (q x e, p) = q x INIT E,1 (e, p), INIT E,L (q -1 y e, p) = q -1 y INIT E,L (e, p), and for the edge updates: UPD E,1 (q x x, y, q x e, p x , p y , p e ) = q x UPD E,1 (x, y, e, p x , p y , p e ), UPD E,L (x, q y y, q -1 y e, p x , p y , p e ) = q -1 y UPD E,L (x, y, e, p x , p y , p e ). Therefore, edge initialisation should be equivariant to the edge scale and therefore can remain as is -Eq. ( 20). The same holds for the edge updates -Eq. ( 26), which allows us to retain the same architecture in our experiments, although the interested reader may observe that one can use more powerful architectures since there is no need for rescaling here.Regarding the readout, we devise a permutation and scale invariant aggregator as follows:EQUATIONThis means that we aggregate all scale canonicalised/symmetrised representations of hidden neurons with a permutation invariant aggregator (Deepsets [81] ), concatenate the result with all i/o neuron representations (that cannot be permuted or scaled) and transform the result with a universal function approximator. A simpler alternative is to only concatenate the output neuron representations (or also the input for the bidirectional version) since after sufficient message-passing iterations they are expected to have collected information from the entire graph. We include and evaluate both choices in our hyperparameter grid.Undoubtedly, using only forward neighbourhoods, although convenient and aligned with the computational graph of the input datapoint, may restrict the expressive power of ScaleGMN. This is self-explanatory since vertices receive information only from parts of the graph that belong to earlier layers. This might be detrimental, especially for equivariant tasks (operators). Additionally, it is contrary to the first works on weight space networks [54, 86] , where vertices/edges collect information from the entire graph, while backward neighbourhoods are crucial for GMNs to express NP-NFN [86] as shown in Proposition 3 in Lim et al. [44] .However, making bidirectional message passing scale equivariant is challenging since backward messages have different symmetries than forward ones. In particular, preserving the same assumptions for the symmetries of vertex/edge representations, the input triplet to a message function h V (i), h V (j), h E (j, i), with j ∈ N B (i) and ℓ = layer (i) is equivalent to q ℓ (i)h V (i), q ℓ+1 (j)h V (j), q ℓ+1 (j)h E (j, i)q -1 ℓ (i) (we have omitted permutations here for simplicity). Therefore, doing the same rescaling as before is inappropriate since it does not preserve the desired scaling coefficient q ℓ (i).There are several options one may follow to circumvent this discrepancy. However, most involve an elementwise division of a certain representation, which we experimentally observed to face numerical instabilities. The simplest one is to add backward edges and initialise them with inverted features:EQUATIONIn this case, we have that INIT E q ℓ+1 (j)q -1 ℓ (i)x E (j, i) = q ℓ (i)q -1 ℓ+1 (j)INIT E (x E (j, i)), and thus (j, i) are initialised with the same symmetries as (i, j). Now one can proceed as before with the message and update functions. In particular, we introduce a backward message function to discern the two message directions (alternatively this can be done with edge positional encodings to reduce the number of parameters):m t V,BW (i) = j∈N B (i) MSG t V,BW (h t-1 V (i), h t-1 V (j), h t-1 E (j, i), p V (i), p V (j), p E (j, i))MSG V,BW (x, y, e, p x , p y , p e ) = AugScaleEq ([x, ReScaleEq (y, e)] , [p x , p y , p e ]) .It is easy to see that: MSG V,BW (q x x, q y y, q -1 y q x e, p x , p y , p e ) = q x MSG V,BW (x, y, e, p x , p y , p e ). Now to incorporate backward messages into the update function we simply concatenate the outputs of the two message functions and apply a scale equivariant layer as usual:EQUATIONOnce, again the desired symmetries are preserved:UPD V (q x x, q x m FW , q x m BW , p x ) = q x UPD V (x, m FW , m BW , p x ).Bidirectional ScaleGMN for sign symmetries. Importantly, the above edge weight inversion of Eq. ( 31) is not necessary when dealing with sign symmetries. In particular, for any q ∈ {-1, 1} it holds that 1 q = q. Therefore, one can proceed with the above formulation but initialise the edge weights as before -Eq. ( 12). We found this crucial in our experimental evaluation since weight inversion led to an unusual distribution of input features: since typically the weight distribution was nearly Gaussian, the distribution of the inverses was close to reciprocal normal distribution, a bimodal distribution with undefined mean and higher-order moments. This frequently led to numerical instabilities as well (albeit less so compared to computing reciprocals of neuron/edge representations), which explains why it was easier to train bidirectional models for sign symmetric NNs compared to positive-scale symmetric ones.A similar methodology to Section 5 can be applied to Convolutional Neural Networks with only a few changes. As observed by prior works on metanetworks (Zhou et al. [85] , Kofinas et al. [33] ), the permutation symmetries in a CNN arise similarly to MLPs; permuting the filters (output channels) of a hidden convolutional layer, while also permuting the input channels of each filter in the subsequent layer does not alter the underlying function of the network. This is easy to see since convolution is nothing else but a linear layer with weight sharing. Weight sharing constrains the valid permutations as far as pixels/input coordinates are concerned (e.g. one cannot permute the representations corresponding to different pixels), but allows for permutations of input/output channels, except for the input/output layers as always.Similar rules apply to scaling: one may scale the output channels of a hidden convolutional layer, while also scaling the input channels of each filter in the subsequent layer as per Proposition 4.1. Note here that, again, all weights corresponding to the same input channel can only be scaled with the same multiplier for all pixels/input coordinates, due to weight sharing. Below, we describe our implementation for all CNN layers, which closely follows [33] .Convolution. Being the main building block of CNNs, convolutional layers consist of kernels and biases. For a hidden layer i we can write the former as W i ∈ R d ℓ ×d ℓ-1 ×k ℓ and the latter as b i ∈ R d ℓ following the FFNN notation. Aligning with the procedure of [33] , we construct one vertex for each i/o channel and the input graph has the following node and edge features:x V (i) = b(i) ∈ R, x E (i, j) = W(i, j) ∈ R wmax•hmaxrespectively, where w max , h max are the maximum width and height of the kernels across layers (zero-padding is used to allow for fixed-size edge representations across all vertices in the graph). Again here, the positional encodings are responsible for the permutation symmetry breaking mechanisms within the CNN. As with FFNNs, the CNN input and output neurons are not permutable, while the hidden neurons are permutable only within the layer they belong. Given that filter positions are not permutable and are associated with a single neuron, they are represented as vectors on the edge features.Average Pooling. Average pooling is commonly placed after the last convolutional layer (typical in modern CNNs instead of flattening + linear layer to allow for variable-size images) and is responsible for pooling the output feature map in a single feature vector. First off, since it is a linear operation, it is amenable to scaling symmetries. It is easy to see this by perceiving average pooling as a KxK convolution with no learnable parameters (equal to 1 K 2 , where K the dimension of the image, and a single output channel. Hence, no additional modifications are required, similarly to [33] .Linear layer. Commonly used as the last layer of CNNs to extract a vector representation of an image, linear layers require a different approach within CNNs compared to the one in MLPs, as in the former case the edge features are vectors, while in the latter scalars. Aligning with the method proposed by [33] , the simplest solution is to zero-pad the linear layers to the maximum kernel size and consequently flatten them to meet the dimension of the rest of the edge features. By handling linear layers this way, no additional measures are needed to ensure permutation and scale equivariance.Datasets: We evaluated our method on three INR datasets. Provided as open-source by Navon et al. [54] , the datasets MNIST and FashionMNIST contain a single INR for each image of the datasets MNIST [38] and FashionMNIST [76] respectively. The selection of these datasets was encouraged by the fact that they were also selected by prior works, establishing them as a useful first benchmark on INR metanetworks. As a third INR dataset, we use CIFAR-10, publicly available by Zhou et al. [85] , which contains one INR per image from CIFAR10 [35] . Regarding the last dataset, the authors also provide an augmented training dataset, consisting of 20 additional copies of each INR network, but with different initializations. Training on such a bigger dataset allows them to achieve better results, probably because it allows them to counteract overfitting. To demonstrate the capabilities of our method we train ScaleGMN both on the "original" dataset as well as on the augmented one, achieving better results in both cases. In the latter case we carefully follow the training and evaluation procedure used in [85] , i.e. we train for 20000 steps and evaluate every 3000 steps. In all the above datasets, we use the same splits as in prior works, we train for 150 epochs and report the test accuracy based on the validation split, following the same procedure as Navon et al. [54] .Hyperparameters: The GNN used for ScaleGMN is a traditional Message Passing Neural Network [24] , with carefully designed update and message functions that ensure sign equivariance. We optimise the following hyperparameters: batch size in {64, 128, 256}, hidden dimension for node/edge features in {64, 128, 256}. We also search learning rates in {0.001, 0.0005, 0.0001}, weight decay in {0.01, 0.001}, dropout in {0, 0.1, 0.2} and number of GNN layers in {2, 3, 4, 5}. Moreover, we experiment with using only vertex positional encodings or also employing edge positional encodings. We apply layer normalization within each MLP and use skip connections between each GNN layer. The last two steps were proven valuable to stabilise training. Finally, for each MLP within the architecture we use SiLU activation function, one hidden layer and no activation function after the last layer. All the experiments were conducted on NVIDIA GeForce RTX 4090.As mentioned above, the core of all building blocks, i.e. Scale Invariant Net is designed to be invariant to sign flips. In our experiments, we explore two means of designing sign equivariance, either by elementwise absolute value |x| (canonicalization only in 1-dimension), or symmetrisation, symm(x) = MLP(x) + MLP(-x). Although the former leads to reduced expressivity, in some cases we observed better performance. A possible explanation could point to the extra training parameters that are added because of the additional MLP. Table 1 , our method is able to push the state-of-the-art in all the datasets considered. Notably, many additional techniques that were employed by previous works in order to achieve good results were not required in our method. Specifically, probe features when inserted into the node features as in [33] were proven to boost the performance much higher, and could potentially enhance the performance of ScaleGMN too. In the reported results we compare with variants of [33] that do not use probe features and rely solely on the parameters of the input network, although our method outperforms even the probe features variants. Moreover, [85] and [33] also apply random Fourier features during the initialization of the weight and bias features, which in our experiments showcased better performance when used in a traditional MPNN framework, but were not employed for our method. In our framework, we also do not include any normalization of the input as conducted by previous works, where the normalization is applied either on parameter level ( [85] , [54] ) or on layer level [33] , computing the mean and standard deviation. Although this step typically leads to better performance, it could not be applied in our scale and permutation equivariant framework. Nevertheless, even without all the above techniques, ScaleGMN achieves remarkable results. Finally, opting for a Transformer, as in [33] , instead of a GNN layer is also an orthogonal to our work addition that could enhance the performance of ScaleGMN.For evaluating our method on the CNN Generalization task, we select CIFAR-10-GS and SVHN-GS datasets from [74] , following the evaluation of prior works (Zhou et al. [85] , Zhou et al. [86] , Lim et al. [44] , Kofinas et al. [33] ). The above datasets contain CNNs trained with ReLU and Tanh activation functions. In order to assess our method separately for each of these activation functions, as they introduce different symmetries, we distinguish two different types of experiments. In the first case we deviate from the path of previous evaluations and split each of these datasets into two subsets. The first one contains CNNs trained with ReLU activation function and the second networks with Tanh. In order to compare with previous methods, we train them on each subset and report Kendall's correlation τ metric [30] as originally proposed by Zhou et al. [85] .Heterogeneous activation functions: In the second case, we evaluate ScaleGMN on the whole CIFAR-10-GS and SVHN-GS datasets. We are able to do so, as our method does not impose any limitations regarding the homogeneity of the activation functions of the input neural networks. The sole necessary modification lies in the construction of the invariant layer. Specifically, following the notation of Eq. (Scale Equiv. Net), each equivariant layer f k is now equipped with two invariant nets, ScaleInv k relu and ScaleInv k tanh , to be applied on the datapoints with ReLU and Tanh activation functions respectively. The rest of the architecture remains the same.Hyperparameters: For the ReLU datasets we implement scale equivariant update and message functions, while for the Tanh sign equivariant. In all of our experiments we use MSE training loss. Regarding the hyperparameters we search: batch size in {32, 64, 128}, hidden dimension for node/edge features in {64, 128, 256}. We also search learning rates in {0.001, 0.0005, 0.0001}, weight decay in {0.01, 0.001}, dropout in {0, 0.1, 0.2} and number of GNN layers in {2, 3, 4, 5}. Again, we experiment with using only node positional encodings or also employing edge positional encodings and apply layer normalization within each MLP. For the Tanh datasets, we also experiment with the applying canonicalization or symmetrization of the sign symmetry. Finally, for the experiments with heterogeneous activation functions, we apply the same hyperparameter search as above.To evaluate the performance of ScaleGMN on equivariant tasks we opted for the task of INR editing. Aligning with the setup of [33] , we utilize the MNIST INR dataset provided by Navon et al. [54] and evaluate our method on dilating the encoded MNIST digits. Specifically, for each image i of the dataset, where i ∈ [N ], we compute the ground truth dilated image using the OpenCV image processing library and denote it as f i , where f i : R2 → R. Let θ i denote the INR parameters of the i-th image and f SIREN the encoded function. Then, f SIREN (x, y; θ i ) is the output of the INR at the (x, y) coordinates, when parameterized by θ i . The updated INR weights are computed as:θ ′ i = θ i + γ • ScaleGMN(θ i ), where γ a learned scalar initialized to 0.01. Finally, we minimize the mean squared error on the function space, that is:L = 1 N • d 2 N i=1 d x,y ∥f SIREN (x, y; θ ′ i ) -f i (x, y)∥ 2Simply put, we compare the reconstructed new INR with the dilated ground truth image. Again, we do not apply any normalizations nor feed the model samples of the encoded input image (probe features).Hyperparameters: The GNN used for ScaleGMN is a traditional Message Passing Neural Network [24] , with carefully designed update and message functions that ensure sign equivariance. In contrast to the above experiments we do not add a final readout layer, as we do not compute any final graph embedding. We optimise the following hyperparameters: batch size in {64, 128, 256}, hidden dimension for node/edge features in 64, 128, 256 and number of GNN layers in {8, 9, 10, 11}. We also search learning rates in {0.001, 0.0005, 0.0001}, weight decay in {0.01, 0.001} and dropout in {0, 0.1, 0.2}. Moreover, we experiment with using only vertex positional encodings or also employing edge positional encodings. We apply layer normalization within each MLP and use skip connections between each GNN layer. The last two steps were proven valuable to stabilise training. Finally, for each MLP within the architecture we use SiLU activation function, one hidden layer and no activation function after the last layer. The learned scalar γ, also used in previous work Zhou et al. [85] , was always initialized at the same value 0.001 and consistently led to better results. All the experiments were conducted on NVIDIA GeForce RTX 4090.Results: As shown in Table 3 , the bidirectional variant achieves better results than all the previous works on metanetworks, without incorporating any of the additional techniques described above. Notably, ScaleGMN-B surpasses the performance of the graph-based permutation equivariant baseline NG-GNN equipped with 64 probe features [33] . Interestingly, our method demonstrated much better results as we added more GNN layers. This experimental observation possibly corroborates with the findings of Theorem 5.2. Regarding our forward variant, we can see that although it is able to perform on par with the previous baselines, it struggles to match the results achieved by the bidirectional variant. This behaviour, observed exclusively in this task, is theoretically anticipated for equivariant tasks. Specifically, in the forward case, neurons of layer ℓ only receive information from layers ℓ ′ < ℓ, which hinders the computation of meaningful embeddings for nodes of the first layers. Taking into account that the INR models are rather shallow (only 2 hidden layers of 32 neurons each), then a big proportion of the whole set of nodes is being updated with less meaningfull information.Regarding the sine activation function, one can easily show the presence of an additional symmetry due to the periodicity of harmonic functions. To see this, recall from Eq. ( 1) the formula for a feedforward L-layer SIREN:x 0 = x, x ℓ (i) = sin ω 0 W ℓ (i, :)x ℓ-1 + b ℓ (i) .We will prove that the following guarantees function preservation:EQUATIONwith Q ℓ defined as diagonal sign matrices,O ℓ = 2M ℓ + inv-sign(Q ℓ ), where M ℓ = diag m ℓ (1), . . . m ℓ (d ℓ ), m ℓ (i) ∈ Z and we defined inv-sign(q) = 0, q ≥ 0 1, q < 0.for compactness (we have omitted permutation matrices here for brevity). Observe here that the matrix transformations are "single-sided", contrary to Eq. ( 3). We show by induction that, when Eq. ( 34) holds, thenx ′ ℓ = x ℓ . Base case: x ′ 0 = x 0 by defintion. Induction step: Assume x ′ ℓ-1 = x ℓ-1 . In order to have x ′ ℓ = x ℓ , ∀x ∈ X , it suffices that ∃m ℓ (i) ∈ Z, q ℓ (i) ∈ {-1, 1}: ω 0 W ′ ℓ (i, :)x ℓ-1 + b ′ ℓ (i) = (ω 0 W ℓ (i, :)x ℓ-1 + b ℓ (i)) q ℓ (i) + 2m ℓ + inv-sign(q ℓ (i)) π ⇐W ′ ℓ (i, :) = q ℓ (i)W ℓ (i, :), b ′ ℓ (i) = q ℓ (i)b 1 (i) + 2m ℓ (i) + inv-sign(q ℓ (i)) π.This can be straightforwardly extended to include permutations.To account for those symmetries, in this paper, we opted for a simple solution, based on the observation that they cannot exist if the biases are bounded. In particular, if b ′ ℓ (i), b ℓ (i) ∈ (-π/2, π/2], then m must be 0 and q must be 1 in all cases: For q ℓ (i) = 1, since 0 ≤ |b ℓ (i) -b ℓ (i)| < π, it must be that 0 ≤ |2m ℓ (i)| < 1 ⇔ 0 ≤ |m ℓ (i)| < 1/2 ⇒ m = 0. For q ℓ (i) = -1, since -π < b ℓ (i)+b ℓ (i) ≤ π, it must be that -1 < 2m ℓ (i) + 1 ≤ 1 ⇔ -1 < m ℓ (i) ≤ 0 ⇒ m ℓ (i) = 0. Further, since theℓ (i), w: weight row W ℓ (i, :) 2: if b < 0 then 3: b 1 ← -b; w 1 ← -w; q 1 ← -1 // sin(ω 0 w ⊤ 1 x ℓ-1 + b 1 ) = -sin(ω 0 w ⊤ x ℓ-1 + b) 4: else 5: b 1 ← b; w 1 ← w; q 1 ← 1 6: end if// Now b 1 ≥ 0. 7: if b 1 > 2π then 8: b 2 ← b 1 mod 2π; w 2 ← w 1 ; q 2 ← 1 // b 1 = 2πk + b 2 ⇒ sin(ω 0 w ⊤ 2 x ℓ-1 + b 2 ) = sin(ω 0 w ⊤ 1 x ℓ-1 + b 1 ) 9: else 10: b 2 ← b 1 ; w 2 ← w 1 ; q 2 ← 1 11: end if// Now 0 ≤ b 2 ≤ 2π. 12: if π < b 2 ≤ 2π then 13: b 3 ← b 2 -π; w 3 ← w 2 ; q 3 ← -1 // sin(ω 0 w ⊤ 3 x ℓ-1 + b 3 ) = -sin(ω 0 w ⊤ 2 x ℓ-1 + b 2 ) 14: else 15: b 3 ← b 2 ; w 3 ← w 2 ;q 3 ← 1 16: end if// Now 0 ≤ b 3 ≤ π. 17: if 0 ≤ b 3 ≤ π/2 then 18: b 4 ← b 3 ; w 4 ← w 3 ; q 4 ← 1 19: Return: (b 4 , w 4 , q 4 ) 20: else 21: b 4 ← b 3 -π; w 4 ← w 3 ; q 4 ← -1 // sin(ω 0 w ⊤ 4 x ℓ-1 + b 4 ) = -sin(ω 0 w ⊤ 3 x ℓ-1 + b 3 ) 22:Return: (b 4 , w 4 , q 4 ) 23: end if// Now -π/2 < b 4 ≤ π/2. 24: Output: New bias bℓ (i) ← q i * b 4 and new weight row Wℓ (i, :) ← q i * w 4 . // sin( q i ω 0 w ⊤ 4 x ℓ-1 + q i b 4 ) = q i q 4 sin(ω 0 w ⊤ 3 x ℓ-1 +b 3 ) = • • • = sin(ω 0 w ⊤ x ℓ-1 +b)equality holds only when both biases are equal to π/2, then the case q ℓ (i) = -1, m = 0 reduces to q ℓ (i) = 1, m = 0.Overall, by constraining the biases in the (-π/2, π/2] interval, these symmetries are eliminated. In all datasets considered, this constraint was already satisfied. For completeness, we provide Algorithm 1 that illustrates a method to shift biases in the desired interval.Empirically, accounting for a given symmetry might not prove fruitful if the given dataset is by construction canonicalised, e.g. if all parameters were already positive or negative in the presence of sign symmetries. To ensure the occurrence of the studied symmetries within the used datasets, we explore their statistics. Regarding the positive-scaling symmetry, the distribution of norms of weights and biases within each network layer in the CIFAR-10-GS-ReLU dataset is shown in Fig. 1 .Concerning sign symmetries, we similarly plot the distribution of the sign values for the MNIST-INR dataset in Fig. 2 and the CIFAR-10-GS-tanh dataset in Fig. 3 . Observe that in the former case, the distribution has relatively high variance, showcasing that the weights/biases are not by training scale-normalised. Even more prominently, in the latter case, the probability of each sign value is close to 1/2 which corroborates our claims on the need for sign equivariance/invariance.In the following section, we explore key questions raised by the reviewers during the evaluation process. These questions touch upon interesting aspects of our methodology that merit in-depth exploration.In the bidirectional case, we add backward edges to our graph and introduce a backward message Trainable parameters: GMN vs ScaleGMN(-B)? Going from a GMN model to ScaleGMN requires adapting the MSG and UPD functions to be scale equivariant, leading to more learnable parameters, as opposed to using plain MLPs. Another design choice of ScaleGMN that introduces more parameters is using different MLPs for the I/O nodes (A.1.4).Are there any limitations on the choice of activations of the ScaleGMN network? Importantly, our method does not impose any limitations on the choice of the activation functions. We are able to select any activation function, because these are only applied within the MLPs of the invariant modules. As discussed in Section 5, the MLPs (equipped with non-linearities) are only applied after the canonicalization/symmetrization function. In case one chose to place activations in a different computational part of ScaleGMN, this would indeed limit their options so as not to compromise scale equivariance. However, this is not the case in our method.Canon/symm mappings for all activation functions. In general, we cannot guarantee that canonicalization/symmetrization mappings are easy to construct for any activation function, since it is currently unknown if we can provide a general characterisation of the possible symmetries that may arise. Our results extend to all positively homogeneous activations, σ(λx) = λσ(x), λ > 0, and all odd ones, σ(-x) = -σ(x). We refer the reviewer to Table 1 of Godfrey et al. [25] , where we can see that LeakyReLU also falls in the first category. Regarding polynomial activations, which demonstrate non-zero scaling symmetries, one option would be: (1) norm division to canonicalise scale, and (2) sign symm/canon as discussed in this paper. The above, cover a fairly broad spectrum of common activation functions.First we restate Proposition 4.1 and provided the detailed conditions for the activation function σ.Proposition A.1 (Proposition 4.1 restated). Consider an activation function σ : R → R.• (Lemma 3.1. [25] .) If the matrix σ(I d ) is invertible, then, for any d ∈ {1, 2, . . . }, there exists a (non-empty) group of matrices, called the intertwiner group of σ d defined as follows:I σ,d = {A ∈ R d×d : invertible | ∃ B ∈ R d×d invertible, such that: σ(Ax) = Bσ(x)},and a group homomorphism ϕ σ,d :I σ,d → GL d (R), such that B = ϕ σ,d (A) = σ(A)σ(I d ) -1 .• (Theorem E.14. [25] .) Additionally, if σ is non-constant, non-linear and differentiable on a dense open set with finite complement, then every A ∈ I σ,d is of the form PQ, where P: permutation matrix andQ = diag q 1 , . . . q d diagonal matrix, with ϕ σ,d (A) = Pdiag ϕ σ,1 (q 1 ), . . . ϕ σ,1 (q d )Note that I d is the identity matrix GL d (R) is the general linear group. The above two statements are proven in [25] . The exact form of ϕ σ,d (A) is not precisely given, but it is easy to derive: ϕ σ,d is a group homomorphism and therefore ϕ σ,d (PQ) = ϕ σ,d (P)ϕ σ,d (Q) = Pdiag ϕ σ,1 (q 1 ), . . . ϕ σ,1 (q d ) , where we also used Lemma E.10. from [25] which states that ϕ σ,d (P) = P for permutation matrices.Proposition A.2 (restated). Hyperbolic tangent activation function σ(x) = tanh(x) and SIREN activation function, i.e. σ(x) = sin(ωx), satisfy the conditions of Proposition 4.1, when (for the latter) ω ̸ = kπ, k ∈ Z. Additionally, I σ,d contains signed permutation matrices of the form PQ, with Q = diag(q 1 , . . . , q d ), q i = ±1 and ϕ σ,d (PQ) = PQ.Proof. Example 1: σ(x) = sin(ωx), with ω > 0 constant: This is the most common activation function for INRs [70] . We can easily find the conditions under which σ(I d ) is invertible invoking Lemma E.7. from Godfrey et al. [25] . In particular, the latter states that if σ(1) ̸ = σ(0), σ(1) ̸ = -(d -1)σ(0), then σ(I d ) is invertible.In our case, σ(0) = sin(ω0) = 0, σ(1) = sin(ω). Therefore, it should hold that ω ̸ = kπ, k ∈ Z, otherwise σ(I d ) is an all-zeros matrix. Additionally, σ is non-constant, non-linear and differentiable everywhere. Thus, as dictated by Proposition 4.1, we can characterise the intertwiner group, by identifying the function ϕ σ,1 (a). In particular, b = ϕ σ,1 (a) if the following holds:sin(ωax) = b sin(ωx) ⇒ sin 2 (ωax) = b 2 sin 2 (ωx) ⇒ 1 -cos 2 (ωax) = b 2 sin 2 (ωx) ωa cos(ωax) = bω cos(ωx) ⇒ a 2 cos 2 (ωax) = b 2 cos 2 (ωx),where in the second line we took the derivative of each side since the equation holds for all x ∈ R and divided by ω ̸ = 0. Now summing up the two equations we obtain:1 + cos 2 (ωax) a 2 -1) = b 2 ⇒ cos 2 (ωax) a 2 -1) = b 2 -1 ⇒ a 2 -1 = b 2 -1 = 0.The above is true because if a 2 ̸ = 1, then one would conclude that cos 2 (ωax) = constant. Thus, a = ±1 and b = ±1. Additionally a cannot be different than b, since in that case we would have e.g. sin(ωx) = -sin(ωx) ⇒ sin(ωx) = 0. Therefore: We can easily verify that σ(I d ) is invertible invoking the same Lemma E.7. from Godfrey et al. [25] as before. We have, σ(0) = tanh(0) = 0, σ(1) = tanh(1) = e 2 -1 e 2 +1 ̸ = 0. Additionally, σ is non-constant, non-linear and differentiable everywhere. Thus, again as dictated by Proposition 4.1, we will characterise the intertwiner group, by identifying the function ϕ σ,1 (a). In particular, b = ϕ σ,1 (a) if the following holds:tanh(ax) = b tanh(x) ⇒ tanh 2 (ax) = b 2 tanh 2 (x) a 1 -tanh 2 (ax) = b 1 -tanh 2 (x),where in the second line we took the derivative of each side since the equation holds for all x ∈ R. Now summing up the two equations we obtain: Proof. We will prove this by induction. In particular, consider two equivalent parameterisations θ ′ , θ as in Eq. ( 3). This implies that vertex/edge features will have the same symmetries, i.e. starting with the hidden neurons:a + tanh 2 (ax) 1 -a) = b + tanh 2 (x) b 2 -b) ⇒ tanh 2 (x) 1 -α -b 2 + b) = b -a ⇒ b = ax ′ V (i) = q ℓ (π ℓ (i)) x V (π ℓ (i)) , ℓ = layer (i) ∈ {1, . . . , L -1} x E (i, j) = q ℓ (π ℓ (i)) x E (π ℓ (i) , π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) , ℓ = layer (i) ∈ {2, . . . , L -1},We will show that these symmetries are propagated to representations for every t ≥ 0, i.e. that Eq. ( 4) and Eq. ( 5). Let us start with the base t = 0.h ′ 0 V (i) = INIT V (x ′ V (i) , p V (i)) = INIT V q ℓ (π ℓ (i)) x V (π ℓ (i)) , p V (i) = q ℓ (π ℓ (i)) INIT V x V (π ℓ (i)) , p V (i) = q ℓ (π ℓ (i)) h 0 V (π ℓ (i)) , h ′ 0 E (i) = INIT E (x ′ E (i, j) , p E (i, j)) = INIT E q ℓ (π ℓ (i)) x E (π ℓ (i) , π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) , p E (i, j) = q ℓ (π ℓ (i)) q -1 ℓ-1 (π ℓ-1 (j)) INIT E x E (π ℓ (i) , π ℓ-1 (j)) , p E (i, j) = q ℓ (π ℓ (i)) q -1 ℓ-1 (π ℓ-1 (j)) h 0 E (π ℓ (i) , π ℓ-1 (j)) .This is due to the properties of the initialisation functions -see Eq. ( 20). Now we proceed with the induction step. Suppose Eq. ( 4) and Eq. ( 5) hold for t -1. We will show this implies they hold for t:m ′ t V (i) = j∈NFW(i) MSG t V h ′ t-1 V (i), h ′ t-1 V (j), h ′ t-1 E (i, j) = j∈NFW(i) MSG t V q ℓ (π ℓ (i)) h t-1 V (π ℓ (i)) , q ℓ-1 (π ℓ-1 (j)) h t-1 V (π ℓ-1 (j)) , q ℓ (π ℓ (i)) h t-1 E (π ℓ (i) , π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) = j∈NFW(i) q ℓ (π ℓ (i)) MSG t V h t-1 V (π ℓ (i)) , h t-1 V (π ℓ-1 (j)) , h t-1 E (π ℓ (i) , π ℓ-1 (j)) = q ℓ (π ℓ (i)) j∈NFW(i) MSG t V h t-1 V (π ℓ (i)) , h t-1 V (j) , h t-1 E (π ℓ (i) , j) = q ℓ (π ℓ (i)) m t V (π ℓ (i)). h ′ t V (i) = UPD t V h ′ t-1 V (i), m ′ t V (i) = UPD t V q ℓ (π ℓ (i)) h t-1 V (π ℓ (i)), q ℓ (π ℓ (i)) m t V (π ℓ (i)) = q ℓ (π ℓ (i)) UPD t V h t-1 V (π ℓ (i)), m t V (π ℓ (i)) = q ℓ (π ℓ (i)) h t V (i) h ′ t E (i, j) = UPD t E h ′ t-1 V (i), h ′ t-1 V (j), h ′ t-1 E (i, j) = UPD t E q ℓ (π ℓ (i)) h t-1 V (π ℓ (i)), q ℓ-1 (π ℓ-1 (j)) h t-1 V (π ℓ-1 (j)), q ℓ (π ℓ (i)) h t-1 E (π ℓ (i) , π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) = q ℓ (π ℓ (i)) q -1 ℓ-1 (π ℓ-1 (j)) UPD t E h t-1 V (π ℓ (i)), h t-1 V (π ℓ-1 (j)), h t-1 E (π ℓ (i) , π ℓ-1 (j)) = q ℓ (π ℓ (i)) q -1 ℓ-1 (π ℓ-1 (j)) h t E (π ℓ (i) , π ℓ-1 (j)) . Therefore we have shown the desideratum for hidden neurons, where again we used the properties of the message and update functions (we omitted positional encodings here for brevity). Now for the input neurons, we have:x ′ V (i) = x V (i), ℓ = layer (i) = 1 x E (i, j) = q ℓ (π ℓ (i)) x E (π ℓ (i) , j) , ℓ = layer (i) = 1,Further, we perform the induction again for Eq. ( 27) and Eq. ( 28) hold. Base case t = 0.h ′ 0 V (i) = INIT V,0 (x ′ V (i) , p V (i)) = INIT V,0 x V (i), p V (i) = INIT V,0 x V (i), p V (i) = h 0 V (i) .Induction step:m ′ t V (i) = 0 = m t V (i). h ′ t V (i) = UPD t V h ′ t-1 V (i), m ′ t V (i) = UPD t V h t-1 V (i), m t V (i) = h t V (i), since input vertices do not have any incoming edge. Finally, for the output neurons, we have:x ′ V (i) = x V (i), ℓ = layer (i) = L x E (i, j) = q -1 ℓ-1 (π ℓ-1 (j)) x E (i, π ℓ-1 (j)) , ℓ = layer (i) = L, Induction for Eq. ( 27) and Eq. ( 29). Base case t = 0.h ′ 0 V (i) = INIT V,L (x ′ V (i) , p V (i)) = INIT V,L x V (i), p V (i) = INIT V,L x V (i), p V (i) = h 0 V (i) , h ′ 0 E (i) = INIT E,L (x ′ E (i, j) , p E (i, j)) = INIT E,L q -1 ℓ-1 (π ℓ-1 (j)) x E (i, π ℓ-1 (j)) , p E (i, j) = q -1 ℓ (π ℓ-1 (j)) INIT E,L x E (i, π ℓ-1 (j)) , p E (i, j) = q -1 ℓ (π ℓ-1 (j)) h 0 E (i, π ℓ-1 (j)) .And the induction step:m ′ t V (i) = j∈NFW(i) MSG t V h ′ t-1 V (i), h ′ t-1 V (j), h ′ t-1 E (i, j) = j∈NFW(i) MSG t V h t-1 V (i) , q ℓ-1 (π ℓ-1 (j)) h t-1 V (π ℓ-1 (j)) , h t-1 E (i, π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) = j∈NFW(i) MSG t V h t-1 V (i) , h t-1 V (π ℓ-1 (j)) , h t-1 E (i, π ℓ-1 (j)) = j∈NFW(i) MSG t V h t-1 V (i) , h t-1 V (j) , h t-1 E (i, j) = m t V (i). h ′ t V (i) = UPD t V h ′ t-1 V (i), m ′ t V (i) = UPD t V h t-1 V (i), m t V (i) = h t V (i) h ′ t E (i, j) = UPD t E h ′ t-1 V (i), h ′ t-1 V (j), h ′ t-1 E (i, j) = UPD t E h t-1 V (i), q ℓ-1 (π ℓ-1 (j)) h t-1 V (π ℓ-1 (j)), h t-1 E (i, π ℓ-1 (j)) q -1 ℓ-1 (π ℓ-1 (j)) = q -1 ℓ-1 (π ℓ-1 (j)) UPD t E h t-1 V (i), h t-1 V (π ℓ-1 (j)), h t-1 E (i, π ℓ-1 (j)) = q -1 ℓ-1 (π ℓ-1 (j)) h t E (i, π ℓ-1 (j)) . Again we used the properties of the message and update functions we defined in Appendix A.1.4. And with that, we conclude the proof.Theorem A.4 (restated). Consider a FFNN as per Eq. (1) with activation functions respecting the conditions of Proposition 4.1. Assume a Bidirectional-ScaleGMN with vertex update functions that can express the activation functions σ ℓ and their derivatives σ ′ ℓ . Further, assume that ScaleGMN has access to the inputs x 0 and the gradients of an (optional) loss function L w.r.t. to the output ∇ x L L, via its positional encodings. Then, ScaleGMN can simulate both a forward and a backward pass of the FFNN, by storing pre-activations, post-activations and their gradients at the vertex representations. In particular, to compute the forward pass at the FFNN layer ℓ, t ≥ ℓ ScaleGMN layers are required, while for the corresponding backward, the requirement is t ≥ 2L -ℓ.Proof. Consider a loss function L : R dout × R dout → R, which is computed on the output of the FFNN L(x L , •). The second argument is optional and is used when we have labelled data -we will omit it in our derivations for brevity. Our proof strategy is based on two observations: First, the pre-activations of each layer (denoted here as z ℓ = W ℓ x ℓ-1 + b ℓ ) have the same symmetries as the biases. This is straightforward to see by induction:z ′ 0 = x 0 = z 0 , x ′ 0 = x 0 (base case 1) z ′ 1 = A 1 W 1 x 0 + A 1 b 1 = A 1 z 1 , x ′ 1 = σ 1 (A 1 z 1 ) = ϕ σ1 (A 1 )x 1 (base case 2) z ′ ℓ = A ℓ W ℓ ϕ σ ℓ-1 A -1 ℓ-1 x ′ ℓ-1 + A ℓ b ℓ = A ℓ W ℓ ϕ σ ℓ-1 A -1 ℓ-1 ϕ σ ℓ-1 (A ℓ-1 )x ℓ-1 + A ℓ b ℓ = A ℓ z ℓ , x ′ ℓ = ϕ σ ℓ (A ℓ )x ℓ (induction step)Second, the gradients of L w.r.t. the pre-activations of each layer (denoted here as∇ z ℓ L(x L ) = ∂z ℓ+1 ∂z ℓ ⊤ ∇ z ℓ+1 L(x L ), where the dependence of x L on z ℓ is implied -sometimes we will omit the argument x L for brevity ) have the inverted + transpose symmetries of the biases. This is due to the invariance of the output to the symmetries considered:x L (z ′ ℓ ) = x L (z ℓ ) ⇐⇒ L (x L (z ′ ℓ )) = L (x L (z ℓ )) ⇐⇒ ∇ z ℓ L (x L (z ′ ℓ )) = ∇ z ℓ L (x L (z ℓ )) ⇐⇒ ∂z ′ ℓ ∂z ℓ ⊤ ∇ z ′ ℓ L (x L (z ′ ℓ )) = ∇ z ℓ L (x L (z ℓ )) ⇐⇒ A ⊤ ℓ ∇ z ′ ℓ L (x L (z ′ ℓ )) = ∇ z ℓ L (x L (z ℓ )) ⇐⇒ ∇ z ′ ℓ L = A ⊤ ℓ -1 ∇ z ℓ L,and similarly ∇ x ′ ℓ L = ϕ σ ℓ (A ℓ ) ⊤ -1 ∇ x ℓ L. Now recall that in the cases we consider in this work, ϕ σ ℓ (A) = A and A = PQ. Therefore, given that (P ⊤ ) -1 = P and (Q ⊤ ) -1 = Q -1 , we have:EQUATION)The latter is what motivates us to enforce vertex representations to have the same symmetries with the biases: Assuming sufficient expressive power of ScaleGMN, we can anticipate that vertex representations will be able to reconstruct (1) the pre-or post-activations and (2) the elementwise-inverse of the pre-or post-activation gradients.Now recall the formulas that we want to recover:z ℓ (i) = j∈N F (i) W ℓ (i, j)x ℓ-1 (j) + b ℓ (i) = j∈N F (i) W ℓ (i, j)σ ℓ-1 z ℓ-1 (j) + b ℓ (i) x ℓ (i) = σ ℓ (z ℓ (i)) = σ ℓ   j∈N F (i) W ℓ (i, j)x ℓ-1 (j) + b ℓ (i)   ∇ z ℓ (i) L = ∂z ℓ+1 ∂z ℓ ⊤ ∇ z ℓ+1 L (i) = σ ′ ℓ z ℓ (i) j∈N B (i) W ℓ (j, i)∇ z ℓ+1 (j) L ∇ x ℓ (i) L = ∂z ℓ+1 ∂x ℓ ⊤ ∇ z ℓ+1 L (i) = j∈N B (i) W ℓ (j, i)∇ z ℓ+1 (j) LNow let us proceed to the proof. We will show that there exists a parameterization of ScaleGMN, such that after t = ℓ layers of message-passing, all vertices i with layer (i) ≤ ℓ will have stored in a part of their representation the pre-/post-activations for one or more inputs to the datapoint NN. Additionally, we will show that after t = 2L -ℓ layers of message-passing, all vertices i with layer (i) ≥ ℓ will have stored in a part of their representation the pre-/post-activation gradients for one or more inputs to the datapoint NN. Therefore, L layers of (forward) message-passing are needed to compute the output of the datapoint NN, and 2L layers of (bidirectional) message passing to calculate the gradients of the output/loss w.r.t. the input.For reasons that will become clearer as we proceed with the proof, we construct the below parameterization of ScaleGMN. First off, recall the input vertex and edge representations are given by:xV (i) = 1, layer (i) = 0, b layer(i) (pos (i)), layer (i) ∈ [L],x E (i, j) = W layer(i) (pos (i) , pos (j)), i > j x E (j, i) = 1 ⊘ W layer(j) (pos (j) , pos (i)), i > jThen, we use the vertex positional encodings to store the information of the input values and the gradients w.r.t the outputs:p V (i) =    [z 0 (pos (i)) , x 0 (pos (i))] , layer (i) = 0 [0, 0], layer (i) ∈ [L -1] [(∇ z L (pos(i)) L) -1 , (∇ x L (pos(i)) L]) -1 , layer (i) = LThroughout the network, the vertex representations will be 5-dimensional: we will use one to relay its neuron's bias across ScaleGMN's layers, two for the pre-and post-activations and two for their respective gradients. One can repeat the process for m inputs, in which case the dimension will be m + 1. The edge representations will be 1-dimensional and will be simply used to relay the NN's weights throughout the network.Now, let us define the vertex initialisation functions.INIT V,0 (x, p x ) = MLP (x, p) = [x[1], p x [1], p x [2], 0, 0] , INIT V,L (x, p x ) = MLP (x, p x ) = [x[1], 0, 0, p x [1], p x [2]] INIT V (x, p) = AugScaleEq (x, p) = [x[1], 0, 0, 0, 0] .Regarding the first two functions, it is obvious that can be expressed by an MLP. Regarding the third one, it can be expressed by AugScaleEq as follows:AugScaleEq(x, p) = Γ x x ⊙ AugScaleInv(x, p) = [1, 0, 0, 0, 0] ⊤ x[1] ⊙ [1, 1, 1, 1, 1] ⊤, where the the constant vector can be expressed by AugScaleInv since it is invariant to the first argument. Further, for all edge initialisation and update functions (forward and backward):INIT E, * (e, p e ) = e, UPD t E, * (x, y, e) = e which can both be expressed by AugScaleEq using the identity as linear transform and an AugScaleInv with constant output.We will not use positional encodings in the message and update functions, therefore we use their definitions of Eq. ( 8), Eq. ( 9). In particular, the forward/backward message functions: Finally, the vertex update functions:MSG t V,FW x, y, e = ScaleEq ([x, ReScaleEq (y, e)]) = y[3] • e[1] MSG t V,UPD t V,0 (x, m FW , m BW ) = MLP(x, m FW , m BW ) = x[1], x[2], x[3], (σ ′ 0 (x[2]) -1 • g U (m BW [1]), g U (m BW [1]) UPD t V,L (x, m FW , m BW ) = MLP(x, m FW , m BW ) = [x[1], x[1] + m FW [1], σ L (x[1] + m FW [1]) , x[4], x[5]] UPD t V (x, m FW , m BW ) = ScaleEq([x, m FW , m BW ]) = x[1], x[1] + m FW [1], σ (x[1] + m FW [1]) , (σ ′ (x[2]) -1 • g U (m BW [1]), g U (m BW [1]) ,where g U (•) = ScaleEq(•) an arbitrarily expressive scale equivariant function. The first two can obviously be expressed by an MLP (note also that usually σ 0 , σ L are the identity). Regarding the third, first, we assumed a common activation function σ across hidden layers. In case the activation is not shared, a different update function should be used per layer. This is because, as we will see below, the activation and its derivative need to be learned for ScaleGMN to simulate the forward and the backward passes. Now, let's see how its element can be expressed by ScaleEq. Recall that ScaleEq(a) = Γa ⊙ ScaleInv(a). Here, Γ ∈ R 5×7 (7 inputs, 5 outputs). The first element (which will be the bias), the second (pre-activation) can be expressed by a ScaleInv with constant output (ones) and Γ(1) an all-zero vector except for the 1st or the 1st and 6th coordinates respectively.The third (post-activation), fourth (pre-activation gradient) and the fifth (post-activation gradient) elements are not straightforward to simulate with elementary operations. However, one can observe that all are scale equivariant. In particular, the third: σ(qx + qy) = qσ(x + y) (by assumption about the symmetries of the activation function). Additionally, the fourth (assuming a scale equivariant g U ):g U (qy) σ ′ (qx) = q•g U (y) σ ′ (x)-this is because the derivative of a scale equivariant activation function is scale invariant. 8 The fifth is by definition scale equivariant. Therefore, if ScaleEq is sufficiently expressive, so as to contain all scale equivariant functions, then it can express the third to fifth elements.It remains to establish a few last statements that will be necessary in our induction. First, the edge representations are constant throughout ScaleGMN and equal to the weights of the input NN:h 0 E (i, j) = INIT t E, * (x E (i, j) , p E (i, j)) = x E (i, j) = W layer(i) (pos (i) , pos (j)) h t E (i, j) = UPD t E, * h t-1 V (i), h t-1 V (j), h t-1 E (i, j) = h t-1 E (i, j) = • • • = h 0 E (i, j).• (Induction step:) Suppose the hypothesis holds for vertices with layer (i) = ℓ + 1. Now, if layer (i) = ℓ, for t ≥ 2L -ℓ we have:m t V,BW (i) = j∈N B (i) MSG t V,BW h t-1 V (i), h t-1 V (j), h t-1 E (i, j) = j∈N B (i) g m (h t-1 V (j)[4] • h t-1 E (i, j)[1]) (t -1 ≥ 2L -(ℓ + 1)) = j∈N B (i) g m (∇ z ℓ+1 (pos(j)) L) -1 • W -1 ℓ (pos (j) , pos (i)) . h t V (i)[4 : 5] = UPD t V h t-1 V (i), m t V (i) [4 : 5] = g U m t V,BW (i) σ ′ ℓ h t-1 V (i)[2] , g U m t V,BW (i) = 1 σ ′ ℓ (z ℓ (pos (i))∇ x ℓ (pos(i)) L , (∇ x ℓ (pos(i)) L) -1 = (∇ z ℓ (pos(i)) L) -1 , (∇ x ℓ (pos(i)) L) -1 .In the last step, we assumed the following: g U g m (x) is a sufficiently expressive scale equivariant and permutation invariant function. This is in order to express the following:(∇ x ℓ (pos(i)) L) -1 = j∈N B (i) ∇ z ℓ+1 (pos(j)) L • W ℓ (pos (j) , pos (i)) -1with:(∇ q ℓ (i)x ℓ (pos(i)) L) -1 = q ℓ+1 (j) -1 ∇ z ℓ+1 (pos(j)) L • q ℓ+1 (j) q ℓ (i) W ℓ (pos (j) , pos (i)) -1 = q ℓ (i) ∇ z ℓ+1 (pos(j)) L • W ℓ (pos (j) , pos (i)) = q ℓ (i)(∇ x ℓ (pos(i)) L) -1 .Therefore if all g U g m (qx) = q • g U g m (x) can be expressed by g U , g m , then so is the inverse of the gradient.Following a similar rationale we can prove the case for the vertices with layer (i) = 0.They can be perceived as the metanetwork analogue of Invariant Graph Nets[50] for conventional graphs.Note the difference with architectures that access u via input-output pairs (xi, u G,θ (xi))[46,34].We use this convention to align with the indexing of the weights W(i, j).See Appendix A.7.1 for the precise statement and more details about ϕ σ,d .This is redundant in the forward ScaleGMN since input neurons do not have any forward neighbours. In the backward case -see Appendix A.2, we use a layer similar to MSGV,L.(33)σ(qx) = qσ(x) =⇒ qσ ′ (qx) = qσ ′ (x) =⇒ σ ′ (qx) = σ ′ (x)for non-zero scalar multipliers. 


Decision: This paper introduces Scale Equivariant Graph Meta Networks (ScaleGMNs), extending metanetwork design to include scaling symmetries. The work presents strong theoretical analysis and empirical improvements in the emerging field of deep weight space learning.
Initial concerns about experimental scope and presentation were adequately addressed in the authors' rebuttal. They provided additional experiments and committed to improving the paper's structure.
Given the novel contributions, theoretical foundation, and empirical results, I recommend accepting this paper for NeurIPS 2024. The authors should incorporate the promised improvements in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This work develops new GNN-based metanetworks that are equivariant to scaling symmetries induced by nonlinearities in input neural networks. Their ScaleGMNs extend metanetworks, which are typically only permutation equivariant (if at all equivariant), to also account for other symmetries in input neural networks' parameters. The architecture is proved to be equivariant to the desired symmetries and also expressive in that it can simulate forward and backward passes of the input. Experiments show improvements over merely-permutation-equivariant metanetworks.
strength_weakness: strength_weakness: 1. It would be interesting to see how ScaleGMNs perform on different tasks, especially an equivariant task (rather than just invariant tasks) such as INR editing.
2. ScaleInv is oddly described. The equation on Page 6 should probably have $\tilde x_i$ or something similar as its arguments, instead of $x_i$ (because if the $\rho^k$ are just general MLPs as you say right after the equation, then this is not scale invariant).
3. At the end of Page 6 and beginning of Page 7, you say that sign canonicalization can only be used for dimension 1, but this is not quite true. In Ma et al. 2023 and Ma et al. 2024, algorithms are given for canonicalizing with respect to the sign group, for use on inputs to a neural network.

References  
* [Ma et al. 2023] Laplacian Canonization: A Minimalist Approach to Sign and Basis Invariant Spectral Embedding. https://arxiv.org/abs/2310.18716
* [Ma et al. 2024] A Canonization Perspective on Invariant and Equivariant Learning.
 https://arxiv.org/abs/2405.18378
questions: questions: 1. Do you have a way to handle translation symmetries in nonlinearities?
limitations: limitations: Limitations are discussed on Page 9
paper_summary: paper_summary: This paper addresses the emerging and fascinating field of deep-weight space learning where neural nets used to process weights and biases of another deep model. The authors have introduced new methods based on GNN architecture called ScaleGMN and ScaleGMNB for Scale Equivariant Graph MetaNetworks. The latter is a bi-directional variant. Both approaches tackle the scale symmetries presented by the input neural model's activation functions.

The authors claim the following contributions:
- Extending the scope of metanetwork design from permutation to scaling symmetries.
- Designing networks that are invariant or equivariant to scalar multiplication from arbitrary scaling groups.
- Theoretical analysis of the expressive power of ScaleGMN. 
- Extensive empirical comparison with recent work on various datasets in the field of weight space learning.
strength_weakness: strength_weakness: - The writing can be significantly improved, particularly by breaking down long sentences that are hard to follow. Additionally, the writing pace is somewhat slow. While this might be beneficial for the average reader, the authors allocate too much space to exposition and problem formulation. As a result, the presentation of the proposed methods begins relatively late in the paper.
- The experimental section focuses only on invariant tasks, i.e. INR classification and NN generalization prediction. It would be interesting to see how well ScaleGMN and ScaleGMNB deal with equivariant tasks like the ones presented in [1,2] which are considered harder for weight space architectures.
- The processed architectures are not diverse dealing only with small-sized feed-forward and CNN architectures. It would be interesting to see more diversity in the processed architectures like deeper nets, attention-based methods, etc.
 
-------
[1] Equivariant Architectures for Learning in Deep Weight Spaces, Navon et al.

[2] Permutation Equivariant Neural Functionals, Zhou et al.
questions: questions: - Why DWS and INR2VEC are missing in Table 1? (CIFAR-10 | Augmented CIFAR-10 experiments).
- In common bi-directional architectures (e.g. LSTM) we see performance (runtime) degradation compared to non-bi-directional design is that the same for ScaleGMN and ScaleGMNB? What is the computational complexity of both methods?
- Can ScaleGMN and ScaleGMNB handle input data with heterogeneous activation functions, i.e. one network with ReLU activations and another with only tanh activations? (as long as they respect Prop. 4.1)
- The results for the Augmented CIFAR-10 experiments are odd. ScaleGMNB performs worse than all baselines except MLP, while in other experiments, it outperforms them. Do the authors have an explanation for this observation? 
- Adding a figure that illustrates ScaleGMN and ScaleGMNB architectures and their design w.r.t permutation and scale symmetries, would be beneficial.
limitations: limitations: See above.
paper_summary: paper_summary: The paper considers the emergent and fascinating field of learning over weight spaces, that is, neural nets that process other neural networks. The processing NN is referred to as a metanetwork. Previous approaches showcased the importance of accounting for the input NN’s symmetries by designing equivariant architectures. However, they were mainly focused on permutation symmetries. This paper proposes a GNN-based metanetwork, which is permutation and scale equivariant. The paper studies the expressiveness (in terms of simulating the forward and backward pass of the input NN) of the proposed arch. The proposed method is evaluated using several INRs and classifiers datasets.
strength_weakness: strength_weakness: My main concern is the limited empirical evaluation and missing natural baselines. While the presented empirical results show significant improvement over baseline methods, at the current state of the learning over weight spaces literature, I would expect a more diverse, challenging, and comprehensive empirical evaluation.

1. The writing and formatting require enhancement and refinement—specifically, long sentences, many slashes, many footnotes, etc. Also, in my view, the proposed method is introduced too late in the paper (page 6).
2. A missing natural baseline is to use a permutation equivariant baseline like DWS/NFN or NG-GNN together with scaling data augmentations as in [1].
3. The experimental section only considers invariant tasks. Additional experiments using equivariant tasks (e.g. INR editing, domain adaptation, etc.) would significantly strengthen the empirical study.
4. Some evaluation and comparison of runtime and memory consumption w.r.t. baselines would be beneficial.
5. Furthermore, adding experiments with larger input networks and diverse input architectures (like a varying number of layers) would again significantly strengthen the empirical study.
6. Also, adding some ablation regarding design choices would be beneficial.
7. Why are the results for DWS and INR2Vec missing in Table 1?
8. Minor:
   - Line 198: should be a^k.
   - Line 336: “transformations We” -> “transformations. We” 

References:

[1] Improved Generalization of Weight Space Networks via Augmentations, ICML 2024.
questions: questions: 1. Is it always feasible and relatively easy to design and implement either the canon or symm mappings for all activation functions?
2. The bidirectional version of the method achieves on-par performance as the non-bidirectional one, except for the Augmented CIFAR-10 dataset, where the performance is much worse. Could you provide some insights regarding this result?
3. Additionally, how does the ScaleGMNB and ScaleGMN compare in terms of runtime?
4. How do ScaleGMN and ScaleGMNB compare to GMN in terms of the number of trainable parameters and runtime?
5. Did you use any augmentation on the input NNs?
6. Are there any limitations on the choice of activations of the ScaleGMN network?
limitations: limitations: Yes.

Comments: title: Reviewer response
Response to rebuttal
References for Author Rebuttal
Additional responses to reviewer Reviewer ugM4
comment: I would like to express my gratitude to the authors for the time and effort they have dedicated to this rebuttal.
Most of my concerns have been addressed and after reading the other reviews and comments I decided to raise my score.
I would like to thank the authors for their rebuttal and for providing additional results and discussion, which addresses some of my concerns. I've read all the reviewers' concerns and the authors' responses. The authors addressed many of my concerns regarding the limited and missing empirical evaluation, so I raised my score accordingly.
We thank the authors for their rebuttal. The new experimental results are also very strong.

I maintain my score of 8, and definitely support acceptance of this paper.
[1] Zhou, Allan, et al. "Permutation equivariant neural functionals." Advances in neural information processing systems 36 (2024).

[2] Kofinas, Miltiadis , et al. "Graph neural networks for learning equivariant representations of neural networks." ICLR, 2024.

[3] Navon, Aviv, et al. "Equivariant architectures for learning in deep weight spaces." International Conference on Machine Learning. PMLR, 2023.

[4] Lim, Derek, et al. "Graph Metanetworks for Processing Diverse Neural Architectures." The Twelfth International Conference on Learning Representations.

[5] Zhou, Allan, Chelsea Finn, and James Harrison. "Universal neural functionals." arXiv preprint arXiv:2402.05232 (2024).

[6] Unterthiner, Thomas, et al. "Predicting neural network accuracy from weights." arXiv preprint arXiv:2002.11448 (2020).

[7] Schürholt, Konstantin, et al. "Model zoos: A dataset of diverse populations of neural network models." Advances in Neural Information Processing Systems 35 (2022): 38134-38148.
>*Weakness 1. Writing.*

Please refer to our global response regarding the structure of our paper and the writing style.

>*Weakness 3. Equivariant tasks.*

Please refer to our global response, where we provide details regarding the INR editing task as well as to Table 1 of the attached PDF.


>*Weakness 4. Runtime and memory consumption.*

We thank the reviewer for suggesting a comparison of runtime and memory consumption. We select the F-MNIST dataset and make two comparisons; at first we report the number of parameters of all the reported models. Regarding the runtime, we fix the number of parameters for fairness of comparison and report the training time, inference time and GPU memory consumption. Please refer to Table 2 of the attached PDF. We can see that ScaleGMN-B does not introduce performance degradation regarding the runtime, while both our methods are quiet slower than the baselines. Since we do not use computationally much heavier operations, this last result indicates that our implementation could be further optimized w.r.t. runtime.


>*Weakness 5. Larger and diverse input architectures.*

Please refer to our global response regarding the datasets and experiments on larger and more complex architectures.
    
    
>*Weakness 7. DWS and INR2VEC on CIFAR-10*

Please refer to our response to reviewer x2pG regarding these two experiments.

>*Question 3. Runtime performance of ScaleGMN and ScaleGMN-B.*

Please refer to Table 2 of the attached PDF. As discussed in *Weakness 4*, ScaleGMN-B does not introduce performance degradation regarding the runtime.