Monomial Matrix Group Equivariant Neural Functional Networks
Authors: Hoang V. Tran
Thieu Vo
Tho Tran Huu
An Nguyen The
Tan Minh Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: neural functional networks
equivariant networks
monomial matrices
symmetry
Primary area: deep_learning_architectures
TL;DR: We extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling symmetries.

Abstract: Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of $\operatorname{ReLU}$ networks, and the weight sign flipping symmetries of $\operatorname{sin}$ or $\operatorname{Tanh}$ networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidences to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency. The code is publicly available at https://github.com/MathematicalAI-NUS/Monomial-NFN.

Monomial Matrix Group Equivariant Neural Functional Networks
Viet-Hoang Tran" Department of Mathematics National University Singapore hoang. tranviet@u. nus edu
Thieu N. Vo* Department of Mathematics National University of Singapore thieuvo@nus.ed .du..eeduuuu.. sg
Tho Tran-Huu
An T. Nguyen FPT Software Al Center
Department Mathematics National University of Singapore thotranhuuêu. nus. nus edu, vn
annt680fpt.com
Tan M. Nguyen
Department of Mathematics National University of Singapore tanmnonus. edu sg
Abstract
Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN de- signs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons hidden layers. However, these designs do not take into account the weight scaling symmetries of ReLU networks, and the weight sign flipping symmetries sin or Tanh net- works. In this paper, we extend the study of the group action on the network weights from the group permutation matrices the group of monomial matrices by incorporating scaling/sign-flipe flipping symmetries. Particularly, we encode these scaling/sign- Saaignn-ppppggggggeoo symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFY) sion of the symmetries, Monomial-NEF parameters compared to the baseline NENs in the literature, thus enhancing the model's 's efficiency. Moreover, for fully connected and convolutional neural net- works, we theoretically prove that all groups that leave these networks invariant while acting their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidences to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency. The code is publicly available at NFN.
Because of the expan- has much fewer independent trainable
1 Introduction
Deep neural networks (DNNs) have become highly versatile modeling tools, finding applications across broad spectrum of fields such Natural Language Processing [15, 29, 51, 66), Computer
Equual contribution. Please correspond thieuvodnus.ed edu.sg.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Vision [27, 37, 61], and the Natural Sciences [31, 49]. There has been growing interest in developing specialized neural networks to process the weights, gradients, or sparsity masks of DNNs as data. These specialized neural networks are called neural functional networks (NFNs) [71]. NFNs have found diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representations. For instance, NFNs have been employed to create learnable optimizers for neural network training [5, 11, 42, 52), extract information from implicit neural representations of data [43, 52, 60), perform corrective editing of network weights [13, 44, 56], evaluate policies [26], and conduct Bayesian inference using networks as evidence [59].
Developing NFNs is inherently challenging due to their high-dimensional nature. Some early methods to address this challenge assume restricted training process that effectively reduced the weight space [9, 19, 41]. More recent efforts have focused on building permutation equivariant NFNs that can process neural network weights without such restrictions [35, 45, 71, 72]. These works construct NFNs that are equivariant to permutations of weights, corresponded to the rearrangement of neurons in hidden layers. Such permutations, known as neuron permutation symmetries, preserve the network's behavior. However, these approaches often overlook other significant symmetries in weight spaces [12, 24]. Notable examples are weight scaling transformations for ReLU networks [7, 12, 46] and sign flipping transformations for sin and tanh networks [14, 22, 38J. Consequently, two weight spaces of a ReLU networks, that differ by a scaling transformation, two weight spaces of sin, or tanh networks that differ by a sign flipping transformation, can produce different results when processed by existing permutation equivariant NFNs, despite representing the same functions. This highlights fundamental limitation of the current permutation equivariant NFNs.
Contribution. In this paper, we extend the study of symmetries in weight spaces of Fully Connected Neural Networks (FCNNs) and Convolution Neural Networks (CNNs) by formally establishing group of symmetries that includes both neuron permutations and scaling/siin--flppinn tions. These symmetries are represented by monomial matrices, which share the nonzero pattern of permutation matrices but allow nonzero entries to be any value rather than just 1. We then introduce novel family of NFNs that are equivariant to groups of monomial matrices, thus incorporating both permutation and scaling/ssinn-fippnn family Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFNs: Due to the expanded set of symmetries, Monomial-NFis requires significantly fewer independent trainable parameters compared to baseline NFNs, enhancing the model's 's efficiency. By incorporating equivari- ance to neuron permutations and weight malling/ign-ffipiing our NFNs demonstrate competitive generalization performance compared to existing models. Our contribution is three-fold:
transforma-
symmetries into the NEN design. We name this new
1.
We formally describe group of monomial matrices satisfying the condition that the transfor- mation of weight spaces of FCNNs and CNNs using these group elements does not change the function defined by the networks. For ReLU networks, this group covers permutation and scaling symmetries of the weight spaces, while for sin or tanh networks, this group covers permutation and sign-flipping symmetries. The group is proved to be maximal in certain cases.
2. We design Monomial- the first family of NFNs that incorporate scaling and syn-flipping symmetries of weight spaces as far as we aware. The main building blocks of Monomial- NFNs are the equivariant and invariant linear layers for processing weight spaces.
3. We show that the number of parameters in our equivariant linear layer is much lower than in recent permutation equivariant NFNs. In particular, our method is linear in the number of layers and dimensions of weights and biases, compared to quadratic as in [71]. This demonstrates that Monomial NNNS have the ability to process weight spaces of large-scale networks.
We evaluate Monomial-NFFi on three tasks: predicting CNN generalization from weights using Small CNN Zoo [64], weight space style editing, and classifying INRs using INRs data [71]. Experimental results show that our model achieves competitive performance and efficiency compared to existing baselines.
Organization. We structure this paper as follows: After summarizing some related work in Section we recall the notions of monomial matrix group and describe their maximal subgroups preserved by some nonlinear activations in Section 3. In Section 4, we formalize the general weight space of FCNNs and CNNs, then discuss the symmetries of these weight spaces using the monomial matrices. In Section 5, we construct monomial matrix group equivariant and invariant layers, which are building blocks for our NNNNss In Section 6, we present our experimental results to justify the advantages of Monomial-NFNs over the existing permutation equivariant NFN baselines. The paper ends with concluding remarks. More experimental details are provided in the Appendix.

---

2 Related Work
Symmetries of Weight Spaces. The challenge of identifying the symmetries in the weight spaces of neural networks, or equivalently, determining the functional equivalence of neural networks, is well-explored area in academic research [3, 10, 16, 23, 47]. This problem was initially posed by Hecht-Nielsen in [28]. Results for various types of networks have been established as in [1, 2, 12, 14,
22, 38, 63].
Neural Functional Networks. Recent research has focused on learning representations for trained classifiers to predict their generalization performance and other insights into neural networks [8, 20, 64, 54, 53, 55J. In particular, now-dimensional encodings for Implicit Neural Representations (INRs) have been developed for downstream tasks [18, 41]. Other studies have encoded and decoded neural network parameters mainly for reconstruction and generation purposes [6, 21, 34, 48).
Equivariant Neural Functional Networks. Permutations and scaling, for ReL.U networks, as well as sign-flipping. for sine or tanh networks, symmetries, are fundamental symmetries of weight networks. cermutation-cquivariin
NFNs are successfully built in [4, 35, 40, 45, 70, 71, 72]. In particular, the authors in [35, 40] carefully construct computational graphs representing the input neural networks' parameters and process the graphs using graph neural networks. In [4], neural network parameters are efficiently encoded by carefully choosing appropriate set-to-set and set-to-vector functions. The authors in [70] view network parameters as a special case of a collection of tensors and then construct maximally expressive equivariant linear layers for processing any collection of tensors given a description of their permutation symmetries. These methods are applicable to several types of networks, including those with branches or transformers. However, these models were not necessarily equivariant to scaling nor sign-fipping transformations, which are important symmetries of the input neural networks.
Our method makes the first step toward incorporating both permutation and non- non-permutation symme- tries into NFNs. In particular, the model proposed our paper is equivariant to permutations and scaling, for ReLU networks, or sign- -lippiiiinng, for sine and tanh networks. This leads to significant reduction in the number of parameters, a property that is particularly useful for large neural networks in modern deep learning, while achieving comparable or better results than those in the literature. The authors in [32, 67] have also developed NFNs that incorporates scaling symmetries.
3 Monomial Matrices Perserved by a Nonlinear Activation
Given two sets X,Y, and a group G acts on them, a function é: X - Y is called G-equivariant ő(g (z) g- o0zz) for all æ E X and g G. If G acts trivially on Y, then say o is G-invariant. In this paper, we consider NFNs which are equivariant with respect to certain symmetries of deep weight spaces. These symmetries will be represented by monomial matrices. In Subsection 3.1, we recall the notion of monomial matrices, as well as their actions on space of matrices. We then formalize the maximal group of matrices preserved by the activations ReLU, sin and Tanh in Subsection 3.2.
3.1 Monomial Matrices and Monomial Matrix Group Actions
All matrices considered in this paper have real entries and n is a positive integer.
Definition 3.1 (See [50, page 46)). A matrix of size n x n is called a monomial matrix (or generalized permutation matrix) if it has exactly one non-zero entry each row and each column, and zeros elsewhere. We will denote by G. the set of such all matrices.
Permutation matrices and invertible diagonal matrices are special cases of monomial matrices. In particular, a permutation matrix a monomial matrix in which the non-zero entries are all equal to 1. In case the nonzero entries of a monomial matrix are in the diagonal, it becomes an invertible diagonal matrix. We will denote by P,, and A, the sets of permutation matrices and invertible diagonal matrices of size n, respectively. It is well-known that the groups Gn. Pn. and A, are subgroups of the general linear group GL.n).
Permutation matrix group P,, is representation of the permutation group Sn, which is the group of all permutations of the set {1, 2, ...,..,. .) with group operator as the composition. Indeed, for each permutation и E Sn, we denote by P= the square matrix obtained by permuting n columns of the identity matrix 7, by n. We call P= the permutation matrix corresponding to म. The correspondence - Ps defines group homomorphism p: Sn - GL(n) with the image Pn, p(S.).
3

---

Each monomial matrix in In is product of an invertible diagonal matrix in A,, and a permutation matrix in P.. i..
9n (DP D A, and P Px)
In general, we have PD * DP. However, for D = diag(d]. dg.... dn) and P Pa. we have PD = (PDP-1) which is again product of the invertible diagonal matrix
PDP-1 diag(d,-1(1)) d2). ... ......,,d--00 dn))(n))
(2)
and the permutation matrix P. As an implication of Eq. (2), there is a group homomorphism :o: Pn Autuuutuan An). defined by the conjugation, i.c. ( PDP-1 for all P E Pn and
De DEA A.. The map defines the group 9n as the semidirect product Gn = An X Pn (see [17]). For convenience, we sometimes denote element DP of Gn as pair (D,
The groups 9n: Pn and A. act on the left and the right of R" and R*xm in a canonical way (by matrix-vector or matrix-matrix multiplications). More precisely, have:
Proposition 3.2. Let E IR and A = (Aiz) E Ruxmm Then for D diag(d),..... ..,, dn) A..
D diag(di..... dm) Ams Px. E Pn. and Po E P we have:
Pj (1)= Fx (2)) ................................ ................................ Ix 1(n)
D (d1 11; d2 T2,..., dn xn))
( . Pa . P.P. P. D) = d, A A ),,
(D. P. A (D Pa) "0)))))),))))))))) ") d ( (1)
The above proposition can be verified by direct computation, and is used in subsequent sections.
3.2 Monomial Matrices Preserved by a Nonlinear Activation
We characterize the maximal matrix groups preserved by the activations a = ReLU, U, sin or lanh. Here, ReLU the rectified linear unit activation function which has been used most of modern neural networks, sin is the sine function which is often used as activation function in implicit neural representations [57]. and tanh the hyperbolic tangent activation function. Different variants of the results in this subsection can also be found in [24, 68). We refine them using the terms of monomial matrices and state explicitly here for the completeness of the paper.
Definition 3.3. A matrix A GL(n) is said to be preserved by activation if and only if (4 x) = A a (x) for all x E IR...
We adopt the matrix group preserved by an activation from [68]. This term is then referred as the intertwiner group of an activation in [24].
Proposition 3.4. For every matrix A E GL(n), we have:
(i) A is preserved by the activation ReLU if and only if A E G2. Here, G2 is the subgroup of Gn containing only monomial matrices whose nonzero entries are positive numbers.
(ii) is preserved by the activation a = sin or tanh if and only if E g Here, g+1 the subgroup of Gn. containing only monomial matrices whose nonzero entries are #1.
detailed proof of Proposition 3.4 can be found in Appendix C.I. As a consequence of the above theorem, G2.0 (respectively, 1 is the maximal matrix subgroup of the general linear group GL(n) that is preserved by the activation RcLU (respectively, sin and tanh).
Remark 3.5. Intuitively, 97.0 is generated by permuting and positive scaling the coordinates of vectors in R", while g is generated by permuting and sign flipping. Formally, these groups can be written as the semidirect products:
A X Pn. G:00 A7" Xx Pn. and g
where
200 (D diag (d) ,,. dn) ) dj 0} and
A=1 (D- diag dn) d, (111
(4)
two subgroups of An.

---

4
Weight Spaces and Monomial Matrix Group Actions on Weight Spaces
In this section, we formulate the general structure of the weight spaces of FCNNs and CNNs. We then determine the group action on these weight spaces using monomial matrices. The activation function a using on the considered FCNNs and CNNs are assumed to be ReLU or sin or Tanh.
4.1 Weight Spaces of FCNNs and CNNs
Weight Spaces of FCNNs. Consider an FCNN with L layers, n, neurons at the i-th layer, and no and bt be the input and output dimensions, together with the activation o, as follows:
f(x; ; U. = W(4, a (...o (w..ew (wey).. (wa) x x + + b(2) ...) B(4)
(5)
Here, U (W, b) is the parameters with the weights W = {W()) R H and the biases
b (b(*) E R 3 The pair U (W,b) belongs to the weight space U W B, where:
W Rur x... R R
(6)
B = = R x x R X R
(7)
Weight Spaces of CNNs. Consider CNN with L convolutional layers, ending with an average pooling layer then fully connected layers, together with activation a. Let n; and W; be the number of channels and the size of the convolutional kernel at the 7th convolutional layer. We will only take account of the L convolutional layers, since the weight space of the fully connected layers are already considered above, and the pooling layer has no learnable parameters:
f(x: U, a (wit) (.... (w(3)= (W(1) * + +x)) + -B(3)
+ +6())
(8)
Here, U (W, b) is the learnable parameters with the weights W (W E R X 1==1 and the biases b = (b(*) E R [RIxxx1]] j.m The convolutional operator * is defined depending on the purpose of the model, and adding b means adding b,5) to all entries of j= th channel at 2th layer. The pair U = bW belongs to the weight space U W B, where:
W- = LL X X x x R x R Xn1 Xno
(9)
B = lxnix1 x x... x TR. 1xxxx1 x R 1xmix1
(10)
Remark 4.1. See Appendix. C.2 for concrete descriptions of weight spaces of FCNNs and CNNs.
4.2 Monomial Matrix Group Action on Weight Spaces
The weight space U/ of an FCNN or CNN with I layers and n, channels at 7th layer has the general form 4 = W B, where:
W- = R. X X X x x... x Ru Xmg x R Xn1 Xno
(11)
B = R Xm, x x... x x xb
(12)
Here, is the number of channels at the 7th layer, in particular, an and nt. are the number of channels of input and output; w, is the dimension of weights and b, is the dimension of the biases in each channel at the i-th layer. The dimension of the weight space U is:
diml
(w, X n, X n: + x x x
(13)
Notation. When working with weight matrices in W, the space Rw; Xn; = (R") ( at the 2h layer will be considered as the space of n. X n4-1 matrices, whose entries are real vectors in R In partieular, the symbol Wi( denotes a matrix in [R
( while
WD jk
E T... denotes the entry at row j and column k of W Similarly, the notion (00 denotes a bias column vector in [g xn;11_ = (R'x)4x1, while b// e I& denotes the entry at row j of 6(0)
To define the group action uf using monomial matrices, denote Gu as the group:
Gu Gng .... x Gng x Gno-
5

---

Ideally, each monomial matrix group Gn, will act on the weights and the biases at the 2thh layer of the network. Each element of Tu will be of the form g = (g(4) ...,), where:
g = D( Ps. = diag ( d Pa. E gn.
(14)
for some invertible diagonal matrix D and permutation matrix Pa.. The action of Gu on U is defined formally as follows.
Definition 4.2 (Group action on weight spaces). With the notation as above, the group action of Gu on l is defined to be the map gu x l - U1 with (g, U) - gU .gWW, gb), where:
(gW) (1() w( (0 and (ab) (1()) 660
(15)
In concrete:
(oW)
W (i) m, ",)0,,, 100 and (ab)
(1) ((0) '6)
(16)
di ((-1)
Remark 4.3. The group Gu is determined only by the number of layers L and the numbers of channels na, not by the dimensions of weights and biases b; at each channel.
The group Gu has nice behaviors when acting on the weight spaces of FCNNs given in Eq. (5) and CNNs given in Eq. (8). In particular, depending on the specific choice of the activation o, the function f built by the given FCNN or CNN is invariant under the action of a subgroup G of Gu, as we will see in the following proposition.
Proposition 4.4 (G-equivariance of neural functionals). Let f = [(:; U, be an FCNN given in Eq. (5) or CNN given in Eq. (8) with the weight space U u and activation a (ReLU, Tanh, sin}. Let us defined a subgroup G of Gu as follows:
(i) If = ReLU, we set G = {idg., x 920 x... x G x fidg.gt-
(ii) If = sin or tanh, then we set G {idg.., 1tt x... x g fidgnng
Then f is G-invariant under the action of G on its weight space, i.e.
f(x; U, fxx gU,
(17)
for all g E G, U EM and R Remark 4.5 (Maximality of G). The proof of Proposition 4.4 can be found in Appendix C.2. The group G defined above is even proved to be the maximal choice in the case:
o = ReLU and L 7 ... > n22 > n1 nu = 1 (see [12, 25]), or
0 tanh (see [14, 221).
Here, G is maximal in the sense that: Uf is another element in li with f(: U(:)= o)),, then there exists an element g E G such that U' = =U. It is natural to ask whether the group G is still maximal in the other case. This question still remains open and we leave it for future exploration.
According to Proposition 4.4, the symmetries of the weight space of an FCNN or CNN must include not only permutation matrices but also other types of monomial matrices resulting from scaling (for ReLU networks) or sign flipping (for sin and lanh networks) the weights. Recent works on NFN design only take into account the permutation symmetries of the weight space. Therefore, is necessary to design a new class of NFNs that incorporates these missing symmetries. We will introduce such a class in the next section.
5 Monomial Matrix Group Equivariant and Invariant NFNs
In this section, we introduce a new family of NFNs, called Monomial-NENs, by incorporating symmetries arising from monomial matrix groups which have been clarified in Proposition 4.4. The main components of Monomial-NFNs are the monomial matrix group equivariant and invariant linear layers between two weight spaces which will be presented in Subsections 5.1 and 5.2, respectively. We will only consider the case of ReLU activation. Network architectures with other activations will be considered in detail in Appendices A and B.
In the following, = (W,B) is the weight space with L. layers, n, channels at 7th layer, and the dimensions of weight and bias are w; and b, respectively (see Eqs. (11) and (12)). Since we consider ReL.U network architectures, according to Proposition 4.4, the symmetries of the weight space G x {idg... of Gu.
given by the subgroup G {idg.. } 9200

---

Table Number of parameters in a linear equivariant layer E: u UL with respect to permutation matrix groups in [71], and monomial matrix groups. Here, = max (;,b;) b,} and c' = max w b/
Subgroups of Su
Number of parameters of E'
Pug PuL X ... Pn1 Pno
((71))
O(ed Olc''L + no + n1)²)
fidgm,.
x PAn-1 X... x Pn, x {idgng }
((71))
fidgn, x 9-0 X... x - x fidgno }
(Ours)
O(cc'L no + n1)) O no + n1))
{idgn,.
9it x... g fidgng
(Ours)
5.1 Equivariant Layers
We now construct linear G-equivariant layer between weight spaces. These layers form the fundamental building blocks for our Mononimal-NANs. Let U and 41' be two weight spaces of the same network architecture described in Eqs. (11) and (12), i.e. they have the same number of layers as well as the same number of channels at each layer. Denote the dimension of weights and biases in each channel at the i-th layer 2t' as w/ and b/. respectively. Note that, in this case, we have Gy = Gu'. We construct G-equivariant afine maps E: l - U' with x - ax + b, where E Rdim 21 xdim and b E Riimmr' x1 are learnable parameters.
To make E to be G-equivarient, and have to satisfy system of constraints (usually called parameter sharing), which are induced from the condition E(gU) = gE(U) for all E G and U E U. We show in details what are these constraints and how derive the concrete formula of E in Appendix A. The formula of E is presented as follows: For U = (W, b) E U. the image E(U) (W", b/) E ll is computed by:
Rg
Ro
Σ
Pijg
1ik
w (1)
jq
15k (1) 919
(1)
0j
67Th Σ
1jg 39
lj W( 1j
$1
w-(())
iik Ligh
wr()
800
o <<i<L
(18)
jk
nr
n.
w
Σ
Pipk W w pk
Likk
69th
L 1 +1 Σ
Here, (p. q. t, t) is the hyperparameter of E. We discuss in detail the dimensions and sharing information between these parameters in Appendix A.I. Note that, we also show that all linear G-equivariant functional are in this form in Appendix A. To conclude, we have:
Theorem 5.1. With notation as above, the linear functional map E: U - U'' defined by Eq. (18) is G-equivariant. Moreover, every G-equivariant linear functional map from U to U' in that form.
Number of parameters and comparison to previous works. The number of parameters in our layer linear in L, fig nL, which is significantly smaller than the number of parameters in layers described in [71], where it is quadratic in L, Ro, (see Table 1). This reduction in parameter count means that our model is suitable for weight spaces of large-scale networks and deep NFNs. Intuitively, the advantage of our layer arises because the group G acting on the weight spaces in our setting is much larger, resulting in significantly smaller number of orbits in the quotient space 4/G. Since the number of orbits is equal to the number of parameters, this leads to a more compact representation. Additionally, the presence of the group A>" forces many coefficients of the linear layer to be zero, further contributing to the efficiency of our model.
5.2 Invariant Layers
We will construct an G-invariant layer T: - R2d for a fixed integer d > In order to do that, we will seek map 7 in the form:
MLP oIp o TA=0,
(19)
where JA U 2 is - -nvaariineee and P. P.-equivariance map, Ip U4 dim is an P. P.-invariant map, and MLP. dd - Rd is an arbitrary multilayer perceptron to adjust the output dimension. Since = G2- A2 x P. (see Remark 3.5), the composition / = MT.P oTp T is clearly G-invariant as expected. The construction of I L and 1p will be presented below.
7

---

Table CNN prediction on Tanh subset of Small CNN Zoo with original and augmented data.
STATNN
NP
HNP
Monomial NFNN (ours)
Gap
Original
0.913 0.001
0.925 + 0.001
0.933 = 0.002
0.939 + 0,001
0.006
Augmented
0.914 0.001
0.928 0.001
0.935 0.001
0.943 0.001
0.008
Construct ~200. To capture A -iniiaaaii,,e, we recall the notion of positively homogeneous
of degree zero maps. For > 0, map from R" is called posifively homogeneous of degree zero if for all > and (fi,..., [na E IR, we have a(AA11,.... Xr = 0.. In We construct 1a>0 U - U by taking collections of positively homogeneous of degree zero functions {ajk R - [R and {a, (0, Rd: - R² each one corresponds to weight and bias of U. The
maps 1a=0: U - U that (W,b) b) - (W", b¹) is defined by simply applying these functions on each
weight and bias entries as follows:
W W a (6) (W and 6501= (b,
(20)
JA is A2 --nnivvaraan by homogeneity of the functions. To make it become P.-equivariant, some functions have to be shared arross any axis that have permutation symmetry. We derive this relation in Appendix B. Some candidates for positively homogeneous of degree zero functions are also presented in Appendix B. They can be fixed or learnable.
Construct Ip. To capture ---invvriance, we simply take summing averaging the weight and bias across any axis that have permutation symmetry as in [71]. concrete, we have Ip: 4 - Rdim is computed follows:
Ir(U) (W!!) W! w W........ W!(-"); (L)
(21)
Here, * denotes summation or averaging over the rows or columns of the weight and bias. Remark 5.2. In our experiments,
averaging operator since is empirically more stable.
Finally we compose an MLP before 1p and IA to obtain an G-invariant map. We summarize the above construction as follows.
Theorem 5.3. The functional map T: U - Rdd defined by Eq. (19) is G-invariant.
5.3 Monomial Matrix Group Equivariant Neural Functionals (Mononial-NENs)
We build Monomial-NFNs by the constructed equivariant and invariant functional layers, with activations and additional layers discussed below. The equivariant NEN is built by simply stacking G-equivariant layers. For the invariant counterpart, we follow the construction in [71]. In particular, we first stack some G-equivariant layers, then a A -in-irrari and P. Pu--quivvrriant layer. This makes our NFN to be - -ivaarian and P. P.-qqiivariant. Then finish the construction by stacking P,-invariant layer and the end. This process makes the whole NFN to be G-invariant as expected.
Activations of G-equivariant functionals. Dealing with equivariance under action of Pa only requires activation of the NEN is enough, since P, acts on only the order of channels in each channel of the weight space. For our G-equivariant NFNs, between each layer that is A? "="=equivarian,, we have to use the same type activations as the activation in the network input (i.e. either ReLU, sin or lanh in our consideration) to maintain the equivariance of the NFN.
Fourier Features and Positional Embedding. As mentioned [35, 71, 72], Fourier Features [30, 62] and (sinusoidal) position embedding play significant role in the performance of their functionals. Also, in [71], position embedding breaks the symmetry input and output neurons, and allows us to use equivariant layers that act on input and output neurons. In our G-equivariant layers, we not consider action on input and output neurons as mentioned. Also, using Fourier Features does not maintain A so we can not use this Fourier layer for our equivariant Monomial-NFNs, and in our invariant Monomial-NFNs, we only can use Fourier layer after the >0 --0-iavariant layer. This can be considered as a limitation of Monomial-NFNs.
6 Experimental Results
In this session, we empirically demonstrate the performance of our Monomial Matrix Group Equiv- ariant Neural Functional Networks (Monomial-NFNs)) various tasks that are either invariant

---

1.0
0.9
Monomial-NFN
0.8
NP
0.7
HNP
0.8
STATNN
0.5
Figure 1: CNN prediction on ReLU subset of Small CNN Zoo with different ranges of augmentations. Here the x-axis is the augment upper scale, presented in log scale. The metric used is Kendall's т.
Table 3: Classification train and test accuracies (%) for implicit neural representations of MNIST, FashionMNIST, and CIFAR-10. Uncertainties indicate standard error over 5 runs.
Monomial-NEN (ours)
NP
HNP
MLP
CIFAR-10
34.23 1 0.33
33.74 + 0.26
31.61 + 0.22
10.48 0.74
MNIST-10
68.43 + 0.51
69.82 = 0.42
66.02 = 0.51
10.62 + 0.54
PaShionMNIST
61.15 + 0.55
58.21 + 0.31
37 = 0.46
9.95 = 0.36
(predicting CNN generalization from weights and classifying INR representations of images) or equivariant (weight space style editing). We aim establish two key points. First, our model exhibits more stable behavior when the input undergoes transformations from the monomial matrix groups. Second, our model, Monomial-NFN, achieves competitive performance compared to other baseline models. Our results are averaged over 5 runs. Hyperparameter settings and the number of parameters can be found in Appendix D.
6.1 Predicting CNN Generalization from Weights
Experiment Setup. In this experiment, we evaluate how our Monomial-NFN predicts the gener- alization of pretrained CNN networks. We employ the Small CNN Zoo [64], which consists of multiple network weights trained with different initialization and hyperparameter settings, together with activations Tanh or ReLU, Since Monomial-NFN- depend on activations of network inputs, we divide the Small CNN Zoo into two smaller datasets based on their activations. The ReLU dataset considers the group G2D0 while the Tanh dataset considers the group G
We construct the dataset with additional weights that undergo random hidden vector permutation and scaling based on their monomial matrix group. For the ReLU dataset with the group g200
we uniformly sample the diagonal indices of D (see Eq. 14) for various ranges: [1, 10), [1, 1
10a)..... [1, x 10%), while belonging to 13 in the case of Tanh dataset with the group g For both datasets, we compare our model with STATNN [65], and with two permutation equivariant neural functional networks from [71], referred to as HNP and NP. To compare the performance of all models, we use Kendall's 7 rank correlation metric [33].
Results. We demonstrate the results of all models on the ReLU subset in Figure 1, showing that model attains stable Kendall's when the scale operators are sampled from different ranges. Specifically, when the log of augmentation upper scale is O, i.e. the data remains unaltered, our model performs well the HNP model. However, the weights undergo more extensive scaling and permutation, the performance of the HNP and STATNN models drops significantly, indicating their lack of scaling symmetry. The NP model exhibits similar trend, albeit to lesser extent. In contrast, our model maintains stable performance throughout.
Table 2 illustrates the performance of all models on both the original and augmented Tanh subsets of CNN Zoo. Our model achieves the highest performance among all models and shows the greatest improvement after training with the augmented dataset. The gap between our model and the second- best model (HNP) increases from 0.006 to 0.008. Additionally, in both experiments, our model utilizes significantly fewer parameters than the baseline models, using only up to 50% of the parameters compared to HNP.
6.2 Classifying implicit neural representations of images
Experiment Setup. In this experiment, our focus on extracting the original data information encoded within the weights of implicit neural representations (INRs). We utilize the dataset from [71],

---

Table 4: Test mean squared error (lower is better) between weight-space editing methods and ground-trun truth mmage-space transformations. Uncertainties indicate standard error over runs.
Monomial NFN (ours)
NP
HNP
MLP
Contrast (CIFAR-10)
0.020 + 0.001 0.069 + 0.002
0.020 + 0.002
0.021 + 0.002
0.031 + 0.001 0.306 + 0.001
Dilate (MNIST)
0.068 - 0.002
0.071 + 0.001
which comprises pretrained INR networks [58] that encode images from the CIFAR- 10 [36], MNIST [39], and FashionMNIST [69] datasets. Each pretrained INR network designed map image coordinates (z, y) to color pixel values- 3-dimensional RGB values for CIFAR- 10 and 1-dimensional grayscale values for MNIST and FashionMNNIST.
Results. We compare our model with NP, HNP, and MLP baselines. The results in Table demonstrate that our model outperforms the second-best baseline, NP, for the MNSHi and CIFAR- 10 datasets by 2.94% and 0.49%, respectively. For the MNIST dataset, our model also obtains comparable performance.
6.3 Weight space style editing.
Experiment setup. In this experiment, we explore altering the weights of the pretrained SIREN model [58] to change the information encoded within the network. We use the network weights provided in the HNP paper for the pretrained SIREN networks on MNIST and CIFAR-10 10 images. Our focus is two tasks: the first involves modifying the network to dilate digits from the MNIST dataset, and the second involves altering the SIREN network weights to enhance the contrast of CIFAR-IO images. The objective is minimize the mean squared error (MSE) training loss between the generated image from the edited SIREN network and the dilated/enhanced contrast image.
Results. Table 4 shows that our model performs par with the best-performing model for increasing the contrast of CIFAR-10 images. For the MNIST digit dilation task, our model also achieves competitive performance compared to the NP baseline. Additionally, Figure 2 presents random samples of the digits that each model encodes for the dilation and contrast tasks, demonstrating that our model's results are visually comparable those of HNP and NP in both tasks.
7 Conclusion
In this paper, we formally describe a group of monomial matrices that preserves FCNNs and CNNs while acting their weight spaces. For ReLU networks, this group includes permutation and scaling symmetries, while for networks with sin or Tanh activations, it encompasses permutation and sign-fipping symmetries. We introduce Monomial-NFNs, first-of-a-kind class of NFNs that incorporates these scaling or sign-flipping symmetries in weight spaces. We demonstrate that the low number of trainable parameters in our equivariant linear layer of Monomial-NFNs compared to previous works on NFNs, highlighting their capability to efficiently process weight spaces of deep networks. Our NFNs exhibit competitive generalization performance and efficiency compared to existing models across several benchmarks.
One limitation of our model is that, due to the large size of the group considered, the resulting linear layers can be limited in terms of expressivity. For example, a weight corresponding to an edge between two neurons will be updated based only on its previous value, ignoring other edges across the same or other layers. To resolve this issue, it is necessary to construct an equivariant nonlinear layer to encode further relations between these weights, thus enhancing the expressivity. Another limitation is that we are uncertain about the maximality of the group G acting on the weight space of the ReLU network. Therefore, other types of symmetries may exist in the weight space beyond neuron permutation and weight scaling, and our model is not equivariant with respect to these symmetries. We leave the problem of identifying such a maximal group for future research.
Acknowledgments and Disclosure of Funding
This research/ / project is supported by the National Research Foundation Singapore under the AI This research / project
Singapore Programme (AISG Award No: AISG2-TC-2023-012-SCC supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2023) (A-8002040-00-00 1-8002039-00-00, This research / project also supported by the NUS Presidential Young Professorship Award (A-0009807-01-000).
10

---

References
[1]
Francesca Albertini and Eduardo D Sontag. For neural networks, function determines form.
Neural networks, 6(7):755990, 1993.
[2] Francesca Albertini and Eduardo D Sontag. Identifiability of discrete-time neural networks, In Proc. European Control Conference, pages 460-465. Springer Berlin, 1993.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-paramettriiaaton.. In International conference on machine learning, pages 242-252. PMLR, 2019.
[4] Bruno Andreis. Bedionita Soro, and Sung Ju Hwang. Set-based neural network encoding. CoRR, abs/2305.16625, 2023.
[5] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[6] Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, and Eran Treister. Nern: Learning neural representations for neural networks. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[7] Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. CoRR, abs/1511 1.01029, 2015.
[8] Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural archi- tecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018.
[9] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. CoRR, abs/2302.03130, 2023.
[10] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias varianee trade-off. Proceedings of the National Academy of Sciences, 116632):18449-15588
2019.
[11] Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synaptic learning rule. In Optimality in Biological and Artificial Networks?, pages 265-287. Routledge, 2013.
[12] Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu networks. In 8th International Conference on Learning Representations, 2020.
[13] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marii-FFancine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6491 6506. Association for Computational Linguistics, 2021.
[14] An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910 19-9977,, 1993.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4 1416. Association for Computational Linguistics, 2019.
11

---

[16] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning. pages 1675-1685. PMIR, 2019.
[17] David Steven Dummit and Richard M Foote. Abstract algebra, volume Wiley Hoboken, 2004.
[18] Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Kamalika Chaudhuri. Stefamie Jegelka, Le Song. Csaba Szepesvári, Gang Niu. and Sivan Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 5694-5725. PMLR, 2022.
[19] Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Research, pages 2022.
PMLR,
[20] Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classi- fying the classifier: Dissecting the weight space of neural networks. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang, editors, ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spuin, August 29 September 8, 2020 Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 1119 1119-1126, IOS Press, 2020.
[21] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner. and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 14300-14310, 2023.
[22] Charles Fefferman and Scott Markel. Recovering a feed feew-forward net from its output. Advances neural information processing systems, 6, 1993.
[23] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse. trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[24] Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:11893-11905 2022.
(25) Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks.
In International Conference on Machine Learning, pages 1 11734-11760. PMLR, 2023.
[26] Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luue Bacon. Policy evaluation networks. arXiv preprint 2.Xx::2002./1833,, 2020.
[27] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[28] Robert Hecht-Niclsen On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129-135. Elsevier, 1990.
[29] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): 1780, 1997.
[30] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances neural information processing systems, 31, 2018.
12

---

[31] John M. Jumper, Richard O. Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russell Bates, Augustin Zidek. Anna Potapenko, Alex Bridgland. Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, R. D. Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David L. Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukeuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 July 2021. httpss//eeroppppcccorrg artiicle/MED332265544
https://www.... nature.
httpps//www.mendeeeeyccomm http://wwweee.......aaaoaaaaaaaa
https://faccllyyopinionns com/
prime/740477161
mcb.
https:
com/
7873: : ottps//ww/wppp.////o......//./.a
[32] loannis Kalogeropoulos, Giorgos Bouritsas, and Yannis Panagakis. Scale equivariant graph metanetworks. arXiv preprint 2rXiv:2406.10682 2024.
[33] M. G. KENDALL. A NEW MEASURE OF RANK CORRELATION. Biometrika, 30(1-2):81-
93, 06 1938.
[34] Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano, Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems, 14:29433-29444, 2021.
[35] Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representutions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[36] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter I.. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106 1006-1116 1114, 2012.
[38] Vera Kurkova and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation, 6(3):543-558, 1994.
[39] Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.
[40] Derek Lim, Haggai Maron, Mare T. Law, Jonathan Lorraine, and James Lucas. Graph metanet- works for processing diverse neural architectures. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[41] Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[42] Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv: arXiv:2211.09760 2022.
13

---

[43] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[44] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net, 2022.
[45] Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces, In International Conference Machine Learning, pages 25790-25816. PMLR, 2023.
[46] Behnam Neyshabur, Russ Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
[47] Roman Novak, Yasaman Bahri, Daniel A. Abolatia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization neural networks: an empirical study. In 6th International Conference on Learning Representationss ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[48] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209. 12892, 2022.
[49] M. Raissi, P. Perdikaris, and G.E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[50] Joseph J Rotman, An introduction to the theory of groups, volume 148, Springer Science & Business Media, 2012.
[51] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representa- tions by eπгог propagation. 1986.
[52] Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pages 59 63. IEEE, 2000.
[53] Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- representations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 35:27906-27920, 2022.
[54] Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, 34: 16481-164999,, 2021.
[55] Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems, 05:38134-38111 2022.
[56] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[57] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
[58] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gor- don Wetzstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
14

---

[59] Samuel Sokota, Hengyuan Hu, David J Wu, J Zico Kolter, Jakob Nicolaus Foerster, and Noam Brown. fine-tuning approach to belief state modeling. In International Conference on Learning Representations, 2021.
[60] Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of develop- ment. Genetic programming and evolvable machines, 8:131-162. 2007.
[61] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 1-9. IEEE Computer Society, 2015.
[62] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kell Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances neural information processing systems, 33:7537-7547, 2020.
[63] Viet-Hoang Tran, Thicu N Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. arXiv preprint MKKi:22100042005 2024.
[64] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002. /448, 2020.
[65] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Iya Tolstikhin. Predicting neural network accuracy from weights, 2021.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 December 4-9, 2017, Long Beach, CA, USA, 2017.
pages
[67] Thieu N Vo, Viet-Hoang Tran, Tho Tran Huu, An Nguyen The, Thanh Tran, Minh-Khi Nguyen- Nhat, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant polynomial functional networks. arXiv preprint 2rXiv:2410.042133 2024.
[68] Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. Discrete applied mathematics, 69(1-2):33-66 60, 1996.
[69] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist mnnis::: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017.
[70] Allan Zhou, Chelsea Finn, and James Harrison. abs/2402.052322, 2024.
Universal neural functionals. CoRR,
[71] Allan Zhou, Kaien Yang. Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems, 36, 2024.
[72] Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Supplement to "Monomial Matrix Group Equivariant Neural Functional Networks"
Table of Contents
Construction Monomial Matrix Group Equivariant Layers
16
A.1 ReLU activation
19
A.2 Sin Tanh activation
20
Construction of Monomial Matrix Group Invariant Layers
B.1 ReLU activation
21
B.2 Sin or Tanh activation
22
C Proofs of Theoretical Results
23
C.I Proof of Proposition 3.4
23
C,2 Proof of Proposition 4.4
24
D Additional experimental details
27
D.I Runtime and Memory Consumption
27
D.2 Comparison of Monomial-NFNs and GNN-based NFNs
27
D.3 Predicting generalization from weights
28
................................
D.4 Classifying implicit neural representations images
................................
29
D.5 Weight space style editing
30
D.6 Ablation Regarding Design Choices
31
A
Construction of Monomial Matrix Group Equivariant Layers
In this appendix, we present how we constructed Monomial Matrix Group Equivariant Layers. We adopt the idea of notation in [71] derive the formula of linear functional layers. For two weight spaces U and U' with the same number of layers L as well as the same number of channels at i-th layer n,:
U = W x where:
(22)
W = IR X2XX XnL-1 X... X RUIXN2XM X R XN1X00 Xno
B x x... Raxxx g
and
u' W x B' where:
(23)
W Re xnz XPI-1 x Th xnxxx xns x IR W xn1 Xno
B' R xn..11
x xx x R xm1x1
our equivariant layer F: U - Ut' will has the form as follows:
E (W,b) = U (W b) where:
(24)
My
1y
L n,
5k
p
ukkW (e)
Pay
jkk
Pq
4 p + 50h
(okg(s)
(25)
p=1
610
W(
((() + 15) sp
(26)
spq
Here, the map E is parameterized by hyperparameter s, () with dimensions of each component as follows:
p I 1/ represents the contribution of pq to w((())
16

---

dep E Ru represents the contribution of t/.²) to W
wok E Re is the bias of the layer for W jk
rij spq
E R represents the contribution of Whe to 1700)
dpp E R represents the contribution of (*) to to,b) (()
E & is the bias of the layer for (/:)
We want to see how an element of the group Gu acts on input and output of layer E. Let
g = (99"))...,g')) E Gnz x... x Ino = Gu
(27)
where
g0 = =00 Px. diag (dh). ".....,,!!! dol Pa. E Gn.
(28)
Recall the definition of the group action gU (gW.gb) gb) where:
(gW) = (9 W00) (9)-11))- and (ab) (1(0 6600)
(29)
or in term of entries:
(gW) (() jk 16-1) m. (8) and (ab)l) d.0) b (1)
(0
(30)
7 ()
gE(U) = -000 (gW', .gb) is computed as follows:
( = d0
W n,
(31)
d 1-11)
Hf 1-
(i)
n2
Papa
in () (1)
W(1) +
pq
(32)
in. in. (1) + 10 (177(01m72,00
9sp
(33)
(ab))5)) d(.). (())))
(34)
())
I.
= d
ΣΣΣ
tn (1)
5 Sapl
+
(35)
pq
F1
+ + (0)
Tap
(36)
ElgU) (gU) ((gW), (gU))) is computed as follows:
17

---

L D2 n
df))
L n
(0
jike
(
1ik d(s)
((())
+ tōk (37)
p spn
dg d(--1)
no
(p) (1)
4ap
P
35 (p'
==1 p=1
igle
(p(p)
W(() +
iik
d(s)
(x(p) 1640) - tijk
(p)m.) (0)
((-11)
98 (p)
19)
A= p=1
(38)
1 11
de
11a
(ab))
( ((0)
12 (39)
spy d
(p) (1)
+
sp
p
(r)
1
1
n2
(p(p)
W(() pq
d(s) (.)
((0)
+ 15.
ST, (p) (g)
(8-1)
ST, (p)
(4)
(40)
We need E is G-equivariant under the action of subgroups of tu as in Theorem 4.4. From the above computation, if gEU) = E(gU), the hyperparameter = (p, 9, 5, t) have to satisfy the system of
constraints as follows:
Pspg
(4))
(N)
-
(p)
(41)
d/ (1)
x(-11) 1)
T--1(())
d(
i =1))m (N)
(11-)
4sp
(0),(() (p)
(42)
d(a)
(
b7. (1)
(.(P)
Kapg
ST. (1)
d<--1)
(43)
ta (U)
Sap
(n) (r)
(44)
pm 01) (N)
(i
(45)
d (i
Ij
1 p (1) 1
(46)
for all possible tuples ((i, k), (s. p, 9 and all g E G. Since the two subgroups G considered in Theorem 4.4 satisfy that: G P; is trivial (for i= or i L) or the whole P. (for < L), so we can simplify the above system of constraints by moving all the permutation IT' to LHS, then
replacing -1 by ग. The system, denoted as (*), now is written as follows:
1-15
(p).(p)
Papp
de (=-1) 1)
P
(*1)
(0)
d(
1((-1)
(epx(p)
9sp
1ik
d(s)
(*2)
(2)
(m)
STT (p)))
ap4
di (0-1)
(43)
(6)
s (p)
1/ (())
(+4)
j
(
j
( 100= 15
(*5)
d
1)
(() = 15
(*6)
We treat each case of activation separately.
18

---

Table Hyperparameter of Equivariant Layers with ReLU activation. Left presents all possible case of tuple ((². k), (s. P. P.))), and Right presents the parameter at the corresponding position. Here, we have three types of notations: means the parameter equal to O; equations with T''S o's in LHS means the equation holds for all possible and a single term with no further information means the term can be arbitrary.
Tuple ((², k), (p,
Hyperparameter (p, 9, r.
and 8
j and p
k and
Ply
PM
Papg
1 1=8=1
0
(a)
Lik
1x(
1ik
(+())
1m(s)
11
p
Pin(ila
Pisy
91x())
415
5 (1m())
51
==s=L
k #
0
Lp
I.
g
P (k))(k)
(k)(()
PLjy
Likk
0
Lp
Lj
I<i=s<h L. j&P
0
==D p
k k
0
(((())
(m(s)
h == ?
Am ( P (im()))' (k)
Piyk
0
(=())
(m())
*
0
Table 6: Construction of equivariant functional layer with ReLU activation. Note that all parameters have satisfy the conditions presented in Table 5.
Layer
Equivariant layer E' (W.b) (W",8)
03(3)
i=1
nh 2==1
P
Pijg jq
jik (2)
+ 913 2.3
Eg=
Fijo
lj W(() + $3i0000
bikk w(())
1 1<i<
Piyk ik
i=L
nL Lik ((L) Σ p=1 Lik pk
Σ p=1 Li
Li (L)
+ 1d
A.1 ReLU activation
Recall that, in this case:
G= (idg,, 920 x ... x 9:00 x fidg.g).
(47)
So the system of constraints (*) holds for:
1. all possible tuples ((i.j.)), (s.
2. all E P, for 0 i < L, all d.0) for < / L,
3. म. = idg., and d
for i = 0 i
By treat each case of tuples ((i,j (8, ()).. we solve Eq. *1, Eq. *2, Eq. *3, Eq. #4 in the system (+) for hyperparameter (p, q. r) as in Table 5. For Litk and {ù by Eq. *5, Eq. *6, we have gikk = for all A), ( Of i L. and { is arbitrary for all j < ng. In conclusion, the formula of equivariant layers E in case of activation ReLU presented as in Table 6.
Example A.1. Let us consider a llos MLP with activation that 70n
ReLU. Assume n22 na 2, i.e. all layers have two neurons. This MLP defines function f: R22 22 given by
f(x) 1 (willo (W('x + b0)) + +b(2) + 6(3),
19

---

W
W
[(0)
where W())
is 2 matrix and 66))
for each 1,2,3, 3. In this case,
W2i
W
b2
the weight space U consists of the tuples
U (WC), W(2), W(s) b()) ((2),8(8))))))) ((3)
and it has dimension 18.
According to Eq. (27), an equivariant layer E over has the form
E(U) (W W(22), w B(a) B(2), B((s))
where
w 15R ((()
Pij1 J1
Pija
1.2 w(
jo2
913 11500011
((2) 55 (/
+r
jo2
+ +
W
2 w
6/2) 52)
211
P2j
W 3/4
w:(3)
3/2 w
6130 53n
31
(3)
31
(3)
jk
P3k1
P3k2
3/2
2
+
These equations can be written in a friendly matrix form as follows.
W
W
11
P112
0
0
qi1b
111
0
w
PIlI
112
P112
qi11
0
w
w
w
0
P121
P122
O
4112
W
0
0
P121
122
P122
122
O
4112
122
w
b(t)
111
111
111
1112
121
F12
w
w
2211
0000
0
O
P212
12
W
0
0
P221
221
0
0
W
21
W
0
0
D222
0
w
(22)
0
9211
(2)
(2)
0
0
0
0
9222
222
(2)
[3)
(3)
P311
P321
0
0
Wr (3)
0
P312
312
O
P322
0
0
W
PS12
P321
0
0
W
W W-3)
P312
P
P322
W
+
(3)
0
0
0
0
8
311
2112
312
b
(3)
(3)
0
5221
5322
(8)
b2
A.2 Sin or Tanh activation
Recall that, in this case:
G= -idg. Git
x Git!! x fidg.g
(48)
So the system of constraints (5) holds for:
1. all possible tuples ((i.j,A), (s,).))).
2. all E P, for 0 < << L, all dfd {+1} for < L
3. = idg., and d 1 for i = 0 or L.
We assume L > 3, the case < can solved similarly. By treat each case of tuples ((i,j, k), (s,P,g)),
solve Eq. *1, Eq. #2, Eq. *3, Eq. #4 in the system (*) for hyperparame- ter (p, r. as in Table 7. For t5k and t by Eq. #5, Eq. #6, we have tü 0 for all (i,j, k), g 0 if I., and { arbitrary for all 1 < & n.. In conclusion, the formula of equivariant layers E in case of sin Tanh activation is presented in Table 8.
20

---

Table 7: Hyperparameter of Equivariant Layers with sin or Tanh activation. Left presents all possible case of tuple ((i. j. k), (s.p.q) 9)), and Right presents the parameter at the corresponding position. Here, we have three types notations: 0 means the parameter equal 0; equations with T's in LHS means the equation holds for all possible æ,; and a single term with no further information means the term can be arbitrary.
Tuple ((t. R), (s. 4))
Hyperparameler (p. r,
Fand s
and
k and o
Pap
Apy
ap
2=8=1
j
0
1 p
P PirGDe
Piye
1 4)=())
413
(m())
P1y
i=s= L
of q
Lp
q
0
2p
I<i=s<L L
itp
0
0
==p p 4
(m
(m)) 0
k ==
(KtE)) (K)
0
(n(s)
(1.0) = (T. 111-11 1.
==q
(2-1)
(pn())
Epps
(1-1))
(1 (L, L 1)
(jR(() 9(2-1)x(8)
9,2--19
Palk
0
0
otherwise
Table 8: Construction of equivariant functional layer with sin or Tanh activation. Note that all parameters have to satisfy the conditions presented in Table 5.
Layer
Equivariant layer E (W, (W", 8)
W jk
83(3
(1 P1jy
1ik
(())
En Fij4 w + $10
i=
+ 41,
<<i<
Payk
lsk
$/66
i=L-1
P 1(5kk jk
(1-
Eps (I. 11/ W (T.)
Lpi
w/(i)
+ (I- (1-
i=b
Epe
Lakk w Lpke pk
I.jk (4-1)
Epe LL up
1 tis
(L--
B Construction of Monomial Matrix Group Invariant Layers
In this appendix, we present how we constructed Monomial Matrix Group Invariant Layers. Let U be a weight spaces with the number of layers L as well as the number of channels at i-th layer n4. We want to construct G-invariint layers 21 Rd for some d > O. We treat each case of activations separately.
B.1 ReLU activation
Recall that, in this case:
G (ide,, G:1l x... x g x {idg..
(49)
Since g200 is the semidirect product of A2" and P. with A20 is the normal subgroup, we will treat these two actions consecutively, A7-0 first then P.. We denote these layers by TA>o and Tp. Note that, since TA s comes before /p, TA required to be -nn-vrrra and P.-equivariant, and Tp is required to be P.- P..iivariiant.
> -ivarriacee and P. -equivariance. To capture -nnivarriance, we recall the notion of pos- itively homogencous of degree zero maps. For > 0, map from I is called positively
21

---

Table Constraints of component invariant functional layer with ReLU, sin, Tanh activations.
Layer
1>> (W,b)
(W)
(1)
w(
(6)
(()
0j
(1) = Ojk
(1) (4)
i=1
x() jj
i<i<L L
(k) jjk
(1)
(1)
(1) a
(4) (1)
(1)
i=
(k(k)
= jjk
homogeneous of degree zero
a a(xt1,..... Azn) = _0x11,..,,
(50)
for all À >> and (II...., E R". We construct Ia>a: U - 14 by taking collections of positively homogeneous of degree zero functions {a corresponds to weight and bias of U. The maps TA=0: U - 21 that (W, (W" Db) is defined by
(e) T - R" and fa) R R } each one
simply applying these functions on each weight and bias entries as follows:
W jk
(wl)) and 670h aj ( (0,)) je
(51)
Ia po is A -nnvariant by homogeneity of the a functions. To make it become P_ P--equivariant, some functions have to be shared arross any axis that have permutation symmetry, presented in Table
Candidates of function We simply choose positively homogeneous of degree zero function a : R" + RR" by taking (0)) = and:
a a mx) 8
(52)
where 3: RD Rd is arbitrary function. The function can be fixed parameterized make to be fixed or learnable.
P.-invariance. To capture P.-invariance, we simply take summing or averaging the weight and bias across any axis that have permutation symmetry in [71]. In concrete, some d > we have Tp: u - Rd computed as follows:
IT(U)
(W!) W( W(2)
F. 1...... ((L--)) (1), v 2
(1-1))
(53)
Here, denotes summation averaging over the
columns of the weight and bias.
G- C-invariance. Now simply compose Ip LA>o to get an G-iivarrant map. We use MLP to complete constructing an G-invariant layer with output dimension d as desired:
= MLP o Ip o TA
(54)
B.2 Sin or Tanh activation
Recall that, in this case:
G= (idg.,) x I x... x 9.ll x fidg..
(55)
Since 9#1 is the semidirect product of =1 and P. with A is the normal subgroup, we will treat these two actions consecutively, A first then P.. We denote these layers by TA and Ip. Note that, since JA comes before /p, TA in required to be A invariant and P.-equivariant, and Tp is required to be P. P.-invariant.
22

---

A invariiace and P, equivaaarieeee. To capture A -invariance. we use even functions, i.c. (x)) a(-z) for all x. We construct L : 4 by taking collections of even functions fajs: I - R and {az (9) R Rh }, each one corresponds to weight and bias of U. The maps 1a+1: 2 - U that (W, + (W", b') is defined by simply applying these functions each weight and bias entries as follows:
(
( and 67
(8)
(56)
+1+1 is A# invariant by design. To make it become P. P.-equivariant, some a functions have be shared arross any axis that have permutation symmetry, presented in Table 9.
Candidates of function a.
We simply choose even function R R by:
o(11,... Fn) = 3 (lail,..., [xnl)
(57)
where 3: R Rd is arbitrary function. The function can be fixed parameterized make to be fixed or learnable.
P. P.invariance. To capture Pr-invariance, simply take summing averaging the weight and bias across any axis that have permutation symmetry as in [71]. In concrete, some >> O, we have Ip: u - Rd is computed as follows:
Ip(U) (w! +, w!t W W((--1), (L))
"
(58)
Here, * denotes summation or averaging over the rows or columns of the weight and bias.
G- invariance. Now we simply compose Ip /A+1 to get an G-invariant map. We use an MLP to complete constructing an G-invariant layer with output dimension d as desired:
I = MLP o 1p A+1.
(59)
C Proofs of Theoretical Results
C.1 Proof of Proposition 3.4
Proof. We simply denote the activation ReLU or sin or tanh by a. Let A E GL(n) that satisfies:
oAA x) A oxx),
for all x E R" This means:
ain
0n
a
0x1
Onn
In
anl
ann
Znn
for all ,1,,.., In E R. We rewrite this equation as:
a11 + 01202 + ... din
Tan
ain
T((2)) )
0n131 + 0,202 + ... + ann or equivalently:
Onl
ann
o (#n)
(ani 01232 ... ain
(nna(()) + (120 ... (xn)
(aa111 + an202 + ... + an)
an1o(x1) + anzo (x2) t... + (In)
Thus,
σ Σ agjz
= (xy))
j=1
for all fn E IR and i = 1,..., n. We will consider the case i i.e.
Σ aijai
Σ
(r))
(60)
and treat the case i > 1 similarly. Now consider the activation case by follows.
23

---

(i Case 1. a = ReLU. We have some observations:
1. Let æ1 1, and 22 In 0. Then from Eq. (60), we have:
σ (an1) d11,
which implies that an 2 O. Similarly, we also have 112 .... Oin 3 0.
2. Since A is an invertible matrix, the entries aj1...., Ain in the first row of A can not be simultaneously equal to 0.
There is most only one nonzero number among the entries alls.... ain. Indeed, = 0, from Eq. (60), we
assume by the contrary that 411, 012 > 0. Let 23 have:
(an1 + 01222) = 0110((11) + a120
Let 22 1, we have:
aannz a12) (())
Now, let 2 be sufficiently large number such that 01131 01 0. (Note that this number exists since 011 > 0). Then we have:
~1 012 = 01131
which implies 01 = 0, a contradiction.
It follows from these three observations that there exactly one non-zero element among the entries d11..... Oin- In other words, matrix A has exactly one nonzero entry in the first row. This applies for every row, so A has exactly one non-zero entry in each row. Since A is invertible, each column of A has least non-zero entry. Thus A also has exactly one non-zero entry in each column. Hence, A is in Ga. Moreover, all entries of A are non-negative, Á is in 9
It is straight forward to check that for
G we have a(A - x) A ox)..
(ii) Case Tanh
sin. We have some observations:
Let 222
Fn = 0. Then from Eq. (60), we have:
(()1=)) = 0110(x1),
which implies 0n E {-1,0, 13. Similarly, we have 012...., Ain E (-1,0,1}.
Since an invertible matrix, the entries ajl:..., Cin in the first of can be simultancously equal to 0.
3. There is at most only one nonzero number among the entries a11..... Oin. Indeed, assume by the contrary that 411, 012 + 0. Let 23 = ... In = 0, from Eq. (60), we have:
(a1ı + 01222) = in ( + (x2).
Note that 011, 012 E {), so by consider all the cases, we will lead to contradic- tion.
It follows from the above three observations that there is exactly one non-zero element among the entries di1:-.., Oin- In other words, matrix A has exactly one nonzero entry in the first row. This applies for every row, A has exactly one non-zero entry in each row. Note that, since A is invertible, each column of A has at least one non-zero entry. Therefore, A also has exactly one non-zero entry in each column. Hence, is in Gn. Moreover, all entries of A are in {-1, 0, 1}, so A is in Git11 It is straight forward check that for all A in gi#1 we have (AA- A oxx).
The proposition is then proved completely.
C.2 Proof of Proposition 4.4
Proof. For both Fully Connected Neural Networks case and Convolutional Neural Networks case, we consider a network f with three layers, with ni, 11, n12 ng are number of channels at each layer, and its weight space U. We will show the proof for part (i) where activation is ReLU, and part (ii) can be proved similarly. For part (i), we prove be G-invariant on weight space 24 for the group G' that is defined by:
G {idg.g) } x G~0 G2D0 x {idgnn Gna Gno Gon x Gno Gui
24

---

Case 1. f is a Fully Connected Neural Network with three layers, with nu, n1. R2, are number of channels at each layer as Eq. 5:
f(x: U,o) = w()) (w(2). (WO)... x x+ + 112) + b(3)
is Convolutional Neural Network with three layers, with nio 11,, r2, ng number of channels at each layer as in Eq. 8:
Case 2.
f(x; U, w(3) * a (w(2) 0 (w(L) * x 1x+b00) + (((3) + b(3)
We have some observations:
For case 1. For W E Imxx x R and 0, we have:
a aWW b) ((aW) x (ab)).
For case 2. For simplicity, we consider * as one-dimentional convolutional operator, and other types of convolutions can be treated similarly. For W (Wi,...., Wmn) m b E R and (f1,..., xx) ) E IR", we have:
W +xx+b=y= (yi.... m 1) E R=-m+t
where:
m yi
w - b.
j=1
So for a >> O, we have:
a o ( * x b) = a ((aW) xx + (ab)).
With these two observations, we can see the proofs for both cases are similar to each other. will show the proof for case 2, when f is convolutional neural network since it is not trivial as case 1.
Now we have U (W, b) with:
W- (W(3), W(2), w b (b1)),13) (19))
Let g be element of G:
(ideng: (2)), ,0))),d ide..))
where:
(22) -D(2). Pr, diag
d
2)
d(2))
P.,, G24
....,
( D(1), Pr, diag
(1)
,...,
(1)
Pa3 9
We compute gU::
gU (gW, gb).
gW (oW)(3)) (2W) ((w))))
gb (g)) (2) (g))
where:
25

---

(gW) (3)
Ww(3)
jk
d 2)
jzz (N)
()
(gW)
W (2)
(1)
(1)m '0))
(11)
(oW)
31 G))
and,
(ab)) 1,5)
(gb),] dzd)
( '''
(ab) d," 1) 0)
Now we show that f(x; ; U, = f(x: GU, σ) for all x = (I1, Ino) E R00 For & i & ng, we compute the i-th entry of f(x ; gU, o) as follows:
f(x; gU,o);
m2
Σ
(gW)
1i52
(3)
Σ
(gW)
(2)
Σ
(gW)
(1)
Z + (ob))
(ob)
+ (ab)
1
W (3)
j2
(2
(2)
2
(1a)
m (2))
32
(1)
# + +
((1)
+ tdd (22) (22)
+ b
m (1i)o
xI TT)
Σ
(3()
j2
(2(2
(2)
(12)
*
(3)
m2 ((a)=
12=1
J2
11=1
mi (.16) ))
(1)
(1))
too + (1)
+d
((2)
+ +b(3)
my ((2)
72
1
W
(3)
R1
(2)
2
(12)
(J2)x
j2
J2
11
mg
w
+ +0
d
(2))
+ +6600)
#
xi 21)
2 22) (ta)
72
1
W
(3)
(2)
(2) (2)
(2)
(iz)
(1))
32
n=1
(1) (1)
(1)
+ (2))
(2))
+b6.30
2j G0)0
1
m. - (3i)
mg (z))
26

---

11g
m1
W
d!2)
Σ
w
Uia)
j2
x2
(2)aa''(1) (1)
=2=1
1=1
15g
))
W
+ (())
+ 6(2)
+ 01.00
G1)jo
* Ijo
mi (i.)
x2 (ca)
J==1
112
Σ
W
(3)
d!22)
(2)
Un)
U2)m
12=1
W (1)
2yo + 6 (1)
11 (i))
(52)
11g
W (3)
(2)
U2)
(2))a (1)) (i))
w (1)
Jj + +b
+ (2)
+ (3)
"I (fi)do
22 ($2)
W tjz
(x2) Jahn
((1)
31
Jjjo
+ (())
51
+ jz
(22)
+ b
==
81=1
J==1
=f(;; U,
End of proof.
D Additional experimental details
D.1 Runtime and Memory Consumption
We provide the runtime and memory consumption of Monomial-NFNs and the previous NFNs in Tables 10 and 11 to compare the computational and memory costs in the task of predicting CNN generalization (see Section 1).. observable that our model runs faster and consumes significantly less memory than NP/HNP in [71] and GNN-based method in [35]. This highlights the benefits of parameter savings in Monomial-NFN.
Table 10: Runtime models.
NP [71]
HNP [71]
GNN [35]
Monomial-NFN (ours)
Tanh subset
35m34s
29m37s
4h25m17s
18m23s
ReLU subset
36m40s
30m06s
4h27m29s
23m47s
Table 11: Memory consumption.
NP [71]
HNP [71]
GNN [35]
Monomial-NFi (ours)
Tanh subset
838MB
856MB
6390MB
582MB
ReLU suhset
838MB
856MB
6390MB
560MB
D.2 Comparison of Monomial-NFN. and GNN-based NFNs
We provide experimental result to compare the efficiency of our model and a permutation equivariant GNN-based NFN [35] in scenarios below.
1. Training the model on augmented train data and testing with the augmented test data (see Tables 12 and 13).
Here, we present the experimental results on the original dataset and the results on the augmented dataset. The augmentation levels for the ReLU subset 1, 2, 3, and 4,
27

---

corresponding to augmentation ranges of [1, 10), [1, 109),
), [1, 10") [1, 100)). The augmented
dataset for the Tanh subset corresponds the augmentation range of [-1,1]
Table 12: Predict CNN generalization ReLU subset (augmented train data)
Original
GNN [35]
0.897
0.892
0.885
0.858
0.851
Monomial-NE (ours)
0.922
0.920
0.919
0.920
0.920
Table 13: Predict CNN generalization on Tanh subset (augmented train data)
Original
Augmented
GNN [35]
0.893
0.902
Monomial-NFFy (ours)
0.939
0.943
The results for GNN exhibit similar trend as other baselines that do not incorporate the scaling symmetry into their architectures. In contrast, our model has stable performance. A notable observation is that the GNN model uses 5.5M parameters (4 times more than our model), occupies 6000MB of memory, and takes 4 hours to train.
2. Training the model on original train data and testing with the augmented test data (see Tables 14 and 15).
Table 14: Predict CNN generalization on ReLU subset (original train data)
Augment level
2
3
GNN [35] 0.794
0.679
0.586
0.562
Monomial-NF (ours)
0.920
0.919
0.920
0.920
Table 15: Predict CNN generalization on Tanh subset (original train data)
Augmented
GNN [35]
0.883
Monomial-NV (aurs)
0.940
In these more challenging scenario, GNN's performance drops significantly, which highlights the lack of scaling symmetry in the model. Our model maintains consistent performance, matching the case in which we train with the augmented data.
D.3 Predicting generalization from weights
Dataset. The original ReLU subset of the CNN Zoo dataset includes 6050 instances for training and 1513 instances for testing. For the Tanh dataset, it includes 5949 training and 1488 testing instances. For the augmented data, we set the augmentation factor to 2, which means that we augment the original data once, resulting in dataset of double the size. The complete size of all datasets is presented in Table 16
Implementation details. Our model follows the same architecture as in [71], comprising three equivariant Monomial-NFN layers with 16, 16, and 5 channels, respectively, each followed by ReLU activation (ReI.U dataset) or Tanh activation (Tanh dataset). The resulting weight space features are input into an invariant Monomial-Ve learnable parameters (ReLU case) or mean pooling (Tanh case). Specifically, the Monomial-NEN pooling layer normalizes the weights across the hidden dimension and takes the average for rows (first layer), columns (last layer), or both (other layers). The output of this invariant Monomial- layer is flattened and projected to 200 (ReLU case) R 1000 (Tanh case). This resulting vector is then passed through an MLP with two hidden layers with ReLU activations. The output is linearly projected sealar and then passed through a sigmoid function. We use the Binary Cross Entropy (BCE) loss function and train the model for 50 epochs, with early stopping based on the validation set, which takes 35 minutes to train on an A100 GPU. The hyperparameters for our model presented in Table 18.
layer with Monomial-NFN pooling (Equation 19) with
28

---

Table 16: Datasets information for predicting generalization task.
Dataset
Train size
Val size
Original ReLU
6050
1513
Original Tanh
5949
1488
Augment ReLU
12100
3026
Augment Tanh
11898
2976
Table 17: Number of parameters of all models for prediciting generalization task.
Model
ReLU dataset
Tanh dataset
STATNN
106M
1.06M
2.03M
2.03M
HNP
2.81M
2.81M
Monomial-NFi (ours)
0.25M
1.41M
Table 18: Hyperparameters for Monomial-NFN on prediciting generalization task.
ReLU
Tanh
MLP hidden neurons
200
1000
Loss Optimizer Learning rate
Binary cross-entropy
Binary cross-entropy
Adam
Adam
0.001
0.001
Batch size
8
Epoch
50
50
Table 19: Dataset size for Classifying INRs task.
Train
Validation
Test
CIFAR-10 MNIST size
45000
5000
10000
45000
5000
10000
Fashion-MNIST
45000
5000
20000
For the baseline models, follow the original implementations deseribed in [71], using the official code (available at: https://itthub.com www./ooollaanaa YangZhou/nfi). For the HNP and NP models, there are 3 equivariant layers with 16. 16, and 5 channels, respectively. The features go through average pooling layer and 3 MIP layers with 1000 hidden neurons. The hyperparameters our model and the number of parameters for all models in this task can be found in Table 17.
D.4 Classifying implicit neural representations of images
Dataset. We utilize the original INRs dataset provided by [71], with no augmentation, The data obtained by implementing single SIREN model for each image in each dataset: CIFAR-10. 10, MNIST, and MiahiO The size of training, validation, and test samples for each dataset is provided in Table 19.
Implementation details. In these experiments, our general architecture includes 2 Monomial- NFN layers with sine activation, followed by Monomial-NFN layer with absolute activation. The choice of hidden dimension in the Monomial-NFN layer depends each dataset and is described in Table 20. The architecture then follows the same design as the NP and HNP models in [71]. where a Gaussian Fourier Transformation is applied encode the input with sine and cosine components, mapping from dimension to 256 dimensions. If the base layer is NP, the features will go through (OSinusoidalEncodding positional encoding designed for the NP layer, with maximum frequency of 10 and 6 frequency bands. After that, the features go through 3 HNP or NP layers with ReLU activation functions. Then, an average pooling is applied, and the output is flattened, and the resulting vector is passed through an MLP with two hidden layers, each containing 1000 units and ReLU activations. Finally, the output is linearly projected to a scalar. For the MNIST dataset, there an additional Channel Dropout layer after the ReLU activation of each HNP layer and a Dropout layer after the ReLU activation each MI. layer, both with dropout rate of O.I. We use the Binary Cross Entropy (BCE) loss function and train the model for 200,000 steps, which takes hour and 35
29

---

Table 20: Hyperparameters Monomial-NFN for each dataset Classify INRs task.
MNIST
Fashion- Cashion-MNiST
CIFAR-10
Monnomill-NFN hidden dimension
64
64
16
Base model
HNNP
NP
HNP
Base model hidden dimension
256
256
MI.P hidden
1000
500
1000
Dropout
0.1
0
Learning rate
0.000075
0.0001
0.0001
Batch size
32
32
32
Step
200000
200000
200000
Loss
Binary cross-entropy
Binary cross-entropy
Binary cross-entropy
Table 21: Number of parameters of all models for classifying INRs task.
CIFAR-10
MNIST
Mahion--NNISTT
MLP NP
2M
2M
2M
16M
15M
15M
HNP
42M
22M
22M
Monomial-Me (ours)
16M
22M
20M
Table 22: Number of parameters of all models for Weight space style editing task.
Model
Number of parameters
MLP
4.5M
NP
4.1M
HNP
12.8M
Monomial-No (ours)
4.1M
Table 23: Hyperparaameters for Monomial-NFN on weight space style editing task.
Name
Value
Monomial-NFis hidden dimension
16
dimension
Optimizer
Adam
Learning rate
0.001
Batch size
32
Steps
50000
minutes A100 GPU. For the baseline models, we follow the same architecture in [71], with minor modifications to the model hidden dimension, reducing it from 512 to 256 to avoid overfitting. We use hidden dimension of 256 for all baseline models and our base model. The number of parameters of all models can be found in Table 21
D.5 Weight space style editing
Dataset. We use the same INRs dataset as used for classification task, which has the size of train, validation and test described in Table 19.
Implementation details. In these experiments, general architecture includes 2 Monomial-NFN layers with 16 hidden dimensions. The architecture then follows the same design as the NP model in [71], where Gaussian Fourier Transformation with mapping size of 256 is applied. After that, the features go through IOSinusoidalEncoding and then through 3 NP layers, each with 128 hidden dimensions and ReLU activation. Finally, the output goes through an NP layer project into scalar and LearnedScale layer described in the Appendix of [71]. We use the Binary Cross Entropy (BCH) loss function and train the model for 50,000 steps, which takes 35 minutes on an A100 GPU. For the baseline models, we keep the same settings the official implementation. Specifically, the HNP or NP model will have layers, each with 128 hidden dimensions, followed by a ReLU activation. An NEN of the same type will be applied to map the output to dimension and pass it through LearnedScale layer. The number parameters of models can be found Table 22. The detailed hyperparameters for model can be found in Table 23.
30

---

Dilate (MNIST)
Contrast (CIFAR-10)
9
2
8
3
4
6
Ground truth
Original INR
9
2
8
3
5
4
6
6
MT.P
9
2
3
5
4
6
6
HNP
9
2
8
3
5
4
6
NP
9
2 2
8
3
5
4
6
6
Monomial-Ni
9
8
3
4
6
6
Figure 2: Random qualitative samples of INR editing behavior on the Dilate (MNIST) and Contrast (CIFAR-10) editing tasks.
D.6 Ablation Regarding Design Choices
We provide the ablation study on the choice of architecture for the task Predict CNN Generalization on ReI.U subset in Table 24. We denote:
Monomial Equivariant Functional Layer (Ours): MNF
Activation: ReLU
Scaling Invariant and Permutation Equivariant Layer (Ours): Norm
Hidden Neuron Permutation Invariant Layer (in [71]): HNP
Permutation Invariant Layer: Avg
Multilayer Perceptron: MLP
Table 24: Ablation study design choices for the task Predict CNN generalization ReLU subset
Original
4
(MFFRcLU))x Norm MENN--RLLUxx Avg MLP
0.917
0.916
0.917
0.917
0.917
NNFF-2 Norm (HNP-ReLU)x Avg MLP
0.918
0.917
0.917
0.917
0.918
MNFFReLU)x? - Norm (HNP-ReLU)x Avg - MLP
0.920
0.919
0.918
0.920
0.920
(MNF-ReLU)x Norm Avg MLP
0.915
0.914
0.917
0.916
0,914
DNNF-ReLU)22 Norm AAg MLP
0.918
0.917
0.918
(MNF ReI.U)x3
Norm Avg MI.P
0.922
0.920
0.919
0.920
0.920
Among these designs, the architecture incorporating three layers of Monomial-NEN with ReLU activation achieves the best performance.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are clearly stated in the Contribution in the Section 1. These claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: All theoretical results in the paper are given together with the full set of assumptions and complete/correct proofs (See Appendix C.2 and Appendix C.I).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the experiment details in the Implementation details section in the Appendix D of our manuscript. We also provide the source code so that the results in the paper can be easily reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the source code in the supplementary resources with detailed guide that the results in the paper can be easily reproduced. We verify proposed methods using public benchmarks (See the Section in our manuscript)
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //iips.co cct..ccccppo wcccpuulcc//aeeeecceeeceeeeeeeee http//uuiie///eeeeeee sondeeSubmissiooPooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details necessary understand the results in the Implementation details section in the Appendix D of our manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars suitably and correctly defined of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources for all experi- ments in our Implementation details Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps://neuu cc/ppblicc/thhiccuuiieee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix ??.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
35

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the githubs and the baselines compare with in the Imple- mentation details part in Appendix D of our manuscript.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "ZOzEdDdo4P", "reviewer": "Reviewer_RAv9", "summary": "- Paper improves neural functional networks by taking into account weight scaling properties of ReLU networks and weight sign flipping symmetries of sin or Tanh networks.\n\n- Monomial matrices are used to represent these symmetries, both permutation and scale/sign-flipping transformations.\n\n- Proposed model has fewer trainable (independent) parameters compared to the original NFN family of models.", "questions": "- Minimal equivariance is the goal for this task as equivariance is generally easy to achieve. How does the proposed model satisfy minimal equivariance while ignoring permutations that are not functionally equivarant?\n\n- Can the model handle a mixed modelzoo with different activation functions? Note that [1] can handle this case and [3] already demonstrates that this is possible. Per my understanding, the proposed model will need different instantiations of the model for each modelzoo with a different activation function.\n\n- How are the input/output layers of the considered networks handled in the proposed framework? Note that these layers do not follow the permutation symmetries of MLPs(only the hidden layers do).\n\n## References\n\n[1] Lim, Derek, et al. \"Graph metanetworks for processing diverse neural architectures.\" arXiv preprint arXiv:2312.04501 (2023).\n\n[2] Kofinas, Miltiadis, et al. \"Graph neural networks for learning equivariant representations of neural networks.\" arXiv preprint arXiv:2403.12143 (2024).\n\n[3] Andreis, Bruno, Soro Bedionita, and Sung Ju Hwang. \"Set-based neural network encoding.\" arXiv preprint arXiv:2305.16625 (2023).\n\n[4] Unterthiner, Thomas, et al. \"Predicting neural network accuracy from weights.\" arXiv preprint arXiv:2002.11448 (2020).\n\n[5] Zhou, Allan, Chelsea Finn, and James Harrison. \"Universal neural functionals.\" arXiv preprint arXiv:2402.05232 (2024).", "limitations": "While limitations are not discussed, i do not see any immediate negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposes a principled way to incorporate activation functions in neural network representations for MLPs and CNNs.", "weaknesses": "- No discussion/comparison with previous works on the subject [1,2,3,5].\n\n- No discussion of extensions to architectures with branches/ transformers?\n\n- Not the first to consider activation functions in representing neural network weights as claimed. In [1] and [2], activation functions are encoded as nodes in a graph.\n\n- The performance gain over the baselines is minimal. Perhaps this suggests that the role of activation function encoding for this task is minimal as shown in the original neural functional works and those of [1, 2, 3, 4].\n\n- There is a focus on very specific activation functions (relu, sin, tanh), admittedly, these are common in many architectures, but the authors do not provide any discussion of how the proposed method can be applied to other activation functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n**W1: Discussion/comparison with previous works in [C1,C2,C3,C5].**\n\n\n**Answer:** Based on the additional references you provided, we have added the following discussion to the revised version of the paper:\n\n- *Previous methods*: Permutations and scaling (for ReLU networks), as well as sign-flipping (for sine or tanh networks) symmetries, are fundamental symmetries of weight networks. Permutation-equivariant NFNs are successfully built in [C1,C2,C3,C5,33,41,64,65]. In particular, the authors in [C1,C2] carefully construct computational graphs representing the input neural networks' parameters and process the graphs using graph neural networks. In [C3], neural network parameters are efficiently encoded by carefully choosing appropriate set-to-set and set-to-vector functions. The authors in [C5] view network parameters as a special case of a collection of tensors and then construct maximally expressive equivariant linear layers for processing any collection of tensors given a description of their permutation symmetries. These methods are applicable to several types of networks, including those with branches or transformers. However, the models in [C1,C2,C3,C5], as well as others mentioned in our paper, were not necessarily equivariant to scaling nor sign-flipping transformations, which are important symmetries of the input neural networks. \n- *Our method* makes the first step toward incorporating both permutation and non-permutation symmetries into NFNs. In particular, the model proposed in our paper is equivariant to permutations and scaling (for ReLU networks) or sign-flipping (for sine and tanh networks). This leads to a significant reduction in the number of parameters, a property that is particularly useful for large NNs in modern deep learning, while achieving comparable or better results than those in the literature.\n\n**W2: Extensions to architectures with branches/ transformers?**\n\n**Answer:** See **Q2** in General Response.\n\n\n**W3: Not the first to consider activation functions in representing neural network weights.**\n\n**W4: The role of activation function encoding for this task is minimal.**\n\n**Answer to W3-W4:** We are certainly not the first to consider activation functions in representing neural network weights as several previous works in the literature, including [C1,C2,C3,C4], have already done. However, we assert that our models are the first family of NFNs to incorporate non-permutations such as scaling and sign-flipping symmetries of weight spaces, which are crucial symmetries of neural network weights. This leads to a significant reduction in the number of parameters, much lower computational cost and memory consumption, while achieving comparable or better results than those in the literature.\n\n**W5: Applied to other activation functions.**\n\n**Answer:** See **Q1** in General Response.\n\n\n**Q1: How does the proposed model satisfy minimal equivariance while ignoring permutations that are not functionally equivariant?**\n\n**Answer:** By Proposition 4.4, we already excluded false permutations (i.e., permutations that are not functionally equivariant) from the symmetry group $G$ of the weight networks. In addition, by Theorem 5.1, we proved that the minimal equivariance is actually achieved for this group $G$. \n\n**Q2: Can the model handle a mixed modelzoo with different activation functions?**\n\n**Answer:** See **Q2** in General Response. In addition, while [C1,C2,C3] can handle these cases, we do not know whether it is possible to incorporate non-permutations, such as scaling and sign-flipping symmetries which are our main focus, into these models.\n\n**Q3: How are the input/output layers of the considered networks handled in the proposed framework?**\n\n**Answer:** The way the input/output layers are handled is described in Proposition 4.4. According to this proposition, the input and output layers are fixed, as any nontrivial permutation or scaling transformation of the input/output layers would result in a different function.\n\n**References**\n\n[C1] Lim, Derek, et al. Graph metanetworks for processing diverse neural architectures. arXiv 2023.\n\n[C2] Kofinas, Miltiadis, et al. Graph neural networks for learning equivariant representations of neural networks. arXiv 2024.\n\n[C3] Andreis, Bruno, Soro Bedionita, and Sung Ju Hwang. Set-based neural network encoding. arXiv 2023.\n\n[C4] Unterthiner, Thomas, et al. Predicting neural network accuracy from weights. arXiv 2020.\n\n[C5] Zhou, Allan et al. Universal neural functionals. arXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ar2uF3VmG0", "reviewer": "Reviewer_MDVG", "summary": "This paper studies the extension of permutation equivariant neural functionals to accommodate the monomial group, which is a generalization of the permutation group. This extension leads to a new class of NFN called monomial NFN that can also handle the scaling symmetry of positively homogenous activation (RELU) and sign-flipping symmetry of activations such as tanh. The paper then characterizes the subset of all monomial matrices that are either *preserved* by either ReLU or sin/tanh ($\\sigma A(x) = A(\\sigma x)$). The paper then proceeds to construct group $G_\\mathcal{U}$ which is the product of all monomial groups that act on a weight space object $U$, and two subgroups of $G_\\mathcal{U}$ under which the models with ReLU or sin / tanh are invariant. Finally, the paper constructs an affine weight space layer that is $G$-equivariant and also $G$-invariant weight space layers, which are more parameter efficient than prior works. Generalization prediction experiments show that the proposed layer performs better than prior works under standard conditions and significantly outperforms when the models are perturbed by scale, corroborating the statement made by the paper. On INR classification and editing tasks, monomial-NFNs are competitive or better than prior works.", "questions": "1. How do the computation cost and memory compare to NFNs? In my opinion, the issue with NFNs' big parameter count is that it's very memory-intensive. While monomial-NFN saves parameters, the benefit is perhaps less significant if it doesn't come with a computational advantage.", "limitations": "Authors have adequately discussed the limitation though it would be good to include some computation cost analysis.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Overall, this is a well-written and technically solid paper with a novel contribution to the neural functional literature. The theoretical results seem sound. While I did not thoroughly check every proof detail, Figure 1 demonstrates that the constructed layer is indeed equivariant to the scale of the weights. The other experimental results are also quite compelling. The parameter saving could also potentially be a big upside of the monomial-NFN which can bring better generalization.", "weaknesses": "I don't see major weaknesses in the paper. Perhaps one shortcoming is that the majority of performance improvement comes when the weights are scaled, which is not a very natural perturbation in reality. On the other hand, many tasks that cannot be \"solved\" by previous NFNs remain difficult for monomial NFNs (e.g., CIFAR10 classification), which might indicate some fundamental limitation of this line of work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n**Weakness: I don't see major weaknesses in the paper. Perhaps one shortcoming is that the majority of performance improvement comes when the weights are scaled, which is not a very natural perturbation in reality. On the other hand, many tasks that cannot be \"solved\" by previous NFNs remain difficult for monomial NFNs (e.g., CIFAR10 classification), which might indicate some fundamental limitation of this line of work.**\n\n**Answer:** We agree with the reviewer on these limitations of this line of work. In addition, the problem of constructing different types of architecture, such as GNN-based or self-attention-based models, that are equivariant to a monomial matrix group is interesting and nontrivial. We leave this open problem for future study.\n\n**Q1: How do the computation cost and memory compare to NFNs? In my opinion, the issue with NFNs' big parameter count is that it's very memory-intensive. While monomial-NFN saves parameters, the benefit is perhaps less significant if it doesn't come with a computational advantage.**\n\n**Answer:** See **Q3** in General Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "84rOoc5XNt", "reviewer": "Reviewer_yJwm", "summary": "The present manuscript concerns the design of Neural Networks capable of processing the weights and biases of other Neural Networks, particularly Fully Connected and Convolutional NNs (known in the literature as *weight space networks, neural functional networks or metanetworks*). Previous works considered only hidden-neuron-permutation symmetries of weights/biases that preserve the function of the NN. In contrast, the authors highlight the existence of symmetries induced by certain activation functions (ReLU – positive scaling symmetries and sine/tanh – sign symmetries – Eq. (20)), following the works of Godfrey et al., NeurIPS’22 and Wood and Shawe-Taylor, Discr. Appl. Math.’96, where in certain cases are mentioned to be maximal (i.e. the only symmetries that preserve the NN function) as proved in the past. Combining these symmetries with permutations leads to the so-called “monomial-matrix groups”. Based on this background, the authors design monomial-matrix equivariant NNs, following the classical NN design strategy: linear layers interleaved with non-linearities. To characterise the former, they identify the weight-sharing pattern that equivariance imposes by solving a system of weight constraints for both the symmetries above (thus fully characterising linear layers), while for the former they use the same non-linearities as the activation functions at hand. Finally, they propose a method for monomial-matrix *invariant* layer design, which is combined with equivariant layers to yield an end-to-end invariant NN. The method is experimentally tested in various tasks: CNN generalisation prediction and Implicit Neural Representation (INR) classification and editing, showing competitive performance against permutation-symmetry-only baselines and a significant reduction in the number of learnable parameters.", "questions": "- As far as I know, Proposition 3.4. and Proposition 4.4 follow from the characterisation of intertwiner groups done by Godfrey et al., NeurIPS’22 (e.g. see Theorem E.14 and Proposition 3.4), but the authors provide alternative proofs. In case these proofs are of independent technical interest, I would recommend that the authors briefly mention a proof sketch in the main paper and the differences with Godfrey et al. (this would also help in properly accrediting this prior work). \n- L210: The authors mention “It is natural to ask whether the group $G$ is still maximal in the other case”. What do they mean by the “other case”?\n- Perhaps an illustrative example of the resulting weight-sharing would help. Or maybe give some intuition on the resulting layers?\n- Maybe adding the exact number of parameters along with the performance in the tables would help in grasping the actual reduction. \n\n**Minor**.\n- L126-L128 are unclear. Perhaps describe in more detail the notation $\\text{Aut}(\\Delta_n)$ and the terms conjugation, semi-direct product etc. (depending on the importance of these statements).\n- L242: hyperparams --> Perhaps you mean learnable params?", "limitations": "Some of the limitations of this work are mentioned in the text (last paragraph of Section 5 and conclusion section). However, I think that some have not been adequately discussed (see weaknesses). I would recommend adding a separate section for this purpose.\n\nI do not foresee any negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Significance/Impact.** The topic of NN processing has been steadily gaining traction in the last year and has the potential to provide significant advantages in various applications such as meta-learning and processing signals of arbitrary nature (encoded into INRs) under a unifying framework. Therefore, improving computational efficiency and incorporating new inductive biases, as done in this work, is an important step towards advancing and popularising the field. \n\n**Presentation**. The paper is in general well-presented, with appropriately chosen notations and clear descriptions of the background concepts involved and the innovations proposed.\n\n**Novelty**. \n   - *Studied problem*. The symmetries discussed in this work, although mentioned in the literature, have not been approached so far in the context of weight space networks. \n   - *Methodology*: This work introduces novel layers that are equivariant or invariant to a group that has been underexplored (monomial matrices).\n   - *Theory*. Additionally, following traditional weight-sharing proof strategies, the layers are characterized as the only linear layers that have the equivariance property, while Remark 4.5. borrows results from relevant papers (that might be not widely known to the community) to highlight the cases where the studied symmetries are the maximally function-preserving ones (however some results are missing – see weaknesses). \n\n**Quality/Execution**. The paper is well-motivated, provides a comprehensive background discussion, follows a rigorous and well-established methodology to design monomial-matrix group equivariant/invariant layers and provides adequate experimental comparisons, including in regimes where previous works fail.\n\n**Computational efficiency**. The proposed method leads to a significant reduction in the number of parameters, a property which is particularly useful for large NNs (a typical use case in modern deep learning).", "weaknesses": "**Limited expressivity (possibly reflected in the experimental results?)**\n- Although characterising the linear layers by solving the weight sharing constraints is a fairly general and rigorous technique for equivariant layer design, I have the impression that the resulting weight sharing, in this case, is severe. I.e. due to the large size of the group considered, the resulting layers seem weak in terms of expressivity. For example, in Eq. (22) all hidden layer weights and bias updates are calculated by linearly processing each element individually, i.e. a weight corresponding to an edge between two neurons will be updated based only on its previous value, ignoring other edges across the same or other layers.\n- Additionally, the activation functions that can be used are quite limited to preserve equivariance (the authors mention that they use the same activation as in the NN that is being processed).\n- Although these design choices are necessary for the current construction, this probably will not be the case for a different one, i.e. by directly designing non-linear layers. In other words, I am concerned that working with the standard NN pattern (linear layers interleaved with non-linearities) might be too limiting for this family of symmetries.\n- I am also wondering if limitations in expressivity are induced by the choice of the invariant layers. Could the authors elaborate on this?\n- I do not consider the above as grounds for rejection, since I believe that even incorporating these symmetries into NFNs and the linear layer characterisation are sufficient contributions. However, they seem like important limitations, which I suspect are reflected in the experimental results, and therefore should be highlighted by the authors.\n\n**Related work and existing results**.  \n- Given that the field is relatively new, I think that the authors should devote more space to a more detailed literature review, e.g. describing the weight symmetries that have been discovered more thoroughly and discussing/comparing the weight space networks that have been proposed so far. Currently, most methods are cited, but not adequately explained. I understand that space might not allow this, but at least adding an extended related work section in the appendix would help.\n- Additionally, the following very related works are missing:  (1) In the topic of weight space networks:\n   - Universal Neural Functionals, Zhou et al., arxiv’24\n   - Graph metanetworks for processing diverse neural architectures, Lim et al., ICLR’23\n- (2) In the topic of weight space symmetries (conditions for maximality):\n   - Reverse-engineering deep ReLU networks, Rolnick et al., ICML’20\n   - Hidden Symmetries of ReLU Networks, Grigsby et al., ICML’23\n- As far as I understand, the works of Chen et al., Neur. Comp.’93, Fefferman et al., NIPS’93 characterise maximal weight space symmetries only for the tanh activation and not for sine, as the authors mention L208 – in case this is true, the statement in Remark 4.5. should be rectified.\n\n**Extensibility**. Judging by the derivation of the weight-sharing patterns, it appears that it is not straightforward to extend this approach to symmetries induced by other activation functions (e.g. some are discussed in Godfrey et al.). Could the authors discuss this (and if true include it in their discussion about limitations?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n-----\n\n\n**W1: Limited Expressivity.**\n\n**Answer:** We agree with the reviewer's discussion on limitations. However, we would also like to share our thoughts on these limitations.\n\n- Although **large size symmetry group might lead to the small number of independent parameters**, we believe that the resulting equivariant layers are still sufficient in terms of expressivity (see Theorem 5.1). Nevertherless, it is necessary to construct equivariant nonlinear layers that can encode more relations between weight network's parameters in order to achieve better results. We leave this interesting topic for future study.\n- **About the activation functions can be used**: See **Q1** in General Response.\n- **Beyond the linear layers**, the problem of characterizing nonlinear layers which are equivariant to both permutations and scaling/sign-flipping, or in particular, the problem of incorporating scaling/sign-flipping symmetries into nonlinear layers (such as self-attentions) is a nontrivial problem. We leave this open problem for future study.\n- **About the choice of the invariant layers**: Unfortunately, the invariant layer constructed in our paper is not expressive enough to express all invariant NFNs. For the ReLU case, one can verify that every invariant layer can be expressed via positively homogenenous functions of degree zero via Eq. (55). However, not all positively homogeneous functions of degree zero can be written in the form of the candidate choice in Eq. (56). Nevertheless, our candidate choice in Eq. (56) already covers a large part of positively homogeneous functions of degree zero. As a result, the invariant layers yield favorable experimental results. The same arguments apply for the sine and tanh networks. We have added this discussion to the limitations.\n\n**W2: Related work and existing results.**\n\n**Answer:** Following the reviewer's suggestion, we have added an extended related work section in the appendix to: (1) include missing related works in the topic of weight space networks and  weight space symmetries, and (2) provide more adequate explanations of cited methods. We have also editted Remark 4.5 regarding the works of Chen et al., Neur. Comp.’93, Fefferman et al., NIPS’93 on characterising maximal weight space symmetries only for the tanh activation and not for sine.\n\n**W3: Extensibility of this approach to other activation functions.**\n\n**Answer:** See **Q1** in General Response.\n\n**Q1: About the proofs of Propositions 3.4. and 4.4.**\n\n**Answer:** While the proofs in Godfrey et al. (NeurIPS '22) are technically involved and apply to very general cases, we provide direct and simple proofs that apply to our considered cases for the convenience of the reader and the completeness of the paper. Our proofs can be seen as simplified versions of those in Godfrey et al. (NeurIPS '22), justified for the considered cases.\n\n**Q2: What do they mean by the \"other case\"?**\n\n**Answer:** By the \"other case\", we mean the case when the network architectures are MLPs with ReLU activation such that the condition $n_L \\geq \\ldots \\geq n_2 \\geq n_1 > n_0 =1$ is not satisfied. In addition, regarding the works of [Chen et al., Neur. Comp.’93], [Fefferman et al., NIPS’93] on characterising maximal weight space symmetries only for the tanh activation and not for sine, the \"other case\" now contains MLPs with sin activation, too. We have added this discussion to the revised version.\n\n**Q3: An illustrative example.**\n\n**Answer:** See **Q4** in General Response.\n\n**Q4: Adding the exact number of parameters along with the performance.**\n\n**Answer:** The exact number of parameters for all models in all tasks have been provided in the Appendix. In addition, we have added runtime and memory usage of our model and the previous ones (see **Q3** in General Response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hwMBWSyoTu", "reviewer": "Reviewer_xQCm", "summary": "The paper explores the important field of learning over weight spaces, where neural networks process other neural networks. Previous research has highlighted the importance of designing equivariant architectures that account for the symmetries of the input neural network, with a primary focus on permutation symmetries. However, previous literature did not account for all symmetries of the input NNs, particularly the weight scaling symmetries of ReLU networks and the weight sign flipping symmetries of sin or Tanh networks. This paper naturally extends previous approaches to account for these activation-based symmetries. \nThis paper first formalizes the group of symmetries that includes both neuron permutations and scaling or sign-flipping transformations using monomial matrices. Next, it proposes a novel architecture for weight space networks that are equivariant to groups of monomial matrices. The new architecture is more efficient in terms of the number of trainable parameters compared to baseline weight space networks (which are only permutation equivariant).", "questions": "1. Is it possible to use the proposed method together with normalization layers?\n2. How easy would it be to extend the method to other activation functions?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is mostly well-written and well-structured.\n2. The paper addresses the important and timely problem of learning in deep weight spaces, presenting a novel architecture that extends previous permutation equivariant networks to also account for scale/sign-flipping symmetries.\n3. The proposed architecture is parameter efficient compared to baseline weight space networks.", "weaknesses": "My main concern is the limited empirical evaluation and missing natural baselines. Also, the presented empirical results, mostly show marginal to no improvement over the limited baseline methods evaluated. The insufficient empirical study of this paper significantly damages the paper’s contribution. Given the current state of the learning over weight spaces literature, I would expect a more diverse, challenging, and comprehensive empirical evaluation.\n\n1. The main text provides very few details on the construction of G-equivariant layers. I suggest the authors to provide at least one concrete example for mapping between a subspace of U, for example, mapping between some bias $b_i$ to a bias $b’_j$. \n2. Since the method is built over NFN, it is limited in the sense that each monomial-NFN can process a specific input architecture. Building over or extending the work to GNN-based weight space networks will allow [1,2] the processing of diverse input architectures.\n3. Missing weight-space baselines, like GNN-based models [1,2], DWSNets [3] (which, while mathematically equiv. to NFNs, obtains better empirical performance, see e.g., [6]) and NFT [4]. \n4. Another important missing natural baseline is to use a permutation equivariant baseline like DWS/NFN or the GNN-based models together with scaling/sign-flipping data augmentations as in [5].\n5. Some evaluation and comparison of runtime and memory consumption w.r.t. baselines would be beneficial.\n6. Also, adding some ablation regarding design choices would be beneficial.\n\nReferences\n\n[1] Graph neural networks for learning equivariant representations of neural networks, ICLR 2024.\n\n[2] Graph metanetworks for processing diverse neural architectures, ICLR 2024.\n\n[3] Equivariant Architectures for Learning in Deep Weight Spaces, ICML 2023.\n\n[4] Neural Functional Transformers, NeurIPS 2024.\n\n[5] Improved Generalization of Weight Space Networks via Augmentations, ICML 2024.\n\n[6] Neural Processing of Tri-Plane Hybrid Neural Fields, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n**W1: Concrete example.**\n\n**Answer:** See an illustrative example in **A4** of General Response.\n\n**W2: Each Monomial-NFN can process a specific input architecture. Building over or extending the work to GNN-based weight space networks will allow [D1,D2] the processing of diverse input architectures.**\n\n**Answer:** See **Q1-Q2** in General response. In addition, while [D1,D2] can handle diverse input architectures, we do not know whether it is possible to incorporate non-permutations, such as scaling and sign-flipping symmetries, into these models.\n\nIn contrast, we take the first step toward incorporating non-permutation symmetries into NFNs. In particular, our proposed model is equivariant to permutations and scaling (for ReLU networks) and sign-flipping (for sin and tanh networks). This leads to a significant reduction in the number of parameters, which is particularly useful for large NNs in modern deep learning, while achieving comparable or better results than those in the literature.\n\n\n\n**W3: Missing weight-space baselines, like GNN-based models [D1,D2], DWSNets [D3] (which, while mathematically equiv. to NFNs, obtains better empirical performance, see e.g., [D6]) and NFT [D4]**\n\n**W4: Another important missing natural baseline is to use a permutation equivariant baseline like DWS/NFN or the GNN-based models together with scaling/sign-flipping data augmentations as in [D5].**\n\n**Answer to W3-W4:**\nThank you for pointing out these related works. Here we provide the experimental results for GNN [D1] in two scenarios: \n1. Training the model on augmented train data and test with the augmented test data \n\n*Table 1: Predict CNN generalization on ReLU subset (augmented train data)*\n| |Original |1|2|3|4|\n|-|:-:|:-:|:-:|:-:|:-:|\n| GNN [D1]|0.897|0.892|0.885|0.858|0.851|\n| Monomial-NF (ours) | **0.922** | **0.920** | **0.919** | **0.920** | **0.920** |\n\n*Table 2: Predict CNN generalization on Tanh subset (augmented train data)*\n\n|| Original| Augmented |\n|-|:-:|:-:|\n| GNN [D1]| 0.893| 0.902|\n| Monomial-NFN (ours) | **0.939** | **0.943** |\n\nThe results for GNN exhibit a similar trend as other baselines that do not incoporate the scaling symmetry into their architectures. In contrast, our model has stable performance. A notable observation is that GNN model uses 5.5M parameters (4 times more than our model), occupies 6000MB of memory and takes 4 hours to train (refer to **Q3** in General Response). \n\n2. Train the model on original train data and test with the augmented test data \n\n*Table 3: Predict CNN generalization on ReLU subset (original train data)*\n|Augment level|1|2|3|4|\n|-|:-:|:-:|:-:|:-:|\n| GNN [D1]|0.794| 0.679|0.586 |0.562|\n| Monomial-NF (ours) |  **0.920** | **0.919** | **0.920** | **0.920** |\n\n*Table 4: Predict CNN generalization on Tanh subset (original train data)*\n| | Augmented |\n|-|:-:|\n| GNN [D1]| 0.883|\n| Monomial-NFN (ours) | **0.940** |\n\n\nIn this more challenging scenario, GNN's performance drops significantly, which highlights the lack of scaling symmetry in the model. Our model maintains consistent performance, matching the case in which we train with the augmented data. \n\n**W5: Comparison of runtime and memory consumption w.r.t. baselines.**\n\n**Answer:** See **Q3** in General Response.\n\n\n**W6: Ablation regarding design choices.**\n\n**Answer:**\n\nHere we provide the ablation study on the choice of architecture for the task Predict CNN Generalization on ReLU subset. We denote: \n\n- Monomial Equivariant Functional Layer (Ours): MNF\n- Activation: ReLU\n- Scaling Invariant and Permutation Equivariant Layer (Ours): Norm \n- Hidden Neuron Permutation Invariant Layer (in [D7]): HNP\n- Permutation Invariant Layer: Avg\n- Multilayer Perceptron: MLP\n\n*Table 5: Ablation study on design choices for the task Predict CNN generalization on ReLU subset*\n| |Original|1|2|3|4|\n|-|:-:|:-:|:-:|:-:|:-:|\n|(MNF → ReLU)x1 → Norm → (HNP → ReLU)x1 → Avg → MLP |0.917|0.916|0.917|0.917|0.917|\n| (MNF → ReLU)x2 → Norm → (HNP → ReLU)x1 → Avg → MLP |0.918|0.917|0.917|0.917|0.918|\n| (MNF → ReLU)x3 → Norm → (HNP → ReLU)x1 → Avg → MLP |0.920|0.919|0.918|0.920|0.920|\n| (MNF → ReLU)x1 → Norm → Avg → MLP |0.915|   0.914|0.917|0.916|0.914|\n| (MNF → ReLU)x2 → Norm → Avg → MLP |0.918|0.919|0.918|0.917|0.918|\n| (MNF → ReLU)x3 → Norm → Avg → MLP | **0.922** | **0.920** | **0.919** | **0.920** | **0.920** |\n    \nAmong these designs, the architecture incoporating three layers of Monomial-NFN with ReLU activation achieves the best performance. \n\n\n**Q1: Is it possible to use the proposed method together with normalization layers?**\n\n**Answer:** See **Q2** in General Response.\n\n**Q2: How easy would it be to extend the method to other activation functions?**\n\n**Answer:** See **Q1** in General Response. \n\n\n**References**\n\n[D1] Graph neural networks for learning equivariant representations of neural networks, ICLR 2024.\n\n[D2] Graph metanetworks for processing diverse neural architectures, ICLR 2024.\n\n[D3] Equivariant Architectures for Learning in Deep Weight Spaces, ICML 2023.\n\n[D4] Neural Functional Transformers, NeurIPS 2023.\n\n[D5] Improved Generalization of Weight Space Networks via Augmentations, ICML 2024.\n\n[D6] Neural Processing of Tri-Plane Hybrid Neural Fields, ICLR 2024.\n\n[D7] Permutation Equivariant Neural Functionals, NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response:**

Dear AC and Reviewers,

Thanks for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that: 1) Our Monomial-NFN is a novel contribution to the neural functional literature with sound theoretical results (Reviewer MDVG) and is an important step toward advancing and popularising the field (Reviewer yJwm); 2) Monomial-NFN is equivariant or invariant to a group that has been underexplored (monomial matrices) (Reviewer yJwm); 3) experimental results are compelling (Reviewer MDVG); 4) the parameter saving is a big upside of the Monomial-NFN which can bring better generalization (Reviewer MDVG, RAv9) and be useful for large NNs in modern deep learning (Reviewer yJwm, xQCm).

We address some of the common comments from Reviewers below.

**Q1: Extensibility to other activation functions (Reviewers RAv9, yJwm and xQCm).**

**Answer:** Our approach can be used to handle the symmetries induced by other activation functions, such as LeakyReLU, by using the same strategy based on weight-sharing patterns. Indeed, it is proven in [24] that the linear groups preserved by some other types of activation functions are certain monomial matrices, which is totally similar to our cases.  The reason why we focus on ReLU, sine, and tanh networks is that: they are commonly used in practice, and it is well-known from the literature that the maximal symmetric group of their weight spaces contains both permutations and non-permutations, such as scaling (for ReLU) and sign-flipping (for sine and tanh) symmetries. Both of these symmetries are fundamental symmetries of the weight spaces.

**Q2: Extensibility to other architectures with normalizations/branches/transformers and mixed activations (Reviewers RAv9 and xQCm).**

**Answer:** Our method is applicable to these architectures, provided that the symmetric group of the weight network is known. The idea is to use the weight-sharing mechanism to redefine the constraints of the learnable parameters. The main concern with the architectures that include normalizations/branches/transformers and mixed activation functions is that we do not know whether their weights are equivariant to any non-permutation symmetries, such as scaling or sign-flipping. Since the aim of this paper is to incorporate scaling and sign-flipping symmetries in addition to permutations into NFNs, these architectures are outside the scope of this paper.


**Q3: Compare computation cost and memory to other NFNs (Reviewers MDVG, yJwm and xQCm).**

**Answer:** We have added the runtime and memory consumption of our model and the previous ones in the two tables below to compare the computational and memory costs in the task of predicting CNN generalization. It is observable that our model runs faster and consumes significantly less memory than NP/HNP in [64] and GNN-based method in [B1]. This highlights the benefits of parameter savings in Monomial-NFN.


*Table 1: Runtime of models*
||NP [64]| HNP [64]|GNN [B1]|Monomial-NFN (ours) |
|-|-|-|-|-|
| Tanh subset | 35m34s | 29m37s  |4h25m17s | **18m23s**|
| ReLU subset | 36m40s | 30m06s |4h27m29s | **23m47s**|


*Table 2: Memory consumption*
| | NP [64] | HNP [64]  |GNN [B1] | Monomial-NFN (ours) |
|-|-|-|-|-|
| Tanh subset | 838MB | 856MB |6390MB|**582MB** |
| ReLU subset | 838MB | 856MB |6390MB|**560MB** |


**Q4: Add an example (Reviewers yJwm and xQCm).**

**Answer:**  Let us consider a two-hidden-layers MLP with activation $\sigma=\operatorname{ReLU}$. Assume that $n_0=n_1=n_2=n_3=2$, i.e. all layers have two neurons. This MLP defines a function $f: \mathbb{R}^2 \to \mathbb{R}^2$ given by
$$f(x) = W^{(3)} \sigma \left( W^{(2)} \sigma \left( W^{(1)} x + b^{(1)} \right) + b^{(2)} \right) + b^{(3)},$$
where 
$W^{(i)} =\left(W^{(i)}\_{jk}\right)$ is a $2 \times 2$ matrix and $b^{(i)}=[b^{(i)}_1,b^{(i)}_2]^{\top}$ for each $i=1,2,3$. In this case, the weight space $\mathcal{U}$ consists of the tuples $U=(W^{(1)},W^{(2)},W^{(3)},b^{(1)},b^{(2)},b^{(3)})$ and it has dimension 18. 

According to Eq. (27), an equivariant layer $E$ over $\mathcal{U}$ has the form $E(U) = (W'^{(1)},W'^{(2)},W'^{(3)},b'^{(1)},b'^{(2)},b'^{(3)})$, where
$$W'^{(1)}\_{jk} = \mathfrak{p}^{1jk}\_{1j1} W^{(1)}\_{j1} + \mathfrak{p}^{1jk}\_{1j2} W^{(1)}\_{j2} + \mathfrak{q}^{1jk}\_{1j} b^{(1)}\_{j}, 
\qquad \text{and} \qquad b'^{(1)}\_j =  \mathfrak{r}^{1j}\_{1j1} W^{(1)}\_{j1} + \mathfrak{r}^{1j}\_{1j2} W^{(1)}\_{j2} + \mathfrak{s}^{1j}\_{1j}b^{(1)}\_{j},$$
$$W'^{(2)}\_{jk} = \mathfrak{p}^{2jk}\_{2jk} W^{(2)}\_{jk}, \qquad \text{and} \qquad  b'^{(2)}\_j = \mathfrak{s}^{2j}\_{2j}b^{(2)}\_{j},$$
$$W'^{(3)}\_{jk}  = \mathfrak{p}^{3jk}\_{31k} W^{(3)}\_{1k} + \mathfrak{p}^{3jk}\_{32k} W^{(3)}\_{2k}, \qquad \text{and} \qquad  b'^{(3)}\_j =\mathfrak{s}^{3j}\_{31}b^{(3)}\_{1} + \mathfrak{s}^{3j}\_{32}b^{(3)}\_{2} + \mathfrak{t}^{3j}.$$
These equations can be written in a friendly matrix form which we included in the attached pdf. (We move the matrix form to the attached pdf due to space constraint.).

**Reference**

[B1] Graph neural networks for learning equivariant representations of neural networks, ICLR 2024.

Meta-review: This paper generalizes the recently introduced Neural Functional Networks (NFNs) to include also the symmetries caused by particular choices of non-linear activations, i.e., the scaling for ReLU and sign flip for Tanh and Sin. The paper shows that any weight symmetry group of fully connected or convolutional network is in particular a subgroup of their group of consideration, i.e., the “monomial matrix group”. Considering this larger symmetry, the NFNs become more parameter efficient. The paper demonstrates competitive performance to previous NFNs. 

The reviewers appreciated the principled approach of the paper and the solid contributions to NFNs. The main limitations raised in the reviews are the worry that these methods suffer from limited expressivity (not improved significantly in the current paper), showing small performance gain over previous work and somewhat limited evaluations. The authors addressed these concerns in rebuttal and offered more experiments and empirical validations including new baselines. The authors also addressed concerns about the extensibility to other activation functions and architectures, and shared some computational and memory cost experiments. Overall the paper seems to offer a strong theoretical advancement and a reasonable empirical advancement to an area of interest and pass the bar for a NeurIPS publication. 

We ask the authors to incorporate all the changes and additions discussed in the rebuttal in the camera ready version.
Decision: Accept (poster)