Equivariant Polynomial Functional Networks
Authors: Thieu Vo
Hoang V. Tran
Tho Tran Huu
An Nguyen The
Thanh Tran
Minh-Khoi Nguyen-Nhat
Duy-Tung Pham
Tan Minh Nguyen
Venue: ICML 2025 poster (2025)
Keywords: neural functional network
equivariant model
permutation equivariance
scaling equivariance
Primary area: deep_learning
TL;DR: We design a family of monomial matrix-equivariant neural functional networks based on a parameter-sharing mechanism that achieves lower memory consumption and running time while preserving expressivity.

Abstract: A neural functional network (NFN) is a specialized type of neural network designed to process and learn from entire neural networks as input data.  Recent NFNs have been proposed with permutation and scaling equivariance based on either graph-based message-passing mechanisms or parameter-sharing mechanisms. Compared to graph-based models, parameter-sharing-based NFNs built upon equivariant linear layers exhibit lower memory consumption and faster running time. However, their expressivity is limited due to the large size of the symmetric group of the input neural networks. The challenge of designing a permutation and scaling equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the development of MAGEP-NFN (**M**onomial m**A**trix **G**roup **E**quivariant **P**olynomial **NFN**). Our approach follows the parameter-sharing mechanism but differs from previous works by constructing a nonlinear equivariant layer represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, enhancing the model's expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demonstrating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.

Equivariant Polynomial Functional Networks
Thieu N. Vo" Viet-Hoang Tran
Tho Tran Huu An Nguyen The? Thanh Tran
Minh-Khoi Nguyen-Nhat 2 Duy-Tung Pham Tan M. Nguyen'
Abstract
such as enhancing training efficiency through learnable op- timizers (Bengio et al., 2013; Runarsson & Jonsson, 2000; Andrychowicz et al., 2016; Metz et al., 2022), capturing features from implicit data representations (Stanley, 2007;
A neural functional network (NFN) is special- ized type of neural network designed to process and learn from entire neural networks as input data. Recent NENs have been proposed with per- mutation and scaling equivariance based on ei- ther graph-based mnessage-passingg mechanisms parameter-sharing mechanisms. However, the challenge of designing a permutation and scal- ing equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the develop- ment of MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial NEN). Our approach fol- lows the sharrin mechanism but differs from previous works by constructing a nonlin- ear equivariant layer represented as polynomial in the input weights. This polynomial formula- tion enables us to incorporate additional relation- ships between weights from different input hid- den layers, enhancing the model's expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demon- strating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.
Mildenhall et al., 2021; Runarsson & Jonsson, 2000), modi- fying network parameters for targeted adjustments (Sinitsin et al., 2020; Cao et al., 2021: Mitchell et al., 2022), assess- ing policies in reinforcement learning (Harb al., 2020), and facilitating Bayesian inference by interpreting neural networks as sources of evidence (Sokota et al., 2021).
A fundamental aspect of NFNs is their ability to respect the inherent symmetries present the weight space of the input neural networks. In the case of multilayer percep- tron (MLP), the weight space exhibits two primary forms of symmetry: permutation symmetry and scaling symme- try. Permutation symmetries arise from the structure of the network itself, since neurons within a hidden layer have no intrinsic ordering. On the other hand, scaling symmetries are induced by the activation functions. For networks with the ReLU activation function, multiplying neuron's bias and all its incoming weights by the same positive scalar scales its output proportionally, leading to scaling-type symmetry (Bui Thi Mai & Lampert, 2020; Neyshabur et al., 2015; Badrinarayanan et al., 2015). Similarly, for sine and tanh activations, flipping the sign of both the bias and all incoming weights of a neuron inverts the sign of its out- put, introducing an alternative form of scaling symmetry (Chen al., 1993; Fefferman & Markel, 1993; Kurkova & Kainen, 1994). These structural symmetries define equiva- lence classes in the weight space, and incorporating them into the design of NFNs ensures that learned representations remain consistent and invariant to these transformations.
1. Introduction
Neural functional networks (NFNs) serve as specialized architectures that operate on fundamental components of deep neural networks, including weights, gradients, and sparsity masks, by treating them as input data (Zhou al., 2024b). These networks have been utilized across various domains of machine learning, contributing to applications
Recent methods have focused on creating permutation equiv- ariant NFNs, such as (Navon et al., 2023; Zhou et al., 2024b; Kofinas et al., 2024: Zhou et al., 2024c). These methods leverage permutation equivariance to respect symmetries arising from neuron reordering within hidden layers. NFNs that are equivariant to both permutations and scaling or sign- flipping have been introduced in (Kalogeropoulos et al., 2024) using a graph-based message- passigg mechanism and in (Tran et al., 2024a) with parameter sharing mechanism. However, similar other graph- based neural functional net- works, treating the entire input neural network graph
Equal contribution ! 'Naiional University of Singapore 'FPT Software AI Center, Vietnam ' UVinUniversity, Vietnam. Correspon- dence to: Viet- HoanH Tran <hoang tranvict wwt.riictttuuu...... edu >.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Equivariant Polynomial Functional Networks
and utilizing graph neural networks causes the graph-based equivariant NFNs in (Kalogeropoulos al., 2024) to have very high memory consumption and running time. In con- trast, the NFNs built upon equivariant linear layers using the parameter sharing mechanism in (Tran et al., 2024a) exhibit much lower memory consumption and running time. Nev- ertheless, the equivariant linear layers introduced in (Tran et al., 2024a) possess weak expressive properties, as the weights of the input hidden layers are updated solety by the corresponding weights of the same input hidden layers. The challenge of designing an equivariant layer based on the parameter-sharing mechanism that maintains both lower memory consumption and running time while preserving expressivity remains unresolved.
nomial layers, we design MAGEP-NEN, a family of monomial matrix equivariant NFNs based on the parameter-sharing mechanism that maintains both lower memory consumption and running time while preserving expressivity.
We evaluate MAGEP- -EENs on three tasks: predicting CNN generalization from weights using Small CNN Zoo (Un- terthiner et al., 2020), weight space style editing, and clas- sifying INRs using INRs data (Zhou et al., 2024b). Exper- imental results show that our model achieves competitive performance and efficiency compared existing baselines.
1.2. Notations
1.1. Contribution
Let n be positive integer. We denote Pa as the set of all permutation matrices, and D, as the set of diagonal matrices in GL,, (RR). We also denote M, as the set of monomial matrices in GL,,, (R), where monomial matrix is product of a diagonal matrix and a permutation matrix. These sets are subgroups of GT,,(R). In addition, we use M and M to denote the set of monomial matrices whose nonzero entries are positive numbers and =1, respectively.
This paper aims to develop a novel NEN that is equivariant to both permutations and scaling/sign-flippin symmetries, called MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial NFN). We follow the parameter-sharing mecha- nism as described in (Tran et al., 2024a); however, unlike (Tran et al., 2024a), we construct nonlinear equivariant layer, which is represented as a polynomial in the input weights. This polynomial formulation enables us to incorpo- rate additional relationships between weights from different input hidden layers, thereby addressing the challenges posed in (Tran et al., 2024a) and enhancing the expressivity of MAGEP-NFN.
For any permutation matrix P E Pr., there exists a unique permutation E S, such that P is obtained by permuting the columns of the identity matrix I, according to т. We denote this as P = Pa and refer to it as the permutation matrix associated with π,, where S, is the symmetric group of all permutations on
However, determining equivariant and invariant layers among generic polynomials in the input weights is chal- lenging due to two main factors: the difficulty in identifying polynomial orbits under group actions and the high com- putational cost of working with generic polynomials. To overcome these issues, we introduce a specialized class of polynomials that remain "stable" under permutations and scaling. Restricting equivariant and invariant layers to linear combinations of these terms ensures computational efficiency and reduced memory consumption.
We reformulate the definitions of weight spaces for MIPs and CNNs, as well as the action of mono- mial matrix groups on these weight spaces. In Section 3, we construct polynomial equivariant and invariant layers, which serve as the main building blocks for our MAGEP-NENS, Several experiments are conducted in Section 4 to verify the applicability and efficiency of our models in comparison with previous ones in the literature. Some related works will be recalled in Section 5. The paper concludes with a summary in Section 6.
Organization.
In particular, our contribution is as follows:
1. We introduce class of polynomials in the input weights, referred to as stable polynomial terms, which remain stable under the group action of the weight space. In addition, we conduct comprehensive study of the linear independence of stable polynomial terms.
2. Background: Weight Spaces and Their Symmetries
Let 44 V be two sets and assume that a group G acts on them. A function ʃ: U - is called G-equívariant if f(g- z) = g- f(r) for all e U and g e G. In case G acts trivially on V/ the function f is called G-invariant. In the context of this paper, Ut and V are weight spaces of fixed neural network architecture, while G is a direct product of the groups of monomial matrices.
2.
We characterize all equivariant and invariant layers among the linear combination of the stable polyno- mial terms. These layers are polynomials of degree at most L - where L is the number of layers of the input neural networks.
From now on, we will fix the activation o to be the rectified linear unit a = ReL.U. The case when a is another typical
3. Build on top of the equivariant and invariant poly-
2

---

Equivariant Polynomial Functional Networks
activation, such semilinear (e.g., LeakyReLU, or odd (e.g., sin, tanh), can be derived similarly.
and [gb]( = (000) [b](0
(5)
or equivalently,
Following (Tran et al., 2024a), we write the weight space of an MLP or CNN with L layers and n; channels at i-th layer in the general form 4 = W x B, where:
(
g jk
[W]
dk
d(-13
Ri (1)
and [abll df
[b] (6)
(6)
W R XPPXXm
RuxXnaxii R X
"00
B = 1 x
X... X glxxaxl Rhxmax
With notation as above, it is well-known that the function f= f(; U, a) be an MLP or CNN given in Equation (1) with the weight space U E li and an activation o = ReLU will be G-invariant under the action of G, i.e.
Here, n4 is the number of channels at the i-th layer, in particular, no and nt are the number of channels of input and output; w; is the dimension of weights and b; is the dimension of the biases in each channel at the i-th layer.
f(x U,o) = [x; gU,
(7)
Each element U of is written as U (WW), [b]), with the weights
for all g E G, U E E and TR
W = (wi)(),.... ,.... (19](,,).,(((()())))) E w,
3. Equivariant and Invariant Polynomial Functional Networks
(2)
and biases
In this section, we construct our MAGEP-NENs, new (3) class of NFNs that exhibit equivariance to the group G of monomial matrices with positive nonzero entries, as de- scribed in the previous section. The core components of MAGEP-NFNS are invariant and equivariant polynomial layers, which will be detailed in Subsection 3.2. At the heart of these layers are the stable polynomial terms, which play crucial role in ensuring low memory consumption and computational efficiency of MAGEP-NENs, These terms will be formally introduced in Subsection 3.1.
[b] = (1) (b] [[]]"),...." ())) E B.
The square brackets will be convenient in the next section when we determine polynomials in the entries of U.
The weight space U/ exhibits two primary forms of symme- try: permutation symmetry and scaling symmetry. Permu- tation symmetries arise from the structure of the network itself, since neurons within hidden layer have no intrinsic ordering. On the other hand, for networks with the ReLU activation function, multiplying neuron's 's bias and all its in- coming weights by the same positive scalar scales its output proportionally. leading to a scaling-type symmetry. Based on the above observation, we define the group G of the form
3.1. Stable polynomial terms
We follow the parameter-sharing mechanism used in (Tran et al., 2024a) for constructing equivariant and invariant lay- ers from 24 to ensure low memory consumption and com- putational efficiency in our model. Unlike the linear layers utilized in (Tran et al., 2024a), we employ polynomial lay- ers. This choice allows us to capture additional relationships between weights from different hidden layers of the input network, thereby enhancing the expressivity of our model.
-==(Inz) M x... x M
(4) }
where I, is the identity matrix of size x n, and M is the set of monomial matrices whose nonzero entries are positive numbers.
Each element E G has the form
However, determining equivariant and invariant layers among generic polynomials in the input weights presents a significant challenge for two key reasons. First, identifying the orbits generic polynomial under the group action is difficult, as such polynomials lack an inherent structured compatibility with the symmetry group of the weight space. Second, computations involving equivariant and invariant layers constructed from generic polynomials are highly in- efficient in both memory usage and computational cost. To address these challenges, we introduce a specialized class of polynomials, referred to as stable polynomial terms. In- tuitively, a stable polynomial term polynomial in the
(g (0))
where (0) D(6) Pa, for some diagonal matrix D diag (d()) ...., a((i) in D., and permutation n, E Sny- The
action of G 4 is defined formally as
(g. gU = ([gW],
where:
(0 = (1(0) (W]()). (da-N)-
3

---

Equivariant Polynomial Functional Networks
entries of U E ll that remains "stable" under the action of G (see Definition 3.1). By restricting equivariant and invariant polynomial layers to linear combinations of these stable polynomial terms, we ensure both computational efficiency and reduced memory consumption.
The above proposition shows that the stable polynomial terms can be viewed as generalization of the entries of the weight matrices [W](t] and bias vectors [6](0. The stable polynomial terms defined above are actually "stable" under the action of G in the sense presented in the following theorem.
Consider the case where the weight spaces have the same number of dimensions across all channels, which means w, = 6, = d for all i.
Theorem 3.3 (Stable polynomial terms are "stable"). With notation as above, let gU (g9W),19bl) be the element of l obtained by acting g (g14),.-- ,....- ())) G on the element U ([W), [6]]. Then we have
Definition 3.1 (Stable polynomial terms), Assume that U = (W)), [b]) is an element of the weight space / with weights [W] ([W](4).... ..... [W](1)) and biases [6] = (16)()) [b] For each L 2 > t > O, we define the following matrices:
[gW](00))
(0000..00 (0(0))
9
[96](*)
= 9
Blo
[w](st]= = [W]6). (W](6-1).... W](++,
[gW g6)46))()))))
= (g(0)
(8)
[W6](501000= = [w](6.0) 16)(0)
[gbgW)
= =(9())
(9(0)
In addition, for each indices & and with L > 8, 3,1> 0, and
[gW WWgg((o00).
(1(5) .W]](..0L.))
(0(1)
matrices (+))(()))) E R xxx and ((a.0)(L.t) E R we
also define the following matrices:
Intuitively, the theorem above asserts that stable polynomi- als exhibit compatibility with the action of the group G. In particular, it provides explicit formulas for efficiently de- termining the transformation of stable polynomials under group actions. This property plays crucial role in enabling the efficient computation of equivariant and invariant layers, especially when utilizing the weight-sharing mechanism.
[BW] (0( = [b]() (())))))) (w)
(9) [WW](s000L..) = [W](+.0) v(LO)L (w](2.))
The entries of the matrices [W](1)), [ (6W7())E.)) and (wW](s defined above are called stable polynomial terms of U under the action of G. Note that we omit the (p()'s in [BW], [WW] for simplicity, they are parameters, and are different for each s,t. t. The denota- tions are foright-forward and reasonable.
Inherited from Proposition 3.2 and Theorem 3.3, we define the polynomial map I: U - R. with maps each element U u to the vector [UU) E I2d of the following form:
In the above definition, use the notation [W] and [WW] to denote products of the weight matrices [W]()) with the appropriate index i. The notation [Wb] indicates that this is product of several weight matrices [W](+) and a bias vector (b))), with appropriate indices and j. For the indices, we use the notation (s, t) to signify that the considered product contains weight matrices with indices ranging from down to t + When the index has two components, for example (W))),)))() the first component (s, t) specifies the range of indices for [W], while the second component (1) indicates the index of the bias vector [b]. Specifically, the last two terms (s)L.) and [W [W]](+.0)(L..) contain the matrices V to multiply two matrices of different sizes from the left and the right.
Σ n1
I(U)= = Σ
4 (x.0);p [W]
L382130 p= 9=1
+ Σ
(s)s (b6)))
1=
Σ
$ (0,01(1) [Wb66000
125>1>0p=1
Σ
(KL.)) [6W](3)(0.))
L3850 L>130
Σ Σ L 120002120
(WW](s00(2.)))) lp
(.,))
Proposition 3.2 (Stable polynomial terms as generalization of weights and biases). With notation as above, then for all
+ +
(10)
L >t>>>>r
>0 O, we have
Intuitively speaking, I(U) is a linear combination of the entries from the input weights [W](*) (from the first sum) and biases [b]() (the second sum), as well as all entries from the stable polynomial terms [W](s.t) (the first sum), [W)((,00())) (the third sum), (16)) (the fourth sum), and (1.)00))) (the last fifth sum) for all appropri- ate indices and t, with a bias. Here, the coefficients
(1]
[w]
E R Iddxnxxm-11
(w]( (W](er))
(wr)
E x
(Bw((>)a (w)(er)
= [bW](es))
E Rdx
=[Wo)((r))
e xx
4

---

Equivariant Polynomial Functional Networks
$_ and the connection matrix V (inside (bW](0)(s.)) and (WW]).L)10.)) are learnable parameters.
polynomial layers serve as the fundamental building blocks for our MAGEP-NFNS.
It is important to note that I(U) is polynomial of degree at most L + in the input weights, encompassing all linear layers as special cases.
We define polynomial map I: 41 - Rd with maps each element U E 24 to the vector I(U) E Rd of the form given in Equation (10). To make / to be G-invariant, the learnable parameters o_ and V must satisfy a system of constraints (usually called parameter sharing). which are induced from the condition (g/)) = I (U) for all g E G and U E U. We show in details what are these constraints and how to derive the concrete formula of / in Appendix C. The formula of I is then determined by
To identify invariant layers among polynomials of the form given in I, an important step is to determine all relations between elements U, U U such that I(U) = I(U")- When the parameters V are fixed, I becomes linear function in the parameters $_, and the difference satisfies I(U) ((V)) = I(U - U"). Thus, the problem of iden- tifying invariant layers within polynomials of the form reduces to determining the linear dependencies among sta- ble polynomials. The following theorem characterizes these dependencies in (U), with formal statement provided in Theorem B.6 in the appendix.
RL
no
I(U)=
(1.0)((T.))3p [WW](L.0)(.0) pq
pm g
a2
ng
((L,0);F (L.0)
Theorem 3.4 (Linear dependence of stable polynomials).
For given pair of coefficients matrix & and V if (U) given in Equation (10) is equal to zero for all input weights U E Ut then we have
p=
Σ
$ (c.0)(L,)):.
(W
レ>8>0p=1
R2 nop
4 (1.0)pg
(L.)) pq
(1))(..))
16W1
Σ
2 (1.0)((.,)) [WW]]a = 0,
Σ Σ
2 PLLO00P [Wb
12000p=1
(11)
Σ
$ (OCE.)))
and
1>1>0p=1
nz
Σ (L.000P (WD))L.))))
+ (L))
16% + 11.
(13)
p=1
In the above formula, the bullet symbol denotes that the corresponding coefficient is independent of the index at that position. The summation terms involving expressions of the form [W] (respectively, [b]. [W6], [6W], [.WW) corre- spond to those present in the summation in Equation (10). The omitted terms are those eliminated during solving the grrameter-sharing process to ensure that the resulting for- mula becomes invariant.
4 (0(1,003 (6T1(0) pg
= 0, L >>t>0 0,
(12)
and all entries of 4 and VT, except those appear in the above two equations, equal to zero.
Intuitively speaking, almost stable polynomial terms are linearly independent over the reals I. except those in Equa- tions (11) and (12). This linear dependence property of the stable polynomials is essential in the computation of equiv- ariant and invariant polynomial layers using weight-sharing mechanism. The proofs of Proposition 3.2, Theorem 3.3, and Thcorem 3.4 can be found in Appendix B.
To conclude, we obtain the following:
Theorem 3.5. With the notation given above, the poly- nomial map T: U: .Rdd defined by Equation (13) is G- invariant. Moreover, if a map takes the form of Equa- tion (10) and is G-invariant, then it has the form given in Equation (13).
Remark (Comparison to the invariant/equivariani linear lay- ers in (Tran et al., 2024a)). Equation (13) describes the in- variant polynomial layer derived from the parameter-sharing mechanism of our MAGEP-NENs. In contrast, the invariant equivariant layer proposed in (Tran al., 2024a) is an ad hoc
3.2. Polynomial Invariant and Equivariant Layers
We now proceed to construct G-invariant polynomial lay- ers. The construction of G-equivariant polynomial layers is similar and will be derived in detail in Appendix C. These
5

---

Equivariant Polynomial Functional Networks
Table 1: Classification train and test accuracies (%) for implicit neural representations of MNIST, FashionMNIST, and
CIFAR-10 10. Uncertainties indicate standard error over 5 runs, baseline results are from (Tran et al., 2024a).
MNIST
CIFAR- 10
FashionMNIST
MLP
10.62 = 0.54
10.48 + 0.74
9.93 + 0.36
NP (Zhou et al., 2024b)
69.82 = 0.42
33.74 0.26
58,21 + 0.31
HNP (Zhou et al., 2024h)
66.02 + 0.51
31.61 + 0.22
57.43 + 0.46
Monomial-NFN (Tran al., 2024a)
68.43 + 0.51
34.23 + 0.33
61.15 + 0.55
MAGEP-NFWS (ours)
77.55 + 0.68
37.18 + 0.30
62.83 0.57
formulation and does not result from parameter-sharing mechanism. Consequently, there is no direct relationship between our invariant layer and the invariant layer in (Tran al., 2024a).
MNIST (Xiao et al., 2017), and MNIST (LeCun & Cortes, 2005). Each INR weight is trained to encode a single im- age from its respective class, capturing the image structure by mapping pixel coordinates (æz y) the corresponding pixel color values represented as 3-channel RGB values for CIFAR-10 10 and 1-channel grayscale values for MNIST and FashionMNIST. The varying complexity and diversity of the datasets provide a robust test for evaluating MAGEP- NFN's performance, demonstrating its effectiveness and benchmarking it against existing models.
However, the equivariant polynomial layer in MAGEP- NENs and the equivariant linear layer from (Tran et al., 2024a) are related. Specifically, the equivariant layer in (Tran et al., 2024a) is exactly the linear component of our equivariant polynomial layer. Due to the lengthy formu- lation and construction process, we have provided the de- tails of the derived equivariant polynomial layers in Ap- pendix D.4.
Results. We present the performance of our model along- side several baseline models, including MLP, NP (Zhou et al., 2024b), HNP (Zhou et al., 2024b), and Monomial- NFN (Tran et al., 2024a). As shown in Table 1, our model achieves the highest test accuracies across all INR datasets. Notably, outperforms the second besond-best model on the MNIST dataset, by significant margin of 7. 7.73%. For the CIFAR-10 and FashionMNIST datasets, our model also demonstrates substantial improvements, with accuracy gains of 2.95% and 1.68%, respectively, over the existing base- lines. These results indicate that our model leverages the embedded information from the pretrained INRs more effec- tively than any of the compared basclines. This consistent superior performance across various INR datasets highlights the effectiveness of MAGEP-NFN, It also suggests that our model generalizes well to INR weights embedded with dif- ferent image structures and complexities.
4. Experimental Results
In this session, we assess the performance of our Monomial Matrix Group Polynomial Equivariant Neural Functionals (MAGEP-NFNs) across a variety of equivariant and invari- ant tasks. For invariant tasks, we implement our model for classifying Implicit Neural Representations of images and predicting CNN generalization based on weights. The equivariant task focuses on weight space style editing. Our experiments are designed to illustrate that MAGEP-NENs ei- ther outperform or match the performance of other bascline models with a similar number of parameters. We perform five independent runs for each experiment and report the average results. Detailed information on hyperparameter settings, training protocols, memory and runtime analysis can be found in Appendix E. Additionally, we present supplementary experiment comparing our approach with GNN-based functional network, which is included in Ap- pendix F.
4.2. Predicting CNN generalization from weights
Experiment setup. For this experiment, we focus on pre- dicting the generalization performance of pretrained CNNs based solely on their weights, without evaluating them on test data. We utilize the Small CNN Zoo dataset (Unterthiner et al., 2020), which contains various pretrained CNN mod- els trained with different combinations of hyperparameters and activation functions. For our study, we split the Small CNN Zoo into two subsets: one comprising networks us- ing ReLU activations and the other using Tanh activations. These two types of CNNs follow different group actions: M²0 for Relu networks (see Equation (4)) and M for Tanh networks.
4.1. Classifying Implicit Neural Representations of images
Experiment setup. In this experiment, we aim to deter- mine which class each pretrained Implicit Neural Represen- tation (INR) weight was trained on. Following (Tran al., 2024a), employ three distinct INR weight datasets (Zhou et al., 2024b), each was trained on a different image dataset: CIFAR- 10 (Krizhevsky & Hinton, 2009), Fashion-
6

---

Equivariant Polynomial Functional Networks
Table 2: Performance prediction of CNNs on the ReLU subset of Small CNN Zoo with varying scale augmentations. We use Kendall's 7 as the evaluation metric. The uncertainty bars indicate the standard deviation across runs.
Augment settings U[1, 10%]
No augment
411,
109]
[1(1,00*]
STATNet (Unterthiner et al., 2020)
0.915 + 0.002
0.894 + 0.0001
0.853 + 0.007
0.523 + 0.02
0.516 + 0.001
NP (Zhou et al., 2024b)
0.920 0.003
0.900 + 0.002
0.898 + 0.003
0.884 + 0.002
0.884 0.002
HNP (Zhou et al., 2024b)
0.926 + 0.003
0.913 + 0.001
0.903 = 0.003
0.891 + 0.003
0.601 = 0.02
Monomial-NEN (Tran et al., 2024a)
0.922 + 0.001
0.920 + 0.001
0.919 + 0.001
0.920 + 0.002
0.920 + 0.001
MAGEP-NFNG (ours)
0.933 - 0.001 0.933 + 0.001
0.933 + 0.001
0.932 + 0.001
0.932 0.001
Table 3: Performance prediction of CNNs on the Tanh sub- set of Small CNN Zoo. We use Kendall's as the evaluation metric. The uncertainty bars indicate the standard deviation across 5 runs.
4.3. Weight space style editing
Experiment setup. In this experiment, we focus on modi- fying the weights of SIREN (Sitzmann et al., 2020) to mod- ify the image encoded within each model. We utilize the pretrained models from paper (Zhou et al., 2024b), which encode images from the CIFAR-10 and MNIST datasets. Specifically, we address two tasks aimed at modifying the embedded information: enhancing the contrast of CIFAR-10 images and dilating MNIST images encoded in the SIREN models. We report the MSE loss between the images en- coded in the modified SIREN network and the ground truth contrast-enhanced CIFAR-10 10 images or dilated MNIST im- ages.
Model
Kendall's
STATNet (Unterthiner et al., 2020)
0.913 = 0.0012
NP (Zhou al., 2024b)
0.925 = 0.0013
HNP (Zhou et al., 2024b)
0.933 = 0.0019
Monomial-NFN (Tran et al., 2024a)
0.939 = 0.0004
MAGEP-NFNs (ours)
0.940 1 0.001
Results. Table 4 demonstrates that our model achieves performance comparable to other baselines. Specifi- cally, MAGEP-NEN: matches the performance of NP and Monomial-NFN in the contrast-enhancing task on the CIFAR-10 dataset. Additionally, our model outperforms Monomial-NEN in the dilation task on the MNIST dataset, while achieving similar results to NP. Interestingly, NP re- mains strong candidate in the weight editing tasks, and our model consistently performs on par with NP across both experiments.
To evaluate the robustness of our model input transforma- tions under group actions, we augment the ReLU dataset by applying randomly sampled group actions M Specifi- cally, we randomly sampling the diagonal elements D of the matrix D with each element drawn from uniform distributions over different ranges, defined as [[11, 10') for i i= 1, 4. To further diversify the transformations, we also randomly sample the permutation matrix P...
4.4. Ablation study on the role of higher-order terms
Results. Table 2 illustrates the performance of all mod- els trained on the ReLU subset, where our MAGEP-NFNs model clearly outperforms all other baselines. Notably, it demonstrates robustness to scale and permutation symmetry, while consistently surpassing its performance across both the original and all augmented dataset settings. This suggests that incorporating polyno- mial layers allows our model to capture more information from the weights across different hidden layers, compared to Monomial-NEN, thereby enhancing expressivity. On the original dataset, our model achieves Kendall's perfor- mance gap of 0.007 over other baselines, and maintaining at least a 0.012 ad advantage in all other augmented settings, Sim- ilarly, Table 3 reveals that MAGEP-NENs achieves the high- est Kendall's 's with Tanh activation, further reinforcing its superior accuracy across different network configurations.
To evaluate the impact of the newly introduced Inter-Layer terms ([W], [WW], [W]], [W6]). we conduct an ablation study focusing on the invariant task of predicting CNN gen- eralization for the ReLU subset, following the same setting outlined in Subsection 4.2. The results presented in Ta- ble 5 clearly show that the inclusion of Inter-Layer terms enhances the network's 's performance. Notably, the perfor- mance improves from a Kendall's r of 0.929 with only Non Inter-Layer terms to 0.933 when both terms combined, highlighting significant boost in overall performance at- tributed to the incorporation of Inter-Layer terms.
similar to Monomial-NFF
7

---

Equivariant Polynomial Functional Networks
Table 4: Test mean squared error (lower is better) between weight-space editing methods and ground-trutt image-space transformations. Uncertainties indicate standard error over runs
(Peebles al., 2022; Ashkenazi al., 2023; Knyazev et al., 2021; Erkoç al., 2023).
Equivariant Neural Functional Networks. To achieve NFNs that are equivariant with respect to permutation symmetries of neural networks weight space, several ap- proaches have been used, such as: weight-sharing mecha- nisms (Navon et al., 2023; Zhou al., 2024b; Kofinas et al.,
Contrast (CIFAR-IO)
Dilate (MNIST)
MLP
0.031 + 0.001
0.306 + 0.001
NP (Zhou et al., 2024b)
0.020 + 0.002
0.068 + 0.002
2024; Zhou et al., 2024c), set-based (Andreis et al., 2023) or graph-based structures (Lim al., 2024; Kofinas et al.,
HNP (Zhou al., 2024b)
0.021 + 0.002
0.071 + 0.001
Monomial-NFi (Tran al., 2024a)
0.020 0.001
0.009 0.002
NFG (ours)
0.020 0.001
0.068 + 0.002
2024; Zhou et al., 2024a).
Table 5: Ablation study assessing the importance of higher- order terms on the task of predicting CNN generalization on the ReLU subset.
Despite these developments, current approaches often over- look additional symmetries present in neural networks, for instance, weight scaling symmetries ReLU networks and weight sign-fipping symmetries in sin and tanh networks. Recently, NFNs that are equivariant both permutations and scaling have been introduced in (Kalogeropoulos et al., 2024; Tran et al., 2024a). These networks leverage ad- vanced techniques such as graph-based message-passing mechanisms (Kalogeropoulos et al., 2024) and parameter- sharing frameworks (Tran et al., 2024a) to extend the scope of equivariant modeling and enhance the expressivity of NFNs. However, the graph-based equivariant NFNs pro- posed in (Kalogeropoulos et al., 2024) suffer from high memory consumption and significant runtime overhead. While, the Monomial- constructed using equivariant linear layers and a parameter-sharing mechanism in (Tran ct al., 2024a) exhibit limited expressive power.
Components
Kendall's 7
Only Non Lnter-Layer terms
0.929
Only Inter-Layerr terms
0.932
Non Lnter-Layer terms + [W]
0,930
Non Inter-Layer terms+ + [WW]
0.930
Non Inter-Layer terms + []
0.931
Non Inter-Layer terms + [6W]
0.931
Non Inter-Layer terms Inter-Layer terms
0.933
5. Related Work
Functional Equivalence of Neural Networks. Hecht- Nielsen was provided a foundational perspective on the rela- tionship between weight symmetries and network function- ality in (Hecht-Nielsen, 1990). This work has been extended to different network architectures, such as ReLU networks (Bui Thi Mai & Lampert, 2020; Neyshabur et al., 2015; Badrinarayanan et al., 2015; Albertini & Sontag, 1993), and sin or (anh networks (Chen al., 1993; Fefferman & Markel, 1993; Kurkova & Kainen, 1994). These studies build on earlier insights into convergence, gradient dynam- ics, and structural properties of neural networks, as explored in (Allen-Zhu et al., 2019; Du et al., 2019; Frankle & Carbin, 2019; Belkin et al., 2019; Novak et al., 2018).
In contrast, our MAGEP-NFNs are built upon equivariant polynomial layers, leveraging parameter-sharing mecha- nism that achieves both lower memory consumption and reduced runtime while preserving strong expressivity.
6. Conclusion
We have developed MAGEP-NEN, a novel NFN that is equivariant to both permutations and scaling symmetries. Our approach follows a parameter- sharring mechanism; how- ever, unlike previous works, we construct an equivariant polynomial layer that incorporates stable polynomial terms. This polynomial formulation enables to capture relation- ships between weights from different input hidden layers, thereby enhancing the expressivity of MAGEP-NFN while maintaining low memory consumption and efficient run- ning time. Experimental results demonstrate that our model achieves competitive performance and efficiency compared to existing baselines.
Neural Functional Networks. Early NFNs have been proposed to evaluate their generalization capabilities and uncover insights into neural network dynamics (Baker et al., 2018; Eilertsen et al., 2020; Unterthiner et al., 2020; Schürholt et al., 2021; 2022a;b). These approaches typically involve either flattening the network parameters or deriving parameter statisties for further processing using standard multi-layer perceptrons (MLPs) (Unterthiner et al., 2020: Dupont et al., 2022; Luigi et al., 2023). To involve the symmetric structure of the input neural networks, Schürholt et al. (2021) introduced neuron permutation augmentations to better align model representations with their functional equivalence. Other studies have expanded on these ideas by focusing on encoding and decoding neural network parame- ters, primarily for reconstruction and generative modeling
One limitation of the equivariant polynomial layers pro- posed in this paper is that they are applied to a specific archi- tecture. However, since our method is based on a parameter- sharing mechanism, it is applicable to other architectures with additional operators (such as layer normalization, soft- max, pooling) and other activation functions, provided that the symmetric group of the weight network is known.
8

---

Equivariant Polynomial Functional Networks
Impact Statement
Bui Thi Mai, P. and Lampert, C. Functional vs. paramet- ric equivalence of relu networks. In 8th International Conference on Learning Representations, 2020.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cao, N. D., Aziz, W., and Titov, I. Editing factual knowl- edge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp. 6491-6506. Association for Computational Linguistics, 2021.
References
Albertini, F, and Sontag, E. D. Identifiability of discrete- time neural networks. In Proc. European Control Confer- ence, pp. 460-465. Springer Berlin, 1993.
Chen, A. M., Lu, H.-m., and Hecht-Nielsen, R. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910-927, 1993.
Allen-Zhu, Z., Li, Y., and Song, Z. A convergence theory for deep learning via prreparamtterizaioon. In International conference on machine learning, pp. 242-252. PMLR, 2019.
Du, S., Lee, J., Li, H., Wang, L.,, and Zhai, X. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675- 1685. PMLR, 2019.
Andreis, B., Bedionita, S., and Hwang, S. J. Set-based neu- ral network encoding. arXiv preprint arXiv:2305. 16625, 2023.
Dupont, E., Kim, H., Eslami, S. M. A., Rezende, D. J., and Rosenbaum, D. From data to functa: Your data point function and you treat it like one. In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5694 5725. PMLR, 2022. URL https: pttettss:/prrceeee httpp//pooeeee.....///// mir. w.r.press//ee 62 dupont22a. html.
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., and De Freitas, N. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Eilertsen, G., Jönsson, D., Ropinski, T., Unger, J., and Ynnerman, A. Classifying the classifier: Dissecting the weight space of neural networks. In ECAI 2020 24th European Conference on Artificial Intelligence, volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 1119-1126. IOS Press, 2020. URL https: /tt.///dii..// org/ 10. 3233
Ashkenazi, M., Rimon, Z., Vainshtein, R., Levi, S., Richard- son, E.,, Mintz, P., and Treister, E. Nemn: Learning neural representations for neural networks. In The Eleventh Inter- national Conference on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
Badrinarayanan, V., Mishra, B., and Cipolla, R. Under- standing symmetries in deep networks. arXiv preprint arXiv:1511.0102. 2015.
Erkoç, Z., Ma, F., Shan, Q.. NieBner, M., and Dai, A. Hyper- diffusion: Generating implicit neural fields with weight- space diffusion. In Proceedings the the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 14300-14310, 2023.
Baker, B., Gupta, O., Raskar, R., and Naik, N. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenRe- view.net, 2018. URL https: /ttpss////eeeeeeeeeee / forum? 3errum?id=Huk3311v
Fefferman, C. and Markel, S. Recovering a feed-forward net from output. In Advances in Neural Information Processing Systems [7th NIPS Conference, Denver, Colorado, USA, 1993], pp. 335-334--303 342. Morgan Kaufmann, 1993.
Frankle, J. and Carbin, M. The lottery ticket hypothe- sis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: httt:s:///ppereviice /wwprevveeeeeeeeeeee/eeeeeeeeeee net forum? -0Fr31-b3RoF..
Belkin, M., Hsu, D., Ma. S., and Mandal. S. Reconeiling modern machine-learning practice and the classical bias- variance trade-off. Proceedings of the National Academy of Sciences, 116(32): (112)))55899915555 2019.
Bengio, S., Bengio, Y., Cloutier, J., and Gecsei, J. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pp. 265-287. Rout- ledge, 2013.
Harb, J., Schaul, T., Precup, D., and Bacon, P.-L. Policy evaluation networks. arXiv preprint arXiv:2002.1183: 2020.

---

Equivariant Polynomial Functional Networks
Hecht-Nielsen, R. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 129-135. Elsevier, 1990.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Man-
ning. C. D. Fast model editing at scale. In The Tenth International Conference Learning Representations, 2022. URL https: http:///ppeeeeeeeeeeeeeeeeeeeeee net forum? i 1d=0Dc7xeWFORE.
Kalogeropoulos, I., Bouritsas, G., and Panagakis, Scale equivariant graph metanetworks. In Advances in Neural Information Processing Systems 38. 2024.
Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., and Maron, H. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pp. 157990255811 PMLR, 2023.
Knyazev, B., Drozdzal, M., Taylor, G. W., and Romero So- riano, A. Parameter prediction for unseen deep archi- tectures. Advances in Neural Information Processing Systems, 34:29433-29448, 2021.
Neyshabur, B., Salakhutdinov, R. R., and Srebro, N. Path- sgd: Path-normalized optimization in deep neural net- works. Advances in neural information processing sys- tems, 28, 2015.
Kofinas, M., Knyazev, B., Zhang, Y., Chen, Y., Burgh- outs, G. J., Gavves, E., Snoek, C. G. M., and Zhang, D. W. Graph neural networks for learning equivari- ant representations of neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https: http//pppereeeeeeeeeeeeeeeeeeeee /aree/foornn Id=o06FaMyDBs sMyDBt.
Novak, R., Bahri, Y., Abolañia, D. A., Pennington, J., and Sohl-Dickstein, J. Sensitivity and generalization in neural networks: an empirical study, arXiv preprint M0Niv:880.00700000 2018
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report O, University of Toronto, Toronto, Ontario, 2009. URL https:/ /wwp::///wwww.e//. cs toronto.......... edu/ farti//larmingg-eettaee
Peebles, W., Radosavovic, I., Brooks, T., Efros, A. A., and Malik, J. Learning to learn with generative mod- els of neural network checkpoints. arXiv preprint arXiv:2209 12892, 2022.
Ruhe, D., Brandstetter, J., and Forré, P. Clifford group equivariant neural networks. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https: https:://penree//// /tppnreveeeeeeeeeeeeeeeeeeeeeeee net forum? id-n84b2MrGUD.
Kurkova, V. and Kainen, P. C. Functionally equivalent feedforward neural networks. Neural Computation, 6(3): 543-558, 1994.
LeCun, Y. and Cortes, C. The mnist database of handwritten digits. 2005. URL https://api. semanticscholes wrtp:o/po//oooopoppooooo 60282629.
Runarsson, T. P. and Jonsson, M. T. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pp. 59-63. IEEE, 2000.
Lim, D., Maron, H., Law, M. T., Lorraine, J., and Lucas, Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.co wtt.://w.eeeeeeeeeeeeeeeeeeeeeee fortt//frmmmmtmm -tt.ooomnmdd-iaassxssee----- jK5hyxsOn.
Schürholt, K., Kostadinov, D., and Borth, D. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural In- formation Processing Systems, 34:16481-16499 2021.
Luigi, L. D., Cardace, A., Spezialetti, R., Ramirez, Z Salti, S., and Stefano, L. D. Deep learning on implicit neural representations of shapes. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, 2023. URL https://te// https:////peeeee..eeeee forum? forrum?dd-oOIW 3uadi.
Schürholt, K., Knyazev, B., Giró-i Nieto, X., and Borth, D. sypeerrrerrssentationss as generative models: Sampling unseen neural network weights. Advances in Neural In- formation Processing Systems, 35:27906 27920, 2022a
Metz, L., Harrison, J.. Freeman, C. D., Merchant, A., Beyer,
L., Bradbury, J., Agrawal, N., Poole, B., Mordatch, I., Roberts, A., et al. Velo: Training versatile learned opti- mizers by scaling up. arXiv preprint 1-Xix:2211.096660 2022.
Schürholt, K., Taskiran, D., Knyazev, B., Giró-i Nieto, X., and Borth, D. Model zoos: A dataset of diverse pop- ulations of neural network models. Advances in Neu- ral Information Processing Systems, 35:38134-38148, 2022b.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communica- tions of the ACM, (5(1):99-1066 2021.
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D. V., Popov, S., and Babenko, A. Editable neural networks. 8th International Conference on Learning Representations,
10

---

Equivariant Polynomial Functional Networks
2020. URL
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist:
id=HJedXaRtvs.
novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http:
Sitzmann, V., Martel, N. P., Bergman, A. W. Lindell, D. B., and Wetzstein, G. Implicit neural representations with periodic activation functions. In Advances in Neu- ral Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
/ //taxxii..rg/aaaaaa..e. www.///s//.//// 0 07747.
Zhou, A., Finn, C., and Harrison, J. Universal neural func- tionals. Advances in neural information processing sys- tems, 2024a.
Zhou, A., Yang, K., Burns, K., Cardace, A., Jiang, Y., Sokota, S., Kolter, J. Z., and Finn, C. Permutation equiv- ariant neural functionals. Advances in Neural Information Processing Systems, 36, 2024b.
Sokota, S., Hu, H., Wu, D. J., Kolter, J. Z., Foerster, J. N., and Brown, N. A fine-tuning approach to belief state modeling. In International Conference on Learning Rep- resentations, 2021.
Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J. Z., and Finn, C. Neural functional transformers. Advances in Neural Information Processing Systems, 36, 2024c.
Stanley, K. O. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:111-162, 2007.
Tran, H., Vo, T., Huu, T., Nguyen The, A., and Nguyen, T. Monomial matrix group equivariant neural functional networks. In Advances in Neural Information Process- ing Systems, volume 37, pp. 48628 48665. Curran Asso- ciates, Inc., 2024a.
Tran, H. V., Nguyen, K. N., Pham, T., Chu, T., Le, T., and Nguyen, T. M. Distance-based tree-sliced wasserstein distance. The Thirteenth International Conference on Learning Representations, 2025a.
Tran, T., Tran, V.H.., Chu, T., Pham, T., Ghaoui, L.. F., Le, T., and Nguyen, T. M. Tree-sliced wasserstein distance with nonlinear projection. arXiv preprint a0X19:2505.00968, 2025b.
Tran, V.-H., Pham, T., Tran, T., Le, T., and Nguyen, M. Tree-sliced wasserstein distance on a system of lines. arXiv preprint arXiv:2406. 1 13725, 2024b.
Tran, V.-.., Vo, T. N., Huu, T. T., and Nguyen, T. M. A clifford algebraic approach (n)-equivariant high-order graph neural networks. arXiv preprint arXiv:2410.0469) 2024c.
Tran, V.-H., Vo, T. N., The, A. N., Huu, T. T., Nguyen- Nhat, M.-K., Tran, T., Pham, D.-T,, and Nguyen, T. M. Equivariant neural functional networks for transformers. arXiv preprint arXiv:2410.04009 2024d.
Tran, V.H.., Chu, T. T., Nguyen, K. N., Pham, T., Le, T., and Nguyen, T. M. Spherical tree-sliced wasserstein distance. arXiv preprint wxXiv:5533.11444 2025c.
Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., and Tolstikhin, Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.1 /448, 2020.

---

Equivariant Polynomial Functional Networks
Supplement to "Equivariant Polynomial Functional Networks"
A. Preliminaries
This section contains notations and basic results on matrices and polynomials that will be used throughout the paper. We will mainly focus on matrices with real entries (real matrices) and polynomials with real coefficients (real polynomials). We will omit almost of the proofs in this section as they are well-known. These results will be use in proofs in the rest the paper.
A.1. Entries of matrices
real matrix with m rows and n columns an element of R m=xx;
011
d1
01n
021
022
02n
A = 166m,1EE
R Ruxn
0ml
am2
Omn
The entry in the -row and the -colummnnn of A, or the (i,j) entry of A, is denoted by Asj = 0zj- The (thw of A and -hhlcolumn of A are, respectively, denoted by:
A; = (4.1 02
ain)
E ER lxn
A.j (a15 daj
amj)
R
Remark. Sometimes, comma added between two subscript indices to make sure there will be no confusion, i.e. Aig; di,j Ai,o,A... Let AL) 12), A be L matrices such that the matrix product:
A A A
well-defined. Proposition A.1. The (i.j) entry of A(t) ................................ A A is equal to:
(A(L) (2)) Aw))
= A (L)
(L-1)
(2) A!
1,
Σ
(L) (L)
(2) (I) -bis"
(1) In the case where L the above equation is simply A dij
We denotation for matrices that have only one nonzero entry with value 1. The matrix with the I in the roo and the jth "oluuum and the rest are 0, is denoted by Eij. Matrix Eg can have any shape, but its shape usually defined by context, and will be omitted without confusion. The product of matrices of this type is presented as below. Proposition A.2. Let Enyniy Eig.ja..... Eigsi be L matrix units such that the product:
Ens. -iz.z
Eir
well-defined. Then:
Ea Eig.ta
Eiiii (8)1.12 8/2.8 .... 85c r.be) Eis
where 6ij is the Kronecker delta:
( i
dij
ifi=
We have direct corollary for E1,'s iss.
Corollary A.3. We have:
F1,1 Fi, ..... F1,1 Fr,1-
12

---

Equivariant Polynomial Functional Networks
A.2. Evaluation of polynomials
[x]] be the ring of all polynomials with real coefficients in indeterminates x Xx. Definition A.4. A monomial of IR [8[x:..... Xn] a polynomial of , Xn]
Denote I[[
that has one term.
Remark. In some contexts, monomial is defined a polynomial that has one term with coefficient We will both of these definitions simultaneously.
Proposition A.5. R(x1,...,Xx, is naturally vector space over R. It is cnffiite-dimeessoonn vector space; moreover, the set of all monomials with coefficient 1 in [[xx..... :x, ;.x,,...,x,) is a basis for the vector space.
Remark. For / E R[(x,,...,, x by saying monomials in f. we refer to all monomials that appeared in the expression of f.
Polynomial evaluation is computing of the value of a polynomial when the indeterminates are substituted for some values. We have the well-known result.
Proposition A.6. Let f,g be two polynomials of TR. [x1, -x.1................ ... x,) If f.g are equal at every evaluations, i.e.
f(11...., Jn) = -(( (o Fn) E R..
(14)
then f = g. In other words, the only polynomial of R[x,,..., x,). that has R as its zero set, is the polynomial E [R[x!...., Xn].
Remark. The result still holds if R is replaced by an arbitrary infinite field, but does not hold if is replaced by a finite field.
We have a direct corollary.
Corollary A.7. Let f be a nonzero polynomial of IE x,). Then there exists (I1,..., Fn) E R such that f nn) + O.
A.3. Entries of tensors
Proposition A.S. Let a = (a:) and b = (b,)igien be two vectors in R If:
a, a. by + ag aa..-bb- = 0,
(15)
for all Eijj then or =0. 0.
Proof. Assume that both of and b are not equal to then there exists ú, such that a, and by are non-zero. From Equation (15), we have:
6. + a. by = O,
(16)
80 0; by = 0. Since a is non-zero, then = 0. implies that:
0g by aj b, a; by 0 a, by + 0,
(17)
which contradicts to Equation (15). So at least one of and b is equal to O.
Proposition A.9. Let A = (igg). and B (b:z) iigii, be two matrices in Rmxn, If:
aij byt + akj bu + aa buj + agi big = O,
(18)
for all 1 < m and & n, then A B Proof. Consider Equation (18) when
< l & n, we have:
0 Oij buj axj big + aij bus + anj bij
(19)
= 2 (a:j buj + akj bıj).
(20)
which means:
Oig bus + azj bij
(21)
13

---

Equivariant Polynomial Functional Networks
This holds for all < i, & m. Apply Proposition A.8, we have 0zj for all <i< & m. bi for all i & m,
which means A.j = 0 or B. = 0. This holds for all 1 & j & Similarly, we have Ai,- 0 or Bi,- 0 for 1 < i & m. Now, assume that, both of A and B are not equal to O, then there exists i,j and k, I such that dij and bu are non-zero. By
previous observation, we have B = B., = Ak A., = 0. It implies that:
Daj bus + Unj ba + Ogs bug + uns bij = aej byt + +0+0+0 - aij bui + O,
(22)
which contradicts to Equation (18). So at least one of A and B is equal to O.
Proposition A.8 and Proposition A.9 are, respectively, one-dimensional and two-dimensional cases. By using the same arguments, we will obtain the d-dimensional version belows.
Proposition A.10. Let d be a positive integer and n1, ....... Rd be d positive integers. Let:
A = (Ai Emn Sit
E R X
(Bopd 1 I Is IEuan
R Xngx...xma
for all 1 < 29 < n1, 1 < 2g E n2,.....
E 12 E nd. we haver
Σ
(A 02 (Ap...g.....3))))
(B 1-02 1-02] 10 O,
(23)
( (e)
then A = or B = 0.
B. Stable Polynomial Terms
Intuitively, stable polynomial ferm is polynomial in the entries of U E l/ that is "stable" under the action of G' (see Definition B.1 below). The equivariant polynomial layers aim to construct are linear combinations of these stable polynomial terms. In Subsection B.1, we provide formal definition for stable polynomial terms as well as their properties. We will study the linear dependence of stable polynomial terms in the language of polynomial rings with real coefficients in Subsections B.2 and B.3. These properties play a central role in determining the parameter-sharing computation of equivariant polynomial layers in the next section.
B.1. Definitions and basic properties
Recall the weight space I given by:
4 = W x B.
where:
W X
p Xnaxnu
R mn gwixn Xm1 Xno
B ROLXREXI
R x
X
Let us consider the case where the weight spaces have the same number of dimensions across all channels, which means w; = b; = d for all i.
Definition B.1 (Stable polynomial terms). LetU = ([W], [6]) be an element of with weights [W] (W](2))
and biases [b] (1)) ..... (6))),,)..)))))))))))).)))) (6) For each L > > 0, define:
[W](+) [W](--)... ..... [W] (e00) [W](+.1) (0)(0)
[W](s.)
Rdxx xn,
(24)
RRdxn
In addition, for each 1.20 8 > O. and matrices (o)(L.)) E RRIxZz and ((..0)(L.1) e Rooxns we also define
(w) = =b](0) (O)L).)) [W](L.))
Rdxa,xn
(25)
(10)0) = [W] ((s.))(L.)) (1](2.)) pdxa,xm.
14

---

Equivariant Polynomial Functional Networks
The entries of the matrices [W](s:0, [Wb](s.00) [6W](()(1))) and [WW](x0Xa defined above are called stable polynomial terms of U under the action of G.
The following observations are direct implications from the definition.
For all L>s>t>r2 0:
[w](m =|w](() E x
(26)
and
[W](x.). [W] = [W](arr E xaxn,xar
(27)
by definition. For g = (g (()) E Su:
[9W](sst) = (s())) [W](600) (0(1)² -1 x
(28)
If E G, then:
(oW)(2.) = [1](20)) (000²= R.ddn X
(29)
[oW)(s00) = (ol+t) [W](s.0) E Rdxa, x
(30)
For all L 8 t> 0, we have
[gW](s.1) [gb](0) = (96)) [W](s pp(0 Rdxn,x
(31)
For all > 8> 0, and ((s.0)(L.1) E xdx we have:
loW](s.0) v(s.0)(1.)) [gW](1.))
= (0(t)) [W](6.0) ((.0))))))) [W](1 (0t))² E Rdxxn xns
(32)
In particular, if 1, we have: (nW)( (m0). v(la.0)1. (gW](
= (1())) [W](s.0) ((s.0)(1,--)), [w](1 (9ht-1)) E Rdxx,
(33)
For all > > 0 and (L)(L)))) e Rdxixma we have: ((()))((())) (o
= =00()) (b](0) (Ol., [W](L.0) (D(1)=
E Iidxx,xn
(34)
Based on the above observations, we can determine the image of the stable polynomial terms under the action of an element E Gu as follows:
(50)
(10)
(0())=
())
(9)
())
[gW 96) (5.))0)
(g())
[Wo](s.0)(0)
(1)
(g())
(BW]))())))
(g(t) -1
[g (W
(g(=)
(g(0)
15

---

Equivariant Polynomial Functional Networks
In concrete, have:
(a.1)
(1,))
d
my (p). (4))
[96i[() (x)
d0
(x)
ma
(p)
loW ob) (1.0(0)
d.
[Wb] (4000))
(n)
d8")
[6
[6
pp
d0
n5 (p)
db4)
LeW (W)(0,00)100
[WW
(s,0)(1.())
dg
3, (p). (g)
B.2. Input weights as indeterminates
To simplify the technical difficulties, we consider the weight space U the case where d = 1, i.e.
u W x B,
where:
W = RELXRE-I
X
X
R
X
RaiXno
= RRAX1
X
X
Rexx1
X
Raix1
We introduce the set / consists of indeterminates defined by:
1=
:
We have [T] l. Denote R = R[T], which is the ring of all polynomials with indeterminates are all elements of I. For
1 < i < L. we define:
(w] =
E R
jk
[b]()
E R
IEfEn.
and
(w)
W) [W] (8-) [W]++) Rm
(W)00.00 [))+..) (b)(1)
R
b (b [b]()) (())) (W)
Rm
(ww](*0XX... = [W] (I.00)(2.)) [W](
RRi
with feasible indices (s, t). The coefficients /i (s's are fixed real matrices and they are omitted from the notations.
Note that the entries of these matrices are stable polynomial terms in which the entries of are now viewed as indeterminates of the polynomial ring R.
B.3. Linear dependence of stable polynomial terms
In this subsection, we derive necessary condition for the coefficients 4__ and V such that the following linear combination of stable polynomial terms are identically zero:
((
Σ
(s.t)om
W (s.t)
T (s)op [b] (*)
L 120 pm1
g=1
L
(b.0s.0))
130512011
16

---

Equivariant Polynomial Functional Networks
T2 14g
Σ
(s) [b
[bW] (L),)) pg
Le4>0 12t20p=1q=
Σ ΣΣΣΣ 136>0
2 (s.0)((..))rr
[WW] pq
(+.0(1)) + +1 1.
(35)
Here, a is parameterized by and V, where + is a collection of real scalars c_'s appeared in the linear combination and is collection real matrices 's that used to define [bw](-)'s and [w [w The index of each scalar & naturally presents its corresponding polynomial in a(Φ, V). This necessary and sufficient condition enables us to determine the equivariant polynomial map via parameter sharing later.
We first take look at entries of [W](-)s, s, [b](-)s, [Wb](-)s, [bw](>)s, [ww](-)s. It clear that for one of these matrices, its entries are homogeneous polynomials with the same degree. For example:
[w]-); The polynomial [W] [W] has degree 8 I. All of its monomial terms consist of one x for each & 3> i>> L.
(*.t)
-)): The polynomial b (+) has degree 1. All of its monomial terms consist of one y
[Wb] The polynomial Wb) (*.1(1)) has degree + 1. All of its monomial terms consist of one x for each
>>> f. and one y
(- The polynomial [bW] (L,))))) has degree L - t 1. All of its monomial terms consist of one y y and one (0) for each L > i > t.
[ww[(-); The polynomial [WW]] (T,0)( has degree L + t. All of its monomial terms consist of one x for each > i > 0 and one x for each L > bis i>t. t.
1: The polynomial R.
By these above observations, have:
[W]-), (W](-); Each of the polynomials [W] and [WW][-"ss is or non-constant element in R, and real polynomial of at least one indeterminate from x! ())s.
Each of the polynomials (b)(-)+s is non-constant element R, and is real polynomial of one indeterminate from "ss..
-b);
()) (-] Each of the polynomials [Wb] and [bw] is non-constant element in R, and real polynomial of at least one indeterminate from x (-)ss and indeterminate from
Therefore, if ( V = 0, we must have
The Tie
The Σ
0
(s.t)=pg
(W,0) (e.1)
pq
+ Σ
Σ
① (s.0)(
(W.(+.0))..) pq
(36)
L3s>120 pm
L2s>0 L>130 pm
Σ
(s)) [b](,)
(37)
12a20p=1
Tl
(..0))t)
[bb (+,)()
Σ
Σ
b (L)(L.))
(38)
pq
126>1>0 p=1
L3820 L>t200 p=1
0 1
(39)
17

---

Equivariant Polynomial Functional Networks
We induce the constraints on & and V in these above equations by using the fact that set of distinct monomials of R is linear independent set (see Proposition A.5).
Equation (36): Observe that.
L > s > t e O and (s,t) * 0), then the monomials x(*) .... x x only appear in the polynomials
[w] They do not appear in the polynomials [w](++>; for pairs (s', + (s, t), and do not appear in the polynomials
for all pairs (s',
If > s 0, L 0, and * t. then the monomials (x!*) (x!").....xx x() (x(". " x)+1)) only appear ................................
in the polynomials ww] They do not appear in the polynomials [W] (o(,))) for all pairs (s', t'), and
do not appear in the polynomials [WW] (ww (=))))())))))) for all pairs (s', (') + (s, t).
So from Equation (36), it implies that
4 (s.();p
(s.))
(40)
9=1
for all ts * (I, 0), and
Yha n2 0
4 (L.))(L)))
[W (L,))L,))
(41)
Pq
for all ts that # t. The rest of the terms in Equation (36) is
0
(L.O);p
(L.0) Pq
+
Σ
(L,))(
(W (s.0)(L.s) pq
[W
(42)
p
1>8>0p=1q=
Equation (37): Since the set of all [b] which is the set of all monomials y (''s,, s, is a linear independent set in R, it implies that
4 (s)opt
(43)
for all >> s >>> 0 and < & nh.
Equation (38): Observe that
If > s >>> > 0, then the monomials x
x 2+1) y
only appear in the polynomials (Wb](.0000ss
They do not appear in the polynomials [Wb] (5' ((())) 's for all L > > <'> O that (s',t) # (s, and do not
appear in polynomials [b for all L > s' > and L > t' > 0.
If s 0. t and + then the monomials ( (x!4) (L) (x.
.... (.....(()) ((+t))): 's only appear in the
polynomials [b (b They do not appear in the polynomials wb)] for all >> 1' > and do not appear in the polynomials (o' for all pairs (s', t') + (s.
If L > t > 0, then the monomials y (1. (......................... ..."))=)+==)==) 's only appear in the polynomials [Wb] [Wb] (1.4)))
and appear in the polynomials (w)](((.)) They do not appear in the polynomials [Wb][4.""0 for all
t' > O that t' * f, and do not appear in polynomials [b (=))(,))) for all pair (s', (') that + t. So from Equation (38). implies that
n2
(o.0))))) [Wb](s.0()
(44)
18

---

Equivariant Polynomial Functional Networks
for all L> 0, and
0
(L)
[bW])o)L.)
(45)
p=
for all ts that + t, and
0
(LL.))())
[bb] (L)0))
+
(c)(CL)))
[b
(46)
p=1==1
for all L > >>>0 0.
Equation (39): Clearly, implies that
11-
(47)
There 8 equations, Equations (40)-(47), that derived. In Equations (43) and (47), the corresponding t_'s are directly characterized, and in Equations (40), (41), (42), (44), (45), (46), the corresponding d_'s not. We will characterize the d_'s 's and V 's in Equations (40), (41), (44) and (45) below by Lemma B.2, Lemma B.3 and Lemma B.5, respectively. These lemma are stated as below (their proofs will be postponed to Section B.4).
Lemma B.2. For pair (s, I) such that L 2 2 0 and (s, 4) + (1,0), if
$ (s.t):pp
(s.0) Pq
(s.0)
(48)
p=1 1=1
then (s,t):pp = 0 for all p.g.
Lemma B.3. For pair ts. such that L 2 s> 0. t2 of
(c..0)(L.1)Ppp (WW](s
(49)
p=
then ((..))(..)) = for all p. or (l=.0)(L.() = O.
Lemma B.4. For a pair (s, f) such that L > s >>t>0,
a
(a00)()))))) (s.00.
(50)
then (ox,))()) = for all p-
Lemma B.5. For pair (s, t) such that L > 8> 0, L 1,
n2
(s) (L)L))
(51)
then (e))())))pp for all p, q or ()()(LL.))
Remark. The reason that we skip the characterizations of $_ and __'s in Equations (42) and (46) is they can be concretely characterized. For instance, consider the case when nt = =... n2 = mn = no From Equation (42), we have
(L.0) pq
ΣΣΣ
WW ( (s.0) pq
(LL0))pp
(s..0)(L.))0n
p= 1=
L p=1
= (L.0).1111
[W]LL
(L.0) (L,0)
+ Σ
(1.0))
[W
(w.) (L.))
111
1.1
1>850
19

---

Equivariant Polynomial Functional Networks
(1) [1,1
(L.0),111
x1,1 X1,1 x1,1)
+ Σ
(x), "....
1,1 x(i) vllo)))))) (x1 .............
xf.11 (
x1,1 x
(x:0)(L.):;11
L>850
(2) x1,1
(L,0)_111
Σ ((.0))(2.):111 (((00)),,))
[ .... x x (x1 x1, xi,1 (0+1))
1>820
(L)
= (I,,0))
X1,1
x1,1
(0.0)(L,)):11 ()..0))L,))
x x1]
+
1>8>0
- ((1.0)a,11
Σ
(x
x x
L>8>0
It implies that
0
(1,0))111
Σ (..0)))))) v(l,0)(L,))
(52)
L>820
From Equation (52), we can not derive a more concrete relation on the 4 and V 's Similarly, from Equation (46), we have
0 Σ (L.0))))))))))
Wb....000 + ΣΣ> (()
bw (L(L,) p7
[Wb]] [wb]L.000 (L,00 + (0),0))) [bw] (X)2,))
P(L.))))
= P(L.)(())
(x),
(L) ((+2)
X1.1
x(f1 (+1) (yi,1
(yl(+"))
(vit (y(tt)) (T0))2.))
x111 ((t1)) 1,1
(t+2)
+ +
x1,1
= (LL,000 (x1.)
((+2)
x((11 (111)
(yl.
(((+1)
1,1
+ +0 VIOLO.
(y(())))
(1+2)
x1,1
x1,1
= ((L))((): (x1,) x1.1
(e+2)
a+1)
(ylit)) ((+1)
111
+ + ((OL))
(L)
111
((+2)
x
((+1))
1,1
(y
(11.000 + (1 w(OLD
(x11
(12)
x1.1 ((1)))
(xit-")
x1,1
implies that
= 0= (LL))(0)) + T (OLL.0) ((O(L)))
(53)
From Equation (53), we can not derive a more concrete relation on the i_'s and 's's
Combining the discussions above, we obtain the following necessary for the coefficients & and v such that a(D, v) = 0. Theorem B.6. Let ax V) be polynomial given in equation 35. If ox O v) 0, then the following condition holds:
1. For all > 1 12 with (8. + (I,, 0), and for all p. g. we have
T (s.));pg
(54)
20

---

Equivariant Polynomial Functional Networks
2. For all s 0, > with + I, we have
(e.0)(L.)) = 0,
(55)
for all p, q, or
((0,0)((.)) = =0.
(56)
3. We have
7o
12 Σ
0
(LL0));p
[W] (L.0)
+
(L,))
[WW] [WW] (s.0)(L,))
(57)
P4
p=
4. For all
> 8 1>> O, and for all p. we have
$ (s.1)(t): = O.
(58)
5. For all >
L > with 8 + t, we have
(c)) = 0,
(59)
for all p, p.q. or
((()(2))) = O.
(60)
6. For all > t> O, we have
nL
R1 (1)L))) [bW]00
D
[Wb] (1))())
(61)
9=
7. For all
0 and for all p. have
4 (s) 0.
(62)
8. We have
1 = 0.
(63)
Proof. By the previous observations, a(4, v/ 0 implies four Equations (36)-(39). These equations imply Equations (40)- (47). The proof of all parts in Theorem B.6 are as follows.
1. It comes from Equation (40) and Lemma B.2.
2. It comes from Equation (41) and Lemma B.3.
3. It comes from Equation (42).
4. It comes from Equation (44) and Lemma B.4
5. It comes from Equation (45) and Lemma B.5.
6. It comes from Equation (46).
7. It comes from Equation (43).
8. It comes from Equation (47).
21

---

Equivariant Polynomial Functional Networks
The proof is finsihed.
The following corollary is direct consequence of Theorem B.6. Corollary B.7. for some b V', and v, have a(T, v ax(0, V) then:
1. For all L > t 0 with (s.t) + (L, 0), and for all have
1 (s,A):pp 4 (9.1))p0'
(64)
2. For all L > 8 0, L >>>>>>>> 12 with + I, we have
(c..)((L.)):p = (10.0))L,1);pp
(65)
for all p, q or
((x.0)(L,)) = O.
(66)
3. We have
(L.0):pg
[ (L.00 pq
(x,0),
[WW(ss.)) (L.)) py
(67)
p=1q=
1>800 p=1 q=1
(L.0)
[W]LL +
Σ
1 (e.0)_
(w.0s.)))))))))
(68)
1>0>0 q=1
4. For all L > s > t >> 0, and all we have
(a.1)()) 06 (ss1))())
(69)
5. For all L > > 0, L > t3 0 with s + t, we have
(s) ((3)(L.0))4
(70)
for all p, 9, or
(p()((,t)
(71)
6. For all L t> 0, we have
4 (L.)))() [Wb]
& (1)
[b (0).))
(72)
p=1
V (L.))))) [Wb][2.000) +
(1)LL.))pp (0(40)
(73)
7. For all L > > 0 and for all we have
4 (x)p 4 ((s):p''
(74)
8. We have
(75)
Proof. Since
0- a( v) a(Φ', v) a( ()
so the results come directly from Theorem B.6.
22

---

Equivariant Polynomial Functional Networks
B.4. Proofs of Lemmas B.2-B.5
The proofs of Lemmas B.2 through B.5 are directly from the coefficient comparison of two equal polynomials and the following lemma. We omit the proofs of Lemmas B.2 through B.5 and show only the proof for Lemma B.8. Lemma B.8. For a feasible tuple (s, (s,t,,,)). 9). if
(W] [W]] ((s.0)(..)) (W](..)
=0E R.
(76)
Pq
Then ((s.0)(L.t) 0 E Rox
Proof. We first consider the case where s and O, then the case 8 & t, and finally the case & > t. Note that, the proof for the last case will by combining the arguments of the first two cases.
Case L and
O. From Equation (76), we have
Ww]/20XL.0 = (WW)LL,) vpLL.))L.)) (W](2.0)
=0,
(77)
pq
which means
(w](L).... [W](), v(L.D)(L.)) (w](2.... (()())...((((())))
= 0.
(78)
pq
Here, we consider three cases, where L = 1, L 2 and L > 3.
Case L = 1. Equation (78) becomes
(w]W) g(.0).. (wj()
(79)
By Proposition A.1, this is equivalent
Whl (1.0)) [ = 0,
(80)
which is:
(1.0)L.))
X. 0.
(81)
i=
Since the LHS of Equation (81) is a linear combination between distinct monomials
x.p
(1)
(1) j.9
(82)
for < < no and < < n1, so it implies that
vp (1.0) 0,
(83)
for all &i< no and 1 <j & n1, which means
VD (1.00(1.0) 0.
(84)
Case L Equation (78) becomes
(W) (1))) (1) (2.0))2.0) (w (w]() pq
0.
(85)
By Proposition A.I, this is equivalent
[ (2) [W]1) (1) (2.00(2.0)
w (2) [W])) (1) b.g 0,
(86)
23

---

Equivariant Polynomial Functional Networks
which is
(Wise (2)
p.
[W]!
W
(2) p.
[W]
2.2
(2)
[W] (2)
p. [W].nn] (f(2.0)(2.0)
(87)
(Wi?? /W]gg
[W] (1)
.g.... .... [ (2) (1)
R2. W1i..)
= 0.
(88)
which
(2.01(2.0)
(W
(2) pp
/w!!) (IW [w]! = O.
(89)
Since the LHS of Equation (89) is linear combination between elements of a linear independent collection of R.
which are:
(Wit) (WI!) (IWIj? (w)!))
(90)
for 1 < no and 1 <j & n2, so it implies that
v (2.0) 0,
(91)
for all 1 < i no and 1 <j & 122, which means
(2.00)2200 = 0.
(92)
Case L > 3. Equation (78) becomes
[w]
WW))) vLL,0)L.0) (w) (w)())
0.
(93)
pg
It is noted that the matrices [W] 1) such that the product
(2] can be substituted for some matrix units at 1"-row and 1
[w]
[W
(94)
becomes a matrix unit row and 1s-column. Substitute this in Equation (93), we have
(w) Fu (W]"), (12.0)(2.) [w]() En . (W]()) pq 0.
(95)
By Proposition A.1, this is equivalent to
W p.
En (W)(). (p(,,0(L..)) (W(1). En [W] (1) 0.
(96)
which can be rewritten as
W (L)
[ (.LL.)((L.)) [w] [W] O,
(97)
p.
or
[ (1) (L..))( [w] (4) (1)
x1. 0.
(98)
Since the LHS of Equation (98) is polynomial in R. we have
w (1) (L.0)(LL.))
(w]4) (4) 0,
(99)
which is
no R2
T
(L.))L.O) ig
(1) (L)
(100)
j.1
24

---

Equivariant Polynomial Functional Networks
Since the LHS of Equation (100) is linear combination between distinct monomials
1,
j.1
(101)
for < & ng and j & n,, so it implies that
w (LL.,)(Z..)) 0,
(102)
for all 1 < no and 1 <J & nz. which means
(LL.0)(L,0) = O.
(103)
In conclusion, for all cases of L, we have
(1L,,0(1.0) O.
(104)
We finish the proof of the case where L and t = 0.
Case & t. From Equation (76), we have
(w) (W]((.)), ((s.0)2..)) (W](2.0) = 0.
(105)
pq
which means
(Iwjo)....
[W ((0.01(1.)) (W] (W
=0
(106)
pg
Since 8 & f, the [W] that are in the product [ (+)
(1), and the [W] 's that are in the product [w]()....
[W](++1, are distinct. By directly multiplying W) products the forms
and [W] ..... [W] we can write these two
[W] W
(W
= ( IgiEn..1EE
E R X
(107)
(WI++)
gi
R
Igignt,
where all (()))s are nonzero and all g" '0's are nonzero. Moreover, f(*)'s are real polynomials of indeterminates x! 0)ss x (2) (s)
(s)ss. Similarly, g" 1f'ss are real polynomials of indeterminates (L), 's x (1-1), ((+1), 's. Now, Equation (106) equal
((t%)
1eisn
(1..01(1.1) (8!07 13s5m pq
0.
(108)
By Proposition A.1, this is equivalent to
(1).) 12
p.no)
(g(0
(81.q 82.9
p(t)
nL.9) = 0,
(109)
which is
ngg mg
v (L.0)LL.))
c(s) (e)
=0. 0. -).9
(110)
Since the LHS Equation (110) is linear combination between elements of linear independent collection R. which
(x) (1) p.1
gj.g
(111)
25

---

Equivariant Polynomial Functional Networks
for < i & no and <J < nr. The linear dependency comes from the distinction between indeterminates of p!
and
It implies that
v (L.0))L..) 0,
(112)
for all 1 < no and 1 & n,, which means
(rc.0)(L.1) O.
(113)
We finish the proof of the case where < t.
Case >> t. From Equation (76), we have:
WW ( [[]](s)) visoo (W](2.t) pq 0,
(114)
which means
(w]lo).... (W g(s.0)(L.)) (w]())... (w](t)))))))))))))))))))))))). =0. (W
(115)
Assume that (L.)) # ReoxRs Observe that
[W](s) (1) g(0.0)(2.1) [W](). (W]++1
(116)
(IW)(0).. [W]++1), [w] "................. ........"))
(117)
(W)) (w /
(W)) (w
(118)
(ww) (w](). v(s.0)((.1) [w]( .... (W](st)).
(W] (w](+)).
In the second term of Equation (118),
[W] [w](), ((s.0)(L)) [w]().... [W](+1))
(119)
Since s > t, all [W] in Equation (119) are distinct. And since (ss00)(L..) #0 "oXL, with the same argument as Case s & f, we have
W] (2)
[W]1), ((o.0)(2.)) (W (1 #0 Ru.xn.
(120)
Moreover, can be written form
[W]).... [W (1.00)(11))) (W
(++1)
(121)
ijj
(at)
E Ruexns
(122)
where all (st) are real polynomials of indeterminates x! 2''s,, (2)
and (L) x ((-11) s. (++1) and at
(2), s. ... (9 and x (1)) s, x! (1-1)) 's.
least one of them is nonzero element in R. By Corollary A.7, indeterminates
1) s can be substituted for some real values to make
[w](.... ..... [W](1) (L.0)))))) (W](.)... [W]++1))
(123)
26

---

Equivariant Polynomial Functional Networks
become a nonzero matrix of R We denote this nonzero matrix by L (s.0)(1A4) E R Note that, since >> ł, the substitution only applies for indeterminates in [W](), [W] (W and [w]66, (W-L))) [W](+1), In other
words, does not apply for [W] ,...., (]++)) So, with the above substitution, Equation (116) becomes
[ (6) [W) ()(,,0)(Le) [w]()...
[W](++)
[[] [W leli)../oooo (10))1) [W] (*) [W](+1)
(124)
Note that, since t, there is at least (W(-) in the product [W] (9) [W Combine with T ((.0)(LL.4)
# by
applying the argument of Case L and 0, we have
(w) [ (1) v (1(.0))..0)
[W](4)... (w](tt))
59
= (W] (+) (W(+)) T (s.0))..)) [W] ......
(w((-1) # O,
(125)
pd
which contradicts to Equation (115). In conclusion
V (a.0)(L))) 0.
(126)
We finish the proof of the case where t.
In summary, we did consider all possible cases. The proof is finished.
C. Equivariant Polynomial Layers
We now proceed to construct a G-equivariant polynomial layer, denoted as F. These layers serve as the fundamental building blocks for our MAGEP-NENs. Our strategy is as follows: we first express E polynomial layer that is linear combination of stable polynomial terms (Subsection C.1). We then find the equivariant maps among these polynomial layers using the parameter sharing mechanism. Equivariance in Machine Learning is used in various context, such as Euclidean equivariance (Tran et al., 2024c; Ruhe et al., 2023), equivariant metanetwork (Tran et al., 2024d; Zhou et al., 2024c:b), Optimal Transport (Tran et al., 2024b; 2025a;c;b), etc.
C.1. Equivariant layer as linear combination of stable polynomial terms
For two weight spaces U and U' with the same number of layers T. as well as the same number of channels at ith layer
4 Wx B,
where:
W RELXDLXOL-1
X
X
R Xn,
Xno
B RRLXXLXX
X
X
pb2X12x1
X Rb xxxx1
and
4' W x B'
where:
W R X XPL-1
Rwi xns Xno,
X
X
B' x
R xxxx1
B xnxx1
We want to build map F:: U - l' such that F is G-equivariant, where:
G= (idg.,
920 x... G2 x {idg.n J.
(127)
Let us consider polynomial map E: 4 u' such that, for input U = ([W), (b), each entry of the output E(U) ([E(W)), [E(b]] is a linear combinations of stable polynomial terms, i.e the entries of [W](..1), [b](s), [W)]),,) (LLL)) [WW] (L.),, together with bias. In concrete:
((W)
Σ
ΣΣ<
(W])),)) + pq
Σ
T (Oyk (b()))
(s.0);p
(s)
1=1
27

---

Equivariant Polynomial Functional Networks
The
Oaik (.,0)))))
(WD)(000)
1
Σ
Σ
(B1]))) Pq
(1)
Las20L
[W (L.0)(..)) + (1)kk)
(c.0)(L.0):p9
15130
n2 E
[E(1)150
E
[W]
(a.1)
$ 10
(b]())
(3.1):p0
(s)
120>t>0p=11
1=
Σ
(WD)(000)
(.,0)))))))
L2s>1>0 p=1
Σ
(.)
pq
1.3500 121 pm
Σ3ΣΣ
(+02L))))L + ((:)s (s.0)(1.)))m
L203001
All sss in Rd''d, except the biases 47's are in Rd'x x1 In summary, E is parameterized by 4_'s and v_'s.
In order to be G-equivariant, the polynomial map E must satisfy the condition E(gU) = ( for all Gu and U e u. In the following subsections, we derive the computations of E(gU) and gE(U), and compare them in order to obtain all possible G-equivariant polynomial maps among those considered.
C.2. Compute EggU)
We have
Σ
T (gik
[WW](x (a.t)
(a.1);p
L2s>1 p=1 g=1
Σ
(1)zk (o):p
1=e30p=1
Diik (c.))())pp
lg Wg gdf(3:0000
+
Lee>t>0 pm1
+
Σ
(gigk (z)(L.)) [gbgWT(o((.)) (L)
Σ
Σ3
(0):f
(s.0))..))>m
53000 p=1
+ $ (1ikk
Orik (.)
Σ
((()))) (a.)
(..1):p
d
m. (p)mm)')))
1 p=1
Σ
T (4:fk
(())
b
(o)ep
p
13830p=1
28

---

Equivariant Polynomial Functional Networks
d(e) (a.0)())p
[Wb]
(7'())
Leest> p=1
d
(W
(L)(L.))
te (pp).T ((4)
13000 1>110p=114
(8)
E ΣEΣΣ
(3)ik (s.0)(L.))pgg
[ W/ (L.0)
(8)
(p))., (()
1.3000 p=
+ 1 (1ikk
E
(p)
[W]664
(p)ml)))))))..4)
L2s>t20p=1
T=(4)
+
Ggik
(())
ma(p) [0]b²)
(p)
L2850 p=1
(x.d)(t):=..p))) =.()) [Wo]|.000
Wgik
L3521>0 p=1
Σ Σ
(0)k (2))
(p)
()(L,)
(())
L p=
[()
(()
(iik (p)))
m (p(
(W
126>0
(e(g)
+ 1
(b):kk
R
[E(gb)]5)
Σ
l9W1(0 (x.t)
(s.t):pq
L283120 p=1 q=1
+ Σ
6):
(s):p [0b]
1
Σ
(a)))t)) low gobfse0000
+
13>1>0p=1
Σ
O0 (.) [obgW](e(LL.)
L p=1
12+
Σ Σ3Σ
ΣΣ
[gW (gWgWW(s00 (WWW
(s.0)(..)< 1.3000 p=1
+ $
(i)
d(a)
Σ
[W](1))
(8.1):p di
(p). 1(0)
L2s>120p=1 g=1
+
(5)
16
(8)
(x)pp
n. (p)
p=1
Σ
Gjj d(a)
[WD
(+)t)())=p
n, (p)
29

---

Equivariant Polynomial Functional Networks
Σ
Lea>0
1
Σ3Σ
(0: (s,0)(L.))pp
p
[WW]] (s.0)(L.1)
"(p),T ((0)
p=
+ 1(10
Σ
ΣΣ
(p) [W]63*))
(p)))))
(1)
(a())
0g
(())
(p) [01))) ())
(p)
10000pp=
(1) _ |Wb66a0000
12s>t>0p=)
Σ3ΣΣ
Σ
T.(p)
(S(LL)
(2)
[((g)
Σ3ΣΣ
(3:3
(p)
[
[WW] pg
(0))
xx(9)
+ 1(103
Note that, we can move around the r's in above equations since the group G satisfy that: G P. is trivial (for i = 0 or
i = I.) or the whole Pa (for 0 < i < T.).
C.3. Compute gE(
We have:
d(
(o(E(W)))
(1)
[--
T (k)
(F (k)
[W](e
1-
(s.1):pg
L3s>1
(1),
100x; (1)) (1)6)
(s)ap
12800p=1 Pp1
Σ
[W6]
(..01(0pp
Les>t>0p=1
Σ 33
B2
B4
(1)x (s)(1.)))pg)
i-1)
pq
12 12:0012120=111 1=
Σ
[WW](a
(c.0)
120>0L>130
a(0)
(1) (0)
1--1
30

---

Equivariant Polynomial Functional Networks
((((())))))) =d,= [b1 (0
())
(a) (s.1):pa
())
[W][a
12s2t30p=1e p=1
(1) (
[b]/2]
(s)pp
p=1
Σ
0 ())
|WB]50000
(s.0)(tp
120>1>0p=1
Σ ΣEΣ
0F (
(1)L..)):P
L3850 1=130ppp0p
Σ Σ3
(G)
[WW](00XZ.))
(s.0)(2.))3pp
L2530 1=:30
a/0 11007 (()
C.4. Compare E(gU) and gE(U)
Since E(gU)= = gE) from Corollary B.7, the parameters 4_'s have satisfy these following conditions:
1. For all L P > t t 0 with ts, # (L, 0), and for all p, we have
((*)
G1ikk
(p)
(0,,
(1)m)
(tt)
dk
(i-1) 1)
(s.1):pg
(1))
((a)
(.())
① (1)77'
(x.4).m (g)
(s,t):pg
(2)))
2. For all L > s 0, L > t + 0 with # t, we have
(0)
d
(p)
$ (0)x (0)
((1)
d
111)
(e.0)(L.))pp
(()))
(0)
T.(p)
(
(0);,, (G)
(1)
(L.))
m(())
3. We have
(LL)
Re No
Ggjk
(z) [W](L.0]
(p)))))()))0(())
To(1)
(+))
+ Σ
1 (i)zkk
T.(p)
[ [WW] (1.
(())
1>550p=1
(.(0)
(1 (1) (k)
[W](20)
(L.U):4
p= 0=1 %
Σ
n2 n,
(1)
()) (1)
[WW](300)L..)))
+
(-1)
(s,0)(L,)))pg
レ>030p=1
g=l
dj
31

---

Equivariant Polynomial Functional Networks
BL Ro
(p)
[W] pq
(L,0) (p))
(0)
+ Σ n2
(1:
F.)
[ [WW](s000
1>820 p=
(.)
(p) (1)
(1.0)
(L,0):P
9=1
Σ レ=
10m (0)
[WW]64
(s,0)(L.0):pg
4. For all L > > t> 0, and for all p, we have
= di"
((t)
① (1x (007211)
(..
mx())
((i-1)
(c.)(())
(0 (n)
m.(D) d
Φ (1),, '()
(s,1)())
5. For all L > s L. 0 with + t, we have
)
(1)
(p)
())
(p)
(()
dz
1-11
(s)(L.));pg
(q)
(s)
$ (0
(p)
(0)
())
(
(2)
dj
①
(s)LL.(;3
7(
6. For all L t > 0, we have
(Wb])
(m)))
(z)
d(1)
Wijk
me (p)
[bW] (4(4))
(q)g)
(1:7 (m)))))))) (8) (L,))())
|WB
(1-1) 1)
p=1
(0) (M)) (O(L.0p
p=1
ny
(z) [Wb] (1)00
(L)
(L.000)ZZZ))
p=l
(0)
4 W8
(p(p)
[6 (L).)))
(7)
(1)
(.(g)
Σ
g
(1) (
(L.)00)
(L):
p
p=1
(1)m G) (K
(O(L.1)pg
p= q=1
32

---

Equivariant Polynomial Functional Networks
7. For all L >> 0 and for all ps we have
d (1),, (077))
Agike ((s) (o)ox.())
x.(p))
d (1-1)
(o):p
00
d)
(1)m '0)
(o):T((p)
=x((p)
(s)-p
8. We have
d
T (0)bk
(0)m, (k)
d d(-11
$1
(1)
$ (1): ())
Also, since the group G satisfy that: G P. is trivial (for i 0 or i L) or the whole P, (for 0 i < L), so we can simplify the above conditions by moving some of the permutation n's the left hand sides. We have
1. For all L 22200 with (s, 1) * (L, 0), and for p.q. we have
(1)
()) (())
(p)
x(())
LOck (s.1):pg
(2)
(1-1)
Me(g)
x.-1(k)
(s)
()x(())
[.(p]
(())
$ (0:
(0)
m.(5)
(a,4):pg
(4)
2. For all I, 3 8 0, L> 1 0 with * we have
d(i
(f): 18k)
(p(p)
=(()) (0yk
(p)))
(4)
(i-1)
(s.0)(L.0)pp
mele)
(k)
(00.(1))
(p))) (0)
(p.0))
(L.))(L.));4
3. We have
No
(L)
(p) [W](L.0)
(L.))xz (p)so(4)
())
#o(9)
d(s)
R2
n
$
(1)m (k)
Tx(p)
[ [WW](e.0)L..)))
d(a)
L>800 p=1 g=1
(x)
70
Wgik
(L.0)
(J.0):p
p= q= 11k)(()
(6)
a,())
T Diik
[WW] (s.))L
+
(--1)
pq
1>000 p=1 1=1
(k)
(L))
ny no
$
(1)
(p)
[W](L.0)
(z)
(0)
TT(()
n2 n
Σ
$ (1)=
(x(P) [WW](e
(p))
p=
(n)
33

---

Equivariant Polynomial Functional Networks
Ro
Of (L.0)-p
[W]LL.0) lg
F((i)
p=
g=1
Σ
(00 (s,0)(2,))3pp
+
m,())
1=
4. For all > t 0, and for all p. we have
d(t)
(3) (k)
(())
x(())
Φ (0sk (s.0)())p
7.(p)
di-1)
(k)
())
(()
(0)
(o.)
(.(p)
m.(1)
(x)))))p
5. For all L > > 0, L 0 with + I, we have
(s)
T.(p)
F(())
(0ak
(p))
(t)
((-1)
(L)
m(())
((k)
(+)
(1),)))
T.(p)
( Oy
(1) m((i)
(L)))p
ne(9)
6. For all L > 8>> > 0, we have
(mmm (L.0())=ZZZ)) Te(p)
(L))
[Wb]
d(t)
())())))))))
me(p)
[bW] (0(2.)
(0)
(Ta(g)
[Wb]
1)
(L.0))pp))))
Tk-1(k)
R4
T Oajk
(-11)
(1(L.)))
p=
1=1
1(K)
(00 (L.0(0:ZZZ))
(LL)
[Wb]
(z())
(e)
[b (L)
(0)
mx(g)
[ [WB](
m())
(L.)))
04
()(C.)
(OLL.00g
7. For all L 0 and for all p, we have
(mm
(55)
7(())
())kk
(p)
r.(P)
1-1
(e)sp
d(3)
(6)
(0
(z):
#0()) 7.))
(6):p'
34

---

Equivariant Polynomial Functional Networks
8. We have
d()
((i-1)
(k)
$ ((:0(1))
(0)
$1 (0)
m.(s)
From the above equalities, we can determine all constraints for the coefficients according to the entries of [E(W)] as follows:
1. . (s, = (i, 1 1) $ {(L, (==
(1)=())((( + ((111))11111
(1isk
(1:-1))sk
If (1,1- 1011- 1) = (T, -- 1),9
[L):jTT (k) (1.11-1)37221166
(L)zk
(L,L- 1)ppk
(s, = (i, - = (1,0),p=j-
(1)= (1):jk
(1)g
(1,0):3
2.
If(,tt = (1,1-1) - 1) $ {(L, L -
(1)
2b (0)zkk
(600(2,ii-)==())(((
(1,)(
. (s, = (i,1-1) - 1) = (L, 9) k
$ (L):ja (k)
(L) (L)))
(s, = (1,1- - 1) = (1, p p=
T (1)
(1)gz
(1)0)(4.0)==11111
(1.0)(L.0)34
3.
7o
(m (L,0)pp
[W](Z.0)
R2 n
(1)m (7=-114)
pq
1=
(.)
B1 7o
[W](Z.0)
(L,0)
p= 1=1
(k))(k)
d(9) ΣΣΣ
a,6))
Giik
[W [WW](&.0 (L,))
(i-1)
p=
(R)
For each L > >>>>0 0, by scaling all the d by the same scalar, we have
PL no
(1)
(L.0
(L.0);pg
p=1 q=1
35

---

Equivariant Polynomial Functional Networks
(())
R2 n1
Σ
(x(
[WW](s,0))).)) pq
(P)
1>220p=19==
(9)
(3ikk (I,0):pg
[W/..0)
1=1
[WW](5.0))..
+
(s.0)(.)s)pp' p=
4. For all 8 > t> 0, and for all p. we have
(0
d(s)
$ (8ih
(s.1(():;ss)))
(.(p)
d (1k)
(s.0)())))
We have
o Glik
= O.
(x,t)(t)spp
5. For all L > 8 3>,,,>>>>>>> 0 with + i, we have
((a)
(00)
Φ (1)
Fp(p)
#(
10
(1))
(1)
(1-1)
(,)(L.();99)
nelg)
(1)
(s, = -1 1) (1
(1)=6)=1( (8) (1 = (())))-1)))
(0:sk
lfi If = (I, T. 1), k
(k)
2 (L)ssk
(L),,L)))))
(L)(L,LL1):pk
Ifi = (1,)))
$ (1)=
(1):jk
(1)(4.0)=+11g
(1)(1,0))3''
6. For all L > <> 0, we have
(1 (L.00Pp
108)
[Wb]
d0)
7p(p)
[6 (L..)
Tx(g)
+(()
(i-1)
(L.0)))))))))
((-1(k)
R1
2 0gfk
[6W](0().)
(0(L,)):
-=
(k)
For each L > / 0, by scaling all the d by the same scalar, we have
(M)
(L.0)0pp
36

---

Equivariant Polynomial Functional Networks
Re
R+
Me(p)
[Dw lp
(2(g)
(i)
1Ogk (L.000pp
[Wb))
(--))
p= (k))
(O(L.)): [6W]00(..)
fgyk
1)
7. = 1, p= = j.
$ (1)(())
(1))kk (1)x())
(1)s
8. We have
$ (0 O.
Similarly, we also determine all constraints for the coefficients according the entries of [E(b] follows:
1. For all L > 12 with (s,1) + (L, 0), and for all p, q, we have
(1) (.()) (p)=(())
(1)
+ (f)g (s.f)spp
(1)
(s,t)= (i,0),
$ (1T())
(c.0)a(()e
(1.0)cig
2. For all L> $>3>000 0, with A we have
(0)
(1)=(())
(.())
0)
(P)=)()))
(1)
x.
(s.0)(1..))p
#((9)
f= 0, j.
(1)(()
+ (1.0)(L.U):3g
(i)
(1,0)(2.))=7())
t= 0, = i= T.
o (D)
(D)s
(L.0)[L.0):p
(L.0)L..))
3. We have
(1)
(2.0)
(L.0)
Σ
1 (1
[ W (s.00)()
(p)
(a)
p=1>0p=1q=
(g)
R1 No
$ (
(L.0)
-())
(L.0):pg
1q
1=1
Σ
(0: (
[WW](s,0)L.
x(6)
(c.00)(2,));p
1>s30p=1g=1
37

---

Equivariant Polynomial Functional Networks
0. For each 1> >> 0, by scaling all the d by the same scalar, we have
0
(1)m()) (L,0):p0
[W] (2.0)
dls)
$ (3)
[p(p)
W WP
WW()) (L.))).
x(0)
L>000 p=
no
[W](L.-0
(L.U)
9=1
The
(9g m.(
[WW](a,0)(L.o)
(L.))(
L>830 pm qm1
i L. We have
n2
(L)
[W](L.0)
(L,0);pg
(0)
Σ
$ (1)
(.)
[WW] (s.0)(L.s)) p7
L>
(g)
Ro
(D)sf
(L,0):pq
1
(s,0)(L.))3p [WW]
(L)
1=150p=19==
which means $ (4)s
can be arbitrary. The rest (L.0);pg
((s)
The Use
Σ
(L)
(x)
(WW)(s.0)L..) pg
(100)
(l0))
L>830 p=1 q=1
(g)
= Σ ΣΣ
[WW] (E.A))
(L.0)(
1>5>0p=1q=1
For an I.>r r > 0, by letting म, be the identity, and ((r) be for all p. we have
(+)
Σ
(L)
WW (L.s) pq
(x)
125200, p=1 q=1
(a(
Σ ΣΣ
(D)= (L.0))L.)))m'
(WT]
L p=1
(x)
fre
(L)
(p(
[WW]
(WW)ro0t (1.0)(
(x0)(
(r)
pq
TT((a)
(D)
[WW]6.0))L..)
(7.0((t.)))m
38

---

Equivariant Polynomial Functional Networks
By Lemma B.3 and Corollary A.7, have
(r)
(L) (00
(p)
(r)
So
$ (L):
(r.0)(L.)))p =0
for p + 9, and
(100))))))))0)))--)) = (LL...j
(Li
(r:0)(L.r)ppp
conclusion, we have + (1)s (L.0)pg
is arbitrary, and for 1,> 1>8>> 0,
① (I.):j (L.3)(
4. For all L s >> 0, and for all p. we have
()
$ 0g
(p.0(()m..
m.(i) (c,))))))
if = s. p
+ (1+())
① (0)s
(i.0(()=(()))
(1.1)())))
5. For all L >>>> 0, 0 with + f, we have
Φ (0m
Ta(p)
(00
$ (0y
(p)=)))
x(())
(s)(L,)))pgg
(((1)
ssi -
T (1)(())
(5):
(L)
(v)(0,0)
s=== T,t t=
(L)
(b))
(L)(L.,)):pq
(I))
6. For all L > t > 0, we have
By
Σ
(1m (L.0)()))))
(()
Tv
(e(p))
[Wjae (L)
(p)
d())
(a(g)
Or (L.)0pp
m0
p=l
Og (10L.))
(b (L))
G.)
I. > >>>00 0. For each T> > 0, by scaling all the a ((')ss by the same scalar, we have
39

---

Equivariant Polynomial Functional Networks
Σ
d mzlp)
[Wb
(L.)(0)=Z(()
n2e
(0)d)) (1)
m((p)
[W (0(.))) p4
( =(()
Gly (L.t)(pppp
[
p=1
(1LL.00 (6
10
i = L. We have
LL (L.00t
[W6]
((1)
(1)
Re(p)
[W] (LL,))) lg
(e)
me(
(1) (L.O))pp
Dij (L(,)))
[bW] (LX,))
p=1==1
which means & (Z)g (L.1))))p))
can be arbitrary. The iest
d(t)
(L) (p)
xx())
[bW] (.))
(e))
xx
nhe nhe
(L)sf (0(1.)):
(6W]
pq
p=1 g=1
By Lemma B.5 and Corollary A.7, we have
(0)
Tx(p)
(L)y
(W))
(1)
(t)(),))
nel9)
So
(L)
(1)(L.t) = 0
for p + q, and
$ (L)s (1)().))..m())xx) (OLL.))pP
(L):
In conclusion, for all L > é > 0, we have 4 (4)g (1.1)())) is arbitrary, and
(L) ()LL)))))))(( (0(..))
(L)y
7. For all L > > 0 and for all p. we have
((*) (00) $ 09
(s) (p) T,)))
m. (6)
40

---

Equivariant Polynomial Functional Networks
Iff - j; j,
① (0)(() (1)=(()
4b (0):
(1)s5'
+ (L)s = 4 (E)s
(L)pp (L)D''
8. We have
4 (())))))) (8) (1)
m1(1))
If i I,, we have
((1) $ (2)5
C.5. G-Equivariant polynomial layers
Based on the above discussions, we conclude that every G-equivariant polynomial layer, which is defined as a linear combination of stable polynomial terms, given as E(U) ([E(W)) [E(b)]), where the entries of [E(W)] and [E(b]] are given case by case folows:
For i L, we have
[E =
$ (Z)g [W](hL-1)
$ +
Σ
(L,0)(,,2-))pp [WW][L.OKLL.-
(Z):
(L.L-1)pp
(L):L_L-1):pp (LLLLL)
(L)j
Ip
p=1
n no
no
[F()))5
(L)sf
[WW] (L.00L,))
T (D)
(W]L200)
(L.0)(..0)mm
(L.0).p
ne no
$ (4):
(s.0)(L.,)::0 [WW](90)1).) pp
(L)(L.0)33g [b pq
1=
1=1==1
ng
(L)d (L.0)(Pp)))
[WB
Σ
Σ
(1)...) (L
(D)s
レ>(>0 p=1
F>1200=1
(1)s
(b)) + (N)s
(4)s
+
(L)s
p=1
[E61)5 =
(L)
[WW]
(L.0)(1.0) [W
$
(D)s (L.0).g
[W](Z.0)
(L.0)
pg
9=1
Σ
(Z)
Σ [WWWJODLL (L.)).))
(D) (L)(L.)):Pg
PWJ (L)(L0))
PW
(c.
L>s>0
p=1
nL T
The
(L)g
[W
(D)
D 1b (1)(E.))
(L)P)())
L> p=1
L
RL
(L)
( + o (1)s
(L)x
p=1
For i = 1, we have
[E(W)
(1)
mgg
[W](1.)
no
(1):ok [WW[(LO000. (L,0)
(1.0)=y
ljy
(1.0)(2.0):=g
.jg
2=1
41

---

Equivariant Polynomial Functional Networks
Bo
(1)okk (1(L.)0)0))
[bW] [BW])LL)) (X)),)) + + 0
[015")
194
[E()]
0
(1)
[WW] (1.0)(2.0)
(1.0)=g
39
(1,0)(1,0)00u
jq
9=1
|b (2)
+ Φ
(1)
11,5
(1)(2.0)0g
(1)
For i> 1, we have
[B(W)]
:0:00
(W](
Φ
(1)) [WW] jk
(11-1):00
16W15K (111)-1)
[E(6)] 1
(1)
[W]Se]]
(2.0)
(1:
[WW] (t.0)(L,0)
(4.0)ag
(1.0)(1.0)0.)
ja
[W
(60700
(.00):
(L.
[6]
In the above formulas, the bullet indicates that the corresponding coefficient is independent of the indices at the bullet.
D. Invariant Polynomial Layers
In this section, we construct a polynomial map I: Rddd that G-invariant, i.e., I(gU) = I(U) for all g E Gu and U e U.
D.1. Invariant layer as linear combination stable polynomial terms
Similar to the equivariant maps, we seek the invariant map among polynomial maps that is linear combination of stable polynomial terms, specifically the entries of [W](*.)) [B](*), [Wb](s.))) (s.1)()), [6W]6)0.. and [ (W)D())))) along with bias. In concrete:
10)= = Σ
$ (s.0
[W] +
$ (s)op [b]{a)
1252130
1 p=1
Σ
Σ
o (s.))())))))) [WBI(
13a2t>0 p=1
n2 ΣΣΣ
Σ
(..L.) [6W](0XL.))
29
L2000 p=130p=1p 1=1
Σ
(s.0))2,,))37 (s.0L. P1.
L3500 L21 p=1
All i are ddd except the bias 9 in Rddx x1, In summary, I is parameterized by t_ 's and T_ We need be G-invariant, which means IgU\) I(U).
D.2. Compute [ggU)
From the definition of I(U), we have:
42

---

Equivariant Polynomial Functional Networks
Ph n4e E
I(gU)=
(s.t):pp [g (g pg
1.392120 p=1g=1
Σ
(s) (gblp-)
123200pl p=1
Σ
(.,00ppp [gW NgWggbffoo (0.00
1e
Σ
(s)2,))) lgbgW769)(2.0))
L 1=t30p=1
Σ n2
Σ
3E (s.0)(())))
(g (W
1 1=:20p=1
+
d0)
Σ
2 (s.0)ap
[W] (3,1)
(0)
mo (p), (g)
12221200 q
Σ
(s)i d(a)
1=
Σ
( [Wb](.000 (p)
(.:000p
125>120p=1
Σ Σ3Σ
d
bW (T.))
(0)(L)))
(t)
n. (p).., (())
L3520 p=
04 Σ3Σ
[W (s.0)(L.)
(s.0))(..))
(t)
(x).
La550 1=120p=1
((s)
T.(p)
(a.))
(a.1)
(1)
(t)
pg
1232130 p= 1 1=
m)(a)
Σ
x0)) (b)
(1)
(p)
L
Σ
(..00)))37)))))) a.(p) [WB
12s2t20 pm1
((s)
Σ E 7ie
(.(p)
(0
())
pg
L 12120p=19=1
m((g)
(+)
Σ Σ3ΣΣ 44
(p)))
[WW]0+0
pg
L 1
m.gg)
D.3. Compare I(gU) and I(U)
Since I(gU)= = I(U), from Corollary B.7, the parameters + "p's have to satisfy these following conditions:
43

---

Equivariant Polynomial Functional Networks
1. For all L 12 with (s,1) (L, 0), and for all p. 9, have
(p)
mx(P)
+ (2.t):pp
(1())
2. For all L 8 0,L>13 with + we have
(c)
(s..)((L.))
(1)
x.(g)
3. We have
(b)
no
(p)
[W]
(W]L.0) (L.0)
(0))01))))
0))
pg
(o())
Ry n2
Σ
[WW] (L.)( pq
1>00 q=1
(n)
(Z.0) pq
(L,0):9
p= 0=1
1 (s.0)(L.))) [WW](s.))(..))
+
L>800 1=1
4. For all L > > t> 0, and for all p, we have
((sd))02. dei) (s.)))()))
5. For all L > s 0, L t with o f, we have
[(c)
Tx(p) (s)((1)))
d(0
m(())
6. For all > 0, we have
BL
Σ
(1.00())))))))
m (WB](
(0)
(p)
(0)
(x(g)
D1
[WD] [Wb (L.00)
(OCLe) lo
(()
7. For all L > > 0 and for all p. we have
4 (x) d.)) (())p
44

---

Equivariant Polynomial Functional Networks
8. We have
1 1
Solve these equations, we have
1. For all I, > 8 12 0 with (8, # (T,, 0), and for all p, we have
4 (o.1)) = 0
2. For all L 2 8 0, >> 1 with + I, we have
mx(p)
(ss.0)(L.))pp
(1)
(x(g)
If (s,t) (T., O, have
(L,0)((,,0)pp (LL,0)(L,0)744
3. We have
(L.0) pq
(L,0)
((*)
+ Σ
T.(p)
[ W (LL))) pg
(s) (p)m.(g)
L>8>0 q=1
(x)
[W](2.0)
(L.0)spg
n2 n2
ΣΣΣ
(w](ss0))t..)))) pq
(c.0)(
208>0pml q
which means (1,00)00 can be arbitrary. The rest is
((s)
[.(p)
WV.00s.0) pg
(1.0)
((s)
1.>5>0p
xx(g)
Σ
$ ((..0)(
(s.).))) pq
レ>8>0 p=
For an L > r > 0, by letting ,, be the identity, and dl be for all p. we have
The
Tia
Σ
n. (p)
WW WWCC (V.)(.0) (2.0)
po)
L2s>0,
g=1
(a)
Bi nx
Σ
(a.0(.
(c.0){L.)))7
lpq
L p=l g=1
SO
d(r)
Ry
Rr
(r(p) [WW](r00X.))
(.)())
45

---

Equivariant Polynomial Functional Networks
By
Re
(WW](n pp
(r.0)(L.))pp
By Lemma B.3 and Corollary A.7, we have
d(r) (p)
((r) (x.0)(1.)))p
(1)
So
(r.,)) O,
for # and
(p) ((r,0)(L.r))pp
In conclusion, we have ((,0)) is arbitrary, and for > >>>0, 0.
(())()))) & (L..))
4. For all L > >> 0, and for all p, have
L (x.))()) = O.
5. For all
2 8 0, >> 1 2 with * I, we have
(x)
T.(p)
@ (pmm
= (s)
(t)
(((g)
If (s, (L, we have
t (LI(LO)) = (L)(L.)))
6. For all L t > 0, we have
d T.(p) [WW](0)2..)
RL
The m1
(L.)0
[WB
(0))
p=l
me(g)
(L.)0) [WB
+
& (X)LL)))
(6W]
1p
which means 4 (L,))()): can be arbitrary. The rest is
(1)
Tie T1
R2 nr
(p)
Σ>
Σ
T (()
(O)L))
(0))
pq
p=l g=1
(n)
p=
By Lemma B.5 and Lemma A.7, we have
(1) a.(p)
(1)
(0)
Tx(g)
(1)
46

---

Equivariant Polynomial Functional Networks
for p A y, and
(p) pp))p))))p
In conclusion, for all L> <>> 0, we have t (2.))()); is arbitrary, and
(1)(,1)))))))) = (P)
7. For all L 3 > 0 and for all p. we have
(p) 7.) = =<(s))
If L, we have
(()) (L)x
8. We have
Φ 11
D.4. G-Invariant polynomial layers
Based on the above discussions, we conclude that every G-invariant polynomial layer, which is defined as a linear combination of stable polynomial terms, given as:
n
7k [W](2-0)
1(0) (L.0)(L.)))pp
(L.0L)) P9
(L,0)pg
1=1
no
[WW] (..0)(.s))
(LL)L.0)= [BW](2)2.0)
(c.0)(L.)))
pp
p=
BL
Σ
$ (Z.0)))
WB]L000
(1)),))) 1=200=1
Σ
Σ
1>1>0
p=1
(L))
(61)5 + 1
p=1
In the above formula, the bullet indicates that the corresponding coefficient is independent of the index at the bullet.
E. Additional Experimental Details
E.1. Predicting generalization from weights
Dataset. The Tanh subset from the CNN Zoo dataset has 5,949 training instances and 1,488 testing instances, while the original ReLU subset consists of 6,050 training instances and 1,513 testing instances. We do the augmentation for ReLU subset, with an augmentation factor of 2, effectively doubling the size of the dataset by adding one augmented version of each original instance. The overall dataset sizes, including both the original and augmented data, are summarized in Table
Baselines
In this experiment, we compare our model with five other baselines:
STATNN (Unterthiner et al., 2020): utilizes statistical features of the weights and biases
Graph-NN (Kofinas et al., 2024): represents input network parameters as graphs and processes using Graph Neural Networks.
NP and HNP (Zhou et al., 2024b): incoporates the permutation symmetries of neurons into neural functional networks.
Monomial-NFY (Tran al., 2024a): extends the group action on weights from group of permutation matrices the group of monomial matrices by incoporates malling/sign-flipping symmetries.
47

---

Equivariant Polynomial Functional Networks
Table 6: Datasets information for predicting generalization task.
Dataset
Train size
Val size
Original ReLU
6050
1513
Augment ReLU Tanh
12100
3026
5949
1488
Table 7: Number of parameters of all models for prediciting generalization task.
Model
ReLU dataset
Tanh dataset
STATNN
1.06M
1.06M
NP
2.03M
2.03M
HNP
2.81M
2.81M
Monomial-NFN
0.25M
1.41M
MAGEP-NEN (ours)
0.99M
0.99M
Table 8: Hyperparameters for MAGEP-NEN on prediciting generalization task.
MLP hidden
Loss
Optimizer
Learning rate
Batch size
Epoch
500
Binary cross-entropy
Adam
0.001
8
50
Table 9: Dataset size for Classifying INRs task.
Train
Validation
Test
CIFAR-10
45000
5000
10000
MNIST size
45000
5000
10000
Pashion-MNIST
45000
5000
20000
Implementation Details. Our architecture begins with a Monomial-NEN layer featuring 20 channels, aligning the dimensions of the weights and biases. This is followed by two equivariant MAGEP-NFN layers, each with 20 channels, using either ReLU activation for the ReLU dataset or Tanh activation for the Tanh dataset. Next, the output is processed by MAGEP-NFF Invariant layer. The final output from this layer is flattened and mapped to 55000 This vector is further processed by fully connected MLP with two hidden layers, both activated by ReLU. The final output undergoes linear projection to a scalar, followed by sigmoid function. The model is trained using Binary Cross Entropy (BCE) loss over 50 epochs, with early stopping determined by a threshold r on the validation set. The entire training process on an 1100 GPU takes 30 minutes. A summary of byperparameters can be found in Table 8.
For the baseline models, we adhere the original implementations outined in (Zhou et al., 2024b), utilizing the official code (available https://github.e com/Allan YangZhou/nfn), and (Tran al., 2024a). In the HNP, NP, and Monomial-NFN models, employ three equivariant layers with channel configurations of 16, 16, and 5, respectively. The extracted features are then passed through an average pooling layer, followed by three MLP layers with hidden dimensions of 200 (Monomial-NFN ReLU case) and 1000 neurons MMonomial-NFN Tanh case and other models). The hyperparameters for our model, along with the parameter counts for all models involved in this task, are detailed in Table 7.
E.2. Classifying implicit neural representations of images
Dataset. We use the original INRs dataset, which contained three dataset: CIFAR-10, MNIST and Fashi obtained by applying a single SIREN model to every image. The detailed information about dataset is described (Zhou et al., 2024b). We use the datasett without any data augmentation as in the settings (Tran et al., 2024a). The breakdown of training, validation, and test sample sizes for each dataset detailed Table
48

---

Equivariant Polynomial Functional Networks
Table 10: Hyperparameters of MAGEP-NEN for each dataset in Classify INRs task.
MNIST
Fashion-MNIST
CIFAR-10
MAGEP-NEN hidden dimension
128x3
64
64 x
Base model
MAGEP-Inv
NP
MAGEP-Inv
Base model hidden dimension
128
128x3
64
MLP hidden neurons
1000
500
1000
Dropout
0.1
0.1
0.1
Learning rate
0.0001
0.0001
0.0001
Batch size
32
32
32 200000
Step
200000
200000
Loss
Binary cross-entropy Binary cross-entropy
Binary cross-entropy
Table 11: Number of parameters of all models for classifying INRs task.
CIFAR-10
MNIST
Fashion-MNIST
MLP
2M
2M
2M
NP
16M
15M
15M
HNP
42M
22M
22M
Monomial-NFN
16M
22M
20M
MAGEP-NEN (ours)
3.4M
4.1M
4.9M
Table 12: Number of parameters of all models for Weight space style editing task.
Model
Number of parameters
MLP NP
4.5M
4.1M
HNP
12.8M
Monomial-NEN
4.1M
MAGEP-NEN (ours)
4.1M
Table 13: Hyperparameters for MAGEP-NEN weight space style editing task.
Name
Value
MAGEP-NFN hidden dimension
16
NP dimension
128
Optimizer
Adam
Learning rate
0.001
Batch size
32
Steps
50000
two different architectures. For both the MNIST and CIFAR datasets, the architecture begins with Monomial-NFN layer to adjust the weight dimensions, followed by three MAGEP-
Implementation Details. In these experiments,
NFN layers, each utilizing sine activation. The resulting weight features are then passed through MAGEP Invariant layer. Finally, the output is flattened and processed by an MLP with two hidden layers, each containing 1,000 units and using ReLU activations.
For the Pashion-MNIST dataset, we begin with a Monomial-NFN layer with sine activation, followed by a MAGEP-NEN
49

---

Equivariant Polynomial Functional Networks
layer also utilizing sine activation, and then a Monomial-NFN layer with absolute activation. The architecture then aligns with the design of the NP model from (Zhou al., 2024b). Specifically, Gaussian Fourier Transformation is applied to encode the input into sine and cosine components, mapping from I dimension to 256 dimensions. The encoded features are processed through IOSinusoidalEncoding positional encoding tailored for the NP layer, which uses a maximum frequency of 10 and 6 frequency bands. Following this, the features pass through three NP layers with ReL.U activations. An average pooling is applied, after which the output is flattened, and the resulting vector is processed by an MLP with two hidden layers, each containing 1,000 units and using ReLU activations. Finally, the output is linearly projected to a scalar. We employ the Binary Cross Entropy (BCE) loss function and train the model for 200,000 steps, taking approximately 2 hours on an A A10 GPU. The parameter counts for all models are presented in Table 11
E.3. Weight space style editing
Dataset. We utilize the same INRs dataset as employed in the classification task, with the sizes of the training, validation, and test sets provided in Table 9.
Implementation Details. In these experiments, our architecture begins with two MAGEP-NEN layers, each with 16 hidden dimensions. The rest of the design follows the NP model outlined in (Zhou et al., 2024b). Specifically, we apply a Gaussian Fourier Transformation with a mapping size of 256, followed by IOSinusoidalEncodingg The features are then processed through three NP layers, each with 128 hidden dimensions and ReLU activation. The final output is passed through an NP layer for scalar projection and a LearnedScale layer as described in the Appendix of (Zhou et al., 2024b). We use the Binary Cross Entropy (BCE) loss function and train the model for 50,000 steps, which takes approximately 35 minutes on an A100 GPU.
For the baseline models, we maintain the same settings as the official implementation. Specifically, the HNP or NP model consists of three layers, each with 128 hidden dimensions, followed by ReLU activations. An NFN of the same type is applied to map the output to one dimension, after which is processed by a LearnedScale layer. The number of parameters for all models is detailed in Table 12, and the hyperparameters for our model are presented in Table 13.
F. Performance comparison with graph-based NFNs
Experiment Setup:
Following the same experiment setup in Appendix E.I, we compare the predictive performance of our model and two graph-baser based baselines: GNN (Kofinas et al., 2024) and ScaleGMN (Kalogeropoulos et al., 2024), using HNP (Zhou et al., 2024b) as a reference.
The results are presented in Table 14. The GNN model exhibits noticeable performance decline when tested on separate activation subsets. Although ScaleGMN significantly improves performance on the Tanh subset, its enhancements on the ReLU subset are comparatively modest. In contrast, our model demonstrates substantial overall improvements across both datasets, highlighting its effectiveness.
Results:
Table 14: Performance comparison with Graph-based NFNs on Small CNN Zoo task.
ReLU subset
Tanh subset
HNP (Zhou et al., 2024b)
0.897
0.934
GNN (Kofinas et al., 2024)
0.897
0.893
ScaleGMN (Kalogeropoulos et al., 2024)
0.928
0.942
MAGEP-NENs (ours)
0.933
0.940
G. Memory and Runtime.
Table 15 and 16 provide runtime and memory consumption data for our model and other baselines in predicting CNN generalization task. For graph-based architectures, we compare with two recent works: GNN (Kofinas et al., 2024) and ScaleGMN (Kalogeropoulos et al.. 2024). Our model runs significantly faster and uses much less memory than these graph-based networks and NP/HNP (Zhou et al., 2024b). Introducing additional polynomial terms slightly increases our model's 's runtime and memory usage compared to Monomial-NFr (Tran et al., 2024a). However, this trade-off results in considerably enhanced expressivity, which evident across many tasks like Predict CNN Generalization or INRs
50

---

Equivariant Polynomial Functional Networks
Table 15: Runtime of models on Small CNN Zoo task.
ReLU subset
Tanh subset
NP (Zhou et al., 2024b)
36m40s
35m34s
HNP (Zhou et al., 2024b)
30m06s
29m37s
GNN (Kofinas al., 2024)
4h27m29s
4h25m 17s
ScaledGMN (Kalogeropoulos al., 2024)
lh20m
1h20m
Monomial-NEN (Tran et al., 2024a)
23m47s
18m23s
MAGEP-NFNs (ours)
28m43s
28m 2s
Table 16: Memory consumption of models Small CNN Zoo task.
ReLU subset
Tanh subset
NP (Zhou et al., 2024b)
838MB
838MB
HNP (Zhou al., 2024b)
856MB
856MB
GNN (Kofinas et al., 2024)
6390MB
6390MB
ScaledGMN (Kalogeropoulos et al., 2024)
2918MB
2918MB
Monomial-NFN (Tran et al., 2024a)
560MB
582MB
MAGEP-NFNs (ours)
584MB
584MB
Classification.
H. Implementation of Equivariant and Invariant Layers
We provide the multi-channel implementations of the Gu-equivariant map E: Ud 4 and the Gu-invariant map I: Ld - Rexd' For uniformity in implementing Equivariant and Invariant layers from Appendix C.5 and Appendix D.4, we employ einops-style pseudocode as a consistent framework.
We summarize the key dimensions in Table 17 and outline the shapes of the input terms in Table 18.
H.1. Equivariant Layers Pseudocode
H.1.1. PSEUDOCODE FOR CASE i = L
From the formula for (E(W))(L)
n2
n2 +
[F(W))
Σ
T
(Mj
[W]ph
(1,--11
(L))
[WW] (L.0)(1,..-) pk
(L.b-))p
pk
(L.0)(1.L-))a'
Table 17: Summary of key dimensions involved in the implementation
Symbol
Description
d
Number of input channels for the equivariant and invariant layer Number of output channels for the equivariant and invariant layer
Batch size
71,
Number of channels at the izh layer
d'
Embedding dimension of the invariant layer's output
51

---

Equivariant Polynomial Functional Networks
Table 18: Shapes of input terms used in the implementation
Term
Shape
[W](6.))
[b, d, ng, nel [b, d, ng]
[W]
[6, d, Rg: ndl
[WW7(000000
[b, d, noy nel
nt.
(L)d
[6W]
(L)
p=1
We define the pseudocode for each term:
For क (L):) (L,L-1);p pk
(L.-1)
with [W](L of shape [b, d, ny, ML-1 and $ (L);f (1,p-1):p 1}:p
of shape (e, d, nz, nzl,
pk
Corresponding pseudocode: e einsum(edp)) bdpk bejk)
For o (Li: (L.)(L,L-))>) [WW] WW]]L.. (L.)),
with [WW](L.OLL,LL) of shape [b, d, n& 1I-1] and 1 (L)z pk
(1.0))2,,L-)ppp of shape (e, d, nr.,
Corresponding pseudocode: ei einsumedd bdpk bejk) For 1b (L):
(L)(L,L1)pp [b (L),L--1)
(L),L_1))) of shape [b, d, n., nI- and 4 (L): (L))p_L111;p of shape [e, d, nz, nz),
with pp Corresponding pseudocode: einsum(edpj bdpk - bejk)
From the formula for (
ne no
RL ng
[(())
Σ>
(1) (LO)LLO)).OO) (L.0)LLO) pq
(1)
[W](L.))
(L.0)_
(L.0)
pq
p=1-=1
9=1
ne ng [WW](s.0)L.))
Σ
(L)
(LI(L.0) (LXLL)
(L)
(s.0)(L,a):
pp
L>s>0p p=1
pm p=1
Σ
(L)s
(L.1)))))))) [WB
Σ Σ
(L)s (I)LL.)) (LLL))
1>1>0 p=1
1=150
(L) (L)xp
[6] + 1 (1)
+
#=1
We define the pseudocode for each term:
For (L)s
[WW](L.0)).00
(L.0)(L.0);gg
with [WW](L.0)) of shape [b, d. n,, no] and 4 (L): (1.00)200) of shape [e, d, nz, no. nz),
pg
Corresponding pseudocode: einsumeedput bdpq - bej)
For (L)s
(1,0)p4 [W](1.0) with [W](7-0) of shape [l, d, nr, no) and $ (L)s
(L.0)3g of shape [e, d, nt, no, nr).
Corresponding pseudocode: e cinsumeedpgj, bdpg bej)
52

---

Equivariant Polynomial Functional Networks
For $ (L) (L))) pp
with [WW] of shape [b, d, na, na] and & (L)) (L,0) of shape [e, d, nay Ray nn.),
Corresponding pseudocode: e nssumedpppj bdpp bej)
For o (L) (L)L..)) [6W7]4))4.0)
with [6W]]] (bW](L)2 of shape [b, d, n,, no] and & (D) (L)
(1) shape [e, d, nz, no, nzl.
py
Corresponding pseudocode: einsum(edpoi) bdpq bej)
For (L))
(L))))))p [Wb]g with (Wb))..t)))) of shape [b, d, nz. and (L) (L,))()p
of shape (e, d, mL]
Corresponding pseudocode: einsum(edpj, bdp - bej)
For &
(1))
(a)(L,)): [bW] with [bW] of shape [b, d, n,, ne] and + (1)
(OLL,)) of shape (e, d, ne, n4 nn.)
Corresponding pseudocode: einsum(edppjj bdpp - bej)
For (D) [b]/ with (b1(4) of shape [b, d, m7] and 1b (1)) of shape [e, d, mz, nL). (L)xp
Corresponding pscudocode: e einsum(edpi) bdp - bej) For (11)) shape [e, nz),
Corresponding pseudocode: einsum(ej, ej) (0nqueeze(()
H. 1.2. PSEUDOCODE FOR CASE i
From the formula for [B(W)
ng
no
[E(W)
(1)
Σ
[W]
(1,0)
$ (1)okk
[WW] (L,0)
(1.0)
jj
(1.0)(.0)):
no
(1(1,) (1)(T,0):0))
(1)aok
[W
+ o (1)akk (
ja
(1):
We define the pseudocode for each term:
For t (1):ok
(1.0) with [W])ho) of shape [b. d, nı, no] and t (1)ok
(1.0)=g of shape [d, e, no. no).
(1.0),n
Corresponding pseudocode: (undjm deqk + bejk)
For q. (1)aok
(L.0) [WW]]L
with [WW](LOXXL.0 of shape [b, d, n, nal and क (1) (L,)) of shape [d, e, no. nol:
Corresponding pseudocode: eins umbbdjq, deqk - bejk)
For क (1)aok [6W](1)2.0)
(1)(1)0):00 lj
with (1) of shape [b, d, nı, no] and $ (1) (I)(L.))=gg of shape [d. e, no: no),
ljg
Corresponding pseudocode: einsum bdjqq, deqk - bejk)
For क
[ with (BI)") of shape [b, d, n1] and t (1)ok of shape (d, no), (1):
(1)
(1)
Corresponding pseudocode: einsum(bc)) (dj, dek - bejk)
From the formula for (E
120
[B))) =
(1) (1)
|WI
Σ
(1)
[WW] (L.))))
(1.0)=g
.jg
(1.0)(
1=1
53

---

Equivariant Polynomial Functional Networks
flg
(1) (I)(L)))=
[WT (1)(.,)) + (()):
34 (1):
We define the pseudocode for each term:
For o [1):
Corresponding pseudocode: e esummbb, deq bej) (1.0)=y of shape [d, nol,
[W]]g]0) with [W151] of shape [b, d, n,, no] and $ (1):
(1.0):.q
For $ (1.0)(2.0); [WW1]] (1.)
(1)=
with [WW]]000 of shape 16, d, nı, no] and & (1):
(1.0)(2.0);09 of shape [d, e, no].
Corresponding pseudocode: sinnumbbdjjg deq bej)
For (1))
(1))(2.0)= [bW] with [bW] of shape [b, d, n1, nol and + (1):
(1)(L,)) of shape [d, e, no):
Corresponding pseudocode: einsum(bdjg, deq bej)
For + (1): " with (B15") of shape [b, d, m1] and (1) of shape (d, e).
Corresponding pseudocode: e einsum(bdjj de - bej)
H. 1.3. PSEUDOCODE FOR CASE 1 <i< < L
From the formula for (E(W))5
((W)
④ (0):00
(1:--))000)
(W7l (
(1)00 (1:000 (20)(1.)-1))0)))
(WW (2,0)(,
① (112.))1)))0)) [BW]
We define the pseudocode for each term:
For 1 (e)e.
[W1s (wi-) with likt
of shape [b, d, n4, 14-1] and o (0):0
(1,1-1):00 of shape [d, e).
(1.--)) 1):00
Corresponding pseudocode: einsum de - bejk)
For & ()):0
(L.0)(,,i-1)=- [WW] (1.))(..i-))
jk
with WWWi (201) of shape [b, d, n, 11 and + (1):00
(1)) of shape [d, c),
Corresponding pseudocode: e coinsummbjjk;;... de bejk) For 4 (1)00
((( (LL.--))
with [6W]] (6w]0LL. (1 of shape [6, d, n4, n- and d (1:00 ()( of shape [d, e),
Corresponding pseudocode: einsum(bdjk, de bejk)
From the formula for [
ng
mg
[E())] (2)
(
(1)c
[W] (2.0)
c (
((0111.0)))))
[WW](1022.00
mr
(00=
[WD
(1)
i1>0
(1)
We define the pseudocode for each term:
For & (
[W])60) with (w](1.0) of shape (b. n4, no] and 4 (0) jj
(F.))eg of shape [d, e, nol,
(1.0):
54

---

Equivariant Polynomial Functional Networks
Corresponding pseudocode: e einsum(bdjj, deq bej)
For 4 (8):
(L.0))10):=r [WW]]o
with [WW]] (1.0) ijq
of shape [b, d, n4, no) and + (8): (L.0) of shape (d, e, no),
Corresponding pseudocode: e einsum(bdjg, deg bej)
For b
[Wo]f with [W6]] of shape [b, d, nz] and (0): ((a)()) of shape [d, e).
((.))()):
Corresponding pscudocode: einsum(bdj, de - bej)
(6): For 4
())(L0)); [6w]5 with (W1() of shape [b, d. ne, no] and o (b)c
())(,000 of shape [d, no]:
jg
Corresponding pseudocode: e einsum(bdjj, deg - bej)
For ob (i):
[ with [ of shape [b, d, ns] and (0)= of shape [d, el,
(s):
Corresponding pseudocode: e einsum(bdjj, de bej)
H.2. Invariant Layers Pseudocode
From the formula for the Invariant layer (U)):
1(0) = (L,0)(LL,))3 [WW](L.0 +
(L.0);p [W](L.0) (L.0)
p=1
(
bW (W (L)(2,0 pg
(c.0)(L.
(L)(L.0)=pg
L>8>0pm
nL
Σ
Σ
2 (1.00(p [WB)L
Σ Σ
(1)C,.)): (bW]
1>t00p=
1=
a1
+ (
(61)40 + 1
We define the pseudocode for each term:
For (L.0)) [WW][L.0XL.0]]
with [WW][2.00) of shape (b, d, nu, nol and (())0)(2.0)) of shape [d, e, nu, no, d'}, Corresponding pseudocode: einsum(bdpq, depgk - bek)
For (1.00); [W](L.0), with [W](L.0) of shape [b, n, nol and ((1,0) of shape [d, e, n2, no, d').
Corresponding pscudocode: eins Lansum(bdpqh depqk + bek) For (s.0))L.)))** [WW]]s.0)(L..)))
with [WW] shape [6, d, n2) and P(c,))(L.)).. of shape [d, e, d'),
Corresponding pseudocode: einsum(bdph: dek - bek)
For ((L))..))''" (LW()))(
with [6W]]2X2200 of shape [b, d, n,, no] and (L)(L,))) of shape [d, e, nı, no, d'),
Corresponding pseudocode: cinsummbbpppp depgk bek)
For ((L,))))P)))) [WD] with [WD))..000) of shape [b, d, ny.] and P(L.))()P of shape [d, e, nr,
Corresponding pseudocode: einsum(bdpp: deijk - bek)
For ())))))): (L(L), with (D(L)) of shape [b, ne] and (1))).)) of shape (d, d'),
pp
Corresponding pseudocode; e ei iinsumuuppp dek - bek)
55

---

Equivariant Polynomial Functional Networks
For (L)) [ with [0]{() of shape [b, mz] and & ((L):p of shape nd, d"], Corresponding pseudocode: e einsum(bdpk: depk: bek) For 1f shape d.]. Corresponding pseudocode: einsum(ek ck).uussquceze()
56


Reviews:
[{"review_id": "kfZc0BcCJz", "reviewer": "Reviewer_HSsC", "summary": "The paper is an extension to Tran et al 2024, by presenting a neural functional network based on stable polynomials. The input to the network are weights of other networks, and the constructed network is equivariant to permutations of its neurons and scaling of the weights. The construction is based on defining certain stable polynomials, where the respective groups (permutations and scalings) are equivariant w.r.t these polynomials. Several experiments and an ablation study are given, beating other constructions on similar tasks.\n\n###############################################################\n\nI thank the authors for their response. I still lean toward acceptance, but the paper seems to require some rewriting and clarifications to make the claims clearer and stand-alone, not relying on reading previous works. For example, stating what are the trainable parameters of their model should be clearly stated in the main part, and not left to the appendices. Also, the models for CNN and MLP should be stated, as well as the training procedure, preferably in the main paper. In the current form, I will maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. I think the main issue is that this paper is written as a follow-up ...** \n\n**Answer Q1.** We appreciate the suggestion from the Reviewer and will include explanations of relevant concepts from prior work in the revised Appendix. Below, we address each part of the question.\n\n**a), b)** In the case of MLPs, all $w_i$'s and $b_i$'s values are equal to 1, whereas in CNNs, the $w_i$'s and $b_i$'s values correspond to the sizes of the convolutional kernels.\n\nWe present the weight space using arbitrary $w_i$'s and $b_i$'s values for an additional reason: within a functional layer, $w_i$'s and $b_i$'s also represent the number of hidden functional units. This generalization allows the framework to accommodate a broader range of architectures and internal structures.\n\nFor MLP's, the way the model is written explicitly is:\n\n$f(\\mathbf{x} ~ ; ~ U, \\sigma) = W^{(L)} \\cdot \\sigma \\left( \\ldots \\sigma \\left(W^{(2)} \\cdot \\sigma \\left (W^{(1)} \\cdot \\mathbf{x}+b^{(1)}\\right ) +b^{(2)}\\right) \\ldots \\right ) + b^{(L)}.$\n\n**c)** The $\\Phi$ and $\\Psi$ are trainable parameters. We mentioned in the appendices that, in constructed functional layers, $\\Phi_{-}$'s and $\\Psi_{-}$'s are trainable parameters (line 1573 for equivariant layers and line 2372 for invariant layers).\n\n**d)** We provide the following examples to clarify the rationale behind our choice of notation in the paper:\n\n$[W], [b]$: Square brackets are used to distinguish whether a term belongs to the weight space or represents learnable weights in functional models.\n\n$\\Psi$: In the expression $[bW]^{(s)(L,t)} = [b]^{(s)} \\cdot \\Psi^{(s)(L,t)} \\cdot [W]^{(L,t)}$, the index $(s)(L,t)$ for $\\Psi$ is chosen to reflect that it connects two indexed components: $(s)$ and $(L,t)$. This way, simply by inspecting the indices of $\\Psi$, one can infer the terms it links—namely, $[b]^{(s)}$ and $[W]^{(L,t)}$.\n\n$\\Phi$: In equivariant layers, the notation $\\Phi_{(s,t):pq}^{(i):jk}$ represents a scalar used to compute the output component $[E(W)]^{(i):jk}$ (indicated by the top index), corresponding to the input component $[W]\\_{(s,t):pq}$ (indicated by the bottom index). In invariant layers, the output does not lie in the weight space, and thus no top index is used—for example, $\\Phi_{(s,t):pq}$.\n\nWhile this indexing system might initially seem verbose, we find it brings clarity and consistency to the paper, particularly when describing the computation of functional layers. We appreciate the Reviewer’s question and will include a paragraph explaining the notation system more thoroughly.\n\n**e)** The dimension $d'$ refers to the output dimension of the invariant layer. It can be chosen arbitrarily, and we use $d'$ to distinguish it from $d$, which denotes dimensions associated with the input.\n\n**Q2. Another major issue is ... or other simple models.**\n\n**Q3. Is there some motivation for why choosing the \\Psi ... they were chosen to be in this form?**\n\n**Answer Q2+Q3.** In our functionals, all the trainable parameters are $\\Phi$'s and $\\Psi$'s. \n\nThe main motivation for introducing the stable polynomial term with parameters $\\Psi$ in our work—compared to the equivariant functionals in Tran et al. (2024)—is to address limitations arising from parameter sharing during the construction of equivariant functionals. Specifically, enforcing equivariance through parameter sharing often results in many parameters being forced to zero or constrained to be equal, thereby reducing expressiveness. In contrast, the parameters $\\Psi$ in our stable polynomial term are inherently equivariant under the considered group action, and thus remain unaffected during the equivariance-enforcing process. As a result, our functionals gain greater representational capacity.\n\nIt is important to note that $\\Psi$ cannot be placed arbitrarily within the polynomial expression, as doing so could break equivariance. Instead, we carefully position $\\Psi$ between terms in a way that ensures the group actions on both sides cancel out, preserving the equivariance of the overall construction.\n\n---\n\n**Methods And Evaluation Criteria.** Is that previous work (e.g. Zhou et al. 2023) ... on MNIST and CIFAR-10.\n\n**Answer.** We do not include experiments on 3D datasets such as ShapeNet or ScanNet because the dataset of pretrained weights used in Zhou et al. 2023, originally from De Luigi et al. 2023, has not been publicly released. Additionally, the implementation code for Zhou et al. 2023 on these datasets is also not publicly available. Given these unfortunate constraints, it is not feasible to include this baseline within the one-week rebuttal period.\n\n---\n\nWe thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.", "claims_and_evidence": "I believe the main claim, that the proposed construction improves on previous ones, is well supported on both the theoretical and empirical sides. However, the paper's presentation has some problems, which I will detail later on.\n\nOn the theoretical side, the proposed polynomials are proved to be “stable” under the group action, and every other G-invariant map has a similar form.\nOn the experimental side, several experiments are given for predicting the model’s accuracy, classifying which image an INR trained on and several ablation studies are given.", "methods_and_evaluation_criteria": "I believe the evaluation criteria are good and follow the baseline of previous related works. One problem I’m not sure about, is that previous work (e.g. Zhou et al. 2023) also tested on 3d datasets, such as ShapeNet and ScanNet, while this work doesn’t. Is there a reason for that? Because I believe this can give a better case for the improvement of performance over previous works, since these are harder datasets than networks trained on MNIST and CIFAR-10.", "theoretical_claims": "From what I went over, the theoretical claims seem sound.", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "I am not very familiar with the neural functionals literature, however, it seems that this paper is a direct followup to this line of work.", "essential_references_not_discussed": "Not that I am aware of", "other_strengths_and_weaknesses": "There are some major issues with the presentation of the paper which I think can be improved with further work:\n\n1) I think the main issue is that this paper is written as a follow-up to Tran et al. 2024. In this way, some details are missing that are difficult to understand without knowledge of the previous work, and make the reading confusing. Some examples below.\n\na) The weight space is defined in Section 2, but not how the model is constructed with those weights. I recommend writing the model explicitly.\n\nb) Are the weight space should be some generalized form to include both MLPs and CNNs? In this case for MLPs the w_i’s should be 1 in Eq. (1)?\n\nc) In Eq. (9), the \\Psi are the trained parameters? They seem to be hidden inside the bracket notation, so it is not clear whether they are trained or some weights of the model.\n\nd) In Eq. (10) the notations are not clear. For example, whether the \\Phi are scalars? what does the (s,t):pq indexing means? \n\ne) What is the dimension d’ in line 193 right side?\n\n2) Another major issue is that the training of the model is not explained in detail. Is it the case that only the coefficients of the polynomials are trained? If so, is it similar to learning a polynomial kernel? If so I think it should be said explicitly. If not, it would be helpful to state the exact training procedure, since the training is different than training standard MLPs or other simple models.\n\n3) Is there some motivation for why choosing the \\Psi in this form as learnable parameters? For \\Psi it makes sense since these are the coefficients of the polynomials. However, for \\Psi it is not clear, since there seem to be many places (i.e. other intermediate layers) where these parameters could be placed. Why they were chosen to be in this form?", "other_comments_or_suggestions": "See above", "questions_for_authors": "I would be happy if the authors could respond to the remarks about the presentation above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KGngF0abA2", "reviewer": "Reviewer_FUc7", "summary": "This paper introduces MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial Neural Functional Network), a novel neural functional network (NFN) designed to process neural networks as input data. Existing NFNs with permutation and scaling equivariance typically rely on either graph-based message-passing or parameter-sharing mechanisms. While parameter-sharing-based NFNs offer lower memory consumption and faster computation, they often suffer from limited expressivity due to the symmetry constraints of the input networks. MAGEP-NFN addresses this limitation by employing a nonlinear equivariant layer represented as a polynomial in the input weights. This design allows the model to capture more complex relationships between weights from different hidden layers while maintaining efficiency. The authors demonstrate through empirical evaluation that MAGEP-NFN achieves competitive performance and efficiency compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The paper could benefit from a more detailed discussion of how their use of stable polynomials ... the broader scientific context.**\n\n**W3: In summary, the paper makes a meaningful contribution to the literature on equivariant neural networks ... would enhance the discussion of its relation to broader scientific advances.**\n\n\n**Answer W1+W3.** We refer to the proposed stable polynomial terms as \"stable\" because they are inherently equivariant under the considered group action and therefore remain unchanged during the equivariance-enforcing process—specifically, the parameter-sharing mechanism discussed in the paper (lines 160-167). The term \"stable\" does not pertain to empirical notions such as stability, robustness, or generalization.\n\n**W2: While the authors mention the potential applicability of their parameter-sharing ... highlight distinctions or potential synergies.**\n\n**Answer W2.** Existing frameworks that involve parameter sharing can be mentioned as follows: \n\n- [1]: NFNs for MLPs or CNNs with permutation equivariant.\n\n- [2]: NFNs for Transformer with permutation equivariant.\n\n- [3]: NFNs for MLPs or CNNs with monomial matrix group equivariant.\n\n- [4]: NFNs for Transformer with equivariance under the maximal symmetry group of Multihead Attention mechanism.\n\n**W4: However, one area for improvement is the theoretical grounding ... strengthen the contribution and enhance its impact.**\n\n**Answer W4.** The rigorous computations in Appendices C and D establish the equivariance and invariance of the proposed functional layers. Additionally, Theorem 3.5 provides an expressivity result, stating that any invariant layer computed via the general formulation (10) must take the form given in Eq. (13). An analogous theorem for equivariant layers can be formulated similarly, given that the computational process for these layers follows the same structure.\n\nIt is worth noting that the proof of this theorem is non-trivial. It is derived through an analysis of linear independence, as detailed in Appendices B.3 and B.4, which we found to be mathematically challenging.\n\n**W5: The abstract is informative and covers the key contributions ... improve readability and make the key messages more impactful.**\n\n**Answer W5.** We sincerely appreciate the Reviewer’s feedback on the abstract and will make the appropriate edits to improve its clarity and accuracy.\n\n**Q1: In Theorem 3.5, you prove the G-invariance of the proposed equivariant polynomial layer. Could you clarify where the main technical challenges lie in establishing this invariance? Understanding the core difficulties in the proof would help appreciate the contribution more fully.**\n\n**Answer Q1.** While it is relatively straightforward to write down a specific formulation of a layer that is equivariant or invariant under the considered group action, it is significantly more challenging to characterize all possible formulations of such layers. The main technical difficulties arise from two aspects: first, the computations presented in Appendices C and D; and second, the theoretical results in Appendices B.3 and B.4 that underpin these computations, which are essential to ensure that no valid layer type is overlooked.\n\n**Q2: In Table 3, the performance of MAGEP-NFN appears ... (e.g., efficiency, scalability) that distinguish your approach in practice?**\n\n**Answer Q2.** We believe the task in Table 3 is comparatively less challenging than other benchmarks, which explains why Monomial-NFN already performs well and our model offers only a slight improvement. This is reflected in the high Kendall’s $\\tau$ values, ranging from $0.913$ to $0.940$ across all models. The strengths of MAGEP-NFN become more evident on more challenging tasks. For instance, in the Classify INR-MNIST task, where the accuracy of other models widely ranges from $10.62\\\\%$ to $69.82\\\\%$, MAGEP-NFN achieves the accuracy of $77.55\\\\%$, surpassing the second-best model by a notable margin of $7.73\\\\%$.\n\n---\nWe thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.\n\n---\n\n*References.*\n\n[1] Allan Zhou et al., Permutation Equivariant Neural Functionals, NeurIPS 2023.\n\n[2] Allan Zhou et al., Neural functional transformers, NeurIPS 2023.\n\n[3] Hoang Tran et al., Monomial matrix group equivariant neural functional networks, NeurIPS 2024.\n\n[4] Hoang Tran et al., Equivariant Neural Functional Networks for Transformers, ICLR 2025.\n\n[5] Derek Lim et al., Graph Metanetworks for Processing Diverse Neural Architectures, ICLR 2024.\n\n[6] Ioannis Kalogeropoulos et al., Scale Equivariant Graph Metanetworks, NeurIPS 2024 Oral.\n\n[7] Miltiadis Kofinas et al., Graph Neural Networks for Learning Equivariant Representations of Neural Networks, ICLR 2024 Oral.", "claims_and_evidence": "Claim 1\n\nThe authors introduce a novel class of stable polynomial terms in the input weights that remain stable under permutation and scaling (sign-flipping) group actions.\n\t•\tEvidence:\nThey conduct a comprehensive study on the linear independence of these stable polynomial terms, ensuring they form a sound basis for constructing equivariant and invariant layers. This approach addresses the challenges associated with identifying polynomial orbits under group actions.\n\nClaim 2\n\nThe authors characterize all equivariant and invariant layers as linear combinations of these stable polynomial terms, with polynomial degree at most L + 1, where L is the number of layers of the input neural networks.\n\t•\tEvidence:\nBy focusing on this restricted class of polynomials, the proposed layers are shown to be both computationally efficient and memory-friendly, overcoming the high computational costs of working with generic polynomials.\n\nClaim 3\n\nThey design MAGEP-NFN, a family of neural functional networks (NFNs) that are permutation and scaling equivariant, and offer improved expressivity while maintaining low memory consumption and running time.\n\nEvidence:\nBuilt upon the parameter-sharing mechanism, MAGEP-NFN leverages the newly introduced nonlinear equivariant polynomial layers. Empirical evaluations on three tasks—predicting CNN generalization (Small CNN Zoo dataset), weight space style editing, and classifying implicit neural representations (INRs)—demonstrate that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.", "methods_and_evaluation_criteria": "Method\n\nThe paper introduces MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial Neural Functional Network), a novel class of Neural Functional Networks (NFNs) designed to process neural networks as input data. The key innovation lies in constructing polynomial invariant and equivariant layers based on stable polynomial terms. These terms are specifically designed to remain stable under the action of the monomial matrix group G, ensuring permutation and scaling (sign-flipping) equivariance.\n\nMAGEP-NFN follows the parameter-sharing mechanism described in (Tran et al., 2024), but extends it by using nonlinear polynomial layers rather than linear ones. This polynomial formulation allows MAGEP-NFN to model more complex inter-layer relationships in the input neural networks, thereby improving expressivity while maintaining low memory usage and computational efficiency.\n\nThe construction of the invariant and equivariant polynomial layers is based on linear combinations of stable polynomial terms, ensuring computational tractability. These layers are polynomials of degree at most L + 1, where L is the number of layers in the input neural networks.\n\nThe authors derive explicit forms of G-invariant layers by solving parameter-sharing constraints that ensure the network is equivariant or invariant under group actions. MAGEP-NFNs are implemented for tasks involving both equivariance (e.g., weight space style editing) and invariance (e.g., classification and generalization prediction from weights).\n\nEvaluation Criteria\n\nThe proposed MAGEP-NFNs are evaluated on three distinct tasks, each designed to test different aspects of the model’s expressivity, efficiency, and equivariance/invariance properties.\n\t1.\tClassification of Implicit Neural Representations (INRs)\n\t•\tObjective: Classify which class each pretrained INR weight was trained on.\n\t•\tDatasets: INR weights trained on MNIST, FashionMNIST, and CIFAR-10 datasets.\n\t•\tMetric: Classification accuracy (%).\n\t•\tSetup: Comparison against baseline models including NP, HNP, and Monomial-NFN over 5 runs. Results include standard error.\n\t2.\tPredicting CNN Generalization from Weights\n\t•\tObjective: Predict the generalization performance of CNNs directly from their weights, without test data evaluation.\n\t•\tDataset: Small CNN Zoo, divided into ReLU and Tanh activation subsets.\n\t•\tMetrics: Kendall’s τ correlation coefficient, assessing the rank correlation between predicted and true generalization performance.\n\t•\tSetup: Evaluation under varying group actions, including scaling augmentations sampled from different uniform distributions and permutations. Performance is reported as the mean Kendall’s τ across 5 runs with standard deviation.\n\t3.\tWeight Space Style Editing\n\t•\tObjective: Modify the weights of pretrained SIREN models to enhance contrast (CIFAR-10) or dilate images (MNIST).\n\t•\tDatasets: Pretrained SIREN models encoding CIFAR-10 and MNIST images.\n\t•\tMetric: Mean Squared Error (MSE) between the image produced by the modified network and the ground truth modified image.\n\t•\tSetup: Comparisons are made with NP and Monomial-NFN models.\n\t4.\tAblation Study on Higher-Order Terms\n\t•\tObjective: Evaluate the impact of including higher-order Inter-Layer terms ([W], [WW], [bW], [Wb]) in MAGEP-NFN.\n\t•\tMetric: Kendall’s τ on CNN generalization prediction (ReLU subset).\n\t•\tSetup: Performance comparison between models with and without Inter-Layer terms, showing improvements in expressivity.", "theoretical_claims": "1. Stable Polynomial Terms Enable Efficient Construction of Equivariant and Invariant Layers\n\t•\tClaim: Determining equivariant and invariant layers from generic polynomials over the input weights is computationally infeasible due to the complexity of identifying polynomial orbits under group actions and the high memory and computational cost.\n\t•\tContribution: The authors introduce a specialized class of polynomials called Stable Polynomial Terms. These terms are carefully designed to remain stable under the action of the monomial matrix group G, which consists of permutations and scaling/sign-flipping transformations.\n\t•\tImpact: By restricting equivariant and invariant polynomial layers to linear combinations of these stable polynomial terms, the model ensures both computational efficiency and reduced memory consumption, making it scalable and practical for NFN tasks.\n\n2. Stable Polynomial Terms Generalize Weight and Bias Entries\n\t•\tClaim: Stable polynomial terms generalize the conventional entries of weight matrices and bias vectors.\n\t•\tProposition (Proposition 3.2):\nFor all  L \\ge s > t > r \\ge 0 , stable polynomial terms satisfy the following recursive relationships:\n\n[W]^{(s,s-1)} = [W]^{(s)} \\in \\mathbb{R}^{d \\times n_s \\times n_{s-1}}\n\nand\n\n[W]^{(s,t)} \\cdot [W]^{(t,r)} = [W]^{(s,r)} \\in \\mathbb{R}^{d \\times n_s \\times n_r}\n\nSimilar relationships hold for terms involving biases.\n\t•\tImpact: These recursive definitions show that stable polynomial terms extend the concept of weights and biases, providing a richer structure that facilitates capturing more complex inter-layer dependencies in the input networks.\n\n3. Stability under Group Actions Guarantees Equivariance\n\t•\tClaim: The stable polynomial terms maintain compatibility with the group action of G, ensuring equivariance.\n\t•\tTheorem (Theorem 3.3):\nWhen the group G acts on an input U = ([W],[b]), the stable polynomial terms transform predictably under G:\n\n[gW]^{(s,t)} = g^{(s)} \\cdot [W]^{(s,t)}\n\n\n[gWgb]^{(s,t)}(t) = g^{(s)} \\cdot [Wb]^{(s,t)}(t) \\cdot (g^{(t)})^{-1}\n\nand similar rules apply to other stable terms.\n\t•\tImpact: These transformation rules guarantee that any layer built from these terms will be equivariant by design, and allow for efficient computation of equivariant and invariant polynomial layers.\n\n5. MAGEP-NFNs Polynomial Layers Include Linear Layers as a Special Case\n\t•\tClaim: The polynomial map I(U), constructed from stable polynomial terms, has a maximum degree of L + 1 in terms of the input weights. This includes linear layers as a special case.\n\t•\tImpact: This ensures that MAGEP-NFNs generalize existing linear methods (such as those in Tran et al., 2024) while offering greater expressivity through higher-order polynomial terms.", "experimental_designs_or_analyses": "The experiments are designed to evaluate the effectiveness, expressivity, and efficiency of the proposed MAGEP-NFNs across both invariant and equivariant tasks. The goal is to demonstrate that MAGEP-NFNs outperform or match baseline models while maintaining low memory consumption and computational efficiency. All experiments are conducted over five independent runs, and results are reported as the mean along with standard error or standard deviation where applicable.\n\nTask 1: Classifying Implicit Neural Representations (INRs)\n\t•\tObjective:\nPredict the class label of pretrained Implicit Neural Representation (INR) weights, which encode images from different datasets.\n\t•\tDatasets:\nINR weights trained on MNIST, FashionMNIST, and CIFAR-10.\n\t•\tBaselines:\nMLP, NP (Zhou et al., 2024b), HNP (Zhou et al., 2024b), and Monomial-NFN (Tran et al., 2024).\n\t•\tEvaluation Metric:\nClassification accuracy (%) on test sets.\n\t•\tProtocol:\nMAGEP-NFNs are trained and evaluated on the INR datasets, with test accuracy compared to baselines.\n\t•\tKey Variation:\nModels are compared in terms of their ability to generalize across different types of image representations encoded in the INRs.\n\nTask 2: Predicting CNN Generalization from Weights\n\t•\tObjective:\nPredict the generalization performance of pretrained CNNs based solely on their weights without using test data.\n\t•\tDataset:\nSmall CNN Zoo (Unterthiner et al., 2020), divided into subsets based on activation functions:\n\t•\tReLU networks (with group action M_{>0}^n)\n\t•\tTanh networks (with group action M_{\\pm1}^n)\n\t•\tBaselines:\nSTATNet, NP, HNP, and Monomial-NFN.\n\t•\tEvaluation Metric:\nKendall’s τ correlation coefficient to measure ranking agreement between predicted and true generalization scores.\n\t•\tProtocol:\n\t•\tFor ReLU CNNs, additional experiments are performed with scale augmentations, where diagonal scaling matrices D_{n, ii}^{>0} are randomly sampled from uniform distributions U[1, 10^i] for i = 1, 2, 3, 4.\n\t•\tPermutation matrices P_n are also randomly applied to assess robustness under group actions.\n\t•\tModels are evaluated both on the original and augmented datasets.\n\t•\tAnalysis:\nComparisons focus on MAGEP-NFN’s ability to maintain high Kendall’s τ scores under varying levels of input transformations.\n\nTask 3: Weight Space Style Editing\n\t•\tObjective:\nModify pretrained SIREN weights to alter the visual characteristics of the encoded images (contrast enhancement and dilation).\n\t•\tDatasets:\nPretrained SIREN models encoding CIFAR-10 and MNIST images (Zhou et al., 2024b).\n\t•\tBaselines:\nNP and Monomial-NFN.\n\t•\tEvaluation Metric:\nMean Squared Error (MSE) between images reconstructed from the modified weights and the target images (contrast-enhanced or dilated versions).\n\t•\tProtocol:\nMAGEP-NFNs are trained to perform weight space edits that correspond to desired image modifications. The quality of the edited model output is compared with baselines by calculating MSE.\n\nTask 4: Ablation Study on Higher-Order Inter-Layer Terms\n\t•\tObjective:\nEvaluate the contribution of higher-order Inter-Layer terms [W], [WW], [bW], [Wb] to model performance.\n\t•\tTask Context:\nCNN generalization prediction (ReLU subset) from Task 2.\n\t•\tEvaluation Metric:\nKendall’s τ correlation coefficient.\n\t•\tProtocol:\nTwo configurations are compared:\n\t•\tMAGEP-NFN with only Non-Inter-Layer terms\n\t•\tMAGEP-NFN with both Non-Inter-Layer and Inter-Layer terms\n\t•\tAnalysis:\nPerformance improvements (from Kendall’s τ of 0.929 to 0.933) are used to demonstrate the positive impact of including higher-order interactions in the model.", "relation_to_broader_scientific_literature": "The paper presents MAGEP-NFN, a neural function network that achieves equivariance to both permutations and scaling symmetries via an innovative parameter-sharing mechanism based on equivariant polynomial layers. This contribution is positioned within the growing body of work on symmetry-aware and equivariant neural networks.\n\nThe authors demonstrate an awareness of prior research on permutation-invariant and permutation-equivariant models, such as Deep Sets [Zaheer et al., 2017] and equivariant graph neural networks [Maron et al., 2018; Keriven and Peyré, 2019]. Additionally, their approach complements existing methods that enforce equivariance through parameter-sharing techniques [Ravanbakhsh et al., 2017]. What distinguishes this work is the incorporation of stable polynomial terms in the construction of equivariant layers—an angle not commonly explored in prior neural network architectures.\n\nHowever, the paper could benefit from a more detailed discussion of how their use of stable polynomials relates to existing literature in both machine learning and the mathematical study of stable polynomials (e.g., Borcea and Brändén, 2009). Clarifying whether this connection offers theoretical guarantees (e.g., in terms of stability, robustness, or generalization) would further strengthen the positioning of the work within the broader scientific context.\n\nMoreover, while the authors mention the potential applicability of their parameter-sharing approach to other architectures (e.g., with normalization layers or alternative activation functions), it would be helpful to reference existing frameworks that tackle such extensions, even if only to highlight distinctions or potential synergies.\n\nIn summary, the paper makes a meaningful contribution to the literature on equivariant neural networks, but a deeper engagement with relevant prior works—particularly regarding the role of stable polynomials—would enhance the discussion of its relation to broader scientific advances.", "essential_references_not_discussed": "The related work section is thorough and clearly articulated. The authors provide a well-organized overview of prior studies on functional equivalence in neural networks, neural functional networks (NFNs), and equivariant NFNs. They carefully outline the strengths and limitations of existing methods, particularly regarding permutation and scaling symmetries. This contextualization effectively highlights the gap addressed by their proposed MAGEP-NFN framework, demonstrating both a solid understanding of the literature and the relevance of their contribution.", "other_strengths_and_weaknesses": "Overall, the paper is well-written, and the proposed method is clearly presented and thoroughly evaluated. The integration of permutation and scaling equivariance into the NFN framework is a notable contribution, and the empirical results are convincing. However, one area for improvement is the theoretical grounding of the method. While the paper introduces an innovative equivariant polynomial layer, providing an additional strong theoretical result—such as a formal expressivity theorem or a rigorous analysis of the equivariant properties—would further strengthen the contribution and enhance its impact.", "other_comments_or_suggestions": "The abstract is informative and covers the key contributions of the paper; however, it feels somewhat lengthy. Condensing the abstract by focusing on the most essential points and streamlining the description of the method and results would improve readability and make the key messages more impactful.", "questions_for_authors": "In Theorem 3.5, you prove the G-invariance of the proposed equivariant polynomial layer. Could you clarify where the main technical challenges lie in establishing this invariance? Understanding the core difficulties in the proof would help appreciate the contribution more fully.\n\nIn Table 3, the performance of MAGEP-NFN appears quite similar to that of the Monomial-NFN across the evaluated tasks. Could you provide further insight into this result? For instance, are there specific scenarios or tasks where the advantages of MAGEP-NFN become more apparent, or are there other factors (e.g., efficiency, scalability) that distinguish your approach in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "blheqvshu4", "reviewer": "Reviewer_wEUN", "summary": "This article proposes the design of specific equivariant polynomial functional network.\nThe authors introduce their notations and some definitions about polynomial terms, that are transformations of the input weights. Their main result states that for some choice of this polynomial map, it is $G$-invariant where $G$ is a group of permutation and scaling of the weights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The authors present intermediate claims on stability ... Functional Neural Networks thereof.**\n\n**Answer W1.** We kindly refer the Reviewer to our response to **W1+W3** in Reviewer FUc7’s review.\n\n**W2: Also, the notation is particularly heavy, ... to clarify this.**\n\n**Answer W2.** We kindly refer the Reviewer to our response to **Q1, part d** in Reviewer HSsC’s review.\n\n**W3: The supplementary material contains ... such as ICML.**\n\n**Q1: I would strongly suggest the authors to include proof sketches as a section of the main text.**\n\n**W4: The presentation of the problem ... at least 15 times in the article.**\n\n**Answer W3+W4+Q1**. While the proofs are extensive, we believe they are essential to ensure the rigor and validity of our method. We thank the Reviewer for the helpful suggestion and will include a proof sketch in the revised version.\n\nIn broad terms, the derivation of the equivariant and invariant layers follows a two-fold structure:\n\n- Computational Derivation: Starting from the general formulation provided in Equation (10), the specific layer structures in Equation (13) are derived through explicit computation, as detailed in Appendices C and D.\n\n- Theoretical Justification: During this computational process, we address several theoretical challenges in Appendices B.3 and B.4, focusing on key properties like linear independence and basis completeness essential for capturing all equivariant and invariant formulations.\n\nIn the revision, we'll include a summary of these steps in the main text to guide readers through the logical structure, and direct them to the appendices for the complete technical details. We'll also refine the writing and references for clarity and readability. We refer to [3] multiple times throughout the paper, as the authors address the same core problem: constructing linear-based functionals with the same equivariance/invariance properties. In contrast, other related works often focus on different types of symmetries or employ alternative methods, such as graph-based approaches. Additionally, their system of notation aligns well with our approach, making it particularly suitable for introducing polynomial terms in our framework.\n\n**Q2: The authors do not confirm experimentally ... equivariant.**\n\n**Answer Q2.** Our equivariant layer is theoretically guaranteed to be equivariant by design, as it is derived through a deterministic and principled framework presented in the paper. Therefore, experimental verification of this property is not strictly necessary.\n\nHowever, for readers who prefer empirical validation over detailed proofs, we provide a simple method to do so. For the layer $E$, we randomly sample multiple input weights $U$ and group actions $g$, and check whether the equivariance condition $E(g(U))=g(E(U))$ holds. This condition will be generally satisfied, though minor deviations may occur due to rounding errors in computation. A verification code is provided in https://sites.google.com/view/polynomialfunctional-rebuttal.\n\n**Q3: Page 3 first column, ... page 1 second column.**\n\n**Answer Q3.** We appreciate the Reviewer’s observation and will revise the relevant part accordingly to address the issue.\n\n**Q4: In my opinion, for readers ... neural networks themselves.**\n\n**Answer Q4.** In the literature on neural functional networks—also known as hypernetworks or metanetworks—it is important to use precise terminology when describing models that operate on data consisting of neural network weights. Since these models are themselves neural network architectures, referring to them explicitly as neural functional networks (or hypernetworks, metanetworks) helps prevent confusion between the model and its input. We believe it is essential to distinguish these concepts through clear and consistent naming. This practice is also reflected in prior works, including but not limited to: [1], [2], [3], [4], [5], [6], [7].\n\n**Q5: How does one go from ... a neural network?**\n\n**Answer Q5.** The implementation details of the functional modules used for various tasks are provided in Appendix E. In summary, to construct equivariant functionals, we stack multiple equivariant functional layers with ReLU activations in between. For invariant functionals used in regression or classification tasks, we append an invariant functional layer to the end of the equivariant stack, followed by a final MLP.\n \n**Q6: Notations for Theorem 3.3 ... composed $s$ times?**\n\n**Answer Q6.** $g^{(s)}$ is the $s$-th component in the group element $g$. The group and its action on weight spaces are defined in Section 2 (see lines 142-164).\n\n---\nDue to space constraints, some responses refer to similar points addressed in other reviews, with references at the end of our response to Reviewer FUc7.\n\nWe appreciate the Reviewer’s feedback and hope our clarifications are satisfactory. If so, we kindly ask you to consider raising the score. We’re happy to address any further concerns in the next discussion phase.", "claims_and_evidence": "The authors present intermediate claims on stability. Although the statement seem to be technically sound, the authors do not explicit the insights they provide or how actionable they can be for the design of Neural Networks, or Functional Neural Networks thereof. \n\nAlso, the notation is particularly heavy, making the evaluation of this article particularly difficult. I would like to enter a detailed discussion phase with the authors to clarify this.", "methods_and_evaluation_criteria": "The benchmark datasets are relevant for the evaluation, although the authors do not confirm experimentally (for example with synthetic datasets) that their Functional Neural Networks are, say, equivariant.", "theoretical_claims": "I could only review the beginning of these proofs (the first 3 pages), and they are correct to me. The supplementary material contains a very large set of proofs (about 40 pages) that are difficult to entirely review for a venue such as ICML. I would recommend this aspect to be taken in account for a final decision in the paper's acceptance.\n\nGiven the above issues above the significance of the results, I would strongly suggest the authors to include proof sketches as a section of the main text, at least for a conference version of the article.", "experimental_designs_or_analyses": "Yes, the design of the experiments is sound (see additional comment in Methods And Evaluation Criteria).", "relation_to_broader_scientific_literature": "The article relates to many references studying Functional Neural Networks, and the design of Equivariant Functional Neural Networks. \nOne comment: The reference [Tran et al., 2024] is cited at least 15 times in the article.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Other weakness: The presentation of the problem that the article is addressing, and the overall writing of the article could be improved. Some references are very redundant.", "other_comments_or_suggestions": "Page 3 first column, there is an entire paragraph that is the same as another one page 1 second column.", "questions_for_authors": "- In my opinion, for readers who are not very familiar with functional neural networks, the article may gain in clarity if the results were presented only as the design of equivariant neural networks for certain group actions: it does not make a difference, to my understanding, that the inputs are weights of neural networks, except to define the symmetries that are being investigated. The results presented (at least, the theoretical ones, are not specific to $\\textit{functional neural networks}$, they are properties of neural networks themselves.\n\n- How does one go from the expressions obtained for $I(U)$ to a neural network?  \n\n- Notations for Theorem 3.3 are unclear. What is $g^{(s)}$ ? each component of $g$ s composed $s$ times?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QDZjS3IFxs", "reviewer": "Reviewer_WSKW", "summary": "A neural functional network takes (the weights and biases of) a neural net as input and predicts some related quantity, such as its expected performance. A central problem is that standard architectures exhibit a variety of symmetries, such as permuting the neurons in fully connected in each layer, or scaling their outputs. \n\nEarlier work proposed functional architectures that are invariant or equivariant to these symmetries, but in the most closely related work [Tran et al. 2024] the constructed invariant/equivariant layers are *linear* in the weights of the input network. The present paper builds on [Tran et al. 2024] to construct higher order equivariant layers. This increases the expressivity of the functional networks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: If you refer to the hyperbolic tangent as \"tanh\" I suggest you refer to \"sine\" as you \"sin\"**\n\n**Q3: I think that in lines 194 and 222 in the right hand column \"with\" should be \"which\"**\n\n**Q4: The sentence on line 118 in the right hand column seems a bit garbled**\n\n**Answer Q1+Q3+Q4**. We are grateful to the Reviewer for highlighting these points and will incorporate the appropriate edits into the manuscript.\n\n**Q2: Overall I like it that the notions used in the paper are relatively compact. However, the \"Wb\" \"bW\" \"WW\" ... here are separate, specific matrices.**\n\n**Answer Q2.** We are pleased that the Reviewer appreciated the choice of notation used in the paper. For example, the notation $[WW]^{(s,t)}$ refers to the product\n\n$[W]^{(s)} \\cdot \\ldots \\cdot [W]^{(t+1)}$ (line 178). \n\nAfter applying a group element $g$, this term becomes\n\n$[gWgW]^{(s,t)} = [gW]^{(s)} \\cdot \\ldots \\cdot [gW]^{(t+1)}$.\n\nWe did consider using $[W]^{(s,t)}$ instead of $[WW]^{(s,t)}$, but ultimately chose the latter for consistency and clarity—particularly in cases like $[Wb]$, where a one-character notation would be ambiguous or insufficient.\n\nTerms after group action, such as $[gWgW]$, $[gWgb]$, etc., can indeed be expressed more concisely as $[gWW]$, $[gWb]$, and so on. Given the notational complexity—though necessary—we will continue refining the notation to improve the overall clarity of the paper\n\n**W1: The main question is how much ... but the difference seems a little incremental.**\n\n**Q5: The main question in my mind, as mentioned above, ... are to a larger number of linear layers.**\n\n**Answer W1+Q5.** The motivation for introducing polynomial terms into the layers is to address limitations caused by parameter sharing in the construction of equivariant functionals. Specifically, enforcing equivariance through parameter sharing often forces many parameters to be zero or identical, which can significantly limit the model’s expressiveness.\n\nTheoretically, the representational capacity of the proposed MAGEP-NFN is greater than that of the Monomial-NFN layers introduced in [Tran et al., 2024]. Empirically, MAGEP-NFN also shows slightly improved performance over Monomial-NFN. However, we have not yet explored whether using a small number of higher-order polynomial layers is more effective than using a larger number of linear layers. From the perspective of representational capacity, this remains an interesting theoretical question, and we plan to investigate it in future work.\n\n**Q6: How many of these layers is it reasonable to use an actual functional net?**\n\n**Answer Q6.** We believe the appropriate number of layers depends on the specific task being considered. For example, in the CNN generalization prediction task, we found that two layers of $E(U)$ followed by one layer of $I(U)$ are sufficient. In contrast, for the INR classification task, we use three layers of $E(U)$ followed by one layer of $I(U)$. Across all experiments, we ensure a fair comparison by keeping the number of parameters comparable across all baselines. Further details can be found in Appendix E.\n\n**Q7: When you mention that the framework is applicable to CNNs as well, do you mean that neglecting the x-y dimensions in the image plane, the way that the CNN mixes channels is actually just like a fully connected network and you can apply the same formalism to it for this reason?**\n\n**Answer Q7.** For CNNs, the weight space described in Eq. (1) includes distinct $w_i$'s and $b_i$'s, in contrast to MLPs where all $w_i$'s and $b_i$'s are set to 1. To normalize these differences, we first applied a Monomial-NFN model from [Tran et al., 2024]—which shares the same structure as our proposed layers but without the polynomial terms—to rescale the values. This was followed by applying the MAGEP-NFN layers. This is mentioned in the implementation details (lines 2684–2690).\n\n**Q8: [Tran et al., 2024] also talks about sign-flipping symmetry. Does that have to be sacrificed in the polynomial case?**\n\n**Answer Q8.** Our method is applicable to sign-flipping symmetries. In fact, the layers computed in the paper are themselves sign-flipping equivariant or invariant. However, to ensure that the overall functional—composed of these layers—remains equivariant or invariant under sign-flipping, it is necessary to use odd activation functions. This requirement is consistent with the approach in [Tran et al., 2024].\n\n---\n\nWe thank the Reviewer for the constructive feedback. If the Reviewer finds our clarifications satisfactory, we kindly ask you to consider raising the score. We would be happy to address any further concerns during the next stage of the discussion.", "claims_and_evidence": "The theoretical claims are well laid out and proved in theorems. The empirical evaluation also appears sound.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proofs appear to be correct but I did not check them line by line.", "experimental_designs_or_analyses": "Experiments seem okay.", "relation_to_broader_scientific_literature": "The authors do a fairly good job of situation their contribution in the wider literature. [Tran et al., 2024] is obviously very closely related and they are upfront about that. Both the setup and the proof strategies appear similar, but the present paper is obviously more general because it considers higher order polynomial invariant/equivariant layers too, not just linear ones. Following [Tran et al., 2024] the present paper's surprise value is limited, but it is still a nice contribution because it carefully derives the form of higher order polynomial layers.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "The main question is how much of a difference it makes to have higher order layers rather than a potentially larger number of linear layers. The authors do present an ablation study, but the difference seems a little incremental. This does not diminish the theoretical significance of the mathematical results presented in the paper, performance differences between competing neural architectures are often very small.", "other_comments_or_suggestions": "- if you refer to the hyperbolic tangent as \"tanh\" I suggest you refer to \"sine\" as you \"sin\"\n- Overall I like it that the notions used in the paper are relatively compact. However, the \"Wb\" \"bW\" \"WW\" notation might be taking things a bit too far because they can be confused with the actual product of W and b, etc.. As far as I understand, \"Wb\", \"bW\" (indexed by some other parameters) are individual matrices. It gets worse when the group action comes in and the authors use notation such as \"gWgW\" which really suggests that the two \"W\"'s here are separate, specific matrices.\n- I think that in lines 194 and 222 in the right hand column \"with\" should be \"which\"\n- the sentence on line 118 in the right hand column seems a bit garbled", "questions_for_authors": "- The main question in my mind, as mentioned above, is what the practical benefit of higher order polynomial layers are to a larger number of linear layers.\n- How many of these layers is it reasonable to use an actual functional net?\n- When you mention that the framework is applicable to CNNs as well, do you mean that neglecting the x-y dimensions in the image plane, the way that that the CNN mixes channels is actually just like a fully connected network and you can apply the same formalism to it for this reason?\n- [Tran et al., 2024] also talks about sign-flipping symmetry. Does that have to be sacrificed in the polynomial case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contributes to the growing literature on neural functional networks by extending the framework of Tran et al. (2024) to include polynomial equivariant layers, enhancing formal expressivity under symmetry constraints.

Reviewers generally recognize the strength of the theoretical contribution, supported by sound empirical evaluation and a principled generalization of prior work. While some noted that the practical gains over linear layers appear incremental, there was consensus that the paper offers a rigorous and well-motivated theoretical advancement. The authors’ rebuttal effectively clarified key concerns, and with the promised revisions, the paper merits acceptance, primarily on the strength of its theoretical insights.

The authors are encouraged to incorporate the important feedback given by the knowledgeable reviewers, particularly regarding notational clarity and the broader discussion of practical implications relative to deeper linear models.
Decision: Accept (poster)