Equivariant Polynomial Functional Networks

## Abstract

Abstract Neural Functional Networks (NFNs) have gained increasing interest due to their wide range of applications, including extracting information from implicit representations of data, editing network weights, and evaluating policies. A key design principle of NFNs is their adherence to the permutation and scaling symmetries inherent in the connectionist structure of the input neural networks. Recent NFNs have been proposed with permutation and scaling equivariance based on either graph-based message-passing mechanisms or parameter-sharing mechanisms. However, graph-based equivariant NFNs suffer from high memory consumption and long running times. On the other hand, parameter-sharing-based NFNs built upon equivariant linear layers exhibit lower memory consumption and faster running time, yet their expressivity is limited due to the large size of the symmetric group of the input neural networks. The challenge of designing a permutation and scaling equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the development of MAGEP-NFN ( M onomial m A trix G roup E quivariant P olynomial NFN ). Our approach follows the parameter-sharing mechanism but differs from previous works by constructing a nonlinear equivariant layer represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, enhancing the model’s expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demonstrating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.

## 1 Introduction

Deep neural networks (DNNs) have emerged as highly adaptable modeling tools, applicable in a wide range of fields such as Natural Language Processing [, , , ], Computer Vision [, , ], and the Natural Sciences [] [].
There is an increasing interest in creating specialized neural networks designed to handle the weights, gradients, or sparsity masks of deep neural networks (DNNs) as data.
These specialized networks are known as neural functional networks (NFNs) [].
NFNs have a wide range of applications, including developing learnable optimizers for training neural networks [, , , ], extracting information from implicit representations of data [, , ], performing corrective editing of network weights [, , ], evaluating policies [], and carrying out Bayesian inference using networks as evidence [].

The design of NFNs poses inherent challenges due to their intricate high-dimensional structures.
Early strategies to mitigate this issue relied on a constrained training process, effectively limiting the weight space [, , ].
In contrast, recent methods have concentrated on developing permutation equivariant NFNs that can handle neural network weights without such limitations [, , , ].
These approaches have introduced NFNs that are equivariant to weight permutations, which correspond to the rearrangement of neurons within hidden layers.
Nonetheless, many of these methods often neglect other important symmetries present in weight spaces [, ], including weight scaling transformations for $\operatorname{ReLU}$ networks [, , ], and sign flipping transformations for $\operatorname{sin}$ and $\operatorname{tanh}$ networks [, , ].

NFNs that are equivariant to both permutations and scaling or sign-flipping have been introduced in [] using a graph-based message-passing mechanism and in [] with a parameter sharing mechanism. However, similar to other graph-based neural functional networks, treating the entire input neural network as a graph and utilizing graph neural networks causes the graph-based equivariant NFNs in [] to have very high memory consumption and running time. In contrast, the NFNs built upon equivariant linear layers using the parameter sharing mechanism in [] exhibit much lower memory consumption and running time. Nevertheless, the equivariant linear layers introduced in [] possess weak expressive properties, as the weights of the input hidden layers are updated solely by the corresponding weights of the same input hidden layers. The challenge of designing an equivariant layer based on the parameter-sharing mechanism that maintains both lower memory consumption and running time while preserving expressivity remains unresolved.

##### Contribution.

This paper aims to develop a novel NFN that is equivariant to both permutations and scaling/sign-flipping symmetries, called MAGEP-NFN (Monomial mAtrix Group Equivariant Polynomial NFN). We follow the parameter-sharing mechanism as described in []; however, unlike [], we construct a nonlinear equivariant layer, which is represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, thereby addressing the challenges posed in [] and enhancing the expressivity of MAGEP-NFN.
In particular, our contribution is as follows:

- 1.
We introduce new polynomials in the input weights, called *stable polynomial terms*, such that these polynomials are stable under the group action of the weight space.
- 2.
We conduct a comprehensive study of the linear independence of stable polynomial terms, which are essential for the parameter-sharing computations of equivariant polynomial layers defined as their linear combinations.
- 3.
We design MAGEP-NFN, a family of monomial matrix equivariant NFNs based on the parameter-sharing mechanism that maintains both lower memory consumption and running time while preserving expressivity.
The main building blocks of MAGEP-NFN are the equivariant and invariant polynomial layers for processing weight spaces.

We evaluate MAGEP-NFNs on three tasks: predicting CNN generalization from weights using Small CNN Zoo [], weight space style editing, and classifying INRs using INRs data [].
Experimental results show that our model achieves competitive performance and efficiency compared to existing baselines.

##### Organization.

After recalling some related works in Section [2](#S2), we reformulate the definitions of weight spaces for MLPs and CNNs, as well as the action of monomial matrix groups on these weight spaces. In Section [4](#S4), we construct polynomial equivariant and invariant layers, which serve as the main building blocks for our MAGEP-NFNs. Several experiments are conducted in Section [5](#S5) to verify the applicability and efficiency of our models in comparison with previous ones in the literature. The paper concludes with a summary in Section [6](#S6).

## 2 Related Work

Symmetries of Weight Spaces. The problem of determining symmetries in the weight spaces of neural networks, which corresponds to evaluating the functional equivalence of these networks, is a well-established area of research [, , , , ].
This problem was initially introduced by Hecht-Nielsen in [].
Subsequent studies have produced results for various network architectures in [, , , , , ].

Neural Functional Networks. Recent research has concentrated on developing representations for trained classifiers to assess their generalization performance and gain insights into neural network behavior [, , , , , ]. Notably, low-dimensional encodings for Implicit Neural Representations (INRs) have been created to facilitate various downstream tasks [, ].
These approaches often involve either flattening the network parameters or computing parameter statistics for processing with standard MLPs [].
Schurholt et al. proposed neuron permutation augmentations []. Other research has focused on encoding and decoding neural network parameters primarily for reconstruction and generative purposes [, , , ].

Equivariant NFNs. Recent research has made significant progress in overcoming the limitations of permutation equivariant neural networks by introducing permutation equivariant layers. These layers utilize complex weight-sharing patterns [, , , ], or set-based [],
or graph-based structure of the network’s weights [, , ]
to ensure equivariance. They facilitate the construction of both permutation invariant and equivariant networks, thereby preserving the permutation symmetry inherent in neural networks that arises from reordering neurons within each layer.
NFNs that are equivariant to both permutations and scaling, which are called monomial equivariant NFNs, have been introduced in [] using a graph-based message-passing mechanism and in [] with a parameter sharing mechanism.

## 3 Weight Space and Group Action on Weight Space

Let $\mathcal{U},\mathcal{V}$ be two sets and assume that a group $G$ acts on them.
A function $f\colon\mathcal{U}\to\mathcal{V}$ is called $G$-equivariant if $f(g\cdot x)=g\cdot f(x)$ for all $x\in\mathcal{U}$ and $g\in G$.
In case $G$ acts trivially on $\mathcal{Y}$, the function $f$ is called $G$-invariant.
In the context of this paper, $\mathcal{U}$ and $\mathcal{V}$ are weight spaces of a fixed neural network architecture, while $\mathcal{G}$ is a direct product of the groups of monomial matrices.

### 3.1 Monomial Matrix Group

Let us start with the definition of monomial and permutation matrices.

###### Definition 3.1 (See [ 47 , page 46] ) .

Let $n$ be a positive integer.

- •
A matrix $M\in\operatorname{GL}_{n}(\mathbb{R})$ is called a *monomial matrix* (or *generalized permutation matrix*) if it has exactly one non-zero entry in each row and each column, and zeros elsewhere. We denote by $\mathcal{M}_{n}$ the set of such all monomial matrices.
- •
A matrix $P\in\operatorname{GL}_{n}(\mathbb{R})$ is called a *permutation matrix* if it is a monomial matrix and all nonzero entries are equal to $1$. We denote by $\mathcal{P}_{n}$ the set of such all permutation matrices.

###### Remark (Semidirect product) .

Let $\mathcal{D}_{n}$ the set of diagonal matrices in $\operatorname{GL}_{n}(\mathbb{R})$.
Then, $\mathcal{M}_{n}$, $\mathcal{P}_{n}$ and $\mathcal{D}_{n}$ are subgroups of of the general linear group $\operatorname{GL}_{n}(\mathbb{R})$.
In addition, it is well-known that
$\mathcal{M}_{n}=\mathcal{D}_{n}\rtimes\mathcal{P}_{n}$, which is a semidirect product of $\mathcal{D}_{n}$ and $\mathcal{P}_{n}$ (see []).
In particular, every monomial matrix can be written as a product of an invertible diagonal matrix and a permutation matrix.
For convenience, we sometimes denote an element of $\mathcal{M}_{n}$ as a pair $(D,P)$, or a product $D\cdot P$, of a diagonal matrix $D$ and a permutation matrix $P$.

###### Remark (Permutation matrix vs permutation) .

For every permutation matrix $P\in\mathcal{P}_{n}$, there exists a unique permutation $\pi\in\mathcal{S}_{n}$ such that $P$ is obtained by permuting the $n$ columns of the identity matrix $I_{n}$ according to $\pi$. In this case, we write $P:=P_{\pi}$ and call it the permutation matrix corresponding to $\pi$. Here, $\mathcal{S}_{n}$ is the group of all permutations of the set $\{1,2,\ldots,n\}$.
In particular, for permutation matrices $P_{\pi}\in\mathcal{P}_{n}$ and $P_{\sigma}\in\mathcal{P}_{m}$, we have

$$ $\displaystyle P_{\pi}\cdot\mathbf{x}=\left(x_{\pi^{-1}(1)},x_{\pi^{-1}(2)},\ldots,x_{\pi^{-1}(n)}\right)^{\top},$ (1) $$

and

$$ $\displaystyle\left(\ P_{\pi}\cdot A\cdot P_{\sigma}\right)_{ij}=A_{\pi^{-1}(i)\sigma(j)},$ (2) $$

for all vector $\mathbf{x}\in\mathbb{R}^{n}$ and matrix $A\in\mathbb{R}^{n\times m}$.

###### Definition 3.1 (See [ 47 , page 46] ) .

###### Remark (Semidirect product) .

###### Remark (Permutation matrix vs permutation) .

### 3.2 Weight space of MLPs and CNNs

An MLP with $L$ layers, $n_{i}$ neurons at the $i$-th layer ($i=0,\ldots,L$), together with an activation $\sigma$ defines a function $f\colon\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{L}}$ as:

$$ $f(\mathbf{x}~{};~{}U,\sigma)=W^{(L)}\cdot\sigma\left(\ldots\sigma\left(W^{(2)}\cdot\sigma\left(W^{(1)}\cdot\mathbf{x}+b^{(1)}\right)+b^{(2)}\right)\ldots\right)+b^{(L)}.$ (3) $$

Here, $U=(W,b)$ is the parameters with the weights

$$ $\displaystyle W=\left(W^{(L)},\ldots,W^{(1)}\right)\in\mathbb{R}^{n_{L}\times n_{L-1}}\times\ldots\times\mathbb{R}^{n_{2}\times n_{1}}\times\mathbb{R}^{n_{1}\times n_{0}}=:\mathcal{W},$ (4) $$

and biases

$$ $\displaystyle b=\left(b^{(L)},\ldots,b^{(1)}\right)\in\mathbb{R}^{n_{L}\times 1}\times\ldots\times\mathbb{R}^{n_{2}\times 1}\times\mathbb{R}^{n_{1}\times 1}=:\mathcal{B}.$ (5) $$

We denote by $\mathcal{U}=\mathcal{W}\times\mathcal{B}$ the weight space of the given MLP.

A CNN, with $L$ convolutional layers together with activation $\sigma$, defines a function

$$ $f(\mathbf{x};U,\sigma)=\sigma\left(W^{(L)}\ast\sigma\left(\ldots\sigma\left(W^{(2)}\ast\sigma\left(W^{(1)}\ast\mathbf{x}+b^{(1)}\right)+b^{(2)}\right)\ldots\right)+b^{(L)}\right).$ (6) $$

Here, $U=(W,b)$ is the learnable parameters with the weights

$$ $\displaystyle W=\left(W^{(L)},\ldots,W^{(1)}\right)\in\mathbb{R}^{w_{L}\times n_{L}\times n_{L-1}}\times\ldots\times\mathbb{R}^{w_{2}\times n_{2}\times n_{1}}\times\mathbb{R}^{w_{1}\times n_{1}\times n_{0}},$ (7) $$

and the biases

$$ $\displaystyle b=\left(b^{(L)},\ldots,b^{(1)}\right)\in\mathbb{R}^{1\times n_{L}\times 1}\times\ldots\times\mathbb{R}^{1\times n_{2}\times 1}\times\mathbb{R}^{1\times n_{1}\times 1}.$ (8) $$

The positive numbers $n_{i}$ and $w_{i}$ represent the number of channels and the size of the convolutional kernel at the $i$-th convolutional layer.
Additionally, by adding $b$, we mean adding $b^{(i)}_{j}$ to all entries of the $j$-th channel at the $i$-th layer. We denote by $\mathcal{U}=\mathcal{W}\times\mathcal{B}$ the weight space of the considered CNN.

It is noted that, in addition to the convolutional layers, the standard design of a CNN includes average pooling layers and fully connected layers at the end of the network. However, we only consider the convolutional layers since the weight space of the fully connected layers has already been discussed above, and the pooling layers have no learnable parameters.

##### Weight Spaces.

We can write the weight space of an MLP or CNN with $L$ layers and $n_{i}$ channels at $i$-th layer in the general form $\mathcal{U}=\mathcal{W}\times\mathcal{B}$, where:

$$ $\displaystyle\begin{array}[]{lllll}\mathcal{W}&=\hskip 10.0pt\mathbb{R}^{w_{L}\times n_{L}\times n_{L-1}}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{w_{2}\times n_{2}\times n_{1}}&\times\hskip 10.0pt\mathbb{R}^{w_{1}\times n_{1}\times n_{0}},\\ \mathcal{B}&=\hskip 10.0pt\mathbb{R}^{b_{L}\times n_{L}\times 1}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{b_{2}\times n_{2}\times 1}&\times\hskip 10.0pt\mathbb{R}^{b_{1}\times n_{1}\times 1}.\end{array}$ (11) $$

Here,
$n_{i}$ is the number of channels at the $i$-th layer,
in particular, $n_{0}$ and $n_{L}$ are the number of channels of input and output;
$w_{i}$ is the dimension of weights and $b_{i}$ is the dimension of the biases in each channel at the $i$-th layer.
The dimension of the weight space $\mathcal{U}$ is:

$$ $\dim{\mathcal{U}}=\sum_{i=1}^{L}\left(w_{i}\times n_{i}\times n_{i-1}+b_{i}\times n_{i}\times 1\right).$ (12) $$

To emphasize the weights and biases of an element $U$ of $\mathcal{U}$, we will write $U=([W],[b])$, with the weights

$$ $\displaystyle[W]=\left([W]^{(L)},\ldots,[W]^{(1)}\right)\in\mathcal{W},$ (13) $$

and biases

$$ $\displaystyle[b]=\left([b]^{(L)},\ldots,[b]^{(1)}\right)\in\mathcal{B}.$ (14) $$

The square brackets will be convenient in the next section when we determine polynomials in the entries of $U$.

###### Remark .

The space $\mathbb{R}^{w_{i}\times n_{i}\times n_{i-1}}=(\mathbb{R}^{w_{i}})^{n_{i}\times n_{i-1}}$ at the $i$-th layer is interpreted as the space of $n_{i}\times n_{i-1}$ matrices, where each entry consists of real vectors in $\mathbb{R}^{w_{i}}$. Specifically, the symbol $[W]^{(i)}$ represents a matrix in $\mathbb{R}^{w_{i}\times n_{i}\times n_{i-1}}=(\mathbb{R}^{w_{i}})^{n_{i}\times n_{i-1}}$, with $[W]^{(i)}_{jk}\in\mathbb{R}^{w_{i}}$ indicating the entry located at row $j$ and column $k$ of $[W]^{(i)}$.
Similarly, the term $[b]^{(i)}$ denotes a bias column vector in $\mathbb{R}^{b_{i}\times n_{i}\times 1}=(\mathbb{R}^{b_{i}})^{n_{i}\times 1}$, while $[b]^{(i)}_{j}\in\mathbb{R}^{b_{i}}$ denotes the entry at row $j$ of the vector $[b]^{(i)}$.
The dimensions of weights $w_{i}$ and biases $b_{i}$ at each channel do not affect the definition of group action in the subsequent sections.

###### Remark .

### 3.3 Action of monomial matrix group on a weight space

The monomial matrix group $\mathcal{M}_{n}$ acts on the left and the right of $\mathbb{R}^{n}$ and $\mathbb{R}^{n\times m}$ in a canonical way (by matrix-vector or matrix-matrix multiplications).
This action induces a canonical action of the larger monomial matrix group

$$ $\displaystyle\mathcal{G}_{\mathcal{U}}:=\mathcal{M}_{n_{L}}\times\mathcal{M}_{n_{L-1}}\times\ldots\times\mathcal{M}_{n_{0}},$ (15) $$

to on the weight space $\mathcal{U}$ of an MLP or CNN defined Equation ([11](#S3.E11)).
Each element $g\in\mathcal{G}_{\mathcal{U}}$ has the form

$$ $g=\left(g^{(L)},\ldots,g^{(0)}\right),$ $$

where $g^{(i)}=D^{(i)}\cdot P_{\pi_{i}}$ for some diagonal matrix $D^{(i)}=\operatorname{diag}(d^{(i)}_{1},\ldots,d^{(i)}_{n_{i}})$ in $\mathcal{D}_{n_{i}}$ and permutation $\pi_{i}\in\mathcal{S}_{n_{i}}$.
The action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$ is defined formally as follows.

###### Definition 3.2 (Group action on weight spaces) .

With the notation as above, the group action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$ is defined to be the map $\mathcal{G}_{\mathcal{U}}\times\mathcal{U}\to\mathcal{U}$ with $(g,U)\mapsto gU=([gW],[gb])$, where:

$$ $\left[gW\right]^{(i)}\coloneqq\left(g^{(i)}\right)\cdot[W]^{(i)}\cdot\left(g^{(i-1)}\right)^{-1}~{}\text{ and }~{}\left[gb\right]^{(i)}\coloneqq\left(g^{(i)}\right)\cdot[b]^{(i)}.$ (16) $$

Or equivalently,

$$ $[gW]^{(i)}_{jk}\coloneqq\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot[W]^{(i)}_{\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}~{}\text{ and }~{}[gb]^{(i)}_{j}\coloneqq d^{(i)}_{j}\cdot[b]^{(i)}_{\pi_{i}^{-1}(j)}.$ (17) $$

In this section, we formally describe a subgroup $G$ of
$\mathcal{G}_{\mathcal{U}}$ such that its action on $\mathcal{U}$ satisfies the condition: elements of $\mathcal{U}$ in the same orbit under the action of $G$ define the same function.

From now on, we will fix the activation $\sigma$ to be the rectified linear unit $\sigma=\operatorname{ReLU}$. The case when $\sigma$ is another typical activation, such as semilinear (e.g., $\operatorname{LeakyReLU}$) or odd (e.g., $\operatorname{sin},\operatorname{tanh}$), can be derived similarly.

To determine the group $G$, we observe that:

- •
Since the neurons in a hidden layer $i=1,\ldots,L-1$ have no inherent ordering, the network is invariant to the symmetric group $\mathcal{S}_{n_{i}}$ of permutations of the neurons in the $i$-th layer. This results in the symmetries of permutation type of $\mathcal{U}$.
- •
Since $\sigma(\lambda x)=\lambda\sigma(x)$ for all positive numbers $\lambda$ and real numbers $x$, multiplying the bias and all incoming weights at a neuron of the MLP by the same positive number $\lambda$ leads to scaling its output by $\lambda$. This results in the symmetries of scaling type of $\mathcal{U}$.

Based on the above observation, we define the group $G$ of the form

$$ $\displaystyle G:=\{I_{n_{L}}\}\times\mathcal{M}^{>0}_{n_{L-1}}\times\ldots\times\mathcal{M}^{>0}_{n_{1}}\times\{I_{n_{0}}\},$ (18) $$

where $I_{n}$ is the identity matrix of size $n\times n$, $\mathcal{M}_{n}^{>0}=\mathcal{D}_{n}^{>0}\rtimes\mathcal{P}_{n}$ which is the semidirect product of $\mathcal{D}_{n}^{>0}$ and $\mathcal{P}_{n}$, and $\mathcal{D}_{n}^{>0}$ is the group of invertible diagonal matrices whose the nonzero entries are all positive.

###### Remark .

Intuitively speaking, $\mathcal{M}_{n}^{>0}$ is generated by permuting and positive scaling the coordinates of vectors in $\mathbb{R}^{n}$.
It is well-known that $\mathcal{M}_{n}^{>0}$ is the maximal intertwiner group of semilinear activation functions, such as: $\operatorname{ReLU}$ and $\operatorname{LeakyReLU}$.
(See [, ] for the definition of intertwiner group of an activation and the proofs).

###### Proposition 3.3 ( $G$ -equivariance of neural functionals) .

With notation as above, let $f=f(\cdot\,;\,U,\sigma)$ be an MLP given in Equation ([3](#S3.E3)) or CNN given in Equation ([6](#S3.E6)) with the weight space $U\in\mathcal{U}$ and an activation $\sigma=\operatorname{ReLU}$.
Then $f$ is $G$-invariant under the action of $G$ on its weight space, i.e.

$$ $f(\mathbf{x}~{};~{}U,\sigma)=f(\mathbf{x}~{};~{}gU,\sigma)$ (19) $$

for all $g\in G,\,U\in\mathcal{U}$ and $\mathbf{x}\in\mathbb{R}^{n_{0}}$.

###### Remark .

The proof of the above proposition can be found in [, ].
This proposition still holds for another semilinear activation functions (such as $\operatorname{LeakyReLU}$).
In case $\sigma$ is an odd activation function (such as $\operatorname{sin}$ or $\operatorname{tanh}$), the scaling symmetry must be replaced by the sign-flipping symmetry. As a consequence, the component $\mathcal{M}_{n_{i}}^{>0}$ in the definition of $G$ in Equation ([18](#S3.E18)) must be replaced by $\mathcal{M}_{n_{i}}^{\pm 1}$, which is the set of all monomial matrices whose nonzero entries are all equal to $\pm 1$ (see [, ]).

###### Definition 3.2 (Group action on weight spaces) .

###### Remark .

###### Proposition 3.3 ( G 𝐺 G -equivariance of neural functionals) .

###### Remark .

## 4 Equivariant and Invariant Polynomial Functional Networks

In this section, we introduce a new family of NFNs, called monomial matrix group equivariant polynomial neural functionals (MAGEP-NFNs).
These NFNs are equivariant to the monomial matrix group described in Proposition [3.3](#S3.Thmtheorem3).
The equivariant and invariant layers, which are the main building blocks of MAGEP-NFNs, will be presented in Subsections [4.2](#S4.SS2) after we introduce stable polynomial terms in Subsection [4.1](#S4.SS1).

We follow the parameter-sharing mechanism in the construction of the equivariant and invariant layers from $\mathcal{U}$ as in [] to keep our model having low memory consumption and running time. However, in contrast to the use of linear layers in [], we will use polynomial layers, which allow us to incorporate further relations of weights from different hidden layers of the input network, thus guaranteeing the expressivity of our model. In order to do that, we will need to determine certain polynomials in the input weights such that they are ”stable” under the action of $G$. These ”stable” polynomials will be determined formally in the following subsection.

### 4.1 Stable polynomial terms

Intuitively, a *stable polynomial term* is a polynomial in the entries of $U\in\mathcal{U}$ such that it is “stable” under the action of $G$ (see Definition [4.1](#S4.Thmtheorem1) below).
These terms are the main components of our equivariant and invariant polynomial layers in the next subsections.

Recall that $\mathcal{U}=(\mathcal{W},\mathcal{B})$ is the weight space with $L$ layers, $n_{i}$ channels at $i^{\text{th}}$ layer, and the dimensions of weight and bias are $w_{i}$ and $b_{i}$, respectively (see Equation ([11](#S3.E11))).
While the symmetries of the weight space is given by the group

$$ $G=\{I_{n_{L}}\}\times\mathcal{M}^{>0}_{n_{L-1}}\times\ldots\times\mathcal{M}^{>0}_{n_{1}}\times\{I_{n_{0}}\}.$ $$

Consider the case where the weight spaces have the same number of dimensions across all channels, which means $w_{i}=b_{i}=d$ for all $i$.

###### Definition 4.1 (Stable polynomial terms) .

Let $U=([W],[b])$ be an element of $\mathcal{U}$ with weights $[W]=\left([W]^{(L)},\ldots,[W]^{(1)}\right)$ and biases
$[b]=\left([b]^{(L)},\ldots,[b]^{(1)}\right)$.
For each $L\geqslant s>t\geqslant 0$, we define:

$$ $\begin{array}[]{lll}[W]^{(s,t)}&\coloneqq[W]^{(s)}\cdot[W]^{(s-1)}\cdot\ldots\cdot[W]^{(t+1)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}},\\ \left[Wb\right]^{(s,t)(t)}&\coloneqq[W]^{(s,t)}\cdot\left[b\right]^{(t)}&\in\mathbb{R}^{d\times n_{s}\times 1}.\end{array}$ (20) $$

In addition, for each $L\geqslant s,t\geqslant 0$, and matrices $\Psi^{(s)(L,t)}\in\mathbb{R}^{1\times n_{L}}$ and $\Psi^{(s,0)(L,t)}\in\mathbb{R}^{n_{0}\times n_{L}}$, we also define

$$ $\begin{array}[]{lll}\left[bW\right]^{(s)(L,t)}&\coloneqq[b]^{(s)}\cdot\Psi^{(s)(L,t)}\cdot[W]^{(L,t)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}},\\ \left[WW\right]^{(s,0)(L,t)}&\coloneqq[W]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[W]^{(L,t)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}}.\end{array}$ (21) $$

The entries of the matrices $[W]^{(s,t)}$, $\left[Wb\right]^{(s,t)(t)}$, $\left[bW\right]^{(s)(L,t)}$ and $\left[WW\right]^{(s,0)(L,t)}$ defined above are called *stable polynomial terms* of $U$ under the action of $G$.

In the above definition, we use the notation $[W]$ and $[WW]$ to denote products of the weight matrices $[W]^{(i)}$ with the appropriate index $i$. The notation $[Wb]$ indicates that this is a product of several weight matrices $[W]^{(i)}$ and a bias vector $[b]^{(j)}$, with appropriate indices $i$ and $j$.
For the indices, we use the notation $(s,t)$ to signify that the considered product contains weight matrices with indices ranging from $s$ down to $t+1$. When the index has two components, for example $[Wb]^{(s,t)(t)}$, the first component $(s,t)$ specifies the range of indices for $[W]$, while the second component $(t)$ indicates the index of the bias vector $[b]$.
Specifically, the last two terms $\left[bW\right]^{(s)(L,t)}$ and $\left[WW\right]^{(s,0)(L,t)}$ contain the matrices $\Psi^{-}$ to multiply two matrices of different sizes from the left and the right.

The following proposition shows that the stable polynomial terms defined above can be viewed as a generalization of the entries of the weight matrices $[W]^{(i)}$ and bias vectors $[b]^{(i)}$.

###### Proposition 4.2 (Stable polynomial terms as generalization of weights and biases) .

With notation as above, then for all $L\geqslant s>t>r\geqslant 0$, we have

$$ $\displaystyle{\begin{array}[]{lll}[W]^{(s,s-1)}&=[W]^{(s)}&\in\mathbb{R}^{d\times n_{s}\times n_{s-1}},\\ \left[W\right]^{(s,t)}\cdot[W]^{(t,r)}&=[W]^{(s,r)}&\in\mathbb{R}^{d\times n_{s}\times n_{r}}.\\ \left[bW\right]^{(s)(s,t)}\cdot[W]^{(t,r)}&=[bW]^{(s,r)}&\in\mathbb{R}^{d\times n_{s}\times n_{r}},\\ \left[W\right]^{(s,t)}\cdot[Wb]^{(t,r)}&=[Wb]^{(s,r)}&\in\mathbb{R}^{d\times n_{s}\times n_{r}}.\end{array}}$ $$

The stable polynomial terms defined above are actually “stable” under the action of $G$ in the sense presented in the following theorem.

###### Theorem 4.3 (Stable polynomial terms are “stable”) .

With notation as above, let $gU=([gW],[gb])$ be the element of $\mathcal{U}$ obtained by acting $g=(g^{(L)},\ldots,g^{(0)})\in G$ on $U=([W],[b])$.
Then we have

$$ $\displaystyle{\begin{array}[]{ll}\left[gW\right]^{(s,t)}&=\left(g^{(s)}\right)\cdot\left[W\right]^{(s,t)}\cdot\left(g^{(t)}\right)^{-1},\\ \left[gb\right]^{(s)}&=\left(g^{(s)}\right)\cdot[b]^{(s)},\\ \left[gWgb\right]^{(s,t)(t)}&=\left(g^{(s)}\right)\cdot[Wb]^{(s,t)(t)},\\ \left[gbgW\right]^{(s)(L,t)}&=\left(g^{(s)}\right)\cdot[bW]^{(s)(L,t)}\cdot\left(g^{(t)}\right)^{-1},\\ \left[gWgW\right]^{(s,0)(L,t)}&=\left(g^{(s)}\right)\cdot[WW]^{(s,0)(L,t)}\cdot\left(g^{(t)}\right)^{-1}.\end{array}}$ $$

The proofs of Proposition [4.2](#S4.Thmtheorem2) and Theorem [4.3](#S4.Thmtheorem3) can be found in Appendix [B](#A2).

###### Definition 4.1 (Stable polynomial terms) .

###### Proposition 4.2 (Stable polynomial terms as generalization of weights and biases) .

###### Theorem 4.3 (Stable polynomial terms are “stable”) .

### 4.2 Polynomial Invariant and Equivariant Layers

We now proceed to construct $G$-invariant polynomial layers.
The construction of $G$-equivariant polynomial layers is similar and will be derived in detail in Appendix [C](#A3).
These polynomial layers serve as the fundamental building blocks for our MAGEP-NFNs.

We define a polynomial map $I\colon\mathcal{U}\rightarrow\mathbb{R}^{d^{\prime}}$ with maps each element $U\in\mathcal{U}$ to the vector $I(U)\in\mathbb{R}^{d^{\prime}}$ of the form

$$ $\displaystyle I(U)\coloneqq$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}+\Phi_{1}.$ (22) $$

In particular, $I(U)$ is a linear combination of the entries from the input weights $[W]^{(s)}$ and biases $[b]^{(s)}$, as well as all entries from the stable polynomial terms $[W]^{(s,t)}$, $[Wb]^{(s,t)(t)}$, $[bW]^{(s)(s,t)}$, and $[WW]^{(s,L)(0,t)}$ for all appropriate indices $s$ and $t$.
Here, the coefficients $\Phi_{-}$ and the connection matrix $\Psi^{-}$ (inside $[bW]^{(s)(s,t)}$ and $[WW]^{(s,L)(0,t)}$) are learnable parameters.

To make $I$ to be $G$-invariant,
the learnable parameters $\Phi_{-}$ and $\Psi^{-}$ must satisfy a system of constraints (usually called *parameter sharing*), which are induced from the condition $I(gU)=I(U)$ for all $g\in G$ and $U\in\mathcal{U}$.
We show in details what are these constraints and how to derive the concrete formula of $I$ in Appendix [C](#A3).
The formula of $I$ is then determined by

$$ $\displaystyle I(U)=$ $\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$ $\displaystyle+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\Phi_{(s,0)(L,s):\bullet\bullet}\cdot[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$ $\displaystyle+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\sum_{p=1}^{n_{t}}\Phi_{(t)(L,t):\bullet\bullet}\cdot[bW]^{(t)(L,t)}_{pp}$ $\displaystyle+\sum_{p=1}^{n_{L}}\Phi_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}.$ (23) $$

In the above formula, the bullet $\bullet$ indicates that the value of the corresponding coefficient is independent of the index at the bullet.

To conclude, we have:

###### Theorem 4.4 .

With notation as above, the polynomial map $I\colon\mathcal{U}\rightarrow\mathbb{R}^{d^{\prime}}$ defined by Equation ([4.2](#S4.Ex17)) is $G$-invariant.
Moreover, if a map given in Equation ([4.2](#S4.Ex14)) is $G$-invariant, then it has the form given in Equation ([4.2](#S4.Ex17)).

The concrete formula for the polynomial $G$-equivariant layer is presented in Appendix [D.4](#A4.SS4).

###### Theorem 4.4 .

## 5 Experimental Results

In this session, we empirically evaluate the performance of our Monomial Matrix Group Polynomial Equivariant NFNs (MAGEP-NFNs) across various equivariant and invariant tasks. For invariant tasks, we apply our model to classifying Implicit Neural Representations of images and predicting CNN generalization from weights. The equivariant task involves weight space style editing. Our experiments aim to demonstrate that MAGEP-NFNs achieves superior or competitive performance compared to other baselines with equivalent parameter counts. We conduct $5$ runs for each experiment and report the averaged results. For comprehensive details on hyperparameter settings and training details, please refer to Appendix [E](#A5).

### 5.1 Classifying Implicit Neural Representations of images

Table: Table 1: Classification train and test accuracies (%) for implicit neural representations of MNIST, FashionMNIST, and CIFAR-10. Uncertainties indicate standard error over 5 runs, baseline results are from [].

##### Experiment setup.

In this experiment, we aim to determine which class each pretrained Implicit Neural Representation (INR) weight was trained on. Following [], we employ three distinct INR weight datasets [], each was trained on a different image dataset: CIFAR-10 [], FashionMNIST [], and MNIST []. Each INR weight is trained to encode a single image from its respective class, capturing the image structure by mapping pixel coordinates $(x,y)$ to the corresponding pixel color values—represented as 3-channel RGB values for CIFAR-10 and 1-channel grayscale values for MNIST and FashionMNIST. The varying complexity and diversity of the datasets provide a robust test for evaluating MAGEP-NFN’s performance, demonstrating its effectiveness and benchmarking it against existing models.

##### Results.

We present the performance of our model alongside several baseline models, including MLP, NP [], HNP [], and Monomial-NFN []. As shown in Table [1](#S5.T1), our model achieves the highest test accuracies across all INR datasets. Notably, it outperforms the second-best model on the MNIST dataset, by a significant margin of $7.73\%$. For the CIFAR-10 and FashionMNIST datasets, our model also demonstrates substantial improvements, with accuracy gains of $2.95\%$ and $1.68\%$, respectively, over the existing baselines. These results indicate that our model leverages the embedded information from the pretrained INRs more effectively than any of the compared baselines. This consistent superior performance across various INR datasets highlights the effectiveness of MAGEP-NFN. It also suggests that our model generalizes well to INR weights embedded with different image structures and complexities.

### 5.2 Predicting CNN generalization from weights

##### Experiment setup.

For this experiment, we focus on predicting the generalization performance of pretrained CNNs based solely on their weights, without evaluating them on test data. We utilize the Small CNN Zoo dataset [], which contains various pretrained CNN models trained with different combinations of hyperparameters and activation functions. For our study, we split the Small CNN Zoo into two subsets: one comprising networks using ReLU activations and the other using Tanh activations. These two types of CNNs follow different group actions: $\mathcal{M}_{n}^{>0}$ for Relu networks (see Equation ([18](#S3.E18))) and $\mathcal{M}_{n_{i}}^{\pm 1}$ for Tanh networks (see Remark [Remark](#Thmremarkx5)).

To evaluate the robustness of our model to input transformations under group actions, we augment the ReLU dataset by applying randomly sampled group actions $\mathcal{M}_{n}^{>0}$. Specifically, we randomly sampling the diagonal elements $\mathcal{D}_{n,ii}^{>0}$ of the matrix $\mathcal{D}_{n}^{>0}$, with each element drawn from uniform distributions over different ranges, defined as $\mathcal{U}[1,10^{i}]$ for $i=1,2,3,4$. To further diversify the transformations, we also randomly sample the permutation matrix $\mathcal{P}_{n}$.

##### Results.

Table [2](#S5.T2) illustrates the performance of all models trained on the $\operatorname{ReLU}$ subset, where our MAGEP-NFNs model clearly outperforms all other baselines. Notably, it demonstrates robustness to scale and permutation symmetry, similar to Monomial-NFN, while consistently surpassing its performance across both the original and all augmented dataset settings. This suggests that incorporating polynomial layers allows our model to capture more information from the weights across different hidden layers, compared to Monomial-NFN, thereby enhancing expressivity. On the original dataset, our model achieves a Kendall’s $\tau$ performance gap of $0.007$ over other baselines, and maintaining at least a $0.012$ advantage in all other augmented settings. Similarly, Table [3](#S5.T3) reveals that MAGEP-NFNs achieves the highest Kendall’s $\tau$ with $\operatorname{Tanh}$ activation, further reinforcing its superior accuracy across different network configurations.

Table: Table 2: Performance prediction of CNNs on the $\operatorname{ReLU}$ subset of Small CNN Zoo with varying scale augmentations. We use Kendall’s $\tau$ as the evaluation metric. The uncertainty bars indicate the standard deviation across 5 runs.

Table: Table 3: Performance prediction of CNNs on the $\operatorname{Tanh}$ subset of Small CNN Zoo. We use Kendall’s $\tau$ as the evaluation metric. The uncertainty bars indicate the standard deviation across 5 runs.

### 5.3 Weight space style editing

Table: Table 4: Test mean squared error (lower is better) between weight-space editing methods and ground-truth image-space transformations. Uncertainties indicate standard error over 5 runs

##### Experiment setup.

In this experiment, we focus on modifying the weights of SIREN [] to modify the image encoded within each model. We utilize the pretrained models from paper [], which encode images from the CIFAR-10 and MNIST datasets. Specifically, we address two tasks aimed at modifying the embedded information: enhancing the contrast of CIFAR-10 images and dilating MNIST images encoded in the SIREN models. We report the MSE loss between the images encoded in the modified SIREN network and the ground truth contrast-enhanced CIFAR-10 images or dilated MNIST images.

##### Results.

Table [4](#S5.T4) demonstrates that our model achieves performance comparable to other baselines. Specifically, MAGEP-NFNs matches the performance of NP and Monomial-NFN in the contrast-enhancing task on the CIFAR-10 dataset. Additionally, our model outperforms Monomial-NFN in the dilation task on the MNIST dataset, while achieving similar results to NP. Interestingly, NP remains a strong candidate in the weight editing tasks, and our model consistently performs on par with NP across both experiments.

## 6 Conclusion

We have developed MAGEP-NFN, a novel NFN that is equivariant to both permutations and scaling symmetries. Our approach follows a parameter-sharing mechanism; however, unlike previous works, we construct an equivariant polynomial layer that incorporates stable polynomial terms. This polynomial formulation enables us to capture relationships between weights from different input hidden layers, thereby enhancing the expressivity of MAGEP-NFN while maintaining low memory consumption and efficient running time.
Experimental results demonstrate that our model achieves competitive performance and efficiency compared to existing baselines.

One limitation of the equivariant polynomial layers proposed in this paper is that they are applied to a specific architecture design. However, since our method is based on a parameter-sharing mechanism, it is applicable to other architectures with additional operators (such as layer normalization, softmax, pooling) and different activation functions, provided that the symmetric group of the weight network is known.

## Ethics Statement.

Given the nature of the work, we do not foresee any negative societal and ethical
impacts of our work.

## Reproducibility Statement.

Source codes for our experiments are provided in the supplementary materials of the paper. The details of our experimental settings are given in Section [5](#S5) and the Appendix [E](#A5). All datasets that we used in the paper are published, and they are easy to access in the Internet.

## Appendix A Preliminaries

This section contains notations and basic results on matrices and polynomials that will be used throughout the paper. We will mainly focus on matrices with real entries (real matrices) and polynomials with real coefficients (real polynomials). We will omit almost all of the proofs in this section as they are well-known. These results will be use in proofs in the rest of the paper.

### A.1 Entries of matrices

A real matrix $A$ with $m$ rows and $n$ columns is an element of $\mathbb{R}^{m\times n}$:

$$ $\displaystyle A=\left(a_{ij}\right)_{1\leqslant i\leqslant m,1\leqslant j\leqslant n}=\begin{pmatrix}a_{11}&a_{12}&\dots&a_{1n}\\ a_{21}&a_{22}&\dots&a_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ a_{m1}&a_{m2}&\dots&a_{mn}\end{pmatrix}\in\mathbb{R}^{m\times n}.$ $$

The entry in the $i^{\text{th}}$-row and the $j^{\text{th}}$-column of $A$, or the $(i,j)$ entry of $A$, is denoted by $A_{ij}=a_{ij}$. The $i^{\text{th}}$-row of $A$ and $j^{\text{th}}$-column of $A$ are, respectively, denoted by:

$$ ${\begin{array}[]{lll}A_{i\ast}&=\left(a_{i1}~{},~{}a_{i2}~{},~{}\ldots~{},~{}a_{in}\right)&\in\mathbb{R}^{1\times n},\\ A_{\ast j}&=\left(a_{1j}~{},~{}a_{2j}~{},~{}\ldots~{},~{}a_{mj}\right)^{\top}&\in\mathbb{R}^{m\times 1}.\\ \end{array}}$ $$

###### Remark .

Sometimes, a comma is added between two subscript indices to make sure there will be no confusion, i.e. $A_{i,j},a_{i,j},A_{i,\ast},A_{\ast,j}$.

Let $A^{(L)},\ldots,A^{(2)},A^{(1)}$ be $L$ matrices such that the matrix product:

$$ $A^{(L)}\cdot\ldots\cdot A^{(2)}\cdot A^{(1)},$ $$

is well-defined.

###### Proposition A.1 .

The $(i,j)$ entry of $A^{(L)}\cdot\ldots\cdot A^{(2)}\cdot A^{(1)}$ is equal to:

$$ $\displaystyle\left(A^{(L)}\cdot\ldots\cdot A^{(2)}\cdot A^{(1)}\right)_{ij}$ $\displaystyle=A^{(L)}_{i,\ast}\cdot A^{(L-1)}\cdot\ldots\cdot A^{(2)}\cdot A^{(1)}_{\ast,j}$ $\displaystyle=\sum_{k_{L-1},\ldots,k_{2},k_{1}}a^{(L)}_{i,k_{L-1}}\cdot a^{(L)}_{k_{L-1},k_{L-2}}\cdot\ldots\cdot a^{(2)}_{k_{2},k_{1}}\cdot a^{(1)}_{k_{1},j}.$ $$

In the case where $L=1$, the above equation is simply $A^{(1)}_{ij}=a^{(1)}_{ij}$.

We set a denotation for matrices that have only one nonzero entry with value $1$. The matrix with the $1$ in the $i^{\text{th}}$-row and the $j^{\text{th}}$-column, and the rest are 0 0, is denoted by $E_{ij}$. Matrix $E_{ij}$ can have any shape, but its shape are usually defined by context, and will be omitted without confusion. The product of matrices of this type is presented as below.

###### Proposition A.2 .

Let $E_{i_{1},j_{1}},E_{i_{2},j_{2}},\ldots,E_{i_{L},j_{L}}$ be $L$ matrix units such that the product:

$$ $\displaystyle E_{i_{1},j_{1}}\cdot E_{i_{2},j_{2}}\cdot\ldots\cdot E_{i_{L},j_{L}},$ $$

is well-defined. Then:

$$ $\displaystyle E_{i_{1},j_{1}}\cdot E_{i_{2},j_{2}}\cdot\ldots\cdot E_{i_{L},j_{L}}=\left(\delta_{j_{1},i_{2}}\cdot\delta_{j_{2},i_{3}}\cdot\ldots\cdot\delta_{j_{L-1},i_{L}}\right)\cdot E_{i_{1},j_{L}},$ $$

where $\delta_{ij}$ is the Kronecker delta:

$$ $\delta_{ij}=\begin{cases}0&\text{if $i\neq j$,}\\ 1&\text{if $i=j$.}\end{cases}$ $$

We have a direct corollary for $E_{1,1}$’s.

###### Corollary A.3 .

We have:

$$ $\displaystyle E_{1,1}\cdot E_{1,1}\cdot\ldots\cdot E_{1,1}=E_{1,1}.$ $$

###### Remark .

###### Proposition A.1 .

###### Proposition A.2 .

###### Corollary A.3 .

### A.2 Evaluation of polynomials

Denote $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$ be the ring of all polynomials with real coefficients in $n$ indeterminates $\mathbf{x}_{1},\ldots,\mathbf{x}_{n}$.

###### Definition A.4 .

A monomial of $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$ is a polynomial of $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$ that has one term.

###### Remark .

In some contexts, a monomial is defined as a polynomial that has one term with coefficient $1$. We will use both of these definitions simultaneously.

###### Proposition A.5 .

$\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$ is naturally a vector space over $\mathbb{R}$. It is an infinite-dimensional vector space; moreover, the set of all monomials with coefficient $1$ in $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$ is a basis for the vector space.

###### Remark .

For $f\in\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$, by saying monomials in $f$, we refer to all monomials that appeared in the expression of $f$.

Polynomial evaluation is computing of the value of a polynomial when the indeterminates are substituted for some values. We have the well-known result.

###### Proposition A.6 .

Let $f,g$ be two polynomials of $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$. If $f,g$ are equal at every evaluations, i.e.

$$ $\displaystyle f(x_{1},\ldots,x_{n})=g(x_{1},\ldots,x_{n})~{},~{}\forall(x_{1},\ldots,x_{n})\in\mathbb{R}^{n},$ (24) $$

then $f=g$. In other words, the only polynomial of $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$, that has $\mathbb{R}^{n}$ as its zero set, is the polynomial $0\in\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$.

###### Remark .

The result still holds if $\mathbb{R}$ is replaced by an arbitrary infinite field, but does not hold if $\mathbb{R}$ is replaced by a finite field.

We have a direct corollary.

###### Corollary A.7 .

Let $f$ be a nonzero polynomial of $\mathbb{R}[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$. Then there exists $(x_{1},\ldots,x_{n})\in\mathbb{R}^{n}$ such that $f(x_{1},\ldots,x_{n})\neq 0$.

###### Definition A.4 .

###### Remark .

###### Proposition A.5 .

###### Remark .

###### Proposition A.6 .

###### Remark .

###### Corollary A.7 .

### A.3 Entries of tensors

###### Proposition A.8 .

Let $a=(a_{i})_{1\leqslant i\leqslant n}$ and $b=(b_{i})_{1\leqslant i\leqslant n}$ be two vectors in $\mathbb{R}^{n}$. If:

$$ $\displaystyle a_{i}\cdot b_{j}+a_{j}\cdot b_{i}=0,$ (25) $$

for all $1\leqslant i,j\leqslant n$, then $a=0$ or $b=0$.

###### Proof.

Assume that both of $a$ and $b$ are not equal to 0 0, then there exists $i,j$ such that $a_{i}$ and $b_{j}$ are non-zero. From Equation ([25](#A1.E25)), we have:

$$ $\displaystyle a_{i}\cdot b_{i}+a_{i}\cdot b_{i}=0,$ (26) $$

so $a_{i}\cdot b_{i}=0$. Since $a_{i}$ is non-zero, then $b_{i}=0$. It implies that:

$$ $\displaystyle a_{i}\cdot b_{j}+a_{j}\cdot b_{i}=a_{i}\cdot b_{j}+0=a_{i}\cdot b_{j}\neq 0,$ (27) $$

which contradicts to Equation ([25](#A1.E25)). So at least one of $a$ and $b$ is equal to 0 0.
∎

###### Proposition A.9 .

Let $A=(a_{ij})_{1\leqslant i\leqslant m,1\leqslant j\leqslant n}$ and $B=(b_{ij})_{1\leqslant i\leqslant m,1\leqslant j\leqslant n}$ be two matrices in $\mathbb{R}^{m\times n}$. If:

$$ $\displaystyle a_{ij}\cdot b_{kl}+a_{kj}\cdot b_{il}+a_{il}\cdot b_{kj}+a_{kl}\cdot b_{ij}=0,$ (28) $$

for all $1\leqslant i,k\leqslant m$ and $1\leqslant j,l\leqslant n$, then $A=0$ or $B=0$.

###### Proof.

Consider Equation ([28](#A1.E28)) when $1\leqslant j=l\leqslant n$, we have:

$$ $\displaystyle 0$ $\displaystyle=a_{ij}\cdot b_{kj}+a_{kj}\cdot b_{ij}+a_{ij}\cdot b_{kj}+a_{kj}\cdot b_{ij}$ (29) $\displaystyle=2\cdot\left(a_{ij}\cdot b_{kj}+a_{kj}\cdot b_{ij}\right),$ (30) $$

which means:

$$ $\displaystyle a_{ij}\cdot b_{kj}+a_{kj}\cdot b_{ij}=0.$ (31) $$

This holds for all $1\leqslant i,k\leqslant m$. Apply Proposition [A.8](#A1.Thmtheorem8), we have $a_{ij}=0$ for all $1\leqslant i\leqslant m$, or $b_{ij}=0$ for all $1\leqslant i\leqslant m$, which means $A_{\ast,j}=0$ or $B_{\ast,j}=0$. This holds for all $1\leqslant j\leqslant n$. Similarly, we have $A_{i,\ast}=0$ or $B_{i,\ast}=0$ for $1\leqslant i\leqslant m$. Now, assume that, both of $A$ and $B$ are not equal to 0 0, then there exists $i,j$ and $k,l$ such that $a_{ij}$ and $b_{kl}$ are non-zero. By previous observation, we have $B_{i,\ast}=B_{\ast,j}=A_{k,\ast}=A_{\ast,l}=0$. It implies that:

$$ $\displaystyle a_{ij}\cdot b_{kl}+a_{kj}\cdot b_{il}+a_{il}\cdot b_{kj}+a_{kl}\cdot b_{ij}=a_{ij}\cdot b_{kl}+0+0+0=a_{ij}\cdot b_{kl}\neq 0,$ (32) $$

which contradicts to Equation ([28](#A1.E28)). So at least one of $A$ and $B$ is equal to 0 0.
∎

Proposition [A.8](#A1.Thmtheorem8) and Proposition [A.9](#A1.Thmtheorem9) are, respectively, one-dimensional and two-dimensional cases.
By using the same arguments, we will obtain the $d$-dimensional version belows.

###### Proposition A.10 .

Let $d$ be a positive integer and $n_{1},n_{2},\ldots,n_{d}$ be $d$ positive integers. Let:

$$ $\displaystyle{\begin{array}[]{lll}A&=(A_{i_{1},i_{2},\ldots,i_{d}})_{1\leqslant i_{1}\leqslant n_{1},1\leqslant i_{2}\leqslant n_{2},\ldots,1\leqslant i_{d}\leqslant n_{d}}&\in\mathbb{R}^{n_{1}\times n_{2}\times\ldots\times n_{d}},\\ B&=(B_{i_{1},i_{2},\ldots,i_{d}})_{1\leqslant i_{1}\leqslant n_{1},1\leqslant i_{2}\leqslant n_{2},\ldots,1\leqslant i_{d}\leqslant n_{d}}&\in\mathbb{R}^{n_{1}\times n_{2}\times\ldots\times n_{d}}.\end{array}}$ $$

If for all $1\leqslant i_{1}^{0},i_{1}^{1}\leqslant n_{1},1\leqslant i_{2}^{0},i_{2}^{1}\leqslant n_{2},\ldots,1\leqslant i_{d}^{0},i_{d}^{1}\leqslant n_{d}$, we have:

$$ $\displaystyle\sum_{(\alpha_{1},\ldots,\alpha_{d})\in\{0,1\}^{d}}\left(A_{i^{\alpha_{1}}_{1},i^{\alpha_{2}}_{2},\ldots,i^{\alpha_{d}}_{d}}\right)\cdot\left(B_{i^{1-\alpha_{1}}_{1},i^{1-\alpha_{2}}_{2},\ldots,i^{1-\alpha_{d}}_{d}}\right)=0,$ (33) $$

then $A=0$ or $B=0$.

###### Proposition A.8 .

###### Proof.

###### Proposition A.9 .

###### Proof.

###### Proposition A.10 .

## Appendix B Stable Polynomial Terms

Intuitively, a *stable polynomial term* is a polynomial in the entries of $U\in\mathcal{U}$ that is “stable” under the action of $G$ (see Definition [B.1](#A2.Thmtheorem1) below).
The equivariant polynomial layers we aim to construct are linear combinations of these stable polynomial terms.
In Subsection [B.1](#A2.SS1), we provide a formal definition for stable polynomial terms as well as their properties.
We will study the linear dependence of stable polynomial terms in the language of polynomial rings with real coefficients in Subsections [B.2](#A2.SS2) and [B.3](#A2.SS3).
These properties play a central role in determining the parameter-sharing computation of equivariant polynomial layers in the next section.

### B.1 Definitions and basic properties

Recall the weight space $\mathcal{U}$ given by:

$$ $\displaystyle\begin{array}[]{lllll}\mathcal{U}&=\hskip 10.0pt\mathcal{W}\times\mathcal{B},&\text{ where:}&&\\ \mathcal{W}&=\hskip 10.0pt\mathbb{R}^{w_{L}\times n_{L}\times n_{L-1}}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{w_{2}\times n_{2}\times n_{1}}&\times\hskip 10.0pt\mathbb{R}^{w_{1}\times n_{1}\times n_{0}},\\ \mathcal{B}&=\hskip 10.0pt\mathbb{R}^{b_{L}\times n_{L}\times 1}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{b_{2}\times n_{2}\times 1}&\times\hskip 10.0pt\mathbb{R}^{b_{1}\times n_{1}\times 1}.\end{array}$ $$

Let us consider the case where the weight spaces have the same number of dimensions across all channels, which means $w_{i}=b_{i}=d$ for all $i$.

###### Definition B.1 (Stable polynomial terms) .

Let $U=([W],[b])$ be an element of $\mathcal{U}$ with weights $[W]=\left([W]^{(L)},\ldots,[W]^{(1)}\right)$ and biases
$[b]=\left([b]^{(L)},\ldots,[b]^{(1)}\right)$.
For each $L\geqslant s>t\geqslant 0$, we define:

$$ $\begin{array}[]{lll}[W]^{(s,t)}&\coloneqq[W]^{(s)}\cdot[W]^{(s-1)}\cdot\ldots\cdot[W]^{(t+1)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}},\\ \left[Wb\right]^{(s,t)(t)}&\coloneqq[W]^{(s,t)}\cdot\left[b\right]^{(t)}&\in\mathbb{R}^{d\times n_{s}\times 1}.\end{array}$ (34) $$

In addition, for each $L\geqslant s,t\geqslant 0$, and matrices $\Psi^{(s)(L,t)}\in\mathbb{R}^{1\times n_{L}}$ and $\Psi^{(s,0)(L,t)}\in\mathbb{R}^{n_{0}\times n_{L}}$, we also define

$$ $\begin{array}[]{lll}\left[bW\right]^{(s)(L,t)}&\coloneqq[b]^{(s)}\cdot\Psi^{(s)(L,t)}\cdot[W]^{(L,t)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}},\\ \left[WW\right]^{(s,0)(L,t)}&\coloneqq[W]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[W]^{(L,t)}&\in\mathbb{R}^{d\times n_{s}\times n_{t}}.\end{array}$ (35) $$

The entries of the matrices $[W]^{(s,t)}$, $\left[Wb\right]^{(s,t)(t)}$, $\left[bW\right]^{(s)(L,t)}$ and $\left[WW\right]^{(s,0)(L,t)}$ defined above are called *stable polynomial terms* of $U$ under the action of $G$.

The following observations are direct implications from the definition.

- •
For all $L\geqslant s>t>r\geqslant 0$:
$\displaystyle[W]^{(s,s-1)}=[W]^{(s)}\in\mathbb{R}^{d\times n_{s}\times n_{s-1}},$
(36)
and
$\displaystyle[W]^{(s,t)}\cdot[W]^{(t,r)}=[W]^{(s,r)}\in\mathbb{R}^{d\times n_{s}\times n_{r}},$
(37)
by definition. For $g=\left(g^{(L)},\ldots,g^{(0)}\right)\in\mathcal{G}_{\mathcal{U}}$:
$[gW]^{(s,t)}=\left(g^{(s)}\right)\cdot[W]^{(s,t)}\cdot\left(g^{(t)}\right)^{-1}\in\mathbb{R}^{d\times n_{s}\times n_{t}}.$
(38)
- •
If $g\in G$, then:
$\displaystyle[gW]^{(L,t)}$
$\displaystyle=[W]^{(L,t)}\cdot\left(g^{(t)}\right)^{-1}\in\mathbb{R}^{d\times n_{L}\times n_{t}}$
(39)
$\displaystyle[gW]^{(s,0)}$
$\displaystyle=\left(g^{(s)}\right)\cdot[W]^{(s,0)}\in\mathbb{R}^{d\times n_{s}\times n_{0}}.$
(40)
- •
For all $L\geqslant s>t>0$, we have
$\displaystyle[gW]^{(s,t)}\cdot[gb]^{(t)}=\left(g^{(s)}\right)\cdot[W]^{(s,t)}\cdot[b]^{(t)}\in\mathbb{R}^{d\times n_{s}\times 1}$
(41)
- •
For all $L\geqslant s>0$, $L>t\geqslant 0$ and $\Psi^{(s,0)(L,t)}\in\mathbb{R}^{d\times n_{0}\times n_{L}}$, we have:
$\displaystyle[gW]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[gW]^{(L,t)}$
$\displaystyle\qquad=\left(g^{(s)}\right)\cdot[W]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[W]^{(L,t)}\cdot\left(g^{(t)}\right)^{-1}\in\mathbb{R}^{d\times n_{s}\times n_{t}}.$
(42)
In particular, if $t=s-1$, we have:
$\displaystyle[gW]^{(s,0)}\cdot\Psi^{(s,0)(L,s-1)}\cdot[gW]^{(L,s-1)}$
$\displaystyle\qquad=\left(g^{(s)}\right)\cdot[W]^{(s,0)}\cdot\Psi^{(s,0)(L,s-1)}\cdot[W]^{(L,s-1)}\cdot\left(g^{(s-1)}\right)^{-1}\in\mathbb{R}^{d\times n_{s}\times n_{s-1}}.$
(43)
- •
For all $L\geqslant s>0$ and $\Psi^{(s)(L,t)}\in\mathbb{R}^{d\times 1\times n_{L}}$, we have:
$\displaystyle[gb]^{(s)}\cdot\Psi^{(s)(L,t)}\cdot[gW]^{(L,t)}$
$\displaystyle\qquad=\left(g^{(s)}\right)\cdot[b]^{(s)}\cdot\Psi^{(s)(L,t)}\cdot[W]^{(L,t)}\cdot\left(g^{(t)}\right)^{-1}\in\mathbb{R}^{d\times n_{s}\times n_{t}}.$
(44)

Based on the above observations, we can determine the image of the stable polynomial terms under the action of an element $g\in\mathcal{G}_{\mathcal{U}}$ as follows:

$$ $\displaystyle{\begin{array}[]{ll}\left[gW\right]^{(s,t)}&=\left(g^{(s)}\right)\cdot\left[W\right]^{(s,t)}\cdot\left(g^{(t)}\right)^{-1},\\ \left[gb\right]^{(s)}&=\left(g^{(s)}\right)\cdot[b]^{(s)},\\ \left[gWgb\right]^{(s,t)(t)}&=\left(g^{(s)}\right)\cdot[Wb]^{(s,t)(t)},\\ \left[gbgW\right]^{(s)(L,t)}&=\left(g^{(s)}\right)\cdot[bW]^{(s)(L,t)}\cdot\left(g^{(t)}\right)^{-1},\\ \left[gWgW\right]^{(s,0)(L,t)}&=\left(g^{(s)}\right)\cdot[WW]^{(s,0)(L,t)}\cdot\left(g^{(t)}\right)^{-1}.\end{array}}$ $$

In concrete, we have:

$$ $\displaystyle{\begin{array}[]{ll}[gW]^{(s,t)}_{pq}&=\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[W]^{(s,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)},\\ \left[gb\right]^{(s)}_{p}&=d^{(s)}_{p}\cdot[b]^{(s)}_{\pi^{-1}_{s}(p)},\\ \left[gWgb\right]^{(s,t)(t)}_{p}&=d^{(s)}_{p}\cdot[Wb]^{(s,t)(t)}_{\pi^{-1}_{s}(p)},\\ \left[gbgW\right]^{(s)(L,t)}_{pq}&=\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[bW]^{(s)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)},\\ \left[gWgW\right]^{(s,0)(L,t)}&=\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[WW]^{(s,0)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}.\end{array}}$ $$

###### Definition B.1 (Stable polynomial terms) .

### B.2 Input weights as indeterminates

To simplify the technical difficulties, we consider the weight space $\mathcal{U}$ in the case where $d=1$, i.e.,

$$ $\displaystyle\begin{array}[]{lllll}\mathcal{U}&=\hskip 10.0pt\mathcal{W}\times\mathcal{B},&\text{ where:}&&\\ \mathcal{W}&=\hskip 10.0pt\mathbb{R}^{n_{L}\times n_{L-1}}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{n_{2}\times n_{1}}&\times\hskip 10.0pt\mathbb{R}^{n_{1}\times n_{0}},\\ \mathcal{B}&=\hskip 10.0pt\mathbb{R}^{n_{L}\times 1}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{n_{2}\times 1}&\times\hskip 10.0pt\mathbb{R}^{n_{1}\times 1}.\end{array}$ $$

We introduce the set $I$ consists of indeterminates defined by:

$$ $I\coloneqq\{\mathbf{x}^{(i)}_{jk}~{}\colon~{}1\leqslant i\leqslant L,1\leqslant j\leqslant n_{i},1\leqslant k\leqslant n_{i-1}\}\cup\{\mathbf{y}^{(i)}_{j}~{}\colon~{}1\leqslant i\leqslant L,1\leqslant j\leqslant n_{i}\}.$ $$

We have $|I|=\operatorname{dim}\mathcal{U}$. Denote $R=\mathbb{R}[I]$, which is the ring of all polynomials with indeterminates are all elements of $I$. For $1\leqslant i\leqslant L$, we define:

$$ ${\begin{array}[]{lll}\left[\mathbf{W}\right]^{(i)}&\coloneqq\left(\mathbf{x}^{(i)}_{jk}\right)_{1\leqslant j\leqslant n_{i},1\leqslant k\leqslant n_{i-1}}&\in R^{n_{i}\times n_{i-1}},\\ \left[\mathbf{b}\right]^{(i)}&\coloneqq\left(\mathbf{y}^{(i)}_{j}\right)_{1\leqslant j\leqslant n_{i}}&\in R^{n_{i}\times 1},\end{array}}$ $$

and

$$ ${\begin{array}[]{lll}\left[\mathbf{W}\right]^{(s,t)}&\coloneqq[\mathbf{W}]^{(s)}\cdot[\mathbf{W}]^{(s-1)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}&\in R^{n_{s}\times n_{t}},\\ \left[\mathbf{Wb}\right]^{(s,t)(t)}&\coloneqq[\mathbf{W}]^{(s,t)}\cdot[\mathbf{b}]^{(t)}&\in R^{n_{s}\times 1},\\ \left[\mathbf{bW}\right]^{(s)(L,t)}&\coloneqq[\mathbf{b}]^{(s)}\cdot\Psi^{(s)(L,t)}\cdot[\mathbf{W}]^{(L,t)}&\in R^{n_{s}\times n_{t}},\\ \left[\mathbf{WW}\right]^{(s,0)(L,t)}&\coloneqq[\mathbf{W}]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L,t)}&\in R^{n_{s}\times n_{t}}.\end{array}}$ $$

with feasible indices $(s,t)$. The coefficients $\Psi^{(-)}$’s are fixed real matrices and they are omitted from the notations.

Note that the entries of these matrices are stable polynomial terms in which the entries of $U$ are now viewed as indeterminates of the polynomial ring $R$.

### B.3 Linear dependence of stable polynomial terms

In this subsection, we derive a necessary condition for the coefficients $\Phi_{-}$ and $\Psi^{-}$ such that the following linear combination of stable polynomial terms are identically zero:

$$ $\displaystyle\alpha(\Phi,\Psi)\coloneqq$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[\mathbf{W}]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot[\mathbf{b}]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[\mathbf{Wb}]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[\mathbf{bW}]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[\mathbf{WW}]^{(s,0)(L,t)}_{pq}+\Phi_{1}\cdot 1.$ (45) $$

Here, $\alpha$ is parameterized by $\Phi$ and $\Psi$, where $\Phi$ is a collection of real scalars $\Phi_{-}$’s appeared in the linear combination and $\Psi$ is a collection of real matrices $\Psi^{-}$’s that be used to define $[\mathbf{bW}]^{(-)}$’s and $[\mathbf{WW}]^{(-)}$’s. The index of each scalar $\Phi_{-}$ naturally presents its corresponding polynomial in $\alpha(\Phi,\Psi)$.
This necessary and sufficient condition enables us to determine the equivariant polynomial map via parameter sharing later.

We first take a look at entries of $[\mathbf{W}]^{(-)}$’s, $[\mathbf{b}]^{(-)}$’s, $[\mathbf{Wb}]^{(-)}$’s, $[\mathbf{bW}]^{(-)}$’s, $[\mathbf{WW}]^{(-)}$’s. It is clear that for one of these matrices, its entries are homogeneous polynomials with the same degree. For example:

- •
$[\mathbf{W}]^{(-)}$: The polynomial $[\mathbf{W}]^{(s,t)}_{pq}$ has degree $s-t$. All of its monomial terms consist of one $\mathbf{x}^{(i)}_{-}$ for each $s\geqslant i>t$.
- •
$[\mathbf{b}]^{(-)}$: The polynomial $[\mathbf{b}]^{(s)}_{p}$ has degree $1$. All of its monomial terms consist of one $\mathbf{y}^{s}_{-}$.
- •
$[\mathbf{Wb}]^{(-)}$: The polynomial $[\mathbf{Wb}]^{(s,t)(t)}_{p}$ has degree $s-t+1$. All of its monomial terms consist of one $\mathbf{x}^{(i)}_{-}$ for each $s\geqslant i>t$ and one $\mathbf{y}^{(t)}_{-}$.
- •
$[\mathbf{bW}]^{(-)}$: The polynomial $[\mathbf{bW}]^{(s)(L,t)}_{p}$ has degree $L-t+1$. All of its monomial terms consist of one $\mathbf{y}^{(s)}_{-}$ and one $\mathbf{x}^{(i)}_{-}$ for each $L\geqslant i>t$.
- •
$[\mathbf{WW}]^{(-)}$: The polynomial $[\mathbf{WW}]^{(s,0)(L,t)}_{pq}$ has degree $L+s-t$. All of its monomial terms consist of one $\mathbf{x}^{(i)}_{-}$ for each $s\geqslant i>0$ and one $\mathbf{x}^{(i)}_{-}$ for each $L\geqslant i>t$.
- •
$1$: The polynomial $1\in R$.

By these above observations, we have:

- •
$[\mathbf{W}]^{(-)}$, $[\mathbf{WW}]^{(-)}$ : Each of the polynomials $[\mathbf{W}]^{(-)}_{-}$’s and $[\mathbf{WW}]^{(-)}_{-}$’s is 0 0 or a non-constant element in $R$, and it is a real polynomial of at least one indeterminate from $\mathbf{x}^{(-)}_{-}$’s.
- •
$[\mathbf{b}]^{(-)}$: Each of the polynomials $[\mathbf{b}]^{(-)}_{-}$’s is a non-constant element in $R$, and it is a real polynomial of one indeterminate from $\mathbf{y}^{(-)}_{-}$’s.
- •
$[\mathbf{Wb}]^{(-)}$, $[\mathbf{bW}]^{(-)}$: Each of the polynomials $[\mathbf{Wb}]^{(-)}_{-}$’s and $[\mathbf{bW}]^{(-)}_{-}$’s is 0 0 or a non-constant element in $R$, and it is a real polynomial of at least one indeterminate from $\mathbf{x}^{(-)}_{-}$’s and one indeterminate from $\mathbf{y}^{(-)}_{-}$’s.

Therefore, if $\alpha(\Phi,\Psi)=0$, we must have

$$ $\displaystyle 0$ $\displaystyle=\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[\mathbf{W}]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[\mathbf{WW}]^{(s,0)(L,t)}_{pq},$ (46) $\displaystyle 0$ $\displaystyle=\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot[\mathbf{b}]^{(s)}_{p},$ (47) $\displaystyle 0$ $\displaystyle=\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[\mathbf{Wb}]^{(s,t)(t)}_{p}+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[\mathbf{bW}]^{(s)(L,t)}_{pq},$ (48) $\displaystyle 0$ $\displaystyle=~{}\Phi_{1}\cdot 1.$ (49) $$

We induce the constraints on $\Phi$ and $\Psi$ in these above equations by using the fact that a set of distinct monomials of $R$ is a linear independent set (see Proposition [A.5](#A1.Thmtheorem5)).

- •
Equation ([46](#A2.E46)): Observe that
–
If $L\geqslant s>t\geqslant 0$ and $(s,t)\neq(L,0)$, then the monomials $\mathbf{x}^{(s)}_{-}\cdot\ldots\cdot\mathbf{x}^{(t+1)}_{-}$’s only appear in the polynomials $[\mathbf{W}]^{(s,t)}_{-}$’s. They do not appear in the polynomials $[\mathbf{W}]^{(s^{\prime},t^{\prime})}_{-}$’s for all pairs $(s^{\prime},t^{\prime})\neq(s,t)$, and do not appear in the polynomials $[\mathbf{WW}]^{(s^{\prime},0)(L,t^{\prime})}_{-}$’s for all pairs $(s^{\prime},t^{\prime})$.
–
If $L\geqslant s>0$, $L>t\geqslant 0$, and $s\neq t$, then the monomials $\left(\mathbf{x}^{(s)}_{-}\cdot\ldots\cdot\mathbf{x}^{(1)}_{-}\right)\cdot\left(\mathbf{x}^{(L)}_{-}\cdot\ldots\cdot\mathbf{x}^{(t+1)}_{-}\right)$’s only appear in the polynomials $[\mathbf{WW}]^{(s,0)(L,t)}_{-}$’s. They do not appear in the polynomials $[\mathbf{W}]^{(s^{\prime},t^{\prime})}_{-}$’s for all pairs $(s^{\prime},t^{\prime})$, and do not appear in the polynomials $[\mathbf{WW}]^{(s^{\prime},0)(L,t^{\prime})}_{-}$’s for all pairs $(s^{\prime},t^{\prime})\neq(s,t)$.
So from Equation ([46](#A2.E46)), it implies that
$\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[\mathbf{W}]^{(s,t)}_{pq},$
(50)
for all $(s,t)\neq(L,0)$, and
$\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[\mathbf{WW}]^{(s,0)(L,t)}_{pq},$
(51)
for all $(s,t)$ that $s\neq t$. The rest of the terms in Equation ([46](#A2.E46)) is
$\displaystyle 0=\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[\mathbf{W}]^{(L,0)}_{pq}+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{pq}.$
(52)
- •
Equation ([47](#A2.E47)): Since the set of all $[\mathbf{b}]^{(-)}_{-}$’s, which is the set of all monomials $\mathbf{y}^{(-)}_{-}$’s, is a linear independent set in $R$, it implies that
$\displaystyle 0=\Phi_{(s):p},$
(53)
for all $L\geqslant s>0$ and $1\leqslant p\leqslant n_{s}$.
- •
Equation ([48](#A2.E48)): Observe that
–
If $L>s>t>0$, then the monomials $\left(\mathbf{x}^{(s)}_{-}\cdot\ldots\cdot\mathbf{x}^{(t+1)}_{-}\right)\cdot\mathbf{y}^{(t)}_{-}$’s only appear in the polynomials $[\mathbf{Wb}]^{(s,t)(t)}_{-}$’s. They do not appear in the polynomials $[\mathbf{Wb}]^{(s^{\prime},t^{\prime})(t^{\prime})}_{-}$’s for all $L\geqslant s^{\prime}>t^{\prime}>0$ that $(s^{\prime},t^{\prime})\neq(s,t)$, and do not appear in polynomials $[\mathbf{bW}]^{(s^{\prime})(L,t^{\prime})}_{-}$’s for all $L\geqslant s^{\prime}>0$ and $L>t^{\prime}\geqslant 0$.
–
If $L\geqslant s>0$, $L>t\geqslant 0$, and $s\neq t$, then the monomials $\mathbf{y}^{(s)}_{-}\cdot\left(\mathbf{x}^{(L)}_{-}\cdot\ldots\cdot\mathbf{x}^{(t+1)}_{-}\right)$’s only appear in the polynomials $[\mathbf{bW}]^{(s)(L,t)}_{-}$’s. They do not appear in the polynomials $[\mathbf{Wb}]^{(L,t^{\prime})(t^{\prime})}_{-}$’s for all $L>t^{\prime}>0$, and do not appear in the polynomials $[\mathbf{bW}]^{(s^{\prime})(L,t^{\prime})}_{-}$’s for all pairs $(s^{\prime},t^{\prime})\neq(s,t)$.
–
If $L>t>0$, then the monomials $\mathbf{y}^{(t)}_{-}\cdot\left(\mathbf{x}^{(L)}_{-}\cdot\ldots\cdot\mathbf{x}^{(t+1)}_{-}\right)$’s only appear in the polynomials $[\mathbf{Wb}]^{(L,t)(t)}_{-}$’s and appear in the polynomials $[\mathbf{bW}]^{(t)(L,t)}_{-}$’s. They do not appear in the polynomials $[\mathbf{Wb}]^{(L,t^{\prime})(t^{\prime})}_{-}$’s for all $L>t^{\prime}>0$ that $t^{\prime}\neq t$, and do not appear in polynomials $[\mathbf{bW}]^{(s^{\prime})(L,t^{\prime})}_{-}$’s for all pair $(s^{\prime},t^{\prime})$ that $t^{\prime}\neq t$.
So from Equation ([48](#A2.E48)), it implies that
$\displaystyle 0=\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[\mathbf{Wb}]^{(s,t)(t)}_{p},$
(54)
for all $L>s>t>0$, and
$\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[\mathbf{bW}]^{(s)(L,t)}_{pq}$
(55)
for all $(s,t)$ that $s\neq t$, and
$\displaystyle 0=\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[\mathbf{bW}]^{(t)(L,t)}_{pq}$
(56)
for all $L>t>0$.
- •
Equation ([49](#A2.E49)): Clearly, it implies that
$\displaystyle 0=\Phi_{1}.$
(57)

There are $8$ equations, Equations ([50](#A2.E50))-([57](#A2.E57)), that are derived. In Equations ([53](#A2.E53)) and ([57](#A2.E57)), the corresponding $\Phi_{-}$’s are directly characterized, and in Equations ([50](#A2.E50)), ([51](#A2.E51)), ([52](#A2.E52)), ([54](#A2.E54)), ([55](#A2.E55)), ([56](#A2.E56)), the corresponding $\Phi_{-}$’s are not. We will characterize the $\Phi_{-}$’s and $\Psi_{-}$’s in Equations ([50](#A2.E50)), ([51](#A2.E51)), ([54](#A2.E54)) and ([55](#A2.E55)) below by Lemma [B.2](#A2.Thmtheorem2), Lemma [B.3](#A2.Thmtheorem3) and Lemma
[B.5](#A2.Thmtheorem5), respectively.
These lemma are stated as below (their proofs will be postponed to Section [B.4](#A2.SS4)).

###### Lemma B.2 .

For a pair $(s,t)$ such that $L\geqslant s>t\geqslant 0$ and $(s,t)\neq(L,0)$, if

$$ $\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[\mathbf{W}]^{(s,t)}_{pq},$ (58) $$

then $\Phi_{(s,t):pq}=0$ for all $p,q$.

###### Lemma B.3 .

For a pair $(s,t)$ such that $L\geqslant s>0$, $L>t\geqslant 0$, if

$$ $\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[\mathbf{WW}]^{(s,0)(L,t)}_{pq},$ (59) $$

then $\Phi_{(s,0)(L,t):pq}=0$ for all $p,q$ or $\Psi^{(s,0)(L,t)}=0$.

###### Lemma B.4 .

For a pair $(s,t)$ such that $L\geqslant s>t>0$, if

$$ $\displaystyle 0=\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[\mathbf{Wb}]^{(s,t)(t)}_{p},$ (60) $$

then $\Phi_{(s,t)(t):p}=0$ for all $p$.

###### Lemma B.5 .

For a pair $(s,t)$ such that $L\geqslant s>0$, $L>t\geqslant 0$, if

$$ $\displaystyle 0=\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[\mathbf{bW}]^{(s)(L,t)}_{pq},$ (61) $$

then $\Phi_{(s)(L,t):pq}=0$ for all $p,q$ or $\Psi^{(s)(L,t)}=0$.

###### Remark .

The reason that we skip the characterizations of $\Phi_{-}$’s and $\Psi_{-}$’s in Equations ([52](#A2.E52)) and ([56](#A2.E56)) is they can be concretely characterized. For instance, consider the case when $n_{L}=\ldots=n_{2}=n_{1}=n_{0}=1$. From Equation ([52](#A2.E52)), we have

$$ $\displaystyle 0$ $\displaystyle=\sum_{p=1}^{1}\sum_{q=1}^{1}\Phi_{(L,0):pq}\cdot[\mathbf{W}]^{(L,0)}_{pq}+\sum_{L>s>0}\sum_{p=1}^{1}\sum_{q=1}^{1}\Phi_{(s,0)(L,s):pq}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{pq}$ $\displaystyle=\Phi_{(L,0):1,1}\cdot[\mathbf{W}]^{(L,0)}_{1,1}+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{1,1}$ $\displaystyle=\Phi_{(L,0):1,1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)$ $\displaystyle\hskip 20.0pt+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot\left(\mathbf{x}^{(s)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)\cdot\Psi^{(s,0)(L,s)}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(s+2)}_{1,1}\cdot\mathbf{x}^{(s+1)}_{1,1}\right)$ $\displaystyle=\Phi_{(L,0):1,1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)$ $\displaystyle\hskip 20.0pt+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot\Psi^{(s,0)(L,s)}\cdot\left(\mathbf{x}^{(s)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(s+2)}_{1,1}\cdot\mathbf{x}^{(s+1)}_{1,1}\right)$ $\displaystyle=\Phi_{(L,0):1,1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)$ $\displaystyle\hskip 20.0pt+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot\Psi^{(s,0)(L,s)}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right)$ $\displaystyle=\left(\Phi_{(L,0):1,1}+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot\Psi^{(s,0)(L,s)}\right)\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(2)}_{1,1}\cdot\mathbf{x}^{(1)}_{1,1}\right).$ $$

It implies that

$$ $\displaystyle 0=\Phi_{(L,0):1,1}+\sum_{L>s>0}\Phi_{(s,0)(L,s):1,1}\cdot\Psi^{(s,0)(L,s)}.$ (62) $$

From Equation ([62](#A2.E62)), we can not derive a more concrete relation on the $\Phi_{-}$’s and $\Psi_{-}$’s. Similarly, from Equation ([56](#A2.E56)), we have

$$ $\displaystyle 0$ $\displaystyle=\sum_{p=1}^{1}\Phi_{(L,t)(t):p}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{p}+\sum_{p=1}^{1}\sum_{q=1}^{1}\Phi_{(t)(L,t):pq}\cdot[\mathbf{bW}]^{(t)(L,t)}_{pq}$ $\displaystyle=\Phi_{(L,t)(t):1}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{1}+\Phi_{(t)(L,t):1,1}\cdot[\mathbf{bW}]^{(t)(L,t)}_{1,1}$ $\displaystyle=\Phi_{(L,t)(t):1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)$ $\displaystyle\hskip 60.0pt+\Phi_{(t)(L,t):1,1}\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)\cdot\Psi^{(t)(L,t)}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)$ $\displaystyle=\Phi_{(L,t)(t):1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)$ $\displaystyle\hskip 60.0pt+\Phi_{(t)(L,t):1,1}\cdot\Psi^{(t)(L,t)}\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)$ $\displaystyle=\Phi_{(L,t)(t):1}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)$ $\displaystyle\hskip 60.0pt+\Phi_{(t)(L,t):1,1}\cdot\Psi^{(t)(L,t)}\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)$ $\displaystyle=\left(\Phi_{(L,t)(t):1}+\Phi_{(t)(L,t):1,1}\cdot\Psi^{(t)(L,t)}\right)\cdot\left(\mathbf{x}^{(L)}_{1,1}\cdot\ldots\cdot\mathbf{x}^{(t+2)}_{1,1}\cdot\mathbf{x}^{(t+1)}_{1,1}\right)\cdot\left(\mathbf{y}^{(t+1)}_{1,1}\right)$ $$

It implies that

$$ $\displaystyle 0=\Phi_{(L,t)(t):1}+\Phi_{(t)(L,t):1,1}\cdot\Psi^{(t)(L,t)}.$ (63) $$

From Equation ([63](#A2.E63)), we can not derive a more concrete relation on the $\Phi_{-}$’s and $\Psi_{-}$’s.

Combining the discussions above, we obtain the following necessary for the coefficients $\Phi$ and $\Psi$ such that $\alpha(\Phi,\Psi)=0$.

###### Theorem B.6 .

Let $\alpha(\Phi,\Psi)$ be a polynomial given in equation [B.3](#A2.Ex56).
If $\alpha(\Phi,\Psi)=0$, then the following condition holds:

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi_{(s,t):pq}=0.$
(64)
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s,0)(L,t):pq}=0,$
(65)
for all $p,q$, or
$\displaystyle\Psi^{(s,0)(L,t)}=0.$
(66)
- 3.
We have
$\displaystyle 0=\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[\mathbf{W}]^{(L,0)}_{pq}+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{pq}.$
(67)
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi_{(s,t)(t):p}=0.$
(68)
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s)(L,t):pq}=0,$
(69)
for all $p,q$, or
$\displaystyle\Psi^{(s)(L,t)}=0.$
(70)
- 6.
For all $L>t>0$, we have
$\displaystyle 0=\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[\mathbf{bW}]^{(t)(L,t)}_{pq}.$
(71)
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi_{(s):p}=0.$
(72)
- 8.
We have
$\displaystyle\Phi_{1}=0.$
(73)

###### Proof.

By the previous observations, $\alpha(\Phi,\Psi)=0$ implies four Equations ([46](#A2.E46))-([49](#A2.E49)). These equations imply Equations ([50](#A2.E50))-([57](#A2.E57)). The proof of all parts in Theorem [B.6](#A2.Thmtheorem6) are as follows.

- 1.
It comes from Equation ([50](#A2.E50)) and Lemma [B.2](#A2.Thmtheorem2).
- 2.
It comes from Equation ([51](#A2.E51)) and Lemma [B.3](#A2.Thmtheorem3).
- 3.
It comes from Equation ([52](#A2.E52)).
- 4.
It comes from Equation ([54](#A2.E54)) and Lemma [B.4](#A2.Thmtheorem4)
- 5.
It comes from Equation ([55](#A2.E55)) and Lemma [B.5](#A2.Thmtheorem5).
- 6.
It comes from Equation ([56](#A2.E56)).
- 7.
It comes from Equation ([53](#A2.E53)).
- 8.
It comes from Equation ([57](#A2.E57)).

The proof is finsihed.
∎

The following corollary is a direct consequence of Theorem [B.6](#A2.Thmtheorem6).

###### Corollary B.7 .

If for some $\Phi,\Phi^{\prime}$, and $\Psi$, we have $\alpha(\Phi,\Psi)=\alpha(\Phi^{\prime},\Psi)$, then:

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi_{(s,t):pq}=\Phi^{\prime}_{(s,t):pq}.$
(74)
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s,0)(L,t):pq}=\Phi^{\prime}_{(s,0)(L,t):pq},$
(75)
for all $p,q$, or
$\displaystyle\Psi^{(s,0)(L,t)}=0.$
(76)
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[\mathbf{W}]^{(L,0)}_{pq}+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{pq}$
(77)
$\displaystyle=~{}$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{\prime}_{(L,0):pq}\cdot[\mathbf{W}]^{(L,0)}_{pq}+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{\prime}_{(s,0)(L,s):pq}\cdot[\mathbf{WW}]^{(s,0)(L,s)}_{pq}.$
(78)
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi_{(s,t)(t):p}=\Phi^{\prime}_{(s,t)(t):p}.$
(79)
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s)(L,t):pq}=\Phi^{\prime}_{(s)(L,t):pq},$
(80)
for all $p,q$, or
$\displaystyle\Psi^{(s)(L,t)}=0.$
(81)
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[\mathbf{bW}]^{(t)(L,t)}_{pq}$
(82)
$\displaystyle=~{}$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{\prime}_{(L,t)(t):p}\cdot[\mathbf{Wb}]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{\prime}_{(t)(L,t):pq}\cdot[\mathbf{bW}]^{(t)(L,t)}_{pq}.$
(83)
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi_{(s):p}=\Phi^{\prime}_{(s):p}.$
(84)
- 8.
We have
$\displaystyle\Phi_{1}=\Phi^{\prime}_{1}.$
(85)

###### Proof.

Since

$$ $\displaystyle 0=\alpha(\Phi,\Psi)-\alpha(\Phi^{\prime},\Psi)=\alpha(\Phi-\Phi^{\prime},\Psi),$ $$

so the results come directly from Theorem [B.6](#A2.Thmtheorem6).
∎

###### Lemma B.2 .

###### Lemma B.3 .

###### Lemma B.4 .

###### Lemma B.5 .

###### Remark .

###### Theorem B.6 .

###### Proof.

###### Corollary B.7 .

###### Proof.

### B.4 Proofs of Lemmas B.2 - B.5

The proofs of Lemmas [B.2](#A2.Thmtheorem2) through [B.5](#A2.Thmtheorem5) are directly from the coefficient comparison of two equal polynomials and the following lemma. We omit the proofs of Lemmas [B.2](#A2.Thmtheorem2) through [B.5](#A2.Thmtheorem5) and show only the proof for Lemma [B.8](#A2.Thmtheorem8).

###### Lemma B.8 .

For a feasible tuple $(s,t,p,q)$, if

$$ $\displaystyle[\mathbf{WW}]^{(s,0)(L,t)}_{pq}=\left([\mathbf{W}]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L,t)}\right)_{pq}=0\in R.$ (86) $$

Then $\Psi^{(s,0)(L,t)}=0\in\mathbb{R}^{n_{0}\times n_{L}}$.

###### Proof.

We first consider the case where $s=L$ and $t=0$, then the case $s\leqslant t$, and finally the case $s>t$. Note that, the proof for the last case will be by combining the arguments of the first two cases.

###### Lemma B.8 .

###### Proof.

##### Case s = L 𝑠 𝐿 s=L and t = 0 𝑡 0 t=0 .

From Equation ([86](#A2.E86)), we have

$$ $\displaystyle[\mathbf{WW}]^{(L,0)(L,0)}_{pq}=\left([\mathbf{W}]^{(L,0)}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L,0)}\right)_{pq}=0,$ (87) $$

which means

$$ $\displaystyle\left([\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\right)_{pq}=0.$ (88) $$

Here, we consider three cases, where $L=1$, $L=2$ and $L\geqslant 3$.

- •
Case $L=1$. Equation ([88](#A2.E88)) becomes
$\displaystyle\left([\mathbf{W}]^{(1)}\cdot\Psi^{(1,0)(1,0)}\cdot[\mathbf{W}]^{(1)}\right)_{pq}=0.$
(89)
By Proposition [A.1](#A1.Thmtheorem1), this is equivalent to
$\displaystyle[\mathbf{W}]^{(1)}_{p,\ast}\cdot\Psi^{(1,0)(1,0)}\cdot[\mathbf{W}]^{(1)}_{\ast,q}=0,$
(90)
which is:
$\displaystyle\sum_{i=1}^{n_{0}}\sum_{j=0}^{n_{1}}\Psi^{(1,0)(1,0)}_{ij}\cdot\mathbf{x}^{(1)}_{p,i}\cdot\mathbf{x}^{(1)}_{j,q}=0.$
(91)
Since the LHS of Equation ([91](#A2.E91)) is a linear combination between distinct monomials
$\displaystyle\mathbf{x}^{(1)}_{p,i}\cdot\mathbf{x}^{(1)}_{j,q},$
(92)
for $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{1}$, so it implies that
$\displaystyle\Psi^{(1,0)(1,0)}_{ij}=0,$
(93)
for all $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{1}$, which means
$\displaystyle\Psi^{(1,0)(1,0)}=0.$
(94)
- •
Case $L=2$. Equation ([88](#A2.E88)) becomes
$\displaystyle\left([\mathbf{W}]^{(2)}\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(2,0)(2,0)}\cdot[\mathbf{W}]^{(2)}\cdot[\mathbf{W}]^{(1)}\right)_{pq}=0.$
(95)
By Proposition [A.1](#A1.Thmtheorem1), this is equivalent to
$\displaystyle[\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(2,0)(2,0)}\cdot[\mathbf{W}]^{(2)}\cdot[\mathbf{W}]^{(1)}_{\ast,q}=0,$
(96)
which is
$\displaystyle\left([\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,1}~{},~{}[\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,2}~{},~{}\ldots~{},~{}[\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,n_{0}}\right)\cdot\Psi^{(2,0)(2,0)}$
(97)
$\displaystyle\hskip 50.0pt\cdot\left([\mathbf{W}]^{(2)}_{1,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,q}~{},~{}[\mathbf{W}]^{(2)}_{2,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,q}~{},~{}\ldots~{},~{}[\mathbf{W}]^{(2)}_{n_{2},\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,q}\right)^{\top}=0,$
(98)
which is
$\displaystyle\sum_{i=1}^{n_{0}}\sum_{j=0}^{n_{2}}\Psi^{(2,0)(2,0)}_{ij}\cdot\left([\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,i}\right)\cdot\left([\mathbf{W}]^{(2)}_{j,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,q}\right)=0.$
(99)
Since the LHS of Equation ([99](#A2.E99)) is a linear combination between elements of a linear independent collection of $R$, which are:
$\displaystyle\left([\mathbf{W}]^{(2)}_{p,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,i}\right)\cdot\left([\mathbf{W}]^{(2)}_{j,\ast}\cdot[\mathbf{W}]^{(1)}_{\ast,q}\right),$
(100)
for $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{2}$, so it implies that
$\displaystyle\Psi^{(2,0)(2,0)}_{ij}=0,$
(101)
for all $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{2}$, which means
$\displaystyle\Psi^{(2,0)(2,0)}=0.$
(102)
- •
Case $L\geqslant 3$. Equation ([88](#A2.E88)) becomes
$\displaystyle\left([\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\right)_{pq}=0.$
(103)
It is noted that the matrices $[\mathbf{W}]^{(L-1)},\ldots,[\mathbf{W}]^{(2)}$ can be substituted for some matrix units at $1^{\text{st}}$-row and $1^{\text{st}}$-column such that the product
$\displaystyle[\mathbf{W}]^{(L-1)}\cdot\ldots\cdot[\mathbf{W}]^{(2)}$
(104)
becomes a matrix unit at $1^{\text{st}}$-row and $1^{\text{st}}$-column. Substitute this in Equation ([103](#A2.E103)), we have
$\displaystyle\left([\mathbf{W}]^{(L)}\cdot E_{11}\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}\cdot E_{11}\cdot[\mathbf{W}]^{(1)}\right)_{pq}=0.$
(105)
By Proposition [A.1](#A1.Thmtheorem1), this is equivalent to
$\displaystyle[\mathbf{W}]^{(L)}_{p,\ast}\cdot E_{11}\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}\cdot E_{11}\cdot[\mathbf{W}]^{(1)}_{\ast,q}=0.$
(106)
which can be rewritten as
$\displaystyle[\mathbf{W}]^{(L)}_{p,1}\cdot[\mathbf{W}]^{(1)}_{1,\ast}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}_{\ast,1}\cdot[\mathbf{W}]^{(1)}_{1,q}=0,$
(107)
or
$\displaystyle\mathbf{x}^{(L)}_{p,1}\cdot[\mathbf{W}]^{(1)}_{1,\ast}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}_{\ast,1}\cdot\mathbf{x}^{(1)}_{1,q}=0.$
(108)
Since the LHS of Equation ([108](#A2.E108)) is a polynomial in $R$, we have
$\displaystyle[\mathbf{W}]^{(1)}_{1,\ast}\cdot\Psi^{(L,0)(L,0)}\cdot[\mathbf{W}]^{(L)}_{\ast,1}=0,$
(109)
which is
$\displaystyle\sum_{i=1}^{n_{0}}\sum_{j=0}^{n_{L}}\Psi^{(L,0)(L,0)}_{ij}\cdot\mathbf{x}^{(1)}_{1,i}\cdot\mathbf{x}^{(L)}_{j,1}=0.$
(110)
Since the LHS of Equation ([110](#A2.E110)) is a linear combination between distinct monomials
$\displaystyle\mathbf{x}^{(1)}_{1,i}\cdot\mathbf{x}^{(L)}_{j,1},$
(111)
for $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{L}$, so it implies that
$\displaystyle\Psi^{(L,0)(L,0)}_{ij}=0,$
(112)
for all $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{L}$, which means
$\displaystyle\Psi^{(L,0)(L,0)}=0.$
(113)

In conclusion, for all cases of $L$, we have

$$ $\displaystyle\Psi^{(L,0)(L,0)}=0.$ (114) $$

We finish the proof of the case where $s=L$ and $t=0$.

##### Case s ⩽ t 𝑠 𝑡 s\leqslant t .

From Equation ([86](#A2.E86)), we have

$$ $\displaystyle[\mathbf{WW}]^{(s,0)(L,t)}_{pq}=\left([\mathbf{W}]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L,t)}\right)_{pq}=0,$ (115) $$

which means

$$ $\displaystyle\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)_{pq}=0.$ (116) $$

Since $s\leqslant t$, the $[\mathbf{W}]^{-}$’s that are in the product $[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}$, and the $[\mathbf{W}]^{-}$’s that are in the product $[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}$, are distinct. By directly multiplying $[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}$ and $[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}$, we can write these two products in the forms

$$ ${\begin{array}[]{lll}[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}&=\left(\mathbf{f}^{(s)}_{ij}\right)_{1\leqslant i\leqslant n_{s},1\leqslant j\leqslant n_{0}}&\in R^{n_{s}\times n_{0}},\\ \left[\mathbf{W}\right]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}&=\left(\mathbf{g}^{(t)}_{ij}\right)_{1\leqslant i\leqslant n_{L},1\leqslant j\leqslant n_{t}}&\in R^{n_{L}\times n_{t}},\end{array}}$ (117) $$

where all $\mathbf{f}^{(s)}_{-}$’s are nonzero and all $\mathbf{g}^{(t)}_{-}$’s are nonzero. Moreover, $\mathbf{f}^{(s)}_{-}$’s are real polynomials of indeterminates $\mathbf{x}^{(1)}_{-}$’s, $\mathbf{x}^{(2)}_{-}$’s, $\ldots$, $\mathbf{x}^{(s)}_{-}$’s. Similarly, $\mathbf{g}^{(t)}_{-}$’s are real polynomials of indeterminates $\mathbf{x}^{(L)}_{-}$’s, $\mathbf{x}^{(L-1)}_{-}$’s, $\ldots$, $\mathbf{x}^{(t+1)}_{-}$’s. Now, Equation ([116](#A2.E116)) is equal to

$$ $\displaystyle\left(\left(\mathbf{f}^{(s)}_{ij}\right)_{1\leqslant i\leqslant n_{s},1\leqslant j\leqslant n_{0}}\cdot\Psi^{(s,0)(L,t)}\cdot\left(\mathbf{g}^{(t)}_{ij}\right)_{1\leqslant i\leqslant n_{L},1\leqslant j\leqslant n_{t}}\right)_{pq}=0.$ (118) $$

By Proposition [A.1](#A1.Thmtheorem1), this is equivalent to

$$ $\displaystyle\left(\mathbf{f}^{(s)}_{p,1}~{},~{}\mathbf{f}^{(s)}_{p,2}~{},~{}\ldots~{},~{}\mathbf{f}^{(s)}_{p,n_{0}}\right)\cdot\Psi^{(s,0)(L,t)}\cdot\left(\mathbf{g}^{(t)}_{1,q}~{},~{}\mathbf{g}^{(t)}_{2,q}~{},~{}\ldots~{},~{}\mathbf{f}^{(t)}_{n_{L},q}\right)^{\top}=0,$ (119) $$

which is

$$ $\displaystyle\sum_{i=1}^{n_{0}}\sum_{j=0}^{n_{L}}\Psi^{(s,0)(L,t)}_{ij}\cdot\mathbf{f}^{(s)}_{p,i}\cdot\mathbf{g}^{(t)}_{j,q}=0.$ (120) $$

Since the LHS of Equation ([120](#A2.E120)) is a linear combination between elements of a linear independent collection of $R$, which are

$$ $\displaystyle\mathbf{f}^{(s)}_{p,i}\cdot\mathbf{g}^{(t)}_{j,q},$ (121) $$

for $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{L}$. The linear dependency comes from the distinction between indeterminates of $\mathbf{f}^{(-)}_{-}$ and $\mathbf{g}^{(-)}_{-}$. It implies that

$$ $\displaystyle\Psi^{(s,0)(L,t)}_{ij}=0,$ (122) $$

for all $1\leqslant i\leqslant n_{0}$ and $1\leqslant j\leqslant n_{L}$, which means

$$ $\displaystyle\Psi^{(s,0)(L,t)}=0.$ (123) $$

We finish the proof of the case where $s\leqslant t$.

##### Case s > t 𝑠 𝑡 s>t .

From Equation ([86](#A2.E86)), we have:

$$ $\displaystyle[\mathbf{WW}]^{(s,0)(L,t)}_{pq}=\left([\mathbf{W}]^{(s,0)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L,t)}\right)_{pq}=0,$ (124) $$

which means

$$ $\displaystyle\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)_{pq}=0.$ (125) $$

Assume that $\Psi^{(s,0)(L,t)}\neq 0\in\mathbb{R}^{n_{0}\times n_{L}}$. Observe that

$$ $\displaystyle~{}[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}$ (126) $\displaystyle=$ $\displaystyle\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\cdot[\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\right)$ (127) $\displaystyle\cdot\Psi^{(s,0)(L,t)}\cdot$ $\displaystyle\left([\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(s+1)}\cdot[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)$ $\displaystyle=$ $\displaystyle\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)$ (128) $\displaystyle\cdot\left([\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(s+1)}\right)\cdot$ $\displaystyle\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right).$ $$

In the second term of Equation ([128](#A2.E128)),

$$ $\displaystyle[\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(s+1)}.$ (129) $$

Since $s>t$, all $[\mathbf{W}]^{(-)}$’s in Equation ([129](#A2.E129)) are distinct. And since $\Psi^{(s,0)(L,t)}\neq 0\in\mathbb{R}^{n_{0}\times n_{L}}$, with the same argument as in Case $s\leqslant t$, we have

$$ $\displaystyle[\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(s+1)}\neq 0\in R^{n_{t}\times n_{s}}.$ (130) $$

Moreover, it can be written in the form

$$ $\displaystyle[\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot$ $\displaystyle\ldots\cdot[\mathbf{W}]^{(s+1)}$ (131) $\displaystyle=\left(\mathbf{c}^{(st)}_{ij}\right)_{1\leqslant i\leqslant n_{t},1\leqslant j\leqslant n_{s}}\in R^{n_{t}\times n_{s}},$ (132) $$

where all $\mathbf{c}^{(st)}_{-}$’s are real polynomials of indeterminates $\mathbf{x}^{(1)}_{-}$’s, $\mathbf{x}^{(2)}_{-}$’s, $\ldots$, $\mathbf{x}^{(t)}_{-}$’s and $\mathbf{x}^{(L)}_{-}$’s, $\mathbf{x}^{(L-1)}_{-}$’s, $\ldots$, $\mathbf{x}^{(s+1)}_{-}$’s, and at least one of them is a nonzero element in $R$. By Corollary [A.7](#A1.Thmtheorem7), indeterminates $\mathbf{x}^{(1)}_{-}$’s, $\mathbf{x}^{(2)}_{-}$’s, $\ldots$, $\mathbf{x}^{(t)}_{-}$’s and $\mathbf{x}^{(L)}_{-}$’s, $\mathbf{x}^{(L-1)}_{-}$’s, $\ldots$, $\mathbf{x}^{(s+1)}_{-}$’s can be substituted for some real values to make

$$ $\displaystyle[\mathbf{W}]^{(t)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(s+1)},$ (133) $$

become a nonzero matrix of $\mathbb{R}^{n_{t}\times n_{s}}$. We denote this nonzero matrix by $\overline{\Psi}^{(s,0)(L,t)}\in\mathbb{R}^{n_{t}\times n_{s}}$. Note that, since $s>t$, the substitution only applies for indeterminates in $[\mathbf{W}]^{(1)},[\mathbf{W}]^{(2)},\ldots,[\mathbf{W}]^{(t)}$ and $[\mathbf{W}]^{(L)},[\mathbf{W}]^{(L-1)},\ldots,[\mathbf{W}]^{(s+1)}$. In other words, it does not apply for $[\mathbf{W}]^{(s)},\ldots,[\mathbf{W}]^{(t+1)}$. So, with the above substitution, Equation ([126](#A2.E126)) becomes

$$ $\displaystyle~{}[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}$ $\displaystyle=$ $\displaystyle~{}[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\cdot\overline{\Psi}^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}.$ (134) $$

Note that, since $s>t$, there is at least one $[\mathbf{W}]^{(-)}$ in the product $[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}$. Combine with $\overline{\Psi}^{(s,0)(L,t)}\neq 0$, by applying the argument of Case $s=L$ and $t=0$, we have

$$ $\displaystyle~{}\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(1)}\cdot\Psi^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(L)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)_{pq}$ $\displaystyle=$ $\displaystyle~{}\left([\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\cdot\overline{\Psi}^{(s,0)(L,t)}\cdot[\mathbf{W}]^{(s)}\cdot\ldots\cdot[\mathbf{W}]^{(t+1)}\right)_{pq}\neq 0,$ (135) $$

which contradicts to Equation ([125](#A2.E125)). In conclusion

$$ $\displaystyle\Psi^{(s,0)(L,t)}=0.$ (136) $$

We finish the proof of the case where $s>t$.

In summary, we did consider all possible cases. The proof is finished.
∎

## Appendix C Equivariant Polynomial Layers

We now proceed to construct a $G$-equivariant polynomial layer, denoted as $E$. These layers serve as the fundamental building blocks for our MAGEP-NFNs. Our strategy is as follows: we first express $E$ as a polynomial layer that is a linear combination of stable polynomial terms (Subsection [C.1](#A3.SS1)). We then find the equivariant maps among these polynomial layers using the parameter sharing mechanism.

### C.1 Equivariant layer as a linear combination of stable polynomial terms

For two weight spaces $\mathcal{U}$ and $\mathcal{U}^{\prime}$ with the same number of layers $L$ as well as the same number of channels at
$i^{\text{th}}$ layer $n_{i}$,

$$ $\displaystyle\begin{array}[]{lllll}\mathcal{U}&=\hskip 10.0pt\mathcal{W}\times\mathcal{B},&\text{ where:}&&\\ \mathcal{W}&=\hskip 10.0pt\mathbb{R}^{w_{L}\times n_{L}\times n_{L-1}}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{w_{2}\times n_{2}\times n_{1}}&\times\hskip 10.0pt\mathbb{R}^{w_{1}\times n_{1}\times n_{0}},\\ \mathcal{B}&=\hskip 10.0pt\mathbb{R}^{b_{L}\times n_{L}\times 1}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{b_{2}\times n_{2}\times 1}&\times\hskip 10.0pt\mathbb{R}^{b_{1}\times n_{1}\times 1},\end{array}$ $$

and

$$ $\displaystyle\begin{array}[]{lllll}\mathcal{U}^{\prime}&=\hskip 10.0pt\mathcal{W}^{\prime}\times\mathcal{B}^{\prime},&\text{ where:}&&\\ \mathcal{W}^{\prime}&=\hskip 10.0pt\mathbb{R}^{w^{\prime}_{L}\times n_{L}\times n_{L-1}}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{w^{\prime}_{2}\times n_{2}\times n_{1}}&\times\hskip 10.0pt\mathbb{R}^{w^{\prime}_{1}\times n_{1}\times n_{0}},\\ \mathcal{B}^{\prime}&=\hskip 10.0pt\mathbb{R}^{b^{\prime}_{L}\times n_{L}\times 1}&\times\hskip 10.0pt\ldots&\times\hskip 10.0pt\mathbb{R}^{b^{\prime}_{2}\times n_{2}\times 1}&\times\hskip 10.0pt\mathbb{R}^{b^{\prime}_{1}\times n_{1}\times 1}.\end{array}$ $$

We want to build a map $E\colon\mathcal{U}\rightarrow\mathcal{U}^{\prime}$ such that $E$ is $G$-equivariant, where:

$$ $\displaystyle G=\{\operatorname{id}_{\mathcal{G}_{n_{L}}}\}\times\mathcal{G}^{>0}_{n_{L-1}}\times\ldots\times\mathcal{G}^{>0}_{n_{1}}\times\{\operatorname{id}_{\mathcal{G}_{n_{0}}}\}.$ (137) $$

Let us consider a polynomial map $E\colon\mathcal{U}\rightarrow\mathcal{U}^{\prime}$ such that, for input $U=([W],[b])$, each entry of the output $E(U)=([E(W)],[E(b)])$ is a linear combinations of stable polynomial terms, i.e the entries of $[W]^{(s,t)}$, $[b]^{(s)}$, $[Wb]^{(s,t)(t)}$, $[bW]^{(s)(L,t)}$, $[WW]^{(s,0)(L,t)}$, together with a bias.
In concrete:

$$ $\displaystyle[E(W)]^{(i)}_{jk}\coloneqq$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}+\Phi^{(i):jk}_{1},$ $\displaystyle[E(b)]^{(i)}_{j}\coloneqq$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}+\Phi^{(i):j}_{1}.$ $$

All $\Phi$’s are in $\mathbb{R}^{d^{\prime}\times d}$, except the biases $\Phi^{-}_{1}$’s are in $\mathbb{R}^{d^{\prime}\times 1}$. In summary, $E$ is parameterized by $\Phi_{-}$’s and $\Psi_{-}$’s.

In order to be $G$-equivariant, the polynomial map $E$ must satisfy the condition $E(gU)=gE(U)$ for all $g\in\mathcal{G}_{\mathcal{U}}$ and $U\in\mathcal{U}$. In the following subsections, we derive the computations of $E(gU)$ and $gE(U)$, and compare them in order to obtain all possible $G$-equivariant polynomial maps among those considered.

### C.2 Compute E ​ ( g ​ U ) 𝐸 𝑔 𝑈 E(gU)

We have

$$ $\displaystyle[E(gW)]^{(i)}_{jk}=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,t):pq}\cdot[gW]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s):p}\cdot[gb]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s,t)(t):p}\cdot[gWgb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s)(L,t):pq}\cdot[gbgW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,0)(L,t):pq}\cdot[gWgW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}^{(i):jk}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[W]^{(s,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s):p}\cdot d^{(s)}_{p}\cdot[b]^{(s)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s,t)(t):p}\cdot d^{(s)}_{p}\cdot[Wb]^{(s,t)(t)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[bW]^{(s)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,0)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[WW]^{(s,0)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\Phi_{1}^{(i):jk}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[W]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):jk}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[WW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}^{(i):jk}$ $\displaystyle[E(gb)]^{(i)}_{j}=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,t):pq}\cdot[gW]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s):p}\cdot[gb]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s,t)(t):p}\cdot[gWgb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s)(L,t):pq}\cdot[gbgW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,0)(L,t):pq}\cdot[gWgW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}^{(i):j}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[W]^{(s,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s):p}\cdot d^{(s)}_{p}\cdot[b]^{(s)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s,t)(t):p}\cdot d^{(s)}_{p}\cdot[Wb]^{(s,t)(t)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[bW]^{(s)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,0)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[WW]^{(s,0)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\Phi_{1}^{(i):j}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[W]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi^{(i):j}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[WW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}^{(i):j}.$ $$

Note that, we can move around the $\pi$’s in above equations since the group $G$ satisfy that: $G\cap\mathcal{P}_{i}$ is trivial (for $i=0$ or $i=L$) or the whole $\mathcal{P}_{i}$ (for $0<i<L$).

### C.3 Compute g ​ E ​ ( U ) 𝑔 𝐸 𝑈 gE(U)

We have:

$$ $\displaystyle[g(E(W))]^{(i)}_{jk}=$ $\displaystyle~{}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot[W^{\prime}]^{(i)}_{\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi_{1}^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}$ $\displaystyle[g(E(b))]^{(i)}_{j}=$ $\displaystyle~{}d^{(i)}_{j}\cdot[b^{\prime}]^{(i)}_{\pi_{i}^{-1}(j)}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+d^{(i)}_{j}\cdot\Phi_{1}^{(i):\pi_{i}^{-1}(j)}$ $$

### C.4 Compare E ​ ( g ​ U ) 𝐸 𝑔 𝑈 E(gU) and g ​ E ​ ( U ) 𝑔 𝐸 𝑈 gE(U)

Since $E(gU)=gE(U)$, from Corollary [B.7](#A2.Thmtheorem7), the parameters $\Phi^{-}_{-}$’s have to satisfy these following conditions:

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi^{(i):jk}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,t):pq}$
$\displaystyle\Phi^{(i):j}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,t):pq}$
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):jk}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,0)(L,t):pq}$
$\displaystyle\Phi^{(i):j}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,0)(L,t):pq}$
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):jk}_{(L,0):\pi_{L}(p)\pi_{0}(q)}\cdot\frac{d^{(L)}_{\pi_{L}(p)}}{d^{(0)}_{\pi_{0}(q)}}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):jk}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):j}_{(L,0):\pi_{L}(p)\pi_{0}(q)}\cdot\frac{d^{(L)}_{\pi_{L}(p)}}{d^{(0)}_{\pi_{0}(q)}}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):j}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi^{(i):jk}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s,t)(t):p}$
$\displaystyle\Phi^{(i):j}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s,t)(t):p}.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):jk}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s)(L,t):pq}$
$\displaystyle\Phi^{(i):j}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s)(L,t):pq}$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):jk}_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):jk}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):j}_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):j}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi^{(i):jk}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}_{(s):p}$
$\displaystyle\Phi^{(i):j}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi^{(i):\pi_{i}^{-1}(j)}_{(s):p}.$
- 8.
We have
$\displaystyle\Phi_{1}^{(i):jk}$
$\displaystyle=\frac{d^{(i)}_{j}}{d^{(i-1)}_{k}}\cdot\Phi_{1}^{(i):\pi_{i}^{-1}(j)\pi_{i-1}^{-1}(k)}$
$\displaystyle\Phi_{1}^{(i):j}$
$\displaystyle=d^{(i)}_{j}\cdot\Phi_{1}^{(i):\pi_{i}^{-1}(j)}$

Also, since the group $G$ satisfy that: $G\cap\mathcal{P}_{i}$ is trivial (for $i=0$ or $i=L$) or the whole $\mathcal{P}_{i}$ (for $0<i<L$), so we can simplify the above conditions by moving some of the permutation $\pi$’s to the left hand sides. We have

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,t):pq}$
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,t):pq}$
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,0)(L,t):pq}$
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,0)(L,t):pq}$
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,0):\pi_{L}(p)\pi_{0}(q)}\cdot\frac{d^{(L)}_{\pi_{L}(p)}}{d^{(0)}_{\pi_{0}(q)}}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)}_{(L,0):\pi_{L}(p)\pi_{0}(q)}\cdot\frac{d^{(L)}_{\pi_{L}(p)}}{d^{(0)}_{\pi_{0}(q)}}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,t)(t):p}$
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,t)(t):p}.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s)(L,t):pq}$
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s)(L,t):pq}$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)}_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s):p}$
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s):p}.$
- 8.
We have
$\displaystyle\Phi_{1}^{(i):\pi_{i}(j)\pi_{i-1}(k)}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi_{1}^{(i):jk}$
$\displaystyle\Phi_{1}^{(i):\pi_{i}(j)}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi_{1}^{(i):j}$

From the above equalities, we can determine all constraints for the coefficients according to the entries of $[E(W)]$ as follows:

- 1.
•
If $(s,t)=(i,i-1)\notin\{(L,L-1),(1,0)\}$, $p=j$, $q=k$,
$\displaystyle\Phi^{(i):\pi(j)\pi^{\prime}(k)}_{(i,i-1):\pi(j)\pi^{\prime}(k)}$
$\displaystyle=\Phi^{(i):jk}_{(i,i-1):jk}$
•
If $(s,t)=(i,i-1)=(L,L-1)$, $q=k$,
$\displaystyle\Phi^{(L):j\pi_{L-1}(k)}_{(L,L-1):p\pi_{L-1}(k)}$
$\displaystyle=\Phi^{(L):jk}_{(L,L-1):pk}$
•
If $(s,t)=(i,i-1)=(1,0)$, $p=j$,
$\displaystyle\Phi^{(1):\pi_{1}(j)k}_{(1,0):\pi_{1}(j)q}$
$\displaystyle=\Phi^{(1):jk}_{(1,0):jq}$
- 2.
•
If $(s,t)=(i,i-1)\notin\{(L,L-1),(1,0)\}$, $p=j$, $q=k$,
$\displaystyle\Phi^{(i):\pi(j)\pi^{\prime}(k)}_{(i,0)(L,i-1):\pi(j)\pi^{\prime}(k)}$
$\displaystyle=\Phi^{(i):jk}_{(i,0)(L,i-1):jk}$
•
If $(s,t)=(i,i-1)=(L,L-1)$, $q=k$,
$\displaystyle\Phi^{(L):j\pi_{L-1}(k)}_{(L,0)(L,L-1):p\pi_{L-1}(k)}$
$\displaystyle=\Phi^{(L):jk}_{(L,0)(L,L-1):pk}$
•
If $(s,t)=(i,i-1)=(1,0)$, $p=j$,
$\displaystyle\Phi^{(1):\pi_{1}(j)k}_{(1,0)(L,0):\pi_{1}(j)q}$
$\displaystyle=\Phi^{(1):jk}_{(1,0)(L,0):jq}$
- 3.
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
For each $L>l>0$, by scaling all the $d^{(l)}_{-}$’s by the same scalar, we have
$\displaystyle 0=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):jk}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):jk}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s,t)(t):p}.$
We have
$\displaystyle\Phi^{(i):jk}_{(s,t)(t):p}=0.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(s)(L,t):pq}.$
•
If $(s,t)=(i,i-1)\notin\{(L,L-1),(1,0)\}$, $p=j$, $q=k$,
$\displaystyle\Phi^{(i):\pi(j)\pi^{\prime}(k)}_{(i)(L,i-1):\pi(j)\pi^{\prime}(k)}$
$\displaystyle=\Phi^{(i):jk}_{(i)(L,i-1):jk}.$
•
If $i=L$, $(s,t)=(L,L-1)$, $q=k$,
$\displaystyle\Phi^{(L):j\pi(k)}_{(L)(L,L-1):p\pi(k)}=\Phi^{(L):jk}_{(L)(L,L-1):pk}$
•
If $i=1$, $(s,t)=(1,0)$, $p=j$,
$\displaystyle\Phi^{(1):\pi(j)k}_{(1)(L,0):\pi(j)q}=\Phi^{(1):jk}_{(1)(L,0):jq}.$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
For each $L>l>0$, by scaling all the $d^{(l)}_{-}$’s by the same scalar, we have
$\displaystyle 0=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)\pi_{i-1}(k)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\frac{d^{(i)}_{\pi_{i}(j)}}{d^{(i-1)}_{\pi_{i-1}(k)}}\cdot\Phi^{(i):jk}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
- 7.
If $i=s=1$, $p=j$,
$\displaystyle\Phi^{(1):\pi(j)k}_{(1):\pi(j)}=\Phi^{(1):jk}_{(1):j}.$
- 8.
We have
$\displaystyle\Phi_{1}^{(i):jk}=0.$

Similarly, we can also determine all constraints for the coefficients according to the entries of $[E(b)]$ as follows:

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,t):pq}$
$(s,t)=(i,0)$, $p=j$,
$\displaystyle\Phi^{(i):\pi(j)}_{(i,0):\pi(j)q}=\Phi^{(i):j}_{(i,0):jq}$
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,0)(L,t):pq}$
•
$t=0$, $s=i<L$, $p=j$,
$\displaystyle\Phi^{(i):\pi(j)}_{(i,0)(L,0):\pi(j)q}\cdot=\Phi^{(i):j}_{(i,0)(L,0):jq}$
•
$t=0$, $s=i=L$,
$\displaystyle\Phi^{(L):j}_{(L,0)(L,0):pq}=\Phi^{(L):j}_{(L,0)(L,0):pq}$
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
•
If $L>i>0$. For each $L>l>0$, by scaling all the $d^{(l)}_{-}$’s by the same scalar, we have
$\displaystyle 0=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(i):\pi_{i}(j)}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 50.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(i):\pi_{i}(j)}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 50.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
•
If $i=L$. We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 50.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 50.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
which means $\Phi^{(L):j}_{(L,0):pq}$ can be arbitrary. The rest is
$\displaystyle\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
For an $L>r>0$, by letting $\pi_{r}$ be the identity, and $d^{(r)}_{p}$ be $1$ for all $p$, we have
$\displaystyle\sum_{L>s>0,s\neq r}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{L>s>0,s\neq r}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq},$
so
$\displaystyle\sum_{p=1}^{n_{r}}\sum_{q=1}^{n_{r}}\Phi^{(L):j}_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(q)}\cdot\frac{d^{(r)}_{\pi_{r}(p)}}{d^{(r)}_{\pi_{r}(q)}}\cdot[WW]^{(r,0)(L,r)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{r}}\sum_{q=1}^{n_{r}}\Phi^{(L):j}_{(r,0)(L,r):pq}\cdot[WW]^{(r,0)(L,r)}_{pq}$
By Lemma [B.3](#A2.Thmtheorem3) and Corollary [A.7](#A1.Thmtheorem7), we have
$\displaystyle\Phi^{(L):j}_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(q)}\cdot\frac{d^{(r)}_{\pi_{r}(p)}}{d^{(r)}_{\pi_{r}(q)}}=\Phi^{(L):j}_{(r,0)(L,r):pq}.$
So
$\displaystyle\Phi^{(L):j}_{(r,0)(L,r):pq}=0$
for $p\neq q$, and
$\displaystyle\Phi^{(L):j}_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(p)}=\Phi^{(L):j}_{(r,0)(L,r):pp}.$
In conclusion, we have $\Phi^{(L):j}_{(L,0):pq}$ is arbitrary, and for $L>s>0$,
$\displaystyle\Phi^{(L):j}_{(s,0)(L,s):\pi(p)\pi(p)}=\Phi^{(L):j}_{(s,0)(L,s):pp}.$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s,t)(t):p}.$
If $i=s$, $p=j$,
$\displaystyle\Phi^{(i):\pi(j)}_{(i,t)(t):\pi(j)}=\Phi^{(i):j}_{(i,t)(t):j}.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s)(L,t):pq}$
•
$s=i<L$, $t=0$, $p=j$,
$\displaystyle\Phi^{(i):\pi(j)}_{(i)(L,0):\pi(j)q}=\Phi^{(i):j}_{(i)(L,0):jq}$
•
$s=i=L$, $t=0$
$\displaystyle\Phi^{(L):j}_{(L)(L,0):pq}=\Phi^{(L):j}_{(L)(L,0):pq}$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)}_{(L,t)(t):p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
•
If $L>i>0$. For each $L>l>0$, by scaling all the $d^{(l)}_{-}$’s by the same scalar, we have
$\displaystyle 0=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(i):\pi_{i}(j)}_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(i):\pi_{i}(j)}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
•
If $i=L$. We have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(L):j}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\cdot\Phi^{(L):j}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
which means $\Phi^{(L):j}_{(L,t)(t):p)}$ can be arbitrary. The rest is
$\displaystyle\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(L):j}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi^{(L):j}_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
By Lemma [B.5](#A2.Thmtheorem5) and Corollary [A.7](#A1.Thmtheorem7), we have
$\displaystyle\Phi^{(L):j}_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}=\Phi^{(L):j}_{(t)(L,t):pq}$
So
$\displaystyle\Phi^{(L):j}_{(t)(L,t):pq}=0$
for $p\neq q$, and
$\displaystyle\Phi^{(L):j}_{(t)(L,t):\pi_{t}(p)\pi_{t}(p)}=\Phi^{(L):j}_{(t)(L,t):pp}$
In conclusion, for all $L>t>0$, we have $\Phi^{(L):j}_{(L,t)(t):p)}$ is arbitrary, and
$\displaystyle\Phi^{(L):j}_{(t)(L,t):\pi(p)\pi(p)}=\Phi^{(L):j}_{(t)(L,t):pp}.$
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi^{(i):\pi_{i}(j)}_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi^{(i):j}_{(s):p}.$
•
If $i=s<L$, $p=j$,
$\displaystyle\Phi^{(i):\pi(j)}_{(i):\pi(j)}=\Phi^{(i):j}_{(i):j}.$
•
If $i=s=L$,
$\displaystyle\Phi^{(L):j}_{(L):p}=\Phi^{(L):j}_{(L):p}.$
- 8.
We have
$\displaystyle\Phi_{1}^{(i):\pi_{i}(j)}$
$\displaystyle=d^{(i)}_{\pi_{i}(j)}\cdot\Phi_{1}^{(i):j}$
•
If $i=L$, we have
$\displaystyle\Phi_{1}^{(L):j}=\Phi_{1}^{(L):j}.$

### C.5 G 𝐺 G -Equivariant polynomial layers

Based on the above discussions, we conclude that every $G$-equivariant polynomial layer , which is defined as a linear combination of stable polynomial terms, is given as $E(U)=([E(W)],[E(b)])$, where the entries of $[E(W)]$ and $[E(b)]$ are given case by case as folows:

- •
For $i=L$, we have
$\displaystyle[E(W)]^{(L)}_{jk}=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi^{(L):j\bullet}_{(L,L-1):p\bullet}\cdot[W]^{(L,L-1)}_{pk}+\sum_{p=1}^{n_{L}}\Phi^{(L):j\bullet}_{(L,0)(L,L-1):p\bullet}\cdot[WW]^{(L,0)(L,L-1)}_{pk}$
$\displaystyle+\sum_{p=1}^{n_{L}}\Phi^{(L):j\bullet}_{(L)(L,L-1):p\bullet}\cdot[bW]^{(L)(L,L-1)}_{pk}$
$\displaystyle[E(b)]^{(L)}_{j}=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):\bullet\bullet}\cdot[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$
$\displaystyle+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\sum_{p=1}^{n_{t}}\Phi^{(L):j}_{(t)(L,t):\bullet\bullet}\cdot[bW]^{(t)(L,t)}_{pp}$
$\displaystyle+\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}^{(L):j}$
$\displaystyle[E(b)]^{(L)}_{j}=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle+\sum_{L>s>0}\Phi^{(L):j}_{(s,0)(L,s):\bullet\bullet}\cdot\sum_{p=1}^{n_{s}}[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$
$\displaystyle+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\Phi^{(L):j}_{(t)(L,t):\bullet\bullet}\cdot\sum_{p=1}^{n_{t}}[bW]^{(t)(L,t)}_{pp}$
$\displaystyle+\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}^{(L):j}$
- •
For $i=1$, we have
$\displaystyle[E(W)]^{(1)}_{jk}=$
$\displaystyle\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq}+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq}$
$\displaystyle+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq}+\Phi^{(1):\bullet k}_{(1):\bullet}\cdot[b]^{(1)}_{j}$
$\displaystyle[E(b)]^{(1)}_{j}=$
$\displaystyle\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq}+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq}$
$\displaystyle+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq}+\Phi^{(1):\bullet}_{(1):\bullet}\cdot[b]^{(1)}_{j}$
- •
For $L>i>1$, we have
$\displaystyle[E(W)]^{(i)}_{jk}=$
$\displaystyle\Phi^{(i):\bullet\bullet}_{(i,i-1):\bullet\bullet}\cdot[W]^{(i,i-1)}_{jk}+\Phi^{(i):\bullet\bullet}_{(i,0)(L,i-1):\bullet\bullet}\cdot[WW]^{(i,0)(L,i-1)}_{jk}$
$\displaystyle+\Phi^{(i):\bullet\bullet}_{(i)(L,i-1):\bullet\bullet}\cdot[bW]^{(i)(L,i-1)}_{jk}$
$\displaystyle[E(b)]^{(i)}_{j}=$
$\displaystyle\sum_{q=1}^{n_{0}}\Phi^{(i):\bullet}_{(i,0):\bullet q}\cdot[W]^{(i,0)}_{jq}+\sum_{q=1}^{n_{0}}\Phi^{(i):\bullet}_{(i,0)(L,0):\bullet q}\cdot[WW]^{(t,0)(L,0)}_{jq}$
$\displaystyle+\sum_{i>t>0}\sum_{p=1}^{n_{i}}\Phi^{(i):\bullet}_{(i,t)(t):\bullet}\cdot[Wb]^{(i,t)(t)}_{j}+\sum_{q=1}^{n_{t}}\Phi^{(i):\bullet}_{(i)(L,0):\bullet q}\cdot[bW]^{(i)(L,0)}_{jq}$
$\displaystyle+\Phi^{(i):\bullet}_{(i):\bullet}\cdot[b]^{(i)}_{j}$

In the above formulas, the bullet $\bullet$ indicates that the corresponding coefficient is independent of the indices at the bullet.

## Appendix D Invariant Polynomial Layers

In this section, we construct a polynomial map $I\colon\mathcal{U}\rightarrow\mathbb{R}^{d^{\prime}}$ that is $G$-invariant, i.e., $I(gU)=I(U)$ for all $g\in\mathcal{G}_{\mathcal{U}}$ and $U\in\mathcal{U}$.

### D.1 Invariant layer as a linear combination of stable polynomial terms

Similar to the equivariant maps, we seek the invariant map among polynomial maps that is a linear combination of stable polynomial terms, specifically the entries of $[W]^{(s,t)}$, $[b]^{(s)}$, $[Wb]^{(s,t)(t)}$, $[bW]^{(s)(L,t)}$, and $[WW]^{(s,0)(L,t)}$, along with a bias.
In concrete:

$$ $\displaystyle I(U)\coloneqq$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[W]^{(s,t)}_{pq}+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[WW]^{(s,0)(L,t)}_{pq}+\Phi_{1}.$ $$

All $\Phi_{-}$’s are in $\mathbb{R}^{d^{\prime}\times d}$, except the bias $\Phi_{1}$ is in $\mathbb{R}^{d^{\prime}\times 1}$. In summary, $I$ is parameterized by $\Phi_{-}$’s and $\Psi_{-}$’s. We need $I$ to be $G$-invariant, which means $I(gU)=I(U)$.

### D.2 Compute I ​ ( g ​ U ) 𝐼 𝑔 𝑈 I(gU)

From the definition of $I(U)$, we have:

$$ $\displaystyle I(gU)=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot[gW]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot[gb]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot[gWgb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot[gbgW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot[gWgW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[W]^{(s,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):p}\cdot d^{(s)}_{p}\cdot[b]^{(s)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):p}\cdot d^{(s)}_{p}\cdot[Wb]^{(s,t)(t)}_{\pi^{-1}_{s}(p)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[bW]^{(s)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):pq}\cdot\frac{d^{(s)}_{p}}{d^{(t)}_{q}}\cdot[WW]^{(s,0)(L,t)}_{\pi_{s}^{-1}(p),\pi_{t}^{-1}(q)}$ $\displaystyle+\Phi_{1}$ $\displaystyle=$ $\displaystyle\sum_{L\geqslant s>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[W]^{(s,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{p=1}^{n_{s}}\Phi_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[b]^{(s)}_{p}$ $\displaystyle+\sum_{L\geqslant s>t>0}\sum_{p=1}^{n_{s}}\Phi_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}\cdot[Wb]^{(s,t)(t)}_{p}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(s)(L,t)}_{pq}$ $\displaystyle+\sum_{L\geqslant s>0}\sum_{L>t\geqslant 0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{t}}\Phi_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[WW]^{(s,0)(L,t)}_{pq}$ $\displaystyle+\Phi_{1}.$ $$

### D.3 Compare I ​ ( g ​ U ) 𝐼 𝑔 𝑈 I(gU) and I ​ ( U ) 𝐼 𝑈 I(U)

Since $I(gU)=I(U)$, from Corollary [B.7](#A2.Thmtheorem7), the parameters $\Phi^{-}_{-}$’s have to satisfy these following conditions:

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi_{(s,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}=\Phi_{(s,t):pq}$
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\Phi_{(s,0)(L,t):pq}$
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):\pi_{L}(p)\pi_{0}(q)}\cdot\frac{d^{(L)}_{\pi_{L}(p)}}{d^{(0)}_{\pi_{0}(q)}}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi_{(s,t)(t):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\Phi_{(s,t)(t):p}.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\Phi_{(s)(L,t):pq}$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):\pi_{L}(p)}\cdot d^{(L)}_{\pi_{L}(p)}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}$
$\displaystyle\hskip 80.0pt+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\Phi_{(s):p}.$
- 8.
We have
$\displaystyle\Phi_{1}$
$\displaystyle=\Phi_{1}.$

Solve these equations, we have

- 1.
For all $L\geqslant s>t\geqslant 0$ with $(s,t)\neq(L,0)$, and for all $p,q$, we have
$\displaystyle\Phi_{(s,t):pq}=0$
- 2.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s,0)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\Phi_{(s,0)(L,t):pq}$
If $(s,t)=(L,0$, we have
$\displaystyle\Phi_{(L,0)(L,0):pq}=\Phi_{(L,0)(L,0):pq}.$
- 3.
We have
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$
$\displaystyle\hskip 80.0pt+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
which means $\Phi_{(L,0):pq}$ can be arbitrary. The rest is
$\displaystyle\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{L>s>0}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq}$
For an $L>r>0$, by letting $\pi_{r}$ be the identity, and $d^{(r)}_{p}$ be $1$ for all $p$, we have
$\displaystyle\sum_{L>s>0,s\neq r}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):\pi_{s}(p)\pi_{s}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(s)}_{\pi_{s}(q)}}\cdot[WW]^{(s,0)(L,s)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{L>s>0,s\neq r}\sum_{p=1}^{n_{s}}\sum_{q=1}^{n_{s}}\Phi_{(s,0)(L,s):pq}\cdot[WW]^{(s,0)(L,s)}_{pq},$
so
$\displaystyle\sum_{p=1}^{n_{r}}\sum_{q=1}^{n_{r}}\Phi_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(q)}\cdot\frac{d^{(r)}_{\pi_{r}(p)}}{d^{(r)}_{\pi_{r}(q)}}\cdot[WW]^{(r,0)(L,r)}_{pq}$
$\displaystyle\hskip 150.0pt=\sum_{p=1}^{n_{r}}\sum_{q=1}^{n_{r}}\Phi_{(r,0)(L,r):pq}\cdot[WW]^{(r,0)(L,r)}_{pq}$
By Lemma [B.3](#A2.Thmtheorem3) and Corollary [A.7](#A1.Thmtheorem7), we have
$\displaystyle\Phi_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(q)}\cdot\frac{d^{(r)}_{\pi_{r}(p)}}{d^{(r)}_{\pi_{r}(q)}}=\Phi_{(r,0)(L,r):pq}.$
So
$\displaystyle\Phi_{(r,0)(L,r):pq}=0,$
for $p\neq q$, and
$\displaystyle\Phi_{(r,0)(L,r):\pi_{r}(p)\pi_{r}(p)}=\Phi_{(r,0)(L,r):pp}.$
In conclusion, we have $\Phi_{(L,0):pq}$ is arbitrary, and for $L>s>0$,
$\displaystyle\Phi_{(s,0)(L,s):\pi(p)\pi(p)}=\Phi_{(s,0)(L,s):pp}.$
- 4.
For all $L>s>t>0$, and for all $p$, we have
$\displaystyle\Phi_{(s,t)(t):p}=0.$
- 5.
For all $L\geqslant s>0$, $L>t\geqslant 0$ with $s\neq t$, we have
$\displaystyle\Phi_{(s)(L,t):\pi_{s}(p)\pi_{t}(q)}\cdot\frac{d^{(s)}_{\pi_{s}(p)}}{d^{(t)}_{\pi_{t}(q)}}$
$\displaystyle=\Phi_{(s)(L,t):pq}$
If $(s,t)=(L,0)$, we have
$\displaystyle\Phi_{(L)(L,0):pq}=\Phi_{(L)(L,0):pq}.$
- 6.
For all $L>t>0$, we have
$\displaystyle\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}$
$\displaystyle=$
$\displaystyle\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
which means $\Phi_{(L,t)(t):p}$ can be arbitrary. The rest is
$\displaystyle\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}\cdot[bW]^{(t)(L,t)}_{pq}=\sum_{p=1}^{n_{t}}\sum_{q=1}^{n_{t}}\Phi_{(t)(L,t):pq}\cdot[bW]^{(t)(L,t)}_{pq}$
By Lemma [B.5](#A2.Thmtheorem5) and Lemma [A.7](#A1.Thmtheorem7), we have
$\displaystyle\Phi_{(t)(L,t):\pi_{t}(p)\pi_{t}(q)}\cdot\frac{d^{(t)}_{\pi_{t}(p)}}{d^{(t)}_{\pi_{t}(q)}}=\Phi_{(t)(L,t):pq}.$
So
$\displaystyle\Phi_{(t)(L,t):pq}=0$
for $p\neq q$, and
$\displaystyle\Phi_{(t)(L,t):\pi_{t}(p)\pi_{t}(p)}=\Phi_{(t)(L,t):pp}.$
In conclusion, for all $L>t>0$, we have $\Phi_{(L,t)(t):p}$ is arbitrary, and
$\displaystyle\Phi_{(t)(L,t):\pi(p)\pi(p)}=\Phi_{(t)(L,t):pp}$
- 7.
For all $L\geqslant s>0$ and for all $p$, we have
$\displaystyle\Phi_{(s):\pi_{s}(p)}\cdot d^{(s)}_{\pi_{s}(p)}$
$\displaystyle=\Phi_{(s):p}.$
If $s=L$, we have
$\displaystyle\Phi_{(L):p}=\Phi_{(L):p}.$
- 8.
We have
$\displaystyle\Phi_{1}$
$\displaystyle=\Phi_{1}.$

### D.4 G 𝐺 G -Invariant polynomial layers

Based on the above discussions, we conclude that every $G$-invariant polynomial layer, which is defined as a linear combination of stable polynomial terms, is given as:

$$ $\displaystyle I(U)=$ $\displaystyle\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$ $\displaystyle+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\Phi_{(s,0)(L,s):\bullet\bullet}\cdot[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$ $\displaystyle+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\sum_{p=1}^{n_{t}}\Phi_{(t)(L,t):\bullet\bullet}\cdot[bW]^{(t)(L,t)}_{pp}$ $\displaystyle+\sum_{p=1}^{n_{L}}\Phi_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}$ $$

In the above formula, the bullet $\bullet$ indicates that the corresponding coefficient is independent of the index at the bullet.

## Appendix E Additional Experimental Details

### E.1 Predicting generalization from weights

##### Dataset.

The $\operatorname{Tanh}$ subset from the CNN Zoo dataset has 5,949 training instances and 1,488 testing instances, while the original $\operatorname{ReLU}$ subset consists of 6,050 training instances and 1,513 testing instances. We do the augmentation for $\operatorname{ReLU}$ subset, with an augmentation factor of 2, effectively doubling the size of the dataset by adding one augmented version of each original instance. The overall dataset sizes, including both the original and augmented data, are summarized in Table [8](#A5.T8).

##### Baselines

In this experiment, we compare our model with five other baselines:

- •
STATNN []: utilizes statistical features of the weights and biases
- •
Graph-NN []: represents input network parameters as graphs and processes using Graph Neural Networks.
- •
NP and HNP []: incoporates the permutation symmetries of neurons into neural functional networks.
- •
Monomial-NFN []: extends the group action on weights from group of permutation matrices to the group of monomial matrices by incoporates scaling/sign-flipping symmetries.

**Table 5: Datasets information for predicting generalization task.**
| Dataset | Train size | Val size |
| --- | --- | --- |
| Original $\operatorname{ReLU}$ | 6050 | 1513 |
| Augment $\operatorname{ReLU}$ | 12100 | 3026 |
| $\operatorname{Tanh}$ | 5949 | 1488 |

**Table 9: Hyperparameters of Monomial-NFN for each dataset in Classify INRs task.**
|  | CIFAR-10 | MNIST | Fashion-MNIST |
| --- | --- | --- | --- |
| MLP | 2M | 2M | 2M |
| NP | 16M | 15M | 15M |
| HNP | 42M | 22M | 22M |
| Monomial-NFN | 16M | 22M | 20M |
| MAGEP-NFN (ours) | 3.4M | 4.1M | 4.9M |

##### Implementation Details.

Our architecture begins with a Monomial-NFN layer featuring 20 channels, aligning the dimensions of the weights and biases. This is followed by two equivariant MAGEP-NFN layers, each with 20 channels, using either a $\operatorname{ReLU}$ activation for the $\operatorname{ReLU}$ dataset or a $\operatorname{Tanh}$ activation for the $\operatorname{Tanh}$ dataset. Next, the output is processed by a MAGEP-NFN Invariant layer. The final output from this layer is flattened and mapped to $\mathbb{R}^{500}$. This vector is further processed by a fully connected MLP with two hidden layers, both activated by $\operatorname{ReLU}$. The final output undergoes a linear projection to a scalar, followed by a sigmoid function. The model is trained using Binary Cross Entropy (BCE) loss over 50 epochs, with early stopping determined by a threshold $\tau$ on the validation set. The entire training process on an A100 GPU takes 30 minutes. A summary of hyperparameters can be found in Table [8](#A5.T8).

For the baseline models, we adhere to the original implementations as outlined in [], utilizing the official code (available at: [https://github.com/AllanYangZhou/nfn](https://github.com/AllanYangZhou/nfn)), and []. In the HNP, NP, and Monomial-NFN models, we employ three equivariant layers with channel configurations of 16, 16, and 5, respectively. The extracted features are then passed through an average pooling layer, followed by three MLP layers with hidden dimensions of 200 (Monomial-NFN $\operatorname{ReLU}$ case) and 1000 neurons (Monomial-NFN $\operatorname{Tanh}$ case and other models). The hyperparameters for our model, along with the parameter counts for all models involved in this task, are detailed in Table [8](#A5.T8).

### E.2 Classifying implicit neural representations of images

##### Dataset.

We use the original INRs dataset, which contained three dataset: CIFAR-10, MNIST and Fashion-MNIST, obtained by applying a single SIREN model to every image. The detailed information about dataset is described in []. We use the datasett without any data augmentation as in the settings of []. The breakdown of training, validation, and test sample sizes for each dataset is detailed in Table [8](#A5.T8).

##### Implementation Details.

In these experiments, we use two different architectures. For both the MNIST and CIFAR datasets, the architecture begins with a Monomial-NFN layer to adjust the weight dimensions, followed by three MAGEP-NFN layers, each utilizing sine activation. The resulting weight features are then passed through a MAGEP Invariant layer. Finally, the output is flattened and processed by an MLP with two hidden layers, each containing 1,000 units and using $\operatorname{ReLU}$ activations.

For the Fashion-MNIST dataset, we begin with a Monomial-NFN layer with sine activation, followed by a MAGEP-NFN layer also utilizing sine activation, and then a Monomial-NFN layer with absolute activation. The architecture then aligns with the design of the NP model from []. Specifically, a Gaussian Fourier Transformation is applied to encode the input into sine and cosine components, mapping from 1 dimension to 256 dimensions. The encoded features are processed through IOSinusoidalEncoding, a positional encoding tailored for the NP layer, which uses a maximum frequency of 10 and 6 frequency bands. Following this, the features pass through three NP layers with $\operatorname{ReLU}$ activations. An average pooling is applied, after which the output is flattened, and the resulting vector is processed by an MLP with two hidden layers, each containing 1,000 units and using $\operatorname{ReLU}$ activations. Finally, the output is linearly projected to a scalar. We employ the Binary Cross Entropy (BCE) loss function and train the model for 200,000 steps, taking approximately 2 hours on an A100 GPU. The parameter counts for all models are presented in Table [12](#A5.T12)

### E.3 Weight space style editing

##### Dataset.

We utilize the same INRs dataset as employed in the classification task, with the sizes of the training, validation, and test sets provided in Table [8](#A5.T8).

##### Implementation Details.

In these experiments, our architecture begins with two MAGEP-NFN layers, each with 16 hidden dimensions. The rest of the design follows the NP model outlined in []. Specifically, we apply a Gaussian Fourier Transformation with a mapping size of 256, followed by IOSinusoidalEncoding. The features are then processed through three NP layers, each with 128 hidden dimensions and $\operatorname{ReLU}$ activation. The final output is passed through an NP layer for scalar projection and a LearnedScale layer as described in the Appendix of []. We use the Binary Cross Entropy (BCE) loss function and train the model for 50,000 steps, which takes approximately 35 minutes on an A100 GPU.

For the baseline models, we maintain the same settings as the official implementation. Specifically, the HNP or NP model consists of three layers, each with 128 hidden dimensions, followed by $\operatorname{ReLU}$ activations. An NFN of the same type is applied to map the output to one dimension, after which it is processed by a LearnedScale layer. The number of parameters for all models is detailed in Table [12](#A5.T12), and the hyperparameters for our model are presented in Table [12](#A5.T12).

## Appendix F Implementation of Equivariant and Invariant Layers

We provide the multi-channel implementations of the $\mathcal{G}_{\mathcal{U}}$-equivariant map $E\colon\mathcal{U}^{d}\to\mathcal{U}^{e}$ and the $\mathcal{G}_{\mathcal{U}}$-invariant map $I\colon\mathcal{U}^{d}\to\mathbb{R}^{e\times d^{\prime}}$. For uniformity in implementing Equivariant and Invariant layers from Appendix [C.5](#A3.SS5) and Appendix [D.4](#A4.SS4), we employ einops-style pseudocode as a consistent framework.

We summarize the key dimensions in Table [13](#A6.T13) and outline the shapes of the input terms in Table [14](#A6.T14).

**Table 13: Summary of key dimensions involved in the implementation**
| Symbol | Description |
| --- | --- |
| $d$ | Number of input channels for the equivariant and invariant layer |
| $e$ | Number of output channels for the equivariant and invariant layer |
| $b$ | Batch size |
| $n_{i}$ | Number of channels at the $i_{t}h$ layer |
| $d^{\prime}$ | Embedding dimension of the invariant layer’s output |

**Table 14: Shapes of input terms used in the implementation**
| Term | Shape |
| --- | --- |
| $[W]^{(s,t)}$ | $[b,d,n_{s},n_{t}]$ |
| $\left[Wb\right]^{(s,t)(t)}$ | $[b,d,n_{s}]$ |
| $\left[bW\right]^{(s)(L,t)}$ | $[b,d,n_{s},n_{t}]$ |
| $\left[WW\right]^{(s,0)(L,t)}$ | $[b,d,n_{s},n_{t}]$ |

### F.1 Equivariant Layers Pseudocode

#### F.1.1 Pseudocode for case i = L 𝑖 𝐿 i=L

From the formula for $[E(W)]^{(L)}_{jk}$:

$$ $\displaystyle[E(W)]^{(L)}_{jk}$ $\displaystyle=\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,L-1):p}\cdot[W]^{(L,L-1)}_{pk}+\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,0)(L,L-1):p}\cdot[WW]^{(L,0)(L,L-1)}_{pk}$ $\displaystyle\quad+\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L)(L,L-1):p}\cdot[bW]^{(L)(L,L-1)}_{pk}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(L):j}_{(L,L-1):p}\cdot[W]^{(L,L-1)}_{pk},$ $\displaystyle\quad\text{ with }[W]^{(L,L-1)}_{pk}\text{ of shape }[b,d,n_{L},n_{L-1}]\ \text{and}\ \Phi^{(L):j}_{(L,L-1):p}\text{ of shape }[e,d,n_{L},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpj,bdpk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L,0)(L,L-1):p}\cdot[WW]^{(L,0)(L,L-1)}_{pk},$ $\displaystyle\quad\text{ with }[WW]^{(L,0)(L,L-1)}_{pk}\text{ of shape }[b,d,n_{L},n_{L-1}]\ \text{and}\ \Phi^{(L):j}_{(L,0)(L,L-1):p}\text{ of shape }[e,d,n_{L},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpj,bdpk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L)(L,L-1):p}\cdot[bW]^{(L)(L,L-1)}_{pk},$ $\displaystyle\quad\text{ with }[bW]^{(L)(L,L-1)}_{pk}\text{ of shape }[b,d,n_{L},n_{L-1}]\ \text{and}\ \Phi^{(L):j}_{(L)(L,L-1):p}\text{ of shape }[e,d,n_{L},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpj,bdpk\rightarrow bejk)$ $$

From the formula for $[E(b)]^{(L)}_{j}$:

$$ $\displaystyle[E(b)]^{(L)}_{j}$ $\displaystyle=\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$ $\displaystyle\quad+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\Phi^{(L):j}_{(s,0)(L,s):}\cdot[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi^{(L):j}_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$ $\displaystyle\quad+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\sum_{p=1}^{n_{t}}\Phi^{(L):j}_{(L)(L,t):}\cdot[bW]^{(t)(L,t)}_{pp}$ $\displaystyle\quad+\sum_{p=1}^{n_{L}}\Phi^{(L):j}_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}^{(L):j}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(L):j}_{(L,0)(L,0):pq}\ \cdot[WW]^{(L,0)(L,0)}_{pq},$ $\displaystyle\quad\text{ with }[WW]^{(L,0)(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi^{(L):j}_{(L,0)(L,0):pq}\text{ of shape }[e,d,n_{L},n_{0},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpqj,bdpq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L,0):pq}\cdot[W]^{(L,0)}_{pq},\text{ with }[W]^{(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi^{(L):j}_{(L,0):pq}\text{ of shape }[e,d,n_{L},n_{0},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpqj,bdpq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(s,0)(L,s):}\cdot[WW]^{(s,0)(L,s)}_{pp},$ $\displaystyle\quad\text{ with }[WW]^{(s,0)(L,s)}_{pp}\text{ of shape }[b,d,n_{s},n_{s}]\ \text{and}\ \Phi^{(L):j}_{(s,0)(L,s):}\text{ of shape }[e,d,n_{s},n_{s},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edppj,bdpp\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq},$ $\displaystyle\quad\text{ with }[bW]^{(L)(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi^{(L):j}_{(L)(L,0):pq}\text{ of shape }[e,d,n_{L},n_{0},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpqj,bdpq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p},\text{ with }[Wb]^{(L,t)(t)}_{p}\text{ of shape }[b,d,n_{L}]\ \text{and}\ \Phi^{(L):j}_{(L,t)(t):p}\text{ of shape }[e,d,n_{L},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpj,bdp\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(t)(L,t):}\cdot[bW]^{(t)(L,t)}_{pp},\text{ with }[bW]^{(t)(L,t)}_{pp}\text{ of shape }[b,d,n_{t},n_{t}]\ \text{and}\ \Phi^{(L):j}_{(t)(L,t):}\text{ of shape }[e,d,n_{t},n_{t},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edppj,bdpp\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(L):j}_{(L):p}\cdot[b]^{(L)}_{p},\text{ with }[b]^{(L)}_{p}\text{ of shape }[b,d,n_{L}]\ \text{and}\ \Phi^{(L):j}_{(L):p}\text{ of shape }[e,d,n_{L},n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(edpj,bdp\rightarrow bej)$ $\displaystyle\text{For }\Phi_{1}^{(L):j}\text{ of shape }[e,n_{L}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(ej,\rightarrow ej).unsqueeze(0)$ $$

#### F.1.2 Pseudocode for case i = 1 𝑖 1 i=1

From the formula for $[E(W)]^{(1)}_{jk}$:

$$ $\displaystyle[E(W)]^{(1)}_{jk}$ $\displaystyle=\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq}+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq}$ $\displaystyle\quad+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet k}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq}+\Phi^{(1):\bullet k}_{(1):\bullet}\cdot[b]^{(1)}_{j}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(1):\bullet k}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq},\text{ with }[W]^{(1,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet k}_{(1,0):\bullet q}\text{ of shape }[d,e,n_{0},n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deqk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(1):\bullet k}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq},$ $\displaystyle\quad\text{ with }[WW]^{(1,0)(L,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet k}_{(1,0)(L,0):\bullet q}\text{ of shape }[d,e,n_{0},n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deqk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(1):\bullet k}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq},$ $\displaystyle\quad\text{ with }[bW]^{(1)(L,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet k}_{(1)(L,0):\bullet q}\text{ of shape }[d,e,n_{0},n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deqk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(1):\bullet k}_{(1):\bullet}\cdot[b]^{(1)}_{j},\text{ with }[b]^{(1)}_{j}\text{ of shape }[b,d,n_{1}]\ \text{and}\ \Phi^{(1):\bullet k}_{(1):\bullet}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdj,dek\rightarrow bejk)$ $$

From the formula for $[E(b)]^{(1)}_{j}$:

$$ $\displaystyle[E(b)]^{(1)}_{j}$ $\displaystyle=\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq}+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq}$ $\displaystyle\quad+\sum_{q=1}^{n_{0}}\Phi^{(1):\bullet}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq}+\Phi^{(1):\bullet}_{(1):\bullet}\cdot[b]^{(1)}_{j}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(1):\bullet}_{(1,0):\bullet q}\cdot[W]^{(1,0)}_{jq},\text{ with }[W]^{(1,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet}_{(1,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(1):\bullet}_{(1,0)(L,0):\bullet q}\cdot[WW]^{(1,0)(L,0)}_{jq},$ $\displaystyle\quad\text{ with }[WW]^{(1,0)(L,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet}_{(1,0)(L,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(1):\bullet}_{(1)(L,0):\bullet q}\cdot[bW]^{(1)(L,0)}_{jq},\text{ with }[bW]^{(1)(L,0)}_{jq}\text{ of shape }[b,d,n_{1},n_{0}]\ \text{and}\ \Phi^{(1):\bullet}_{(1)(L,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(1):\bullet}_{(1):\bullet}\cdot[b]^{(1)}_{j},\text{ with }[b]^{(1)}_{j}\text{ of shape }[b,d,n_{1}]\ \text{and}\ \Phi^{(1):\bullet}_{(1):\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdj,de\rightarrow bej)$ $$

#### F.1.3 Pseudocode for case 1 < i < L 1 𝑖 𝐿 1<i<L

From the formula for $[E(W)]^{(i)}_{jk}$:

$$ $\displaystyle[E(W)]^{(i)}_{jk}$ $\displaystyle=\left(\Phi^{(i):\bullet\bullet}_{(i,i-1):\bullet\bullet}\right)\cdot[W]^{(i,i-1)}_{jk}+\left(\Phi^{(i):\bullet\bullet}_{(i,0)(L,i-1):\bullet\bullet}\right)\cdot[WW]^{(i,0)(L,i-1)}_{jk}$ $\displaystyle\quad+\left(\Phi^{(i):\bullet\bullet}_{(i)(L,i-1):\bullet\bullet}\right)\cdot[bW]^{(i)(L,i-1)}_{jk}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(i):\bullet\bullet}_{(i,i-1):\bullet\bullet}\cdot[W]^{(i,i-1)}_{jk},\text{ with }[W]^{(i,i-1)}_{jk}\text{ of shape }[b,d,n_{i},n_{i-1}]\ \text{and}\ \Phi^{(i):\bullet\bullet}_{(i,i-1):\bullet\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjk,de\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(i):\bullet\bullet}_{(i,0)(L,i-1):\bullet\bullet}\cdot[WW]^{(i,0)(L,i-1)}_{jk},$ $\displaystyle\quad\text{ with }[WW]^{(i,0)(L,i-1)}_{jk}\text{ of shape }[b,d,n_{i},n_{i-1}]\ \text{and}\ \Phi^{(i):\bullet\bullet}_{(i,0)(L,i-1):\bullet\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjk,de\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(i):\bullet\bullet}_{(i)(L,i-1):\bullet\bullet}\cdot[bW]^{(i)(L,i-1)}_{jk},$ $\displaystyle\quad\text{ with }[bW]^{(i)(L,i-1)}_{jk}\text{ of shape }[b,d,n_{i},n_{i-1}]\ \text{and}\ \Phi^{(i):\bullet\bullet}_{(i)(L,i-1):\bullet\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjk,de\rightarrow bejk)$ $$

From the formula for $[E(b)]^{(i)}_{j}$:

$$ $\displaystyle[E(b)]^{(i)}_{j}$ $\displaystyle=\sum_{q=1}^{n_{0}}\left(\Phi^{(i):\bullet}_{(i,0):\bullet q}\right)\cdot[W]^{(i,0)}_{jq}+\sum_{q=1}^{n_{0}}\left(\Phi^{(i):\bullet}_{(i,0)(L,0):\bullet q}\right)\cdot[WW]^{(i,0)(L,0)}_{jq}$ $\displaystyle\quad+\sum_{i>t>0}\left(\Phi^{(i):\bullet}_{(i,t)(t):\bullet}\right)\cdot[Wb]^{(i,t)(t)}_{j}+\sum_{q=1}^{n_{t}}\left(\Phi^{(i):\bullet}_{(i)(L,0):\bullet q}\right)\cdot[bW]^{(i)(L,0)}_{jq}$ $\displaystyle\quad+\left(\Phi^{(i):\bullet}_{(i):\bullet}\right)\cdot[b]^{(i)}_{j}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(i):\bullet}_{(i,0):\bullet q}\cdot[W]^{(i,0)}_{jq},\text{ with }[W]^{(i,0)}_{jq}\text{ of shape }[b,d,n_{i},n_{0}]\ \text{and}\ \Phi^{(i):\bullet}_{(i,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(i):\bullet}_{(i,0)(L,0):\bullet q}\cdot[WW]^{(i,0)(L,0)}_{jq},$ $\displaystyle\quad\text{ with }[WW]^{(i,0)(L,0)}_{jq}\text{ of shape }[b,d,n_{i},n_{0}]\ \text{and}\ \Phi^{(i):\bullet}_{(i,0)(L,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(i):\bullet}_{(i,t)(t):\bullet}\cdot[Wb]^{(i,t)(t)}_{j},\text{ with }[Wb]^{(i,t)(t)}_{j}\text{ of shape }[b,d,n_{i}]\ \text{and}\ \Phi^{(i):\bullet}_{(i,t)(t):\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdj,de\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(i):\bullet}_{(i)(L,0):\bullet q}\cdot[bW]^{(i)(L,0)}_{jq},\text{ with }[bW]^{(i)(L,0)}_{jq}\text{ of shape }[b,d,n_{i},n_{0}]\ \text{and}\ \Phi^{(i):\bullet}_{(i)(L,0):\bullet q}\text{ of shape }[d,e,n_{0}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,deq\rightarrow bej)$ $\displaystyle\text{For }\Phi^{(i):\bullet}_{(i):\bullet}\cdot[b]^{(i)}_{j},\text{ with }[b]^{(i)}_{j}\text{ of shape }[b,d,n_{i}]\ \text{and}\ \Phi^{(i):\bullet}_{(i):\bullet}\text{ of shape }[d,e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdj,de\rightarrow bej)$ $$

### F.2 Invariant Layers Pseudocode

From the formula for the Invariant layer $I(U)$:

$$ $\displaystyle I(U)$ $\displaystyle=\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq}$ $\displaystyle\quad+\sum_{L>s>0}\sum_{p=1}^{n_{s}}\Phi_{(s,0)(L,s):\bullet\bullet}\cdot[WW]^{(s,0)(L,s)}_{pp}+\sum_{p=1}^{n_{L}}\sum_{q=1}^{n_{0}}\Phi_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq}$ $\displaystyle\quad+\sum_{L>t>0}\sum_{p=1}^{n_{L}}\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p}+\sum_{L>t>0}\sum_{p=1}^{n_{t}}\Phi_{(t)(L,t):\bullet\bullet}\cdot[bW]^{(t)(L,t)}_{pp}$ $\displaystyle\quad+\sum_{p=1}^{n_{L}}\Phi_{(L):p}\cdot[b]^{(L)}_{p}+\Phi_{1}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi_{(L,0)(L,0):pq}\cdot[WW]^{(L,0)(L,0)}_{pq},$ $\displaystyle\quad\text{with }[WW]^{(L,0)(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi_{(L,0)(L,0):pq}\text{ of shape }[d,e,n_{L},n_{0},d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,depqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(L,0):pq}\cdot[W]^{(L,0)}_{pq},\text{ with }[W]^{(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi_{(L,0):pq}\text{ of shape }[d,e,n_{L},n_{0},d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpqk,depqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(s,0)(L,s):\bullet\bullet}\cdot[WW]^{(s,0)(L,s)}_{pp},$ $\displaystyle\quad\text{ with }[WW]^{(s,0)(L,s)}_{pp}\text{ of shape }[b,d,n_{s}]\ \text{and}\ \Phi_{(s,0)(L,s):\bullet\bullet}\text{ of shape }[d,e,d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,dek\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(L)(L,0):pq}\cdot[bW]^{(L)(L,0)}_{pq},$ $\displaystyle\quad\text{ with }[bW]^{(L)(L,0)}_{pq}\text{ of shape }[b,d,n_{L},n_{0}]\ \text{and}\ \Phi_{(L)(L,0):pq}\text{ of shape }[d,e,n_{L},n_{0},d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpqk,depqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(L,t)(t):p}\cdot[Wb]^{(L,t)(t)}_{p},\text{ with }[Wb]^{(L,t)(t)}_{p}\text{ of shape }[b,d,n_{L}]\ \text{and}\ \Phi_{(L,t)(t):p}\text{ of shape }[d,e,n_{L},d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,deijk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(t)(L,t):\bullet\bullet}\cdot[bW]^{(t)(L,t)}_{pp},\text{ with }[bW]^{(t)(L,t)}_{pp}\text{ of shape }[b,d,n_{t}]\ \text{and}\ \Phi_{(t)(L,t):\bullet\bullet}\text{ of shape }[d,e,d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,dek\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(L):p}\cdot[b]^{(L)}_{p},\text{ with }[b]^{(L)}_{p}\text{ of shape }[b,d,n_{L}]\ \text{and}\ \Phi_{(L):p}\text{ of shape }[d,e,n_{L},d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,depk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{1}\text{of shape }[e,d^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(ek\rightarrow ek).unsqueeze(0)$ $$

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How difficult is the implementation of the proposed nonlinear layers? Can the authors provide pseudo code?

2. Can the trained MAGEP-NFN be used on the wider networks (e.g. predict generalization for larger CNNs than in the Small CNN Zoo dataset or retraining the model would be necessary?
soundness: 2
strengths: 1. The paper tackles a challenging problem of modeling neural network weights and their symmetries.
2. The proposed nonlinear equivariant layers appear to be original and interesting. 
3. The paper seems theoretically sound, however, I haven't checked the correctness.
4. The results are better than for the linear polynomials (Monomial-NFN).
summary: The paper proposes a new method to model neural network weights based on Equivariant Neural Functional Networks. Compared to previous approaches, this method is claimed to be more expressive while still efficient. This is achieved by adding nonlinear (polynomial) equivariant layers acting on weights. The paper presents an extensive theoretical framework and shows improved empirical performance on the INR-based tasks and generalization prediction.
weaknesses: The paper's main weaknesses are experiments and writing/presentation.

### The experiments are weak in several aspects:
1. The paper claims "low memory consumption and efficient running time", however these claims are not supported by empirical evidence. The previous paper (Tran et al., 2024) shows that it can reduce memory from 6GB to 0.5GB, however, the extra (stable) polynomial terms may increase memory/runtime of the method in this paper. 
2. No comparison to graph-based approaches (Lim et al.,2023; Kofinas et al., 2024) in Table 1-4 (ideally in terms of memory/efficiency vs results), so the significance of the results is hard to estimate.
3. Some inconsistencies in evaluation, which is not explained, making comparison to previous literature difficult. For example, in (Kofinas et al., 2024) for MNIST the INR classification performance was about 90-95% and for FashionMNIST 70-75%, while in this paper it's  77 and 62 respectively. On the Small CNN Zoo dataset (Unterthineretal.,2020) the method achieves 0.933, while graph-based approaches (Kofinas et al., 2024) achieve better results (0.935).
4. The shortcoming of the method is that it requires some manual design of the polynomials for each architecture, so it's unclear how to extend it to other architectures such as transformers or even if it's possible to use a trained MAGEP-NFN on the wider networks.
5. The experiments on INR and generalization prediction for simple small CNNs are not sufficient. Previous works (Lim et al.,2023; Kofinas et al., 2024) added more practical experiments with learning to optimize or diverse transformer architectures. Otherwise, the proposed approach appears to be overengineered for some specific toy cases. 

### Writing and presentation
1. "These specialized networks are known as neural functional networks (NFNs)(Zhouetal.,2024b)" is a questionable statement.
2. "graph-based equivariant NFNs suffer from high memory consumption and long running times" is an overstatement given that Tran et al., 2024 showed the memory consumption of those methods is at maximum 6GB and runtime at max 4.5 hours. So in practice, this does not seem to be a problem for any modern GPUs. Please rewrite or scale the experiments.
2. Related works is not informative, because it just list existing papers without discussing how the current work is connected (improved, different, etc.) to them. 
3. The whole Section 3, if I understand correctly, seems to be background (notation introduction, etc. from Tran 2024), and it's quite lengthy, which makes the amount of the original content in the paper quite small.
4. In section 4 it's not described how the equations 20-21 are different from the previous work (Tran 2024) or section 3. It would be more clear to directly contrast nonlinear and linear layers in this context to understand the contribution better.
5. Contribution 2 of the paper (linear independence) is overstated - there is no comprehensive study in the main text. In contrast, the graph-based approached handled diverse and complex architectures (such as transformers) more easily. 
6. The paper could be more accessible, e.g. by having illustrations of the proposed vs previous method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - How would one apply this approach to methods that support arbitrary computation graphs such as [1]?



**Reference**

[1] Universal neural functionals. Zhou et al.
soundness: 4
strengths: This paper brings both theoretical and empirical contributions to NFNs.
The techniques proposed in the paper are well-motivated and rigorously derived.
Experiments and mathematical details are quite thorough, especially given how mathematically involved these approaches tend to be (I was able to follow the derivation reasonably well).

Overall, I believe MAGEP-NFN provides a concrete improvement in the line of work on NFNs and points towards a direction to improve NFNs further.
summary: This paper proposes monomial matrix group equivariant polynomial neural functional networks (MAGEP-NFN), a new class of neural functionals whose input and output are the weights of neural networks. MAGEP-NFN builds on a prior work Monomial NFN which adds scale equivariance and sign equivariance of the original NFN which is only equivariant to neuron permutations. MAGEP-NDF introduces *stable polynomial terms* which are the product of a series of adjacent weights and biases of a neural networks that are also equivariant under different neuron permutations. These stable polynomial terms can be thought of as some "features" of the original weights, similar to how polynomials are used as features in kernels. MAGEP-NFN layers can process these stable polynomials in addition to the original weights. Experimental results demonstrate that MAGEP-NFNs outperform previous NFNs that use weight-sharing schemes in terms of generalization prediction and perform similarly on INR editing tasks.
weaknesses: The paper makes some unsubstantiated claims about its advantages over prior methods and GNN-based methods in terms of runtime and memory consumption. Specifically, while the paper emphasizes that MAGEP-NFN enjoys superior runtime and memory efficiency, there are no comparisons of these quantities for MAGEP-NFN and relevant baselines (For example, it is not clear to me that MAGEP-NFN should be more efficient than vanilla NFN). It would be good to add a runtime and memory efficiency comparison to the paper which in my opinion would make the paper stronger. 

Furthermore, while the theoretical contribution is nice and sound, MAGEP-NFN does not seem to bring substantial empirical improvement over relevant baselines.

**Minor**
- Use `` and '' for quotations.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: - Due to the lack of insights in the methodological section, several questions arise. For instance,
   - What is the degree of the resulting polynomial? Shouldn’t this be chosen as a hyperparameter by the user?
   - I assume that for different activation functions (or more precisely for different symmetries induced by the activation function) the resulting layers should be different. However, this is not clear to me from the Equations 20 and 21. Could the authors explain what are the differences?
   - What are the matrices $\Phi$ and $\Psi$? They are kind of abruptly introduced.


**Minor:**
- **Related work and background**:  I have the impression that these two sections (2 and 3) are quite similar to those of the Monomial-NFN paper. Especially, regarding the background section, I felt it was too long, which might not be necessary, since the reader can be pointed to Monomial-NFN or ScaleGMN to find a detailed explanation of the background.
- The paper misses an important citation: “Equivariant Polynomials for Graph Neural Networks”, Puny et al., ICML’23. This work discusses equivariant polynomials for permutation symmetries present in graphs.
soundness: 2
strengths: - **Significance and Motivation**. Similarly to many equivariant machine learning tasks, designing models that are equivariant to parameter symmetries, typically requires dealing with a computational complexity vs expressivity dilemma. The authors correctly identify this dilemma, promptly (since NFNs/metanetworks are increasingly becoming more popular), and with their approach, they attempt to find a better trade-off.
- **Novelty**. This is the first work that proposes polynomials equivariant to parameter symmetries and may potentially mark a first step towards improving this paradigm as an alternative to linear layers + non-linearities, as well as GNNs.
- **Empirical performance**. In practice, the improvement in expressivity is shown to be reflected in the downstream performance in most of the tested cases.
summary: This paper follows a recent line of work on *neural functional networks (NFNs)*, i.e. neural networks that are specifically designed to process other neural networks. In specific, the first NFNs are composed of linear layers (interleaved with non-linearities) that are equivariant to the *permutation* symmetries of the input NNs, while a recent follow-up called Monomial-NFN follows the same strategy, but additionally accounting for the *scaling* symmetries. However, the latter turned out to suffer from limited expressivity, due to the extensive parameter sharing that resulted from the existing symmetries. 

To overcome this limitation, the authors in the present manuscript propose a Polynomial-NFN, i.e. a combination of polynomial layers (potentially interleaved with non-linearities) that are equivariant to permutation + scaling symmetries. The paper mainly focuses on the characterisation of polynomial equivariant layers, using a construction named stable polynomials. Experimentally, the method is tested on a variety of common NFN benchmarks showing improved performance in the majority of the cases.
weaknesses: - **Presentation and Exposition**. My major concern with this paper is that the methodological/technical contributions are not well-presented. In particular,
   - The methodological part (Section 4) is quite hard to follow and the information provided is highly technical. This is not necessarily an issue, but currently, the authors do not provide any intuition/insights or simpler explanations of their thought process (why did they choose this proof technique, i.e. via stable polynomials), derivations (e.g. a proof sketch) and resulting equivariant/invariant layers (why do they look like that). 
   - Therefore, I am afraid that this problem might be reflected in **reproducibility/extensibility**, as it will be hard for the interested reader to adopt/re-implement this method, and even harder to deeply understand the structure of the resulting layers. 
   - I  would recommend the authors invest more in explanations and intuition about each part of their methodological section, most importantly in explaining the terms involved in the equivariant/invariant layer equations.


- **Claims and Theory**. Another major concern is that the authors have made a few claims that are not adequately supported with evidence:
   - First, they argue that their method improves expressivity compared to Monomial-NFN. Intuitively, probably this is indeed the case, but the authors have not provided any theoretical argument to support this claim.
   - Additionally, the authors mention that GNNs have increased memory consumption and runtime. As far as I can see, this claim is also not supported with, at least, experimental evidence. I suggest the authors report the above metrics and compare those of their method to GNNs (metanetworks), NFN, Monomial-NFN, and ScaleGMN (see below), to improve our understanding of the trade-offs. Additionally, a computational complexity table with comparisons would help a lot.
   - Finally, the theoretical contributions need some improvements. For example, it is unclear what is the role of Proposition 4.2. and Theorem 4.3. (why are they important?). Additionally, as far as I understand Theorem 4.4. characterises all G-invariant polynomials (by the way, of what degree?). Is there a similar argument for G-equivariant polynomials?


- **Method**. Additionally, examining the method per se, even if it improves expressivity, I believe it inherits several of the limitations of Monomial-NFN:
   - Only certain types of activation functions can be used.
   - It is hard to extend to other activation functions
   - It cannot operate on diverse architectures.
I advise the authors to mention those clearly and discuss potential options to address them.

- **Experiments**.
   - Apart from the memory/runtime comparisons that this paper is missing, I think that the authors should include another baseline: “Scale Equivariant Meta Networks”, Kalogeropoulos et al., NeurIPS’24. This is going to be a fairer competitor as opposed to Monomial-NFN and it would be interesting to observe the performance vs runtime tradeoffs.
   - I am wondering why in the editing experiment we don’t observe any performance improvement.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: [1] What does the outer sum  $\sum_{L\geq s>t\geq 0}$ in equation 20 denote? Is it a double sum over $s$ and $t$?

[2] Related to question 1, in equations 20-21, is it necessary to sum over the whole range of $s$ and $t$? This seems like a very inefficient operation for very deep networks (e.g. ResNet 101). Isn't it possible to sum over a few adjacent layer indices? An ablation study on the number of hops would be useful here.

[3] What is the computational complexity of the method compared to having linear layers?

[4] Can the method work in a combined Small CNN Zoo dataset with both ReLU and tanh activations (and potentially other similar groups)?

**Minor questions**:

[5] In Eq. 15, shouldn't it be $\pi_{i}(j)$ instead of $\pi_{i}^{-1}(j)$?
soundness: 2
strengths: The manuscript is fairly clear and easy to follow, despite being math heavy.

NFNs are a very promising research direction, and making them faster and more efficient is key to scaling them up to more real-world applications.
Hence, the fact that the proposed method can reach good performance with a reduced number of parameters
compared to previous works is very promising, and a significant step forward.
summary: This work proposes a novel Neural Functional Network (NFN), i.e. a neural network that takes the weights of neural networks as input data. The method, termed MAGEP-NFN, belongs to the family of parameter-sharing NFNs and incorporates both the permutation symmetries of neural networks and the scaling symmetries of common non-linearities, such as ReLU and tanh. The proposed method alleviates the reduced expressivity of previous works by introducing non-linear equivariant layers through polynomials of the input weights and biases. MAGEP-NFN achieves competitive results in INR tasks and CNN generalization tasks, while having far fewer parameters than the compared baselines.
weaknesses: The numbers reported for NFN in the INR classification experiments are considerably worse than
the official ones. Namely, NFN scores $46.6 \pm 0.072$ in CIFAR10, $92.9 \pm 0.218$ on
MNIST and $75.6 \pm 1.07$ on FashionMNIST. This also means that the performance of
MAGEP-NFN is quite poor. According to the appendix E, the authors use the same datasets as Zhou et al. Is there a different setup used in the experiments?

Almost no explanation is given for equations 20 and 21, that comprise the core functionality of the proposed method.
These equations are very math-heavy, and it is very hard for the reader to understand the usefulness of all individual components.
Further explanation of each module could provide more intuition about the method. Further, both formulas could benefit from some visualizations that show the connections between layers and the flow of information. Finally, it is unclear if all the individual components contribute to the effectiveness of the method. An ablation study on those components could make the method more efficient, without sacrificing accuracy.

Meta-review: The paper is the first to propose polynomials equivariant to parameter symmetries in neural field networks, allowing for a good balance between computational complexity vs expressivity dilemma. There were also concerns regarding comparisons. There were questions regarding the presentation that was too dense and could lead to challenging reproducibility, which still persisted by the reviewers, regarding deeper intuition and additional explanations for choosing stable polynomials and possible limitations of this approach with respect to the expressivity of the method, by not including all invariant/equivariant polynomials, the degree of the polynomial the limitations of monomials, and not discussing sufficiently the differences that arise for different activation functions.

By far the most controversial point was whether the paper committed plagiarism, as raised by one of the reviewers. After checking myself, it appears that even with the edits, there is quite some similarity. I do not want to go too much into the reasons why this similarity could be (for instance, maybe it's the same group of people co-authoring) but in general, plagiarism is a hard issue to address, especially when it comes to very large conferences with many unknown parties involved (at the time of the reviewing), since more often than not the truth is somewhere in the middle. The only thing that is certain, to avoid problems with other papers in the future (and how much is something or not a plagiarism), it is better to be strict to ascertain that there is no doubt.

As far as the content is concerned, I think the paper is good. However, considering that there are also some unclarities to improve, I suggest the authors to do one more round of serious rewriting, and resubmit.

Decision: Reject