Equivariant Neural Functional Networks for Transformers

## Abstract

Abstract This paper systematically explores neural functional networks (NFN) for transformer architectures. NFN are specialized neural networks that treat the weights, gradients, or sparsity patterns of a deep neural network (DNN) as input data and have proven valuable for tasks such as learnable optimizers, implicit data representations, and weight editing. While NFN have been extensively developed for MLP and CNN, no prior work has addressed their design for transformers, despite the importance of transformers in modern deep learning. This paper aims to address this gap by providing a systematic study of NFN for transformers. We first determine the maximal symmetric group of the weights in a multi-head attention module as well as a necessary and sufficient condition under which two sets of hyperparameters of the multi-head attention module define the same function. We then define the weight space of transformer architectures and its associated group action, which leads to the design principles for NFN in transformers. Based on these, we introduce Transformer-NFN, an NFN that is equivariant under this group action. Additionally, we release a dataset of more than 125,000 Transformers model checkpoints trained on two datasets with two different tasks, providing a benchmark for evaluating Transformer-NFN and encouraging further research on transformer training and performance.

## 1 Introduction

Deep neural networks (DNNs) have become highly flexible and adaptable modeling tools, widely employed in domains such as natural language processing [, , , ], computer vision [, , ], and natural science [, ]. Increasing attention is being given to constructing specialized neural networks that treat the weights, gradients, or sparsity patterns of DNNs as data inputs. These specialized networks are referred to as neural functional networks (NFNs) [].
NFNs have proven useful in various applications, such as designing learnable optimizers for neural network training [, , , ], extracting information from implicit representations of data [, , ], performing targeted weight editing [, , ], evaluating policies [], and applying Bayesian inference with neural networks as evidence [].

The construction of NFNs presents significant challenges due to the complexity and high dimensionality of their structures.
Earlier approaches sought to address this by constraining the training process, thereby restricting the weight space [, , ].
More recently, research has shifted towards the development of permutation-equivariant NFNs, capable of operating on neural network weights without such constraints [, , , ].
These advancements have resulted in NFNs that exhibit equivariance to weight permutations, which correspond to neuron rearrangements within hidden layers.
Additionally, NFNs that are equivariant to both permutations and operations such as scaling or sign-flipping have been proposed. These include graph-based message-passing methods as described in [] and parameter sharing techniques in [].
However, most of the recent neural functional networks are designed for Multilayer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs).

Transformers have achieved remarkable success in the field of natural language processing (NLP) [, ], powering numerous large-scale language models such as GPT-2, GPT-3, GPT-4, AlbertAGPT, Claude, BERT, XLNet, RoBERTa, and ChatGPT. These models highlight the transformer architecture’s capacity to perform a broad range of NLP-related tasks and its applicability to real-world scenarios.
The influence of Transformers extends beyond NLP, finding applications in areas such as computer vision through Vision Transformers, reinforcement learning [, ], audio processing [], multi-modal data integration, and robotics [].
Despite the significance of Transformers, no systematic development of NFNs for Transformers has been realized to date. Addressing the design principle of NFNs for Transformers, as well as constructing equivariant NFNs capable of processing transformer models, has thus emerged as a critical research problem.

### 1.1 Background

Self-attention is the core component of a transformer block [, , ]. Self-attention learns to align tokens within an input sequence by assessing the relative importance of each token in relation to all other tokens. It then transforms each token into a weighted average of the feature representations of other tokens, with the weights determined by importance scores between token pairs. These importance scores allow each token to attend to others in the sequence, effectively capturing contextual representations [, , ].

##### Self-attention.

Let $X=[X_{1},\ldots,X_{L}]^{\top}\in\mathbb{R}^{L\times D}$ be an input sequence of $L$ tokens with $D$ features. Each token is a vector $X_{i}\in\mathbb{R}^{D}$.
The self-attention, denoted by $\operatorname{Head}$, transforms $X$ into the output sequence $\operatorname{Head}(X)\in\mathbb{R}^{L\times D_{v}}$ defined as

$$ $\displaystyle\operatorname{Head}(X;W^{(Q)},W^{(K)},W^{(V)})=\operatorname{softmax}\left(\frac{(XW^{(Q)})(XW^{(K)})^{\top}}{\sqrt{D_{k}}}\right)XW^{(V)},$ (1) $$

where the parameters $W^{(Q)},W^{(K)}\in\mathbb{R}^{D\times D_{k}}$ and $W^{(V)}\in\mathbb{R}^{D\times D_{v}}$ are called the query, key and value matrices.
Here, $D_{k}$ and $D_{v}$ are given positive integers.

##### Multihead attention.

To jointly attend to information from different representation subspaces at different positions, a multihead attention is used.
Let $h$ be a positive integer which represents the number of head.
The multihead attention transforms the input sequence $X\in\mathbb{R}^{L\times D}$ to an output sequence in $\mathbb{R}^{L\times D}$ defined by:

$$ $\displaystyle\operatorname{MultiHead}$ $\displaystyle\left(X;W^{(O)},\{W^{(Q,i)},W^{(K,i)},W^{(V,i)}\}_{i=1}^{h}\right)$ $\displaystyle=\left(\bigoplus\limits_{i=1}^{h}\operatorname{Head}\left(X;W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right)\right)W^{(O)},$ (2) $$

where each head is a self-attention defined in equation [1](#S1.E1), and $\bigoplus$ is the concatenation operator. Here, the matrices $W^{(Q,i)},W^{(K,i)}\in\mathbb{R}^{D\times D_{k}}$, $W^{(V,i)}\in\mathbb{R}^{D\times D_{v}}$ and $W^{(O)}\in\mathbb{R}^{hD_{v}\times D}$ are learnable from input data for some postive integers $D_{k}$ and $D_{v}$.

### 1.2 Contribution

This paper provides a systematic study on the development of a neural functional network (NFN) for transformer architectures.
To achieve this, we present three essential components for the study: (1) a design principle of NFNs for Transformers that incorporates the maximal symmetric group for the multi-head attention module, (2) an equivariant NFN for Transformers, which we will call Transformer-NFN, and (3) a benchmark dataset for testing the applicability and efficiency of Transformer-NFN. In particular, our contributions are four-fold:

- 1.
We determine the maximal symmetric group of the weights in a multi-head attention module, establishing the necessary and sufficient conditions under which two sets of hyperparameters for a multi-head attention module define the same function.
- 2.
We formally define the weight space of a transformer architecture, along with a group action on this weight space.
In conjunction with the maximal symmetric group of the weights in multi-head attention modules, we characterize the design principles for NFNs in transformer architectures.
- 3.
We introduce Transformer-NFN, an NFN for transformer architectures that is equivariant under the specified group action.
The main building block of Transformer-NFN is an equivariant polynomial layer derived from a parameter-sharing strategy.
- 4.
We release Small Transformer Zoo dataset, which consists of more than 125,000 Transformers model checkpoints trained on two different tasks: digit image classification on MNIST and text topic classificaction on AGNews. To our knowledge, this the first dataset of its kind.
This dataset serves as a benchmark for testing the applicability and efficiency of our Transformer-NFN, while also encouraging further research in this field to gain a deeper understanding of transformer network training and performance.

We empirically demonstrate that Transformer-NFN consistently outperforms other baseline models on our constructed datasets. Through comprehensive ablation studies, we emphasize Transformer-NFN’s ability to effectively capture information within the transformer block, establishing it as a robust predictor of model generalization.

Organization. Following a review of related works in Section [2](#S2), we derive the maximal symmetric group of multihead attention in Section [3](#S3). In Section [4](#S4), we construct the weight space of a standard transformer block and define a corresponding group action. Section [5](#S5) introduces a family of equivariant NFNs for Transformers, referred to as Transformer-NFNs. We then present the setting and details of the Small Transformer Zoo dataset in Section [6](#S6). Finally, in Section [7](#S7), we evaluate the applicability and efficiency of Transformer-NFNs on this dataset.

## 2 Related Work

Symmetries of weight spaces. The exploration of symmetries within the weight spaces of neural networks, which relates to assessing the functional equivalence of these networks, is a well-established field of study [, , , , ]. This topic was first introduced by Hecht-Nielsen in []. Following this, numerous studies have yielded insights for various network architectures, as detailed in [, , , , , ].

Neural functional networks. Recent studies have aimed to develop representations for trained classifiers to evaluate their generalization performance and understand neural network behavior [, , , , , ]. Notably, low-dimensional encodings for Implicit Neural Representations (INRs) have been created for various downstream tasks [, ].
[] proposed neuron permutation augmentations, and other research has focused on encoding and decoding neural network parameters for reconstruction and generative purposes [, , , ].

Equivariant NFNs for MLPs and CNNs. Recent advancements have made considerable strides in addressing the limitations of permutation equivariant neural networks through the introduction of permutation equivariant layers. These layers employ intricate weight-sharing patterns [, , , ], as well as set-based [] and graph-based structures of the network’s weights [, , ], to maintain equivariance.
Moreover, monomial equivariant NFNs, which are equivariant to both permutations and scaling, have been proposed in [] utilizing a graph-based message-passing mechanism and in [] through a parameter sharing mechanism.

## 3 Maximal Symmetric Group of A Multi-head Attention

As the first step in characterizing a principal design of NFNs for Transformers, we need to decide when two tuples of matrices with appropriate sizes, say $\left(\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right\}_{i=1}^{h},W^{(O)}\right)$ and $\left(\left\{\overline{W}^{(Q,i)},\overline{W}^{(K,i)},\overline{W}^{(V,i)}\right\}_{i=1}^{h},\overline{W}^{(O)}\right)$, define the same $\operatorname{Multihead}$ map. We provide a complete answer for this step in this section, thus characterizing the maximal symmetric group of the weights of $\operatorname{Multihead}$.

Let us consider the $\operatorname{MultiHead}$ map with $h$ heads defined in Equation ([1.1](#S1.Ex1)).
We can rewrite $\operatorname{MultiHead}$ as:

$$ $\displaystyle\operatorname{MultiHead}$ $\displaystyle\left(X;W^{(O)},\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right\}_{i=1}^{h}\right)$ $\displaystyle=\left(\bigoplus\limits_{i=1}^{h}\operatorname{Head}\left(X;W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right)\right)W^{(O)}$ $\displaystyle=\sum\limits_{i=1}^{h}\operatorname{Head}\left(X;W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right)W^{(O,i)}$ $\displaystyle=\sum\limits_{i=1}^{h}\operatorname{softmax}\left(X\cdot\left(\frac{W^{(Q,i)}\cdot\left(W^{(K,i)}\right)^{\top}}{\sqrt{D_{k}}}\right)\cdot X^{\top}\right)\cdot X\cdot\left(W^{(V,i)}\cdot W^{(O,i)}\right),$ $$

where $W^{(O)}=\left(W^{(O,1)},\ldots,W^{(O,h)}\right)$ with each $W^{(O,i)}\in\mathbb{R}^{D_{v}\times D}$.
Here, the matrices ${\left(W^{(Q)}\right)\cdot\left(W^{(K)}\right)^{\top}}/{\sqrt{D_{k}}}$ and $W^{(V,i)}\cdot W^{(O,i)}$ have the same size $D\times D$.
Based on this observation, we define for each positive integer $h$ and each sequence of matrices $\{A_{i},B_{i}\}_{i=1}^{h}$ in $\mathbb{R}^{D\times D}$ a map

$$ $\displaystyle F\left(\cdot;\{A_{i},B_{i}\}_{i=1}^{h}\right)~{}\colon~{}\bigsqcup_{l>0}\mathbb{R}^{l\times D}\rightarrow\bigsqcup_{l>0}\mathbb{R}^{l\times D},$ $$

defined by

$$ $\displaystyle F(X;\{A_{i},B_{i}\}_{i=1}^{h})\coloneqq\sum_{i=1}^{h}\operatorname{softmax}\left(X\cdot A_{i}\cdot X^{\top}\right)\cdot X\cdot B_{i},$ $$

for each $X\in\mathbb{R}^{l\times D}$.
It is noted that, for each integer $l>0$, the image of $\mathbb{R}^{l\times D}$ via $F$ is contained in $\mathbb{R}^{l\times D_{v}}$ and $F$ can be viewed as a generalization of $\operatorname{MultiHead}$.
With this setting at hand, we proved that:

###### Theorem 3.1 (Independence of heads in multi-head attention) .

Let $D$ be a positive integer. Assume that for a positive integer $h$, matrices $A_{1},A_{2},\ldots,A_{h}\in\mathbb{R}^{D\times D}$ and $B_{1},B_{2},\ldots,B_{h}\in\mathbb{R}^{D\times D}$, we have

$$ $\displaystyle F\left(X;\left\{A_{i},B_{i}\right\}_{i=1}^{h}\right)=0,$ (3) $$

for all positive integers $L$ and $X\in\mathbb{R}^{L\times D}$. Then, if $A_{1},A_{2},\ldots,A_{h}$ are pairwise distinct, then

$$ $\displaystyle B_{1}=\ldots=B_{h}=0.$ $$

###### Remark 1 .

Roughly speaking, the above theorem says that, in the multi-head attention, each individual head plays its own unique role.

Based on the above theorem, we characterize the maximal symmetric group of the weights of $\operatorname{MultiHead}$ in the following theorem.

###### Theorem 3.2 (Maximal symmetric group of multi-head attentions) .

Let $h,D,D_{k},D_{v}$ be positive integers.
Let $\left(W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right)$ and $\left(\overline{W}^{(Q,i)},\overline{W}^{(K,i)},\overline{W}^{(V,i)},\overline{W}^{(O,i)}\right)$ be arbitrary elements of $\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}$ with $i=1,\ldots,h$.
Assume that

- (a)
$\max(D_{k},D_{v})\leqslant D$,
- (b)
the matrices $W^{(Q,i)}$, $W^{(K,i)}$, $\overline{W}^{(Q,i)}$, and $\overline{W}^{(K,i)}$ are of full rank,
- (c)
the matrices $W^{(Q,i)}\cdot\left(W^{(K,i)}\right)^{\top}$ with $i=1,\ldots,h$ are pairwise distinct,
- (d)
the matrices $\overline{W}^{(Q,i)}\cdot\left(\overline{W}^{(K,i)}\right)^{\top}$ with $i=1,\ldots,h$ are pairwise distinct.

Then the following are equivalent:

- 1.
For every positive integer $L$ and every $X\in\mathbb{R}^{L\times D}$, we always have
$\displaystyle\operatorname{MultiHead}\left(X;\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right\}_{i=1}^{h}\right)$
$\displaystyle\qquad=\operatorname{MultiHead}\left(X;\left\{\overline{W}^{(Q,i)},\overline{W}^{(K,i)},\overline{W}^{(V,i)},\overline{W}^{(O,i)}\right\}_{i=1}^{h}\right).$
(4)
- 2.
There exist matrices $M^{(i)}\in\operatorname{GL}_{D_{k}}(\mathbb{R})$ and $N^{(i)}\in\operatorname{GL}_{D_{v}}(\mathbb{R})$ for each $i=1,\ldots,h$, as well as a permutation $\tau\in\mathcal{S}_{h}$, such that
$\displaystyle\left(\overline{W}^{(Q,\tau(i))},\overline{W}^{(K,\tau(i))},\overline{W}^{(V,\tau(i))},\overline{W}^{(O,\tau(i))}\right)$
$\displaystyle\qquad=\left(W^{(Q,i)}\cdot(M^{(i)})^{\top},W^{(K,i)}\cdot(M^{(i)})^{-1},W^{(V,i)}\cdot N^{(i)},(N^{(i)})^{-1}\cdot W^{(O,i)}\right).$
(5)

###### Remark 2 (Necessity of the assumptions $(a)$ , $(b)$ , $(c)$ , and $(d)$ ) .

The four assumptions in Theorem [3.2](#S3.Thmtheorem2) are necessary for the implication $(1.)\Rightarrow(2.)$. Indeed, the assumptions $(c)$ and $(d)$ are needed to utilize Theorem [3.1](#S3.Thmtheorem1). In addition, it follows from Theorem [3.1](#S3.Thmtheorem1) and item $(1.)$ that

$$ $\begin{array}[]{ccccl}W^{(Q,i)}\cdot(W^{(K,i)})^{\top}&=&\overline{W}^{(Q,\tau(i))}\cdot\left(\overline{W}^{(K,\tau(i))}\right)^{\top}&=&A_{i},\,\text{and}\\ W^{(V,i)}\cdot W^{(O,i)}&=&\overline{W}^{(V,\tau(i))}\cdot\overline{W}^{(O,\tau(i))}&=&B_{i}\end{array}$ $$

for some matrices $A_{i},B_{i}\in\mathbb{R}^{D\times D}$. At this point, the assumptions $(a)$ and $(b)$ are necessary to utilize the rank decomposition of $A_{i}$ and $B_{i}$ to obtain the symmetries of the factors []. It worth noting that these assumptions are generically satisfied in practical implementations since the parameters of $\operatorname{MultiHead}$ are randomly initialized with dimensions $D_{k}\leqslant D$ and $D_{v}\leqslant D$.

###### Remark 3 (Maximal symmetric group of multi-head attentions) .

Theorem [3.2](#S3.Thmtheorem2) suggests that a multi-head attention in practice is characterized by the products $W^{(Q,i)}\cdot\left(W^{(K,i)}\right)^{\top}$ and $W^{(V,i)}\cdot W^{(O,i)}$ at each head.
As a consequence, we can consider the group $\mathcal{S}_{h}\times\operatorname{GL}_{D_{k}}(\mathbb{R})^{h}\times\operatorname{GL}_{D_{v}}(\mathbb{R})^{h}$ to be the maximal symmetric group of the $\operatorname{MultiHead}$ map.

###### Theorem 3.1 (Independence of heads in multi-head attention) .

###### Remark 1 .

###### Theorem 3.2 (Maximal symmetric group of multi-head attentions) .

###### Remark 2 (Necessity of the assumptions ( a ) 𝑎 (a) , ( b ) 𝑏 (b) , ( c ) 𝑐 (c) , and ( d ) 𝑑 (d) ) .

###### Remark 3 (Maximal symmetric group of multi-head attentions) .

## 4 Weight Space of a Transformer Block and Group Action

In this section, we construct the weight space of a standard transformer block and define a group action on it. Let us first recall necessary notations from permutation matrices before defining the weight space in Section [4.2](#S4.SS2) and the group action in Section [4.3](#S4.SS3).

### 4.1 Permutation Matrices

###### Definition 4.1 .

Let $n$ be a positive integer.
A matrix of size $n\times n$ is called a *permutation matrix* if it has exactly one entry equal to $1$ in each row and each column, and zeros elsewhere. We denote by $\mathcal{P}_{n}$ the set of such all permutation matrices.

###### Remark 4 (Permutation matrix vs. permutation) .

For every permutation matrix $P\in\mathcal{P}_{n}$, there exists a unique permutation $\pi\in\mathcal{S}_{n}$ such that $P$ is obtained by permuting the $n$ columns of the identity matrix $I_{n}$ according to $\pi$. In this case, we write $P:=P_{\pi}$ and call it the permutation matrix corresponding to $\pi$. Here, $\mathcal{S}_{n}$ is the group of all permutations of the set $\{1,2,\ldots,n\}$.
In particular, for permutation matrices $P_{\pi}\in\mathcal{P}_{n}$ and $P_{\sigma}\in\mathcal{P}_{m}$, we have

$$ $\displaystyle P_{\pi}\cdot\mathbf{x}=\left(x_{\pi^{-1}(1)},x_{\pi^{-1}(2)},\ldots,x_{\pi^{-1}(n)}\right)^{\top},$ (6) $$

and

$$ $\displaystyle\left(\ P_{\pi}\cdot A\cdot P_{\sigma}\right)_{ij}=A_{\pi^{-1}(i)\sigma(j)},$ (7) $$

for all vector $\mathbf{x}\in\mathbb{R}^{n}$ and matrix $A\in\mathbb{R}^{n\times m}$.

###### Definition 4.1 .

###### Remark 4 (Permutation matrix vs. permutation) .

### 4.2 Weight space

A standard transformer block, which will be denoted by $\operatorname{Attn}$ from now on, contains a multi-head attention and a two-linear-layer ReLU MLP as well as a layer normalization.
Formally, $\operatorname{Attn}$ transforms an input sequence $X\in\mathbb{R}^{L\times D}$ to an output sequence $\operatorname{Attn}(X)\in\mathbb{R}^{L\times D}$ defined as follows:

$$ $\displaystyle\operatorname{Attn}(X)$ $\displaystyle=\operatorname{LayerNorm}\left(\operatorname{ReLU}\left(\hat{X}\cdot W^{(A)}+\mathbf{1}_{L}\cdot b^{(A)}\right)\cdot W^{(B)}+\mathbf{1}_{L}\cdot b^{(B)}\right),$ (8) $$

where

$$ $\displaystyle\hat{X}$ $\displaystyle=\operatorname{LayerNorm}\left(\operatorname{MultiHead}\left(X;\{W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\}_{i=1}^{h}\right)\right).$ (9) $$

with $\mathbf{1}_{L}=[1,\ldots,1]^{\top}\in\mathbb{R}^{L\times 1}$.
This transformer block is parametrized by the following matrices:

- •
$\left(W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right)_{i=1,\ldots,h}\in\left(\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}\right)^{h}$ inside the multi-head attention, and
- •
weights $(W^{(A)},W^{(B)})\in\mathbb{R}^{D\times D_{A}}\times\mathbb{R}^{D_{A}\times D}$ and biases $(b^{(A)},b^{(B)})\in\mathbb{R}^{1\times D_{A}}\times\mathbb{R}^{1\times D}$ inside the two-linear-layers $\operatorname{ReLU}$ MLP.

The weight space of this transformer block, say $\mathcal{U}$, is the vector space:

$$ $\displaystyle\mathcal{U}=\left(\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}\right)^{h}\times\big{(}\mathbb{R}^{D\times D_{A}}\times\mathbb{R}^{D_{A}\times D}\big{)}\times\big{(}\mathbb{R}^{1\times D_{A}}\times\mathbb{R}^{1\times D}\big{)}.$ (10) $$

We write each element $U\in\mathcal{U}$ in the form

$$ $\displaystyle U=\Bigl{(}\left([W]^{(Q,i)},[W]^{(K,i)},[W]^{(V,i)},[W]^{(O,i)}\right)_{i=1,\ldots,h},\left([W]^{(A)},[W]^{(B)}\right),\left([b]^{(A)},[b]^{(B)}\right)\Bigr{)}.$ (11) $$

To emphasize the weights of each $\operatorname{Attn}$ map, we will write $\operatorname{Attn}(X;U)$ instead of $\operatorname{Attn}(X)$. In particular, each element $U\in\mathcal{U}$ will define a map $\operatorname{Attn}(\cdot;U)\colon\mathbb{R}^{L\times D}\to\mathbb{R}^{L\times D}$, which maps a sequence $X\in\mathbb{R}^{L\times D}$ to a sequence $\operatorname{Attn}(X;U)\in\mathbb{R}^{L\times D}$ defined by equation [8](#S4.E8) and equation [9](#S4.E9). In the next section, we will find a sufficient condition for a pair $U$ and $U^{\prime}$ in $\mathcal{U}$ such that $\operatorname{Attn}(\cdot;U)=\operatorname{Attn}(\cdot;U^{\prime})$.

### 4.3 Group action on weight space

Let us consider the weight space $\mathcal{U}$ of a standard transformer block defined in Equation ([10](#S4.E10)) and Equation ([11](#S4.E11)). We consider the symmetries of $\mathcal{U}$ which arise from the following two sources:

- •
the maximal symmetric group of the multi-head attention, which are characterized in Theorem [3.2](#S3.Thmtheorem2);
- •
the permutation symmetries represented by permutation matrices, which arise from the unordered structure of neurons in the two-layer $\operatorname{ReLU}$ MLP.

Based on these observations, we consider the group

$$ $\displaystyle\mathcal{G}_{\mathcal{U}}=\mathcal{S}_{h}\times\operatorname{GL}_{D_{k}}(\mathbb{R})^{h}\times\operatorname{GL}_{D_{v}}(\mathbb{R})^{h}\times\mathcal{P}_{D}\times\mathcal{P}_{D_{A}}.$ (12) $$

This group will serve as the symmetric group of the standard attention block.
Each element $g\in\mathcal{G}_{\mathcal{U}}$ has the form

$$ $\displaystyle g=\Bigl{(}\tau,(M_{i})_{i=1,\ldots,h},(N_{i})_{i=1,\ldots,h},P_{\pi_{O}},P_{\pi_{A}}\Bigr{)},$ (13) $$

where
$\tau\in\mathcal{S}_{h},\,\pi_{O}\in\mathcal{S}_{D},\,\pi_{A}\in\mathcal{S}_{D_{A}}$ are permutations and $M_{i},N_{i}$ are invertible matrices of appropriate sizes.
The action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$ is defined formally as follows:

###### Definition 4.2 (Group action) .

With notation as above,
the action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$ is defined to be a map $\mathcal{G}_{\mathcal{U}}\times\mathcal{U}\to\mathcal{U}$
which maps an element $(g,U)\in\mathcal{G}_{\mathcal{U}}\times\mathcal{U}$ with $U$ and $g$ given in equation [11](#S4.E11) and equation [13](#S4.E13)
to the element

$$ $\displaystyle gU$ $\displaystyle=\biggl{(}\left([gW]^{(Q,i)},[gW]^{(K,i)},[gW]^{(V,i)},[gW]^{(O,i)}\right)_{i=1,\ldots,h},$ $\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\left([gW]^{(A)},[gW]^{(B)}\right),\left([gb]^{(A)},[gb]^{(B)}\right)\biggr{)},$ (14) $$

where

$$ $\displaystyle[gW]^{(Q,i)}$ $\displaystyle=[W]^{(Q,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{\top},$ $\displaystyle[gW]^{(K,i)}$ $\displaystyle=[W]^{(K,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{-1},$ $\displaystyle[gW]^{(V,i)}$ $\displaystyle=[W]^{(V,\tau(i))}\cdot N^{(\tau(i))},$ $\displaystyle[gW]^{(O,i)}$ $\displaystyle=\left(N^{(\tau(i))}\right)^{-1}\cdot[W]^{(O,\tau(i))}\cdot P_{\pi_{O}},$ $\displaystyle[gW]^{(A)}$ $\displaystyle=P_{\pi_{O}}^{-1}\cdot[W]^{(A)}\cdot P_{\pi_{A}},$ $\displaystyle[gW]^{(B)}$ $\displaystyle=P_{\pi_{A}}^{-1}\cdot[W]^{(B)},$ $\displaystyle[gb]^{(A)}$ $\displaystyle=[b]^{(A)}\cdot P_{\pi_{A}},$ $\displaystyle[gb]^{(B)}$ $\displaystyle=[b]^{(B)}.$ $$

We conclude the construction of the group action on $\mathcal{U}$ in the following theorem which results in a design principle for transformer blocks.

###### Theorem 4.3 (Invariance of $\operatorname{Attn}$ under the action of $\mathcal{G}_{\mathcal{U}}$ ) .

With notations as above, we have

$$ $\displaystyle\operatorname{Attn}(X;gU)=\operatorname{Attn}(X;U)$ (15) $$

for all $U\in\mathcal{U}$, $g\in\mathcal{G}$, and $X\in\mathbb{R}^{L\times D}$.

###### Remark 5 (Other types of symmetries) .

The group $\mathcal{G}_{\mathcal{U}}$ defined above does not cover all symmetries of the weight space $\mathcal{U}$. Indeed, there are symmetries of scaling type arising from the internal architecture of the $\operatorname{ReLU}$ MLP (see [, ]). In addition, layer normalization also offers additional scaling and sign-flipping symmetries for $\mathcal{U}$ (see Appendix [B](#A2)). We leave the problem of designing the maximal symmetric group of a transformer block for future study.

###### Definition 4.2 (Group action) .

###### Theorem 4.3 (Invariance of Attn Attn \operatorname{Attn} under the action of 𝒢 𝒰 subscript 𝒢 𝒰 \mathcal{G}_{\mathcal{U}} ) .

###### Remark 5 (Other types of symmetries) .

## 5 Equivariant Polynomial NFNs for Transformers

In this section, we introduce a family of NFNs specifically designed for Transformers, referred to as Transformer-NFNs, which are equivariant to the group $\mathcal{G}_{\mathcal{U}}$ as described in equation [12](#S4.E12). The main building blocks of Transformer-NFNs consist of $\mathcal{G}_{\mathcal{U}}$-equivariant and $\mathcal{G}_{\mathcal{U}}$-invariant polynomial layers. We will outline the construction of a class of $\mathcal{G}_{\mathcal{U}}$-invariant polynomial layers below. The detailed construction of the $\mathcal{G}_{\mathcal{U}}$-equivariant polynomial layers, which follows a similar approach, will be thoroughly derived in Appendix [D](#A4).

In concrete, we define a polynomial map $I\colon\mathcal{U}\rightarrow\mathbb{R}^{D^{\prime}}$ with maps each element $U\in\mathcal{U}$ to the vector $I(U)\in\mathbb{R}^{D^{\prime}}$ of the form

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,s):p,q}\cdot[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,s):p,q}\cdot[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(Q,s):p,q}\cdot[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(K,s):p,q}\cdot[W]^{(K,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi_{(V,s):p,q}\cdot[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi_{(O,s):p,q}\cdot[W]^{(O,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):p,q}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):p,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):q}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1},$ (16) $$

where

$$ $\displaystyle[WW]^{(QK,s)}:=[W]^{(Q,s)}\cdot\left([W]^{(K,s)}\right)^{\top},\quad\text{and}\quad[WW]^{(VO,s)}:=[W]^{(V,s)}\cdot[W]^{(O,s)},$ (17) $$

and the coefficients $\Phi_{-}$s are matrices of size $D^{\prime}\times 1$.

###### Remark 6 ( $I(U)$ as a quadratic polynomial) .

Intuitively speaking, $I(U)$ is a linear combination of all entries of the matrices $[W]^{(Q,s)}$, $[W]^{(K,s)}$, $[W]^{(V,s)}$, $[W]^{(O,s)}$, $[W]^{(A)}$, $[W]^{(B)}$, $[b]^{(A)}$, and $[b]^{(B)}$ in $U$, as well as all entries of the additional matrices $[WW]^{(QK,s)}$ and $[WW]^{(VO,s)}$ defined in Equation ([17](#S5.E17)). Since the entries of the matrices $[WW]^{(QK,s)}$ and $[WW]^{(VO,s)}$ are polynomials of degree $2$, the map $I(U)$ is indeed a quadratic polynomial in the entries of $U$. These additional quadratic terms help us incorporate more relations between weights inside the input multi-head attention block, thus allowing Transformer-NFN to maintain its expressivity.

The above formula for $I(U)$ is irredundant in the following sense:

###### Proposition 5.1 .

With notation as above, if $I(U)=0$ for all $U\in\mathcal{U}$, then $\Phi_{-}=0$ for all coefficients $\Phi_{-}$.

To make $I$ to be $\mathcal{G}_{\mathcal{U}}$-invariant,
the parameters $\Phi_{-}$ must satisfy a system of constraints (usually called *parameter sharing*), which are induced from the condition $I(gU)=I(U)$ for all $g\in\mathcal{G}_{\mathcal{U}}$ and $U\in\mathcal{U}$.
We show in details what are these constraints and how to derive the concrete formula of $I$ in Appendix [E](#A5).
The formula of $I$ is then determined by

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,\bullet):p,q}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,\bullet):p,\bullet}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet,\bullet}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):\bullet,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1}$ (18) $$

In the above formula, the bullet $\bullet$ indicates that the value of the corresponding coefficient is independent of the index at the bullet.

###### Theorem 5.2 .

With notation as above, the polynomial map $I\colon\mathcal{U}\rightarrow\mathbb{R}^{D^{\prime}}$ defined by Equation ([5](#S5.Ex22)) is $\mathcal{G}_{\mathcal{U}}$-invariant.
Moreover, if a map given in Equation ([5](#S5.Ex17)) is $\mathcal{G}_{\mathcal{U}}$-invariant, then it has the form given in Equation ([5](#S5.Ex22)).

The concrete formula for the polynomial $\mathcal{G}_{\mathcal{U}}$-equivariant layer is presented in detail in Appendix [D](#A4).

###### Remark 6 ( I ​ ( U ) 𝐼 𝑈 I(U) as a quadratic polynomial) .

###### Proposition 5.1 .

###### Theorem 5.2 .

## 6 The Small Transformer Zoo dataset

Large-scale empirical studies have produced datasets of trained classification models with varied hyperparameters [, ], enabling data-driven approaches to modeling generalization. However, a dataset of Transformer-based classification models is absent. Motivated by this shortage, we introduce the Small Transformer Zoo dataset to experimentally demonstrate the efficacy of our proposed method. This dataset contains the weights of a fixed Transformer architecture trained on two distinct datasets, spanning both vision and language tasks. Each entry in the dataset includes a checkpoint weight along with its corresponding accuracy metrics and hyperparameter configurations.

##### General settings.

We focus on two prevalent deep learning tasks: digit image classification using the MNIST dataset [] for vision, and text topic classification using AGNews [] for natural language processing. This selection covers two primary data modalities in deep learning—image and text—while addressing classification tasks, which are among the most common and fundamental in the field. Our model architecture contains three components: an embedding layer, an encoder, and a classifier. The embedding layer processes raw input data to produce initial token embeddings, which the encoder then transforms through self-attention mechanisms to capture contextual relationships. Finally, the classifier generates the classification output based on these enriched embeddings. While we adapt the embedding and classifier components to each specific task, we maintain a consistent encoder architecture across both tasks, consisting of two stacked two-head transformer blocks as defined in Equation ([8](#S4.E8)).

Our resulting datasets, named MNIST-Transformers and AGNews-Transformers, consist of $62756$ and $63796$ model checkpoints, respectively. These models were generated by varying key hyperparameters such as the optimizer, learning rate, weight regularization, weight initialization, and dropout, as detailed in Appendix [F](#A6). By making these datasets available, we aim to facilitate and inspire further research into the inner workings and behavior of Transformer models.

## 7 Experimental Results

We empirically evaluate the performance of the proposed Transformer-NFN model on two datasets: MNIST-Transformers and AGNews-Transformers. Additionally, we conduct ablation studies to examine the contribution of each component within the Transformer architecture in predicting network generalization, and investigate the impact of varying the Transformer-NFN dimension and the number of layers on the overall performance. Our analysis yields three key findings: (1) Transformer-NFN, with its enhanced layers for processing transformer block parameters, outperforms existing baselines in both vision and NLP Transformers datasets, maintains consistent performance across different accuracy thresholds of the data (2) the information embedded in the weights of Transformer blocks provides a strong predictor for the performance of transformer model, and (3) good performance for Transformer-NFN can be obtained with a compact setting.

We use Kendall’s $\tau$ rank correlation [], ranging from $[-1,1]$, as the evaluation metric to assess how closely predicted accuracy rankings align with ground truth accuracy rankings. A value near $1$ indicates strong agreement, as shown in the scatterplot in Figure [2](#A7.F2) (in Appendix [G](#A7)). All results in this section are averaged over 5 runs with different random seeds, with details on hyperparameters and training settings provided in Appendix [G](#A7).

### 7.1 Predicting Vision Transformers Generalization from pretrained weights

Experiment Setup. In this experiment, we focus on predicting the test accuracy of pretrained Vision Transformer models using only their weights, without access to the test set. To perform this task, we utilize our MNIST-Transformers dataset. We evaluate our model against 5 models: MLP, STATNN [], XGBoost [], LightGBM [], and Random Forest  []. As shown in Figure [1](#A6.F1) (in Appendix [G](#A7)), the accuracy distribution of the MNIST-Transformers dataset is highly skewed (notice the log scale on the y-axis). Therefore, we evaluate each model’s prediction performance not only on the entire dataset but also on four smaller subsets, each filtered by accuracy thresholds of $20\%$, $40\%$, $60\%$, and $80\%$. As a significant portion of pretrained models in the dataset exhibit higher accuracy, achieving a strong Kendall’s $\tau$ correlation becomes increasingly challenging as the accuracy thresholds increase.

Results. Table [1](#S7.T1) illustrates the results of all models when predicting generalization of Vision Transformer networks trained on MNIST-Transformer dataset. As expected, Kendall’s $\tau$ generally decreases as the accuracy threshold increases. In addition, our Transformer-NFN consistently outperforms all four baseline models across all dataset settings with performance gap ranging from $0.004$ to $0.026$, demonstrating the effectiveness of our model’s design in capturing the information within each transformer block.

Table: Table 1: Performance measured by Kendall’s $\tau$ of all models on MNIST-Transformers dataset. Uncertainties indicate standard error over 5 runs.

Table: Table 2: Performance measured by Kendall’s $\tau$ of all models on AGNews-Transformers dataset. Uncertainties indicate standard error over 5 runs.

### 7.2 Predicting Text classification Transformers generalization

##### Experiment Setup.

In this experiment, we utilize the AGNews-Transformers dataset to predict the performance of pretrained transformer models in text classification. The goal is to evaluate the effectiveness of Transformer-NFN in predicting the performance of pretrained models specifically trained on language tasks. Similar to Experiment [7.1](#S7.SS1), we assess our model’s capabilities across different dataset configurations by using five subsets: the entire dataset without any accuracy threshold, and four subsets with accuracy thresholds of $20\%$, $40\%$, $60\%$, and $80\%$, respectively.

##### Results.

Table [2](#S7.T2) shows that our model consistently outperforms all baseline models across all text classification dataset configurations. Compared to the MNIST-Transformers dataset, the performance improvement in this experiment is even more significant, with Kendall’s $\tau$ gaps ranging from $0.018$ to $0.033$. Since language tasks inherently involve more complex syntactic structures, the transformer encoder can capture richer information in its weights, enabling Transformer-NFN to more effectively predict performance in NLP tasks than in vision tasks. Figure [2](#A7.F2) (in Appendix [G](#A7)) highlights our model’s superior performance compared to other baselines. Unlike other models that struggle with predicting generalization in low accuracy networks, Transformer-NFN effectively captures weight feature information, demonstrating consistent superior performance.

### 7.3 Importance of encoder in predicting the generalization

A interesting question arises is how much information about the network generalization ability is embedded in each component of the transformer model. To investigate this, we restrict our Transformer-NFNs to access only certain subsets of the Transformer’s components and train the model on MNIST-Transformers dataset and AGNews-Transformers dataset. Our goal is to determine the importance of each component, both individually and in combination, for predicting generalization of the input network.

Table [3](#S7.T3) shows that, for both the MNIST-Transformers and AGNews-Transformers datasets, the transformer blocks alone provide a strong prediction for the performance. The classifier is the second most important, followed by the embedding. Even with only access to the transformer blocks weights, our model achieves a Kendall’s $\tau$ score almost equal to when all components are used: $0.902$ compared to $0.905$ for MNIST-Transformers, and $0.909$ compared to $0.91$ for AGNews-Transformers.

Table: Table 3: Ablation study the important of each component of the input networks on predicting generalization of the input network, the metric being used is Kendall’s $\tau$.

### 7.4 Abalation study on varying the dimension and number of layers

In this section, we conduct an ablation study to explore the impact of varying the hidden dimension and the number of equivariant layers in Transformer-NFN. We evaluate different configurations of hidden dimensions and layer counts on the AGNews-Transformer dataset, with dimensions $\in[3,5,10,15]$, and number of layer $\in[1,2]$. These configurations allow us to assess how changes in model size affect performance and efficiency.

Table [5](#A7.T5) (in Appendix [G](#A7)) demonstrates that strong performance can be achieved with a relatively small dimension and few parameters. For example, with dimension of $15$ and a single layer, the model reaches a Kendall’s $\tau$ score of $0.913$, matching the best performance across all settings. These results suggest that good performance for our Transformer-NFN can be obtained with a compact setting.

## 8 Conclusion

In this work, we made significant contributions to the understanding and application of NFNs in transformer architectures. We determined the maximal symmetric group of the weights in a multi-head attention module. We also formally defined the weight space of a transformer architecture and introduced a group action on this weight space, thereby characterizing the design principles for NFNs. Additionally, we presented Transformer-NFN, an NFN designed for transformer architectures that is equivariant under the specified group action. Finally, we released a dataset of more than 125,000 transformers model checkpoints trained on two datasets with two different tasks, marking a significant resource for benchmarking the applicability and efficiency of Transformer-NFN and promoting further research to enhance our understanding of transformer network training and performance.

## Ethics Statement.

Given the nature of the work, we do not foresee any negative societal and ethical
impacts of our work.

## Reproducibility Statement.

Source codes for our experiments are provided in the supplementary
materials of the paper. The details of our experimental settings are given in Section [7](#S7) and the Appendix [G](#A7). All datasets used in this paper are publicly available through an anonymous link provided in the README file of the supplementary material.

## Appendix A Maximal Symmetric Group of MultiHead MultiHead \operatorname{MultiHead}

This section aims to provide a complete answer to the question of when two sets of parameters of a multi-head attention mechanism will define the same function. The answer to this question will serve as the core of the design principle of NFNs for Transformers. We recall the formulation of $\operatorname{Head}$ and $\operatorname{MultiHead}$ attention maps in Section [A.1](#A1.SS1). Then, the maximal symmetric group of $\operatorname{Head}$ is derived in Section [A.2](#A1.SS2).

### A.1 Multi-head Attention

Let $D$ be a positive integer. Recall the notion of the parameterized map $\operatorname{Head}$ as follows: For a positive integer $L$ and $X\in\mathbb{R}^{L\times D}$, we have

$$ $\displaystyle\operatorname{Head}\left(X;W^{(Q)},W^{(K)},W^{(V)}\right)$ $\displaystyle=\operatorname{softmax}\left(\frac{\left(XW^{(Q)}\right)\cdot\left(XW^{(K)}\right)^{\top}}{\sqrt{D_{k}}}\right)\cdot\left(XW^{(V)}\right),$ $$

where $W^{(Q)},W^{(K)}\in\mathbb{R}^{D\times D_{k}}$ and $W^{(V)}\in\mathbb{R}^{D\times D_{v}}$. By definition, we have

$$ $\displaystyle\operatorname{Head}\left(\cdot;W^{(Q)},W^{(K)},W^{(V)}\right)~{}\colon~{}\bigsqcup_{l>0}\mathbb{R}^{l\times D}\rightarrow\bigsqcup_{l>0}\mathbb{R}^{l\times D_{v}},$ $$

and for all $l>0$, the image of $\mathbb{R}^{l\times D}$ is contained in $\mathbb{R}^{l\times D_{v}}$. By combining $\left(W^{(Q)}\right)\cdot\left(W^{(K)}\right)^{\top}/\sqrt{D_{k}}\in\mathbb{R}^{D\times D}$, we can rewrite the map $\operatorname{Head}$ as follows

$$ $\displaystyle\operatorname{Head}\left(X;W^{(Q)},W^{(K)},W^{(V)}\right)$ $\displaystyle=\operatorname{softmax}\left(X\cdot\frac{\left(W^{(Q)}\right)\cdot\left(W^{(K)}\right)^{\top}}{\sqrt{D_{k}}}\cdot X^{\top}\right)\cdot\left(XW^{(V)}\right),$ $$

By this observation, we define a class of parameterized maps as follows: For $X\in\mathbb{R}^{L\times D}$, we have

$$ $\displaystyle f(X;A)\coloneqq\operatorname{softmax}\left(XAX^{\top}\right)\cdot X,$ $$

where $A\in\mathbb{R}^{D\times D}$. Similarly, for the $\operatorname{MultiHead}$ map, we have

$$ $\displaystyle\operatorname{MultiHead}$ $\displaystyle\left(X;W^{(O)},\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right\}_{i=1}^{h}\right)$ $\displaystyle=\left(\bigoplus\limits_{i=1}^{h}\operatorname{Head}\left(X;W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right)\right)W^{(O)}$ $\displaystyle=\sum\limits_{i=1}^{h}\operatorname{Head}\left(X;W^{(Q,i)},W^{(K,i)},W^{(V,i)}\right)W^{(O,i)}$ $\displaystyle=\sum\limits_{i=1}^{h}\operatorname{softmax}\left(\frac{\left(XW^{(Q,i)}\right)\cdot\left(XW^{(K,i)}\right)^{\top}}{\sqrt{D_{k}}}\right)\cdot X\cdot\left(W^{(V,i)}\cdot W^{(O,i)}\right),$ $$

where $h$ is a positive integer, $W^{(O)}=\left(W^{(O:1)},\ldots,W^{(O:h)}\right)$ with each $W^{(O:i)}\in\mathbb{R}^{D_{v}\times D}$. Considering $W^{(V,i)}\cdot W^{(O,i)}$ as a matrix $B_{i}\in\mathbb{R}^{D\times D}$, we define a new class of parameterized maps as follows: For $X\in\mathbb{R}^{L\times D}$, we have

$$ $\displaystyle F\left(X;\left\{A_{i},B_{i}\right\}_{i=1}^{h}\right)=\sum_{i=1}^{h}f(X;A_{i})\cdot B_{i},$ $$

where $h$ is a positive integer and $A_{i},B_{i}\in\mathbb{R}^{D\times D}$. Note that,

$$ $\displaystyle\operatorname{rank}\left(\left(W^{(Q)}\right)\cdot\left(W^{(K)}\right)^{\top}/\sqrt{D_{k}}\right)\leqslant\min(D,D_{k})\leqslant D$ $\displaystyle\operatorname{rank}\left(W^{(V,i)}\cdot W^{(O,i)}\right)\leqslant\min(D,D_{v})\leqslant D$ $$

So, in general, the new class of $F$ maps contains the class of $\operatorname{MultiHead}$ maps. Note that, $F$ simply is a weighted summation of some $f$ maps and is linear with respect to $\{B_{i}\}_{i=1}^{h}$.

###### Remark 7 .

We can see, in $f$, the matrix $A$ plays the role as a core matrix that defines $f$. Similarly, in $F$, each $A_{i}$ defines each $f_{i}$, and $B_{i}$ is the weight of each component contributes to $F$.

###### Remark 7 .

### A.2 Maximal symmetric group of multi-head attention

In this section, we present a theoretical result that shows the following: Roughly speaking, in the multi-head scenario, each individual head plays its own unique role. The main results are Theorems [A.1](#A1.Thmtheorem1) (which corresponds to Theorem [3.1](#S3.Thmtheorem1) in the main text) on the unique role of each head and Theorem [A.7](#A1.Thmtheorem7) (which corresponds to Theorem [3.2](#S3.Thmtheorem2) in the main text) on the maximal symmetric group of multi-head attention.

###### Theorem A.1 .

Let $D$ be a positive integer. Assume that for a positive integer $h$, matrices $A_{1},A_{2},\ldots,A_{h}\in\mathbb{R}^{D\times D}$ and $B_{1},B_{2},\ldots,B_{h}\in\mathbb{R}^{D\times D}$, we have

$$ $\displaystyle F\left(X;\left\{A_{i},B_{i}\right\}_{i=1}^{h}\right)=\sum_{i=1}^{h}f(X;A_{i})\cdot B_{i}=0$ (19) $$

for all positive integers $L$ and $X\in\mathbb{R}^{L\times D}$. Then, if $A_{1},A_{2},\ldots,A_{h}$ are pairwise distinct, then

$$ $\displaystyle B_{1}=\ldots=B_{h}=0.$ $$

###### Proof.

From Equation ([19](#A1.E19)), we have

$$ $\displaystyle\sum_{i=1}^{h}\operatorname{softmax}\left(XA_{i}X^{\top}\right)\cdot X\cdot B_{i}=0$ (20) $$

Consider $L>1$. We write $X=(x,y,y,\ldots,y)^{\top}\in\mathbb{R}^{L\times D}$ where $x=(x_{1},\ldots,x_{D})$ and $y=(y_{1},\ldots,y_{D})$ in $\mathbb{R}^{1\times D}$ (So in $X$, $y$ appears $L-1$ times). We will consider the entry in the first row and first column of both sides of Equation ([20](#A1.E20)). Let $b_{1},b_{2},\ldots,b_{h}\in\mathbb{R}^{D\times 1}$ be the first column of matrices $B_{1},B_{2},\ldots,B_{h}$. From Equation ([20](#A1.E20)), we have

$$ $\displaystyle\sum_{i=1}^{h}\left(\dfrac{e^{xA_{i}x^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot x+\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot y\right)\cdot b_{i}=0$ (21) $$

By substituting $x=y=(1,0,\ldots,0)\in\mathbb{R}^{D}$ to Equation ([21](#A1.E21)), we have the sum of all first entries of $b_{1},\ldots,b_{h}$ is equal to 0 0. Similarly, for every $j=1,\ldots,D$, the sum of $j^{\text{th}}$ entries of $b_{1},\ldots,b_{h}$ is equal to 0 0. It shows that

$$ $\displaystyle b_{1}+b_{2}+\ldots+b_{h}=0.$ (22) $$

From Equation ([21](#A1.E21)), we have

$$ $\displaystyle 0$ $\displaystyle=\sum_{i=1}^{h}\left(\dfrac{e^{xA_{i}x^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot x+\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot y\right)\cdot b_{i}$ (23) $\displaystyle=\sum_{i=1}^{h}\left(\dfrac{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot x+\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot(y-x)\right)\cdot b_{i}$ (24) $\displaystyle=\sum_{i=1}^{h}\left(x+\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot(y-x)\right)\cdot b_{i}$ (25) $\displaystyle=\sum_{i=1}^{h}x\cdot b_{i}+\sum_{i=1}^{h}\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot(y-x)\cdot b_{i}$ (26) $\displaystyle=x\cdot\left(\sum_{i=1}^{h}b_{i}\right)+(y-x)\cdot\left(\sum_{i=1}^{h}\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot b_{i}\right)$ (27) $\displaystyle=(y-x)\cdot\left(\sum_{i=1}^{h}\dfrac{Le^{xA_{i}y^{\top}}}{e^{xA_{i}x^{\top}}+Le^{xA_{i}y^{\top}}}\cdot b_{i}\right)$ (28) $\displaystyle=(y-x)\cdot\left(\sum_{i=1}^{h}\dfrac{L}{e^{xA_{i}x^{\top}-xA_{i}y^{\top}}+L}\cdot b_{i}\right)$ (29) $\displaystyle=(y-x)\cdot\left(\sum_{i=1}^{h}\dfrac{L}{e^{xA_{i}(x-y)^{\top}}+L}\cdot b_{i}\right)$ (30) $$

By let $z=x-y$, from the above equations, we have

$$ $\displaystyle z\cdot\left(\sum_{i=1}^{h}\dfrac{L}{e^{xA_{i}z^{\top}}+L}\cdot b_{i}\right)=0,$ (31) $$

or

$$ $\displaystyle z\cdot\left(\sum_{i=1}^{h}\dfrac{1}{e^{xA_{i}z^{\top}}+L}\cdot b_{i}\right)=0,$ (32) $$

for all $x=(x_{1},\ldots,x_{D})$ and $z=(z_{1},\ldots,z_{D})$ in $\mathbb{R}^{D}$. Now, each $xA_{i}z^{\top}$ can be viewed as a polynomial in $2D$ indeterminates $x_{1},\ldots,x_{D}$ and $z_{1},\ldots,z_{D}$ as follows

$$ $\displaystyle xA_{i}z^{\top}=\sum_{p=1}^{D}\sum_{q=1}^{D}(A_{i})_{p,q}\cdot x_{p}z_{q}.$ (33) $$

Since $A_{1},\ldots,A_{h}$ are pairwise distinct, so $xA_{1}z^{\top},\ldots,xA_{h}z^{\top}$ are pairwise distinct polynomials. By Lemma [A.3](#A1.Thmtheorem3), there exists $u\in\mathbb{R}^{D}$ and a non-empty open set $V\in\mathbb{R}^{D}$, such that for all $z\in V$, we have $uA_{1}z^{\top},\ldots,uA_{h}z^{\top}$ are pairwise distinct real numbers. Obviously $0\notin V$. Now fix an $v\in V$ and let $z=t\cdot v$ for $t\in\mathbb{R}$. Denote $uA_{i}v^{\top}=s_{i}$, from Equation ([33](#A1.E33)), we have

$$ $\displaystyle t\cdot v\cdot\left(\sum_{i=1}^{h}\dfrac{1}{e^{t\cdot s_{i}}+L}\cdot b_{i}\right)=0$ (34) $$

for all $t\in\mathbb{R}$, or

$$ $\displaystyle v\cdot\left(\sum_{i=1}^{h}\dfrac{1}{e^{t\cdot s_{i}}+L}\cdot b_{i}\right)=0$ (35) $$

for all $t\in\mathbb{R}\setminus\{0\}$. By continuity, this still holds for $t=0$. So we have

$$ $\displaystyle v\cdot\left(\sum_{i=1}^{h}\dfrac{1}{e^{t\cdot s_{i}}+L}\cdot b_{i}\right)=0$ (36) $$

for all $t\in\mathbb{R}$. Now, consider the set

$$ $\displaystyle S\coloneqq\left\{\left(\dfrac{1}{e^{t\cdot s_{1}}+L},\dfrac{1}{e^{t\cdot s_{2}}+L},\ldots,\dfrac{1}{e^{t\cdot s_{h}}+L}\right)\in\mathbb{R}^{h}~{}\colon~{}t\in\mathbb{R}\right\}.$ (37) $$

From Equation ([36](#A1.E36)), we have the linear span of $S$, i.e. $\operatorname{span}(S)$, satisfies that: For all $s=(s_{1},s_{2},\ldots,s_{h})$ $\in\operatorname{span}(S)$, we have

$$ $\displaystyle v\cdot\left(\sum_{i=1}^{h}s_{i}\cdot b_{i}\right)=0.$ (38) $$

In other words, $v$ and $\sum_{i=1}^{h}s_{i}\cdot b_{i}$ are orthogonal to each other as two vectors in $\mathbb{R}^{D}$. Since $s_{1},\ldots,s_{h}\in\mathbb{R}$ are pairwise distinct, by Lemma [A.4](#A1.Thmtheorem4), there exist $t_{1},\ldots,t_{h}\in\mathbb{R}$ such that their $h$ corresponding vectors in $S$ form a basis of $\mathbb{R}^{h}$. This implies $\operatorname{span}(S)=\mathbb{R}^{h}$. We have the set

$$ $\displaystyle\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)$ $\displaystyle=\left\{\sum_{i=1}^{h}s_{i}\cdot b_{i}~{}\colon~{}s=(s_{1},\ldots,s_{h})\in\mathbb{R}^{h}\right\}$ (39) $\displaystyle=\left\{\sum_{i=1}^{h}s_{i}\cdot b_{i}~{}\colon~{}s=(s_{1},\ldots,s_{h})\in\operatorname{span}(S)\right\}.$ (40) $$

By the previous observation, it implies that $v$ is orthogonal to every vectors in $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)$. In other words, the orthogonal complement of $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)$, which is denoted by $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)^{\perp}$, contains $v$. This holds for every vectors $v$ in the non-empty open set $V$, so

$$ $\displaystyle V\subset\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)^{\perp}.$ (41) $$

Since $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)^{\perp}$ is a linear subspace of $\mathbb{R}^{D}$ that contains a non-empty open set of $\mathbb{R}^{d}$, so by Lemma [A.2](#A1.Thmtheorem2), we have $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)^{\perp}=\mathbb{R}^{D}$. This implies $\operatorname{span}\left(\{b_{1},\ldots,b_{h}\}\right)=0$, which means $b_{1}=\ldots=b_{h}=0$. So the first column of $B_{1},\ldots,B_{h}$ are all equal to 0 0, and a similar proof is applied for every other columns of $B_{1},\ldots,B_{h}$. So $B_{1}=\ldots=B_{h}=0$.
∎

###### Lemma A.2 .

Let $D$ be a positive integer, and $V$ is a non-empty open set of $\mathbb{R}^{D}$ with the usual topology. If $U$ is a linear subspace of $\mathbb{R}^{D}$ that contains $V$, then $U=\mathbb{R}^{D}$.

###### Proof.

Since $V$ is non-empty, let $x\in V$. Since $V$ is open, there exists $r>0$ such that the closed ball

$$ $\displaystyle\bar{B}_{r}(x)=\{y\in\mathbb{R}^{D}~{}\colon~{}\|x-y\|\leqslant r\}\subset V.$ (42) $$

Then for all $y\in\mathbb{R}^{D}$ that $y\neq 0$, we have

$$ $\displaystyle\left\|x-\left(x+r\cdot\dfrac{y}{\|y\|}\right)\right\|=r,$ (43) $$

which means

$$ $\displaystyle x+r\cdot\dfrac{y}{\|y\|}\in\bar{B}_{r}(x)\subset V\subset U.$ (44) $$

Since $x$ is also in $V\subset U$, and $U$ is a linear subspace, then $y\in U$. So, for $y\in\mathbb{R}^{D}$ that $y\neq 0$, we have $y\in U$, and clearly $0\in U$, so $U=\mathbb{R}^{D}$.
∎

###### Remark 8 .

Lemma [A.2](#A1.Thmtheorem2) still holds if we replace $\mathbb{R}^{D}$ by a normed vector space.

###### Lemma A.3 .

Let $n,h$ be two positive integers, and $f_{1},\ldots,f_{h}\in\mathbb{R}[x_{1},\ldots,x_{n}]$ be $h$ pairwise distinct real polynomials in $n$ variables. Then there exists a non-empty open subset $U$ of $\mathbb{R}^{n}$ such that $f_{1}(x),\ldots,f_{h}(x)$ are pairwise distinct for all $x\in U$.

###### Proof.

Consider the polynomial

$$ $\displaystyle f=\prod_{1\leqslant i<j\leqslant h}(f_{i}-f_{j})\in\mathbb{R}[x_{1},\ldots,x_{n}].$ (45) $$

Since $f_{1},\ldots,f_{h}$ are pairwise distinct, then $f_{i}-f_{j}$ is non-zero for all $1\leqslant i<j\leqslant h$. It is well-known that the real polynomial ring is an integral domain, so $f$ is non-zero. There exists $a\in\mathbb{R}^{n}$ such that $f(a)\neq 0$. Since $\mathbb{R}$ is Hausdorff, we can choose a open set $V$ of $\mathbb{R}$ that contains $a$ and does not contain 0 0. Let $U=f^{-1}(V)$, and since $f$ is continuous, $U$ is open. We have $f(x)\neq 0$ for all $x\in U$, which means $f_{1}(x),\ldots,f_{n}(x)$ are pairwise distinct for all $x\in U$
∎

###### Lemma A.4 .

Let $h$ be a positive integer, and $h$ distinct real numbers $s_{1},\ldots,s_{h}$. Then there exists $h$ real numbers $t_{1},\ldots,t_{h}$ and a positive integer $L$ such that the matrix

$$ $\displaystyle A=(a_{ij})_{1\leqslant i\leqslant h,1\leqslant j\leqslant h}\in\mathbb{R}^{h\times h},~{}~{}\text{ where }a_{ij}=\dfrac{1}{e^{t_{i}\cdot s_{j}}+L},$ (46) $$

is full rank. In other words, there exists $h$ real numbers $t_{1},\ldots,t_{h}$ and a positive integer $L$ such that $h$ vectors

$$ $\displaystyle\left(\dfrac{1}{e^{t_{i}\cdot s_{1}}+L},\dfrac{1}{e^{t_{i}\cdot s_{2}}+L},\ldots,\dfrac{1}{e^{t_{i}\cdot s_{h}}+L}\right)\in\mathbb{R}^{h}$ (47) $$

for $i=1,\ldots,h$, form a basis of $\mathbb{R}^{h}$.

###### Proof.

We first have an observations. Let $t_{1},\ldots,t_{h}$ be fixed and $x$ be a variable. Consider the matrix

$$ $\displaystyle A(x)=(a_{ij})_{1\leqslant i\leqslant h,1\leqslant j\leqslant h},~{}~{}\text{ where }a_{ij}=\dfrac{1}{e^{t_{i}\cdot s_{j}}+x}$ (48) $$

Then the determinant of $A(x)$, denoted by $\operatorname{det}(A(x))$ can be viewed as a real rational function, i.e. a function that can be written as the ratio of two real polynomials. So, in the case that this rational function is zero, $\operatorname{det}(A(x))=0$ for all $x\in\mathbb{R}$, and in the case that this rational function is non-zero, there are a finite number of $x\in\mathbb{R}$ such that $\operatorname{det}(A(x))=0$ or $\operatorname{det}(A(x))$ is not defined. In other words, $A(x)$ is not full rank for all $x$, or for only a finite number of $x$. So, in Lemma [A.4](#A1.Thmtheorem4), if there exists $h$ real numbers $t_{1},\ldots,t_{h}$ and a real number $L$ that makes $A$ becomes full rank, then $L$ can be made to be a positive integer.

Back to the problem, we will prove by mathematical induction. We will show that for every $h$, it is possible to choose $t_{1},\ldots,t_{h}$ and $L$ to make $\operatorname{det}(A)$ becomes non-zero. For $h=1$, then the matrix $A$ is full rank since its single entry is always positive for $t=L=1$. Assume that the result holds for a positive integer $h-1$, we will show it holds for $h$. For $j=1,\ldots,h$, let $B_{j}$ is the $(h-1)\times(h-1)$ matrix obtained by removing the first row and the $j^{\text{th}}$ column of matrix $A$. By computing the determinant of $A$ via the Laplace expansion along the first row, we have

$$ $\displaystyle\operatorname{det}(A)=\sum_{j=1}^{h}(-1)^{1+j}\cdot a_{1j}\cdot\operatorname{det}(B_{j})=\sum_{j=1}^{h}(-1)^{1+j}\cdot\dfrac{1}{e^{t_{1}\cdot s_{j}}+2}\cdot\operatorname{det}(B_{j})$ (49) $$

Denote $c_{j}=(-1)^{1+j}\cdot\operatorname{det}(B_{j})$, and note that $c_{j}$ depends on the choice of $t_{2},\ldots,t_{h}$. Without loss of generality, assume $s_{1}\neq 0$. Since $s_{2},\ldots,s_{h}$ are $h-1$ pairwise distinct real numbers, by the induction hypothesis, there exists $t_{2},\ldots,t_{h}$ such that $c_{1}$ is non-zero for at least one $L\in\mathbb{R}$. So with this choice of $t_{2},\ldots,t_{h}$, there exists $\alpha\in\mathbb{R}$ such that

$$ $\displaystyle c_{1}\text{ is non-zero for all }L<\alpha.$ (50) $$

Since $s_{1},\ldots,s_{h}$ are pairwise distinct and $s_{1}\neq 0$, we can choose $t=t_{1}\in\mathbb{R}$ such that:

- 1.
$e^{t_{1}\cdot s_{1}}>1-\alpha$; and,
- 2.
$|e^{t_{1}\cdot s_{1}}-e^{t_{i}\cdot s_{j}}|>3$; for all $(i,j)\neq(1,1)$.

With this choice of $t_{1}$, let $\Delta=[-1-e^{t_{1}\cdot s_{1}},1-e^{t_{1}\cdot s_{1}}]$. Then for $L\in\Delta$

- 1.
We have $e^{t_{1}\cdot s_{1}}+L\in[-1,1]$.
- 2.
For $(i,j)\neq(1,1)$, since $|e^{t_{1}\cdot s_{1}}-e^{t_{i}\cdot s_{j}}|>3$, then $e^{t_{i}\cdot s_{j}}+L\notin[-1,1]$.
- 3.
We have $L\leqslant 1-e^{t_{1}\cdot s_{1}}<1-(1-\alpha)=\alpha$.

We show that with this choice of $t_{1},t_{2},\ldots,t_{h}$, there exists $L\in\mathbb{R}$ such that $\operatorname{det}(A)$ is non-zero. Assume the contrary that

$$ $\sum_{j=1}^{h}\dfrac{1}{e^{t_{1}\cdot s_{j}}+L}\cdot c_{j}=0$ (51) $$

for all $L\in\mathbb{R}$. This implies that

$$ $\displaystyle\dfrac{1}{e^{t_{1}\cdot s_{1}}+L}\cdot c_{1}=\sum_{j=2}^{h}\dfrac{1}{e^{t_{1}\cdot s_{j}}+L}\cdot c_{j},$ (52) $$

so

$$ $\displaystyle\left|\dfrac{1}{e^{t_{1}\cdot s_{1}}+L}\right|\cdot\left|c_{1}\right|=\left|\sum_{j=2}^{h}\dfrac{1}{e^{t_{1}\cdot s_{j}}+L}\cdot c_{j}\right|\leqslant\sum_{j=2}^{h}\left|\dfrac{1}{e^{t_{1}\cdot s_{j}}+L}\right|\cdot\left|c_{j}\right|$ (53) $$

Considering $c_{1},\ldots,c_{h}$ as functions in $L$, we have these functions are well-defined on the closed interval $\Delta$, since they are determinants of matrices where their entries are

$$ $\displaystyle\dfrac{1}{e^{t_{i}\cdot s_{j}}+L}$ (54) $$

for $1<i\leqslant h$ and $1\leqslant j\leqslant h$, and these entries are defined on $\Delta$ by the choice of $t_{1},\ldots,t_{h}$. Moreover, $c_{1},\ldots,c_{h}$ are continuous on $\Delta$. Since a continuous function on a compact set is bounded, so there exists $\delta_{1}>0$ such that

$$ $\displaystyle|c_{1}|,\ldots,|c_{h}|<\delta_{1},~{}\text{ for all }L\in\Delta.$ (55) $$

Moreover, since $L\in\Delta$ implies $L<\alpha$, then $|c_{1}|>0$ for all $L<\alpha$, which means there exists $\delta_{2}>0$ such that

$$ $\displaystyle|c_{1}|>\delta_{2},~{}\text{ for all }L\in\Delta.$ (56) $$

Similarly, for $1<j\leqslant h$,

$$ $\displaystyle\dfrac{1}{e^{t_{1}\cdot s_{j}}+L},$ (57) $$

considered as functions in $L$, is well-defined and continuous on $\Delta$, so there exist $\delta_{3}>0$ such that

$$ $\displaystyle\left|\dfrac{1}{e^{t_{1}\cdot s_{2}}+L}\right|,\ldots,\left|\dfrac{1}{e^{t_{1}\cdot s_{h}}+L}\right|<\delta_{3}~{}\text{ for all }L\in\Delta.$ (58) $$

From Equation ([53](#A1.E53)), since we have Equations ([55](#A1.E55)), ([56](#A1.E56)), ([58](#A1.E58)), then for all $L\in\Delta\setminus\{-e^{t_{1}\cdot s_{1}}\}$,

$$ $\displaystyle\delta_{2}\cdot\left|\dfrac{1}{e^{t_{1}\cdot s_{1}}+L}\right|\leqslant(h-1)\cdot\delta_{1}\cdot\delta_{3}.$ (59) $$

As $L\rightarrow-e^{t_{1}\cdot s_{1}}$, the LHS of Equation ([59](#A1.E59)) goes to $\infty$, but the RHS is a constant, which is a contradiction. So with the choice of $t_{1},\ldots,t_{h}$, there exists $L\in\mathbb{R}$ such that $\operatorname{det}(A)\neq 0$. The result holds for $h$. By mathematical induction, it holds for every positive integers $h$. The proof is done.
∎

###### Remark 9 .

If we fix a positive integer $L$, there might not exist $t_{1},\ldots,t_{h}$ satisfy the condition. For instance, if $h\geqslant 4$, and $s_{1}+s_{2}=s_{3}+s_{4}=0$, then the matrix $A$ is not full rank for all $t_{1},\ldots,t_{h}$.

We have two direct corollaries of Theorem [A.1](#A1.Thmtheorem1).

###### Corollary A.5 .

Let $D$ be a positive integer. Assume that, for two positive integers $h,h^{\prime}$, collections $\{A_{i}\}_{i=1}^{h}$, $\{A^{\prime}_{i}\}_{i=1}^{h^{\prime}}$, $\{B_{i}\}_{i=1}^{h}$, $\{B^{\prime}_{i}\}_{i=1}^{h^{\prime}}$ of matrices in $\mathbb{R}^{D\times D}$, we have

$$ $\displaystyle F\left(X;\left\{A_{i},B_{i}\right\}_{i=1}^{h}\right)=F\left(X;\left\{A^{\prime}_{i},B^{\prime}_{i}\right\}_{i=1}^{h^{\prime}}\right)$ (60) $$

for all positive integers $L$ and $X\in\mathbb{R}^{L\times D}$. Then, for all $A\in\mathbb{R}^{D\times D}$, we have

$$ $\displaystyle\sum_{i~{}:~{}A_{i}=A}B_{i}=\sum_{i~{}:~{}A^{\prime}_{i}=A}B^{\prime}_{i}.$ (61) $$

###### Proof.

From Equation ([60](#A1.E60)), we have

$$ $\displaystyle\sum_{A\in\mathbb{R}^{D\times D}}f(XAX^{\top})\cdot\left(\sum_{i~{}:~{}A_{i}=A}B_{i}-\sum_{i~{}:~{}A^{\prime}_{i}=A}B^{\prime}_{i}\right)=0,$ (62) $$

then the result is directly from Corollary [A.5](#A1.Thmtheorem5).
∎

###### Corollary A.6 .

Let $D,D_{k},D_{v}$ and $h,h^{\prime}$ be positive integers. For $1\leqslant i\leqslant h$ and $1\leqslant j\leqslant h^{\prime}$, let

$$ $\displaystyle\left(W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right)$ (63) $\displaystyle\left(\overline{W}^{(Q,j)},\overline{W}^{(K,j)},\overline{W}^{(V,j)},\overline{W}^{(O,j)}\right)$ (64) $$

be elements of $\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}$. Assume that, the two corresponding MultiHead’s are identical, i.e.

$$ $\displaystyle\operatorname{MultiHead}\left(X;\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right\}_{i=1}^{h}\right)$ $\displaystyle\qquad\qquad=\operatorname{MultiHead}\left(X;\left\{\left(\overline{W}^{(Q,j)},\overline{W}^{(K,j)},\overline{W}^{(V,j)},\overline{W}^{(O,j)}\right)\right\}_{j=1}^{h^{\prime}}\right).$ (65) $$

for all positive integers $L$ and $X\in\mathbb{R}^{L\times D}$. Then, for all $A\in\mathbb{R}^{D\times D}$, we have

$$ $\displaystyle\sum_{i~{}:~{}W^{(Q,i)}\cdot\left(W^{(K,i)}\right)^{\top}=A}W^{(V,i)}\cdot W^{(O,i)}=\sum_{j~{}:~{}\overline{W}^{(Q,j)}\cdot\left(\overline{W}^{(K,j)}\right)^{\top}=A}\overline{W}^{(V,j)}\cdot\overline{W}^{(O,j)}.$ (66) $$

We characterize the symmetries of the weights of $\operatorname{MultiHead}$ in the following theorem.

###### Theorem A.7 .

Let $h,D,D_{k},D_{v}$ be positive integers.
Let $\left(W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right)$ and $\left(\overline{W}^{(Q,i)},\overline{W}^{(K,i)},\overline{W}^{(V,i)},\overline{W}^{(O,i)}\right)$ be arbitrary elements of $\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}$ with $i=1,\ldots,h$.
Assume that

- (a)
$\max(D_{k},D_{v})\leqslant D$,
- (b)
the matrices $W^{(Q,i)}$, $W^{(K,i)}$, $\overline{W}^{(Q,i)}$, and $\overline{W}^{(K,i)}$ are of full rank,
- (c)
the matrices $W^{(Q,i)}\cdot\left(W^{(K,i)}\right)^{\top}$ with $i=1,\ldots,h$ are pairwise distinct,
- (d)
the matrices $\overline{W}^{(Q,i)}\cdot\left(\overline{W}^{(K,i)}\right)^{\top}$ with $i=1,\ldots,h$ are pairwise distinct.

Then the following are equivalent:

- 1.
For every positive integer $L$ and every $X\in\mathbb{R}^{L\times D}$, we always have
$\displaystyle\operatorname{MultiHead}\left(X;\left\{W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\right\}_{i=1}^{h}\right)$
$\displaystyle\qquad\qquad=\operatorname{MultiHead}\left(X;\left\{\overline{W}^{(Q,i)},\overline{W}^{(K,i)},\overline{W}^{(V,i)},\overline{W}^{(O,i)}\right\}_{i=1}^{h}\right).$
- 2.
There exist matrices $M^{(i)}\in\operatorname{GL}_{D_{k}}(\mathbb{R})$ and $N^{(i)}\in\operatorname{GL}_{D_{v}}(\mathbb{R})$ for each $i=1,\ldots,h$, as well as a permutation $\tau\in\mathcal{S}_{h}$, such that
$\displaystyle\left(\overline{W}^{(Q,\tau(i))},\overline{W}^{(K,\tau(i))},\overline{W}^{(V,i)},\overline{W}^{(O,\tau(i))}\right)$
$\displaystyle\qquad\qquad=\left(W^{(Q,i)}\cdot(M^{(i)})^{\top},W^{(K,i)}\cdot(M^{(i)})^{-1},W^{(V,i)}\cdot N^{(i)},(N^{(i)})^{-1}\cdot W^{(O,i)}\right).$

###### Proof.

The implication $(2.)\Rightarrow(1.)$ is clear.
Let us consider the implication $(1.)\Rightarrow(2.)$.
For each $s=1,\ldots,h$, we set

$$ $\displaystyle A^{(s)}$ $\displaystyle=W^{(Q,s)}\cdot\left(W^{(K,s)}\right)^{\top},$ $\displaystyle\overline{A}^{(s)}$ $\displaystyle=\overline{W}^{(Q,s)}\cdot\left(\overline{W}^{(K,s)}\right)^{\top},$ $\displaystyle B^{(s)}$ $\displaystyle=W^{(V,s)}\cdot W^{(O,s)},$ $\displaystyle\overline{B}^{(s)}$ $\displaystyle=\overline{W}^{(V,s)}\cdot\overline{W}^{(O,s)},$ $$

which are matrices in $\mathbb{R}^{D\times D}$.
By applying Corollary [A.6](#A1.Thmtheorem6) for $A=A^{(s)}$, we see that

$$ $\displaystyle B^{(s)}=\sum_{j~{}:~{}\overline{A}^{(j)}=A^{(s)}}\overline{B}^{(j)}.$ (67) $$

However, since the matrices $\overline{A}^{(j)}$ with $j=1,\ldots,h$ are pairwise distinct, there exist a unique index $j_{s}\in\{1,\ldots,h\}$ such that $\overline{A}^{(j_{s})}=A^{(s)}$.
The correspondence $s\mapsto j_{s}$ is a permutation of $\{1,\ldots,h\}$.
Therefore, we can write $j_{s}=\tau(s)$ for some $\tau\in\mathcal{S}_{h}$.
Hence, $\overline{A}^{(\tau(s))}=A^{(s)}$, and thus equation [67](#A1.E67) becomes

$$ $\overline{B}^{(\tau(s))}=B^{(s)}.$ $$

The theorem is then followed by the rank factorization of the matrices $A^{(s)}$ and $B^{(s)}$ [].
∎

###### Theorem A.1 .

###### Proof.

###### Lemma A.2 .

###### Proof.

###### Remark 8 .

###### Lemma A.3 .

###### Proof.

###### Lemma A.4 .

###### Proof.

###### Remark 9 .

###### Corollary A.5 .

###### Proof.

###### Corollary A.6 .

###### Theorem A.7 .

###### Proof.

## Appendix B Matrix Group Preserved by LayerNorm LayerNorm \operatorname{LayerNorm}

In our setting, layer normalization is a row-wise operator.
In particular, for a row vector $x=(x_{1},\ldots,x_{D})\in\mathbb{R}^{D}$, the standard layer normalization of $x$, denoted by $\operatorname{LayerNorm}(x)$, is determined as

$$ $\displaystyle\operatorname{LayerNorm}(x)=\sqrt{D}\cdot\frac{x-\bar{x}\cdot\mathbf{1}_{D}}{||x-\bar{x}\cdot\mathbf{1}_{D}||_{2}},$ (68) $$

where $\bar{x}=\frac{1}{D}(x_{1}+\ldots+x_{D})$ is the mean of the coordinates of $x$ and $\mathbf{1}_{D}$ is a row vector in $\mathbb{R}^{D}$ whose coordinates are all equal to $1$.
Geometrically, we can view $\operatorname{LayerNorm}$ as the composition

$$ $\displaystyle\operatorname{LayerNorm}=\tau\circ\rho,$ (69) $$

of two transformations $\tau$ and $\rho$ on $\mathbb{R}^{D}$ defined belows.

- •
The perpendicular projection map $\rho\colon\mathbb{R}^{D}\to\left<\mathbf{1}_{D}\right>^{\perp}$ which maps $x\mapsto x-\bar{x}\cdot\mathbf{1}_{D}$ for each row vector $x\in\mathbb{R}^{D}$.
Here, the hyperplane $\left<\mathbf{1}_{D}\right>^{\perp}$ is the orthogonal complement of $\left<\mathbf{1}_{D}\right>$ in $\mathbb{R}^{D}$ with respect to the standard dot product. It is noted that $\rho$ is a linear map with the kernel $\operatorname{Ker}(\rho)=\left<\mathbf{1}_{D}\right>$, which is the one-dimensional vector subspace of $\mathbb{R}^{D}$ generated by the row vector $\mathbf{1}_{D}$.
- •
The scaling map $\tau\colon\mathbb{R}^{D}\setminus\{0\}\to\mathbb{R}^{D}$ which maps $x\mapsto\sqrt{D}\frac{x}{||x||_{2}}$.
The image of $\tau$ is the sphere centered at the origin of radius $\sqrt{D}$.

It is noted that, the $\operatorname{LayerNorm}$ operator is not defined on the whole $\mathbb{R}^{D}$.
Indeed, since $\tau$ is not defined at the origin, it is necesarily that $\rho(x)\neq 0$.
This means that $x$ should not lie in the kernel of $\rho$.
Therefore, the $\operatorname{LayerNorm}$ operator actually defines a map from the set $\mathbb{R}^{D}\setminus\left<\mathbf{1}_{D}\right>$ to $\mathbb{R}^{D}$.

Under the view of the $\operatorname{LayerNorm}$ as a composition above, it is natural to ask which matrix (or linear map represented by this matrix) will commute with both the projection $\rho$ and the scaling $\tau$. The following theorem gives a complete answer to this question.

###### Theorem B.1 .

Let $D$ be a positive integer.
Let $\rho$ and $\tau$ be the projection and scaling map defined above.
The following are equivalent for an arbitrary matrix $M\in\operatorname{GL}_{D}(\mathbb{R})$:

- 1.
$M$ commutes with $\rho$ and $\tau$, i.e. $\rho(xM)=\rho(x)M$ and $\tau(xM)=\tau(x)M$ for all row vector $x\in\mathbb{R}^{D}$.
- 2.
$M$ is an orthogonal matrix such that its row sums and column sums are all equal.

###### Proof.

One can direct verify (2.) implies (1.). To show (1.) implies (2.), let $x=(x_{1},\ldots,x_{D})\in\mathbb{R}^{D}$ and $M=(m_{ij})_{1\leqslant i,j\leqslant D}\in\mathbb{R}^{D\times D}$. From $\rho(xM)=\rho(x)M$, we have

$$ $\displaystyle xM-\overline{xM}\cdot\mathbf{1}_{D}=(x-\overline{x}\cdot\mathbf{1}_{D})M,$ (70) $$

which means

$$ $\displaystyle\overline{xM}\cdot\mathbf{1}_{D}=\overline{x}\cdot\mathbf{1}_{D}\cdot M.$ (71) $$

So

$$ $\displaystyle\left(\sum_{i=1}^{D}\left(x_{i}\cdot\left(\sum_{j=1}^{D}m_{ij}\right)\right)\right)\cdot\mathbf{1}_{D}=\left(\sum_{i=1}^{D}x_{i}\right)\cdot\left(\sum_{i=1}^{D}m_{i1},~{}\sum_{i=1}^{D}m_{i2},~{}\ldots,~{}\sum_{i=1}^{D}m_{iD}\right).$ (72) $$

It implies that

$$ $\displaystyle\sum_{i=1}^{D}x_{i}\cdot\left(\sum_{j=1}^{D}m_{ij}\right)=\left(\sum_{i=1}^{D}x_{i}\right)\cdot\left(\sum_{i=1}^{D}m_{i1}\right)=\ldots=\left(\sum_{i=1}^{D}x_{i}\right)\cdot\left(\sum_{i=1}^{D}m_{iD}\right).$ (73) $$

The above equation holds for all feasible $x$ (as $\rho$ and $\tau$ are not defined on the whole $\mathbb{R}^{d}$). But since the set of feasible $x$ is dense in $\mathbb{R}^{d}$, by continuity, the equation holds for all $x\in\mathbb{R}^{d}$. It implies row sums and column sums of $M$ are all equal.

In addition, since $\tau(xM)=\tau(x)M$, we have $\|xM\|_{2}=\|x\|_{2}$. By the same argument, it holds for all $x\in\mathbb{R}^{d}$. Hence, $M$ is orthogonal.
∎

###### Remark 10 (generalized doubly stochastic matrix) .

A matrix whose row sums and column sums are all equal to one is called a generalized doubly stochastic matrix.
[] characterized all orthogonal generalized doubly stochastic matrices. In particular, let $\operatorname{O}(D)$ is the set of $D\times D$ orthogonal matrices, and $\mathcal{U}_{D}$ is the subset of $\operatorname{O}(D)$ consists of all orthogonal matrices with the first column is $(1/\sqrt{D})\cdot(\mathbf{1}_{D})^{\top}$, then every orthogonal generalized doubly stochastic matrix can be writen in the form

$$ $\displaystyle\begin{pmatrix}1&0^{\top}\\ 0&X\end{pmatrix}$ (74) $$

for some $(D-1)\times(D-1)$ orthogonal matrix $X$.

There are invertible matrices that do not permute $\phi$ and $\tau$, but still have nice behavior with $\operatorname{LayerNorm}$, as we see in the following theorem.

###### Theorem B.2 .

Let $D$ be a positive integer.
Then for arbitrary permutation matrix $P\in\mathcal{P}_{D}$ and real number $\lambda\neq 0$, we have

$$ $\displaystyle\operatorname{LayerNorm}(\lambda xP)=\operatorname{sign}(\lambda)\operatorname{LayerNorm}(x)P,$ (75) $$

for every row vector $x\in\mathbb{R}^{D}$.

###### Proof.

We have

$$ $\displaystyle\operatorname{LayerNorm}(\lambda xP)$ $\displaystyle=\tau\circ\rho(\lambda xP)$ (76) $\displaystyle=\tau(\lambda xP-\overline{\lambda xP}\cdot\mathbf{1}_{D})$ (77) $\displaystyle=\tau(\lambda xP-\lambda\overline{x}\cdot\mathbf{1}_{D}\cdot P)$ $\displaystyle=\operatorname{sign}(\lambda)\tau(x-\overline{x}\cdot\mathbf{1}_{D})P$ (78) $\displaystyle=\operatorname{sign}(\lambda)\operatorname{LayerNorm}(x)P$ (79) $$

We finish the proof.
∎

###### Theorem B.1 .

###### Proof.

###### Remark 10 (generalized doubly stochastic matrix) .

###### Theorem B.2 .

###### Proof.

## Appendix C Weight Space and Group Action on Weight Space

In this section, we recall the weight space of a transformer block and the group action on it. Then, we prove Theorem [C.5](#A3.Thmtheorem5) (which corresponds to Theorem [4.3](#S4.Thmtheorem3) in the main text).

### C.1 Weight space

Recall that a standard transformer block, which is denoted by $\operatorname{Attn}$, is defined as follows: for each $X\in\mathbb{R}^{L\times D}$, we have

$$ $\displaystyle\operatorname{Attn}(X)$ $\displaystyle=\operatorname{LayerNorm}\left(\operatorname{ReLU}(\hat{X}\cdot W^{(A)}+\mathbf{1}_{L}\cdot b^{(A)})\cdot W^{(B)}+\mathbf{1}_{L}\cdot b^{(B)}\right),$ (80) $$

where

$$ $\displaystyle\hat{X}$ $\displaystyle=\operatorname{LayerNorm}\left(\operatorname{MultiHead}(X;\{W^{(Q,i)},W^{(K,i)},W^{(V,i)},W^{(O,i)}\}_{i=1}^{h})\right).$ (81) $$

with $\mathbf{1}_{L}=[1,\ldots,1]^{\top}\in\mathbb{R}^{L\times 1}$.

The weight space $\mathcal{U}$ of the above transformer block is the vector space:

$$ $\displaystyle\mathcal{U}=\left(\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{k}}\times\mathbb{R}^{D\times D_{v}}\times\mathbb{R}^{D_{v}\times D}\right)^{h}\times\big{(}\mathbb{R}^{D\times D_{A}}\times\mathbb{R}^{D_{A}\times D}\big{)}\times\big{(}\mathbb{R}^{1\times D_{A}}\times\mathbb{R}^{1\times D}\big{)}.$ (82) $$

An element $U\in\mathcal{U}$ is of the form

$$ $\displaystyle U=\Bigl{(}\left([W]^{(Q,i)},[W]^{(K,i)},[W]^{(V,i)},[W]^{(O,i)}\right)_{i=1,\ldots,h},\left([W]^{(A)},[W]^{(B)}\right),\left([b]^{(A)},[b]^{(B)}\right)\Bigr{)}.$ (83) $$

To emphasize the weights of $\operatorname{Attn}$, we will write $\operatorname{Attn}(X;U)$ instead of $\operatorname{Attn}(X)$.

### C.2 Group action on weight space

Consider the weight space $\mathcal{U}$ defined in Equation ([82](#A3.E82)) and Equation ([83](#A3.E83)), set

$$ $\displaystyle\mathcal{G}_{\mathcal{U}}=\operatorname{S}_{h}\times\operatorname{GL}_{D_{k}}(\mathbb{R})^{h}\times\operatorname{GL}_{D_{v}}(\mathbb{R})^{h}\times\mathcal{P}_{D}\times\mathcal{P}_{D_{A}}.$ (84) $$

Each element $g\in\mathcal{G}_{\mathcal{U}}$ has the form

$$ $\displaystyle g=\Bigl{(}\tau,(M_{i})_{i=1,\ldots,h},(N_{i})_{i=1,\ldots,h},P_{\pi_{O}},P_{\pi_{A}}\Bigr{)},$ (85) $$

where
$\tau,\pi_{O},\pi_{A}$ are permutations and $M_{i},N_{i}$ are invertible matrices of appropriate sizes.

###### Definition C.1 (Group action on $\mathcal{G}_{\mathcal{U}}$ ) .

The action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$ is defined to be a map $\mathcal{G}_{\mathcal{U}}\times\mathcal{U}\to\mathcal{U}$
which maps an element $(g,U)\in\mathcal{G}_{\mathcal{U}}\times\mathcal{U}$ with $U$ and $g$ given in Equation ([83](#A3.E83)) and Equation ([85](#A3.E85))
to the element

$$ $\displaystyle gU$ $\displaystyle=\biggl{(}\left([gW]^{(Q,i)},[gW]^{(K,i)},[gW]^{(V,i)},[gW]^{(O,i)}\right)_{i=1,\ldots,h},$ (86) $\displaystyle\qquad\qquad\left([gW]^{(A)},[gW]^{(B)}\right),\left([gb]^{(A)},[gb]^{(B)}\right)\biggr{)},$ (87) $$

where

$$ $\displaystyle[gW]^{(Q,i)}$ $\displaystyle=[W]^{(Q,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{\top},$ $\displaystyle[gW]^{(K,i)}$ $\displaystyle=[W]^{(K,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{-1},$ $\displaystyle[gW]^{(V,i)}$ $\displaystyle=[W]^{(V,\tau(i))}\cdot N^{(\tau(i))},$ $\displaystyle[gW]^{(O,i)}$ $\displaystyle=\left(N^{(\tau(i))}\right)^{-1}\cdot[W]^{(O,\tau(i))}\cdot P_{\pi_{O}},$ $\displaystyle[gW]^{(A)}$ $\displaystyle=P_{\pi_{O}}^{-1}\cdot[W]^{(A)}\cdot P_{\pi_{A}},$ $\displaystyle[gW]^{(B)}$ $\displaystyle=P_{\pi_{A}}^{-1}\cdot[W]^{(B)},$ $\displaystyle[gb]^{(A)}$ $\displaystyle=[b]^{(A)}\cdot P_{\pi_{A}},$ $\displaystyle[gb]^{(B)}$ $\displaystyle=[b]^{(B)}.$ $$

In addition to the above definition, we will also need the following ones for the construction of the equivariant and invariant maps later.

###### Definition C.2 .

With notations as above, for each $i=1,\ldots,h$, we denote:

$$ $\displaystyle[WW]^{(QK,i)}:=[W]^{(Q,i)}\cdot\left([W]^{(K,i)}\right)^{\top},\quad\text{and}\quad[WW]^{(VO,i)}:=[W]^{(V,i)}\cdot[W]^{(O,i)}.$ (88) $$

###### Remark 11 .

The terms $[WW]^{(QK,i)}$ and $[WW]^{(VO,i)}$ are equivariant under the action of $g$, since

$$ $\displaystyle[gWgW]^{(QK,i)}$ $\displaystyle:=\left([gW]^{(Q,i)}\cdot[gW]^{(K,i)}\right)^{\top}$ $\displaystyle=\left([W]^{(Q,{\tau(i)})}\cdot(M^{\tau(i)})^{\top}\right)\cdot\left([W]^{(K,{\tau(i)})}\cdot(M^{\tau(i)})^{-1}\right)^{\top}$ $\displaystyle=[W]^{(Q,{\tau(i)})}\cdot\left([W]^{(K,{\tau(i)})}\right)^{\top}$ $\displaystyle=[WW]^{(QK,{\tau(i)})},$ $$

and

$$ $\displaystyle[gWgW]^{(VO,i)}$ $\displaystyle:=[gW]^{(V,i)}\cdot[gW]^{(O,i)}$ $\displaystyle=\left([W]^{(V,{\tau(i)})}\cdot M^{(\tau(i))}\right)\cdot\left((M^{(\tau(i))})^{-1}\cdot[W]^{(O,{\tau(i)})}\cdot P_{\pi_{O}}\right)$ $\displaystyle=[W]^{(V,{\tau(i)})}\cdot[W]^{(O,{\tau(i)})}\cdot P_{\pi_{O}}$ $\displaystyle=[WW]^{(VO,{\tau(i)})}\cdot P_{\pi_{O}}.$ $$

Therefore, we can intuitively say that these terms are compatible under the action of $\mathcal{G}_{\mathcal{U}}$.

###### Proposition C.3 .

With notations as above, we have

$$ $\displaystyle[gWgW]^{(QK,i)}_{j,k}$ $\displaystyle=[WW]^{(QK,{\tau(i)})}_{j,k},$ $\displaystyle[gWgW]^{(VO,i)}_{j,k}$ $\displaystyle=[WW]^{(VO,{\tau(i)})}_{j,\pi_{O}(k)},$ $\displaystyle[gW]^{(Q,i)}_{j,k}$ $\displaystyle=\left[[W]^{(Q,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{\top}\right]_{j,k},$ $\displaystyle[gW]^{(K,i)}_{j,k}$ $\displaystyle=\left[[W]^{(K,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{-1}\right]_{j,k},$ $\displaystyle[gW]^{(V,i)}_{j,k}$ $\displaystyle=\left[[W]^{(V,\tau(i))}\cdot N^{(\tau(i))}\right]_{j,k},$ $\displaystyle[gW]^{(O,i)}$ $\displaystyle=\left[\left(N^{(\tau(i))}\right)^{-1}\cdot[W]^{(O,\tau(i))}\right]_{j,\pi_{O}(k)},$ $\displaystyle[gW]^{(A)}_{j,k}$ $\displaystyle=[W]^{(A)}_{\pi_{O}(j),\pi_{A}(k)},$ $\displaystyle[gW]^{(B)}_{j,k}$ $\displaystyle=[W]^{(B)}_{\pi_{A}(j),k},$ $\displaystyle[gb]^{(A)}_{k}$ $\displaystyle=[b]^{(A)}_{\pi_{A}(k)},$ $\displaystyle[gb]^{(B)}_{k}$ $\displaystyle=[b]^{(B)}_{k}.$ $$

for all appropriate indices $j$ and $k$.

###### Proof.

This proposition follows immediately from the definition of the group action on $\mathcal{U}$.
∎

###### Lemma C.4 .

Let $n$ be a positive integer.
The following are equivalent for a matrix $M\in\operatorname{GL}_{n}(\mathbb{R})$:

- 1.
$\operatorname{ReLU}(x\cdot M)=\operatorname{ReLU}(x)\cdot M$ for all row vector $x\in\mathbb{R}^{n}$.
- 2.
$M$ is a monomial matrix whose nonzero entries are positive numbers.

###### Proof.

See [].
∎

###### Theorem C.5 (Invariance of $\operatorname{Attn}$ under the action of $\mathcal{G}_{\mathcal{U}}$ ) .

With notations as above, we have

$$ $\displaystyle\operatorname{Attn}(X;gU)=\operatorname{Attn}(X;U)$ (89) $$

for all $U\in\mathcal{U}$, $g\in\mathcal{G}$, and $X\in\mathbb{R}^{L\times D}$.

###### Proof.

According to Equation ([80](#A3.E80)) and Equation ([81](#A3.E81)), we have

$$ $\displaystyle\operatorname{Attn}(X;gU)$ (90) $\displaystyle=\operatorname{LayerNorm}\big{(}\operatorname{ReLU}(\hat{X}\cdot[gW]^{(A)}+\mathbf{1}_{L}\cdot[gb]^{(A)})\cdot[gW]^{(B)}+\mathbf{1}_{L}\cdot[gb]^{(B)}\big{)}$ (91) $\displaystyle=\operatorname{LayerNorm}\big{(}\operatorname{ReLU}(\hat{X}\cdot P_{\pi_{O}}^{-1}\cdot[W]^{(A)}P_{\pi_{A}}+\mathbf{1}_{L}\cdot[b]^{(A)}P_{\pi_{A}})\cdot P_{\pi_{A}}^{-1}\cdot[W]^{(B)}+\mathbf{1}_{L}\cdot[b]^{(B)}\big{)}$ (92) $\displaystyle=\operatorname{LayerNorm}\big{(}\operatorname{ReLU}(\hat{X}P_{\pi_{O}}^{-1}[W]^{(A)}+\mathbf{1}_{L}[b]^{(A)})P_{\pi_{A}}P_{\pi_{A}}^{-1}[W]^{(B)}+\mathbf{1}_{L}[b]^{(B)}\big{)}$ (93) $\displaystyle=\operatorname{LayerNorm}\big{(}\operatorname{ReLU}(\hat{X}P_{\pi_{O}}^{-1}[W]^{(A)}+\mathbf{1}_{L}[b]^{(A)})[W]^{(B)}+\mathbf{1}_{L}[b]^{(B)}\big{)}.$ (94) $$

In the above equalities, Equation ([92](#A3.E92)) following from the definition of the group action of $\mathcal{G}_{\mathcal{U}}$ on $\mathcal{U}$, while Equation ([93](#A3.E93)) follows from Lemma [C.4](#A3.Thmtheorem4).
In addition, we proceed the term $\hat{X}P_{\pi_{O}}^{-1}$ inside Equation ([94](#A3.E94)) as:

$$ $\displaystyle\hat{X}P_{\pi_{O}}^{-1}$ (95) $\displaystyle=\operatorname{LayerNorm}\big{(}\operatorname{MultiHead}(X;\{[gW]^{(Q,i)},[gW]^{(K,i)},[gW]^{(V,i)},[gW]^{(O,i)}\}_{i=1}^{h})\big{)}\cdot P_{\pi_{O}}^{-1}$ (96) $\displaystyle=\operatorname{LayerNorm}\left(\sum\limits_{i=1}^{h}\operatorname{Head}(X;\{[gW]^{(Q,i)},[gW]^{(K,i)},[gW]^{(V,i)}\}_{i=1}^{h})[gW]^{(O,i)}\right)\cdot P_{\pi_{O}}^{-1}$ (97) $\displaystyle=\operatorname{LayerNorm}\left(\sum\limits_{i=1}^{h}\operatorname{Head}\left(X;\{[W]^{(Q,\tau(i))},[W]^{(K,\tau(i))},[W]^{(V,\tau(i))}\}_{i=1}^{h}\right)[W]^{(O,\tau(i))}\right)$ (98) $\displaystyle=\operatorname{LayerNorm}\left(\sum\limits_{i=1}^{h}\operatorname{Head}\left(X;\{[W]^{(Q,i)},[W]^{(K,i)},[W]^{(V,i)}\}_{i=1}^{h}\right)[W]^{(O,i)}\right)$ (99) $\displaystyle=\hat{X}.$ (100) $$

In the above equalities, we used the notations

$$ $\displaystyle\overline{[W]}^{(Q,\tau(i))}$ $\displaystyle=[W]^{(Q,\tau(i))}\cdot(M^{(\tau(i))})^{\top},$ $\displaystyle\overline{[W]}^{(K,\tau(i))}$ $\displaystyle=[W]^{(K,\tau(i))}\cdot(M^{(Q,\tau(i))})^{-1},$ $\displaystyle\overline{[W]}^{(V,\tau(i))}$ $\displaystyle=[W]^{(V,\tau(i))}\cdot M^{(V,\tau(i))},$ $\displaystyle\overline{[W]}^{(O,\tau(i))}$ $\displaystyle=(M^{(V,\tau(i))})^{-1}\cdot[W]^{(O,\tau(i))}.$ $$

In the above equalities,
Equation ([98](#A3.E98)) follows from Theorem [A.1](#A1.Thmtheorem1) and Corollary [B.2](#A2.Thmtheorem2).
In addition, Equation ([99](#A3.E99)) obtained by permuting terms in the sum by $\tau$.
From Equation ([94](#A3.E94)) and Equation ([100](#A3.E100)), we see that

$$ $\operatorname{Attn}(X;gU)=\operatorname{Attn}(X;U),$ $$

for all $X,g$ and $U$.
The theorem is then proved.
∎

###### Definition C.1 (Group action on 𝒢 𝒰 subscript 𝒢 𝒰 \mathcal{G}_{\mathcal{U}} ) .

###### Definition C.2 .

###### Remark 11 .

###### Proposition C.3 .

###### Proof.

###### Lemma C.4 .

###### Proof.

###### Theorem C.5 (Invariance of Attn Attn \operatorname{Attn} under the action of 𝒢 𝒰 subscript 𝒢 𝒰 \mathcal{G}_{\mathcal{U}} ) .

###### Proof.

## Appendix D An Equivariant Polynomial Layer

We now proceed to construct a $\mathcal{G}_{\mathcal{U}}$-equivariant polynomial layer, denoted as $E$. These layers serve as the fundamental building blocks for our Transformer-NFNs. Our method is based on parameter sharing mechanism.

### D.1 A general form with unknown coefficients

We want to build a map $E\colon\mathcal{U}\rightarrow\mathcal{U}$ such that $E$ is $\mathcal{G}_{\mathcal{U}}$-equivariant.
We select $E(U)$ to be a linear combination of entries of the matrices $[W]^{(Q,s)}$, $[W]^{(K,s)}$, $[W]^{(V,s)}$, $[W]^{(O,s)}$, $[W]^{(A)}$, $[W]^{(B)}$, $[b]^{(A)}$, $[b]^{(B)}$ as well as the matrices $[WW]^{(QK,s)}$ and $[WW]^{(VO,s)}$, i.e

$$ $\displaystyle E(U)$ $\displaystyle=\biggl{(}\left([E(W)]^{(Q,i)},[E(W)]^{(K,i)},[E(W)]^{(V,i)},[E(W)]^{(O,i)}\right)_{i=1,\ldots,h},$ $\displaystyle\qquad\qquad\left([E(W)]^{(A)},[E(W)]^{(B)}\right),\left([E(b)]^{(A)},[E(b)]^{(B)}\right)\biggr{)},$ (101) $$

where

$$ $\displaystyle[E(W)]^{(Q,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,i):j,k},$ (102) $\displaystyle[E(W)]^{(K,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(K,i):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(K,i):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(K,i):j,k},$ (103) $\displaystyle[E(W)]^{(V,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(V,i):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(V,i):j,k},$ (104) $\displaystyle[E(W)]^{(O,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,i):j,k},$ (105) $\displaystyle[E(W)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k},$ (106) $\displaystyle[E(W)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):j,k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):j,k},$ (107) $\displaystyle[E(b)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(A):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):k},$ (108) $\displaystyle[E(b)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k},$ (109) $$

and the constants $\Phi^{-}_{-}$s are parameters.

In the following subsections, we will determine the constraints on the coefficients $\Phi^{-}_{-}$ such that $E(gU)=gE(U)$ for all $U\in\mathcal{U}$ and $g\in\mathcal{G}_{\mathcal{U}}$.

### D.2 Auxiliary lemmas

We will need the following auxiliary lemmas in order to determine the constraints of the coefficients of $E$ later.

###### Lemma D.1 .

Assume that $E\colon\mathcal{U}\to\mathcal{U}$ is a function defined as in Equation ([D.1](#A4.Ex75)) for some coefficients $\Phi^{-}_{-}$.
If $E(U)=0$ for all $U\in\mathcal{U}$, then all coefficients are equal to zero.

###### Proof.

We view the entries of $U$ as indeterminates over $\mathbb{R}$.
Under this view, since $E(U)=0$ for all $U\in\mathcal{U}$, we can view $E(U)$ as a zero polynomial.
Moreover, the set including all entries of the matrices $[W]^{(Q,s)}$, $[W]^{(K,s)}$, $[W]^{(V,s)}$, $[W]^{(O,s)}$, $[W]^{(A)}$, $[W]^{(B)}$, $[b]^{(A)}$, $[b]^{(B)}$ as well as the matrices $[WW]^{(QK,s)}$ and $[WW]^{(VO,s)}$ is a linear independent set over $\mathbb{R}$.
Therefore, the coefficients of $E$, which are $\Phi^{-}_{-}$, must be equal to zero.
∎

###### Lemma D.2 .

Let $h$ and $D$ be positive integers.
Let $f^{(1)}_{s},\,f^{(2)}_{s}\colon\mathbb{R}^{D\times D}\to\mathbb{R}$ be $\mathbb{R}$-linear functions for each $s=1,\ldots,h$.
Assume that there exists a constant $\lambda\in\mathbb{R}$ such that

$$ $\displaystyle\sum\limits_{s=1}^{h}f^{(1)}_{s}\left(M^{(s)}\right)+f^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)=\lambda,$ (110) $$

for all $\left(M^{(1)},\ldots,M^{(h)}\right)\in\operatorname{GL}_{D}(\mathbb{R})^{h}$.
Then

$$ $f^{(1)}_{s}\left(M\right)=f^{(2)}_{s}\left(M\right)=\lambda=0$ $$

for all $s=1,\ldots,h$ and $M\in\operatorname{GL}_{D}(\mathbb{R})$.

###### Proof.

Fix an arbitrary element $\left(M^{(1)},\ldots,M^{(h)}\right)$ of $\operatorname{GL}_{D}(\mathbb{R})^{h}$.
Then for arbitrary nonzero numbers $\lambda_{1},\ldots,\lambda_{h}$, the tuple
$\left(\lambda_{1}M^{(1)},\ldots,\lambda_{h}M^{(h)}\right)$ is also an element of $\operatorname{GL}_{D}(\mathbb{R})^{h}$.
Since $f^{(1)}_{s},f^{(2)}_{s}$ are linear, it follows from Equation ([110](#A4.E110)) that

$$ $\displaystyle\sum\limits_{s=1}^{h}\lambda_{s}f^{(1)}_{s}\left(M^{(s)}\right)+\frac{1}{\lambda_{s}}\cdot f^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)=\lambda.$ (111) $$

Consider the function

$$ $\displaystyle P(t_{1},\ldots,t_{h}):=\left(\prod\limits_{i=1}^{h}t_{i}\right)\cdot\left(\sum\limits_{s=1}^{h}t_{s}f^{(1)}_{s}\left(M^{(s)}\right)+\frac{1}{t_{s}}\cdot f^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)-\lambda\right)$ (112) $$

as a polynomial in the variables $t_{1},\ldots,t_{h}$ with coefficients in $\mathbb{R}$.
Then, according to Equation ([111](#A4.E111)), we have $P(\lambda_{1},\ldots,\lambda_{h})=0$ for every nonzero numbers $\lambda_{1},\ldots,\lambda_{h}$.
This happens only when $P$ is a zero polynomial.
Thus all coefficients of $P$ must be equal to zero.
In particular, we have $\lambda=0$ and $f^{(1)}_{s}\left(M^{(s)}\right)=f^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)=0$ for all $s=1,\ldots,h$ and all $\left(M^{(1)},\ldots,M^{(h)}\right)\in\operatorname{GL}_{D}(\mathbb{R})^{h}$.
The lemma is then proved.
∎

###### Corollary D.3 .

Let $h$ and $D$ be a positive integers.
Let $f^{(1)}_{s},\,f^{(2)}_{s},g^{(1)}_{s},\,g^{(2)}_{s}\colon\mathbb{R}^{D\times D}\to\mathbb{R}$ be linear functions for each $s=1,\ldots,h$.
Assume that there are constants $\lambda_{1},\lambda_{2}\in\mathbb{R}$ such that:

$$ $\displaystyle\sum\limits_{s=1}^{h}f^{(1)}_{s}\left(M^{(s)}\right)+f^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)+\lambda_{1}=\sum\limits_{s=1}^{h}g^{(1)}_{s}\left(M^{(s)}\right)+g^{(2)}_{s}\left(\left(M^{(s)}\right)^{-1}\right)+\lambda_{2},$ (113) $$

for all $\left(M^{(1)},\ldots,M^{(h)}\right)\in\operatorname{GL}_{D}(\mathbb{R})^{h}$.
Then $f^{(1)}_{s}(M)=g^{(1)}_{s}(M)$ and $f^{(2)}_{s}(M)=g^{(2)}_{s}(M)$ and $\lambda_{1}=\lambda_{2}$ for all $s=1,\ldots,h$ and $M\in\operatorname{GL}_{D}(\mathbb{R})$.

###### Proof.

Apply Lemma [D.2](#A4.Thmtheorem2) with $f^{(1)}_{s}$ is replace by $f^{(1)}_{s}-g^{(1)}_{s}$, $f^{(2)}_{s}$ is replace by $f^{(2)}_{s}-g^{(2)}_{s}$, and $\lambda$ is replaced by $\lambda_{2}-\lambda_{1}$.
∎

###### Lemma D.1 .

###### Proof.

###### Lemma D.2 .

###### Proof.

###### Corollary D.3 .

###### Proof.

### D.3 Finding the constraints for the unknown coefficients

In the following, we will find necessary and sufficient conditions for the coefficients $\Phi^{-}_{-}$ such that $E(gU)=gE(U)$ for all $U\in\mathcal{U}$ and $g\in\mathcal{G}_{\mathcal{U}}$.
We follow the parameter-sharing strategy.
In particular, we first determine the entries of $E(gU)$ and $gE(U)$.
Then, we compare the corresponding entries to determine the exact constraints on the coefficients.

#### D.3.1 Computing E ​ ( g ​ U ) 𝐸 𝑔 𝑈 E(gU)

Following Equation ([86](#A3.E86)), we have

$$ $\displaystyle E(gU)$ $\displaystyle=\left(([E(gW)]^{(Q,i)},[E(gW)]^{(K,i)},[E(gW)]^{(V,i)},[E(gW)]^{(O,i)})_{i=1,\ldots,h},\right.$ $\displaystyle\qquad\qquad\left.([E(gW)]^{(A)},[E(gW)]^{(B)}),([E(gb)]^{(A)},[E(gb)]^{(B)})\right),$ (114) $$

where

$$ $\displaystyle[E(gW)]^{(Q,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(Q,i):j,k},$ (115) $\displaystyle[E(gW)]^{(K,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(K,i):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(K,i):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(K,i):j,k},$ (116) $\displaystyle[E(gW)]^{(V,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(V,i):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(V,i):j,k},$ (117) $\displaystyle[E(gW)]^{(O,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(O,i):j,k},$ (118) $\displaystyle[E(gW)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(A):j,k},$ (119) $\displaystyle[E(gW)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):j,k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(B):j,k},$ (120) $\displaystyle[E(gb)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(A):k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(A):k},$ (121) $\displaystyle[E(gb)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[gWgW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[gWgW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(Q,s):p,q}[gW]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(K,s):p,q}[gW]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):k}_{(V,s):p,q}[gW]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(O,s):p,q}[gW]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[gW]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[gW]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[gb]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[gb]^{(B)}_{q}+\Phi^{(B):k}.$ (122) $$

Using Proposition [C.3](#A3.Thmtheorem3), we can proceed the right hand side of the above equation further as:

$$ $\displaystyle[E(gW)]^{(Q,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,i):j,k},$ (123) $\displaystyle[E(gW)]^{(K,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(K,i):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(K,i):j,k},$ (124) $\displaystyle[E(gW)]^{(V,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(V,i):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(V,i):j,k},$ (125) $\displaystyle[E(gW)]^{(O,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,i):j,k},$ (126) $\displaystyle[E(gW)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k},$ (127) $\displaystyle[E(gW)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ (128) $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):j,k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):j,k},$ (129) $\displaystyle[E(gb)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):k},$ (130) $\displaystyle[E(gb)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[WW]^{{(QK,\tau(s))}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[WW]^{(VO,\tau(s))}_{p,\pi_{O}(q)}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(Q,s):p,q}\left[[W]^{{(Q,\tau(s))}}\cdot\left(M^{(\tau(s))}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(K,s):p,q}\left[[W]^{(K,\tau(s))}\cdot\left(M^{(\tau(s))}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):k}_{(V,s):p,q}\left[[W]^{{(V,\tau(s))}}\cdot N^{(\tau(s))}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(O,s):p,q}\left[\left(N^{(\tau(s))}\right)^{-1}\cdot[W]^{(O,\tau(s))}\right]_{p,\pi_{O}(q)}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[W]^{(A)}_{\pi_{O}(p),\pi_{A}(q)}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[W]^{(B)}_{\pi_{A}(p),q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[b]^{(A)}_{\pi_{A}(q)}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k}.$ (131) $$

Using the symmetries of the indices, we can rearrange the right hand sides of the above equations as:

$$ $\displaystyle[E(gW)]^{(Q,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,i):j,k},$ (132) $\displaystyle[E(gW)]^{(K,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(K,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(K,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(K,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(K,i):j,k},$ (133) $\displaystyle[E(gW)]^{(V,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(V,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(V,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(V,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(V,i):j,k},$ (134) $\displaystyle[E(gW)]^{(O,i)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,i):j,k},$ (135) $\displaystyle[E(gW)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k},$ (136) $\displaystyle[E(gW)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ (137) $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):j,k},$ (138) $\displaystyle[E(gb)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):k},$ (139) $\displaystyle[E(gb)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k}.$ (140) $$

#### D.3.2 Computing g ​ E ​ ( U ) 𝑔 𝐸 𝑈 gE(U) .

According to Equation ([86](#A3.E86)) and Equation ([D.4](#A4.Ex438)), we have:

$$ $\displaystyle gE(U)$ $\displaystyle=\left(\left([gE(W)]^{(Q,i)},[gE(W)]^{(K,i)},[gE(W)]^{(V,i)},[gE(W)]^{(O,i)}\right)_{i=1,\ldots,h},\right.$ $\displaystyle\qquad\qquad\left.\left([gE(W)]^{(A)},[gE(W)]^{(B)}\right),\left([gE(b)]^{(A)},[gE(b)]^{(B)}\right)\right),$ $$

where

$$ $\displaystyle[gE(W)]^{(Q,i)}_{j,k}$ $\displaystyle=\left[[E(W)]^{(Q,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{\top}\right]_{j,k},$ $\displaystyle[gE(W)]^{(K,i)}_{j,k}$ $\displaystyle=\left[[E(W)]^{(K,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{-1}\right]_{j,k},$ $\displaystyle[gE(W)]^{(V,i)}_{j,k}$ $\displaystyle=\left[[E(W)]^{(V,\tau(i))}\cdot N^{(\tau(i))}\right]_{j,k},$ $\displaystyle[gE(W)]^{(O,i)}_{j,k}$ $\displaystyle=\left[\left(N^{(\tau(i))}\right)^{-1}\cdot[E(W)]^{(O,\tau(i))}\right]_{j,\pi_{O}(k)},$ $\displaystyle[gE(W)]^{(A)}_{j,k}$ $\displaystyle=[E(W)]^{(A)}_{\pi_{O}(j),\pi_{A}(k)},$ $\displaystyle[gE(W)]^{(B)}_{j,k}$ $\displaystyle=[E(W)]^{(B)}_{\pi_{A}(j),k},$ $\displaystyle[gE(b)]^{(A)}_{k}$ $\displaystyle=[E(b)]^{(A)}_{\pi_{A}(k)},$ $\displaystyle[gE(b)]^{(B)}_{k}$ $\displaystyle=[E(b)]^{(B)}_{k}.$ $$

We calculate these entries in detail below.

$$ $\displaystyle[gE(W)]^{(Q,i)}_{j,k}=\left[[E(W)]^{(Q,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{\top}\right]_{j,k}$ $\displaystyle=\sum\limits_{l=1}^{D_{k}}[E(W)]^{(Q,\tau(i))}_{j,l}\cdot\left(M^{(\tau(i))}\right)^{\top}_{l,k}$ $\displaystyle=\sum\limits_{l=1}^{D_{k}}M^{(\tau(i))}_{k,l}\cdot\left\{{\small\begin{aligned} &\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,\tau(i)):j,l}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,\tau(i)):j,l}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}\\ &\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,\tau(i)):j,l}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,\tau(i)):j,l}_{(K,s):p,q}[W]^{(K,s)}_{p,q}\\ &\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,\tau(i)):j,l}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,\tau(i)):j,l}_{(O,s):p,q}[W]^{(O,s)}_{p,q}\\ &\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(Q,\tau(i)):j,l}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,\tau(i)):j,l}_{(B):p,q}[W]^{(B)}_{p,q}\\ &\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,\tau(i)):j,l}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,\tau(i)):j,l}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,\tau(i)):j,l}\end{aligned}}\right\}.$ $$

$$ $\displaystyle[gE(W)]^{(K,i)}_{j,k}=\left[[E(W)]^{(K,\tau(i))}\cdot\left(M^{(\tau(i))}\right)^{-1}\right]_{j,k}$ $\displaystyle=\sum\limits_{l=1}^{D_{k}}[E(W)]^{(K,\tau(i))}_{j,l}\cdot\left(\left(M^{(\tau(i))}\right)^{-1}\right)_{l,k}$ $\displaystyle=\sum\limits_{l=1}^{D_{k}}[E(W)]^{(K,\tau(i))}_{j,l}\cdot\left(\left(M^{(\tau(i))}\right)^{-1}\right)_{l,k}$ (141) $\displaystyle=\sum\limits_{l=1}^{D_{k}}\left(\left(M^{(\tau(i))}\right)^{-1}\right)_{l,k}\cdot\left\{{\small\begin{aligned} &\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,\tau(i)):j,l}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(K,\tau(i)):j,l}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}\\ &\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,\tau(i)):j,l}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(K,\tau(i)):j,l}_{(K,s):p,q}[W]^{(K,s)}_{p,q}\\ &\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(K,\tau(i)):j,l}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(K,\tau(i)):j,l}_{(O,s):p,q}[W]^{(O,s)}_{p,q}\\ &\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(K,\tau(i)):j,l}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(K,\tau(i)):j,l}_{(B):p,q}[W]^{(B)}_{p,q}\\ &\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(K,\tau(i)):j,l}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(K,\tau(i)):j,l}_{(B):q}[b]^{(B)}_{q}+\Phi^{(K,\tau(i)):j,l}\end{aligned}}\right\}.$ $$

$$ $\displaystyle[gE(W)]^{(V,i)}_{j,k}=\left[[E(W)]^{(V,\tau(i))}\cdot N^{(\tau(i))}\right]_{j,k},$ (142) $\displaystyle=\sum\limits_{l=1}^{D_{v}}[E(W)]^{(V,\tau(i))}_{j,l}\cdot N^{(\tau(i))}_{l,k}$ (143) $\displaystyle=\sum\limits_{l=1}^{D_{v}}N^{(\tau(i))}_{l,k}\cdot\left\{{\small\begin{aligned} &\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,\tau(i)):j,l}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(V,\tau(i)):j,l}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}\\ &\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,\tau(i)):j,l}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(V,\tau(i)):j,l}_{(K,s):p,q}[W]^{(K,s)}_{p,q}\\ &\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(V,\tau(i)):j,l}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(V,\tau(i)):j,l}_{(O,s):p,q}[W]^{(O,s)}_{p,q}\\ &\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(V,\tau(i)):j,l}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(V,\tau(i)):j,l}_{(B):p,q}[W]^{(B)}_{p,q}\\ &\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(V,\tau(i)):j,l}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(V,\tau(i)):j,l}_{(B):q}[b]^{(B)}_{q}+\Phi^{(V,\tau(i)):j,l}\end{aligned}}\right\}.$ $$

$$ $\displaystyle[gE(W)]^{(O,i)}_{j,k}=\left[\left(N^{(\tau(i))}\right)^{-1}\cdot[E(W)]^{(O,\tau(i))}\right]_{j,\pi_{O}(k)},$ (144) $\displaystyle=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(\tau(i))}\right)^{-1}\right)_{j,l}\cdot[E(W)]^{(O,\tau(i))}_{l,\pi_{O}(k)}$ (145) $\displaystyle=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(\tau(i))}\right)^{-1}\right)_{j,l}\cdot\left\{{\small\begin{aligned} &\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}\\ &\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(K,s):p,q}[W]^{(K,s)}_{p,q}\\ &\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(O,s):p,q}[W]^{(O,s)}_{p,q}\\ &\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(B):p,q}[W]^{(B)}_{p,q}\\ &\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):l,\pi_{O}(k)}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,\tau(i)):l,\pi_{O}(k)}\end{aligned}}\right\}.$ $$

$$ $\displaystyle[gE(W)]^{(A)}_{j,k}$ $\displaystyle=[E(W)]^{(A)}_{\pi_{O}(j),\pi_{A}(k)},$ (146) $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q}[b]^{(B)}_{q}$ $\displaystyle\quad\quad\quad\quad+\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$ (147) $$

$$ $\displaystyle[gE(W)]^{(B)}_{j,k}$ $\displaystyle=[E(W)]^{(B)}_{\pi_{A}(j),k},$ (148) $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):\pi_{A}(j),k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):\pi_{A}(j),k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):\pi_{A}(j),k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):\pi_{A}(j),k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):\pi_{A}(j),k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):\pi_{A}(j),k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):\pi_{A}(j),k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):\pi_{A}(j),k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):\pi_{A}(j),k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):\pi_{A}(j),k}_{(B):q}[b]^{(B)}_{q}$ $\displaystyle\quad\quad\quad\quad+\Phi^{(B):\pi_{A}(j),k}$ (149) $$

$$ $\displaystyle[gE(b)]^{(A)}_{k}$ $\displaystyle=[E(b)]^{(A)}_{\pi_{A}(k)},$ (150) $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{A}(k)}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{A}(k)}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{A}(k)}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{A}(k)}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):\pi_{A}(k)}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{A}(k)}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(A):\pi_{A}(k)}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{A}(k)}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):\pi_{A}(k)}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{A}(k)}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\pi_{A}(k)}.$ (151) $$

$$ $\displaystyle[gE(b)]^{(B)}_{k}$ $\displaystyle=[E(b)]^{(B)}_{k}$ (152) $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(B):k}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(B):k}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k}.$ (153) $$

#### D.3.3 Coefficients comparison in the equation E ​ ( g ​ U ) = g ​ E ​ ( U ) 𝐸 𝑔 𝑈 𝑔 𝐸 𝑈 E(gU)=gE(U)

Since $E(gU)=gE(U)$, we must have:

$$ $\displaystyle[E(gW)]^{(Q,i)}$ $\displaystyle=[gE(W)]^{(Q,i)},$ $\displaystyle[E(gW)]^{(K,i)}$ $\displaystyle=[gE(W)]^{(K,i)},$ $\displaystyle[E(gW)]^{(V,i)}$ $\displaystyle=[gE(W)]^{(V,i)},$ $\displaystyle[E(gW)]^{(O,i)}$ $\displaystyle=[gE(W)]^{(O,i)},$ $\displaystyle[E(gW)]^{(A)}$ $\displaystyle=[gE(W)]^{(A)},$ $\displaystyle[E(gW)]^{(B)}$ $\displaystyle=[gE(W)]^{(B)},$ $\displaystyle[E(gb)]^{(A)}$ $\displaystyle=[gE(b)]^{(A)},$ $\displaystyle[E(gb)]^{(B)}$ $\displaystyle=[gE(b)]^{(B)},$ $$

and these equalities hold for all $i=1,\ldots,h$, $U\in\mathcal{U}$ and $g\in\mathcal{G}_{\mathcal{U}}$.

In the following, we will solve these equalities, one by one, to find the constraints of the parameters $\Phi^{-}_{-}$s of $E(U)$.

##### Case 1. Solve the equation

$[E(gW)]^{(Q,i)}=[gE(W)]^{(Q,i)}$.

For every $j,k$, we have

$$ $[E(gW)]^{(Q,i)}_{j,k}=[gE(W)]^{(Q,i)}_{j,k},$ $$

or equivalently,

$$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,i):j,k}$ $\displaystyle\quad\quad\quad\quad=\sum\limits_{l=1}^{D_{k}}[E(W)]^{(Q,\tau(i))}_{j,l}\cdot\left(M^{(\tau(i))}\right)^{\top}_{l,k}.$ (154) $$

Observe that the right hand side of the above equation is just a linear function on $M^{(\tau(i))}$. By applying Corollary [D.3](#A4.Thmtheorem3), we see that all terms of the left hand side are equal to zero, except those containing $M^{(\tau(i))}$ will be identically equal to the right hand side, i.e.

$$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{\begin{subarray}{c}s=1\\ s\neq\tau(i)\end{subarray}}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(Q,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(Q,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(Q,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(Q,i):j,k}$ $\displaystyle\quad\quad\quad\quad=0,$ (155) $$

and

$$ $\displaystyle\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,i):p,q}\left[[W]^{{(Q,\tau(i))}}\cdot\left(M^{(\tau(i))}\right)^{\top}\right]_{p,q}=\sum\limits_{l=1}^{D_{k}}[E(W)]^{(Q,\tau(i))}_{j,l}\cdot\left(M^{(\tau(i))}\right)^{\top}_{l,k}.$ (156) $$

We choose $g$ to be the identity of $\mathcal{G}_{\mathcal{U}}$ (i.e. $\tau$, $\pi_{O}$, $\pi_{A}$, $M^{(s)}$ and $N^{(s)}$ are all identities) in Equation ([D.3.3](#A4.Ex329)), then substituting the obtained result into Equation ([102](#A4.E102)), we obtain a more compact formula for $[E(W)]^{(Q,i)}_{j,k}$ as follows:

$$ $\displaystyle[E(W)]^{(Q,i)}_{j,k}=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,i):p,q}[W]^{{(Q,i)}}_{p,q}.$ (157) $$

Then by substituting this compact formula of $[E(W)]^{(Q,i)}_{j,k}$ to Equation ([156](#A4.E156)), we obtain the first constraints:

$$ $\displaystyle\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,k}_{(Q,i):p,q}\left[[W]^{{(Q,\tau(i))}}\cdot\left(M^{(\tau(i))}\right)^{\top}\right]_{p,q}$ $\displaystyle\qquad\qquad\qquad=\sum\limits_{l=1}^{D_{k}}\left(\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,\tau(i)):j,l}_{(Q,\tau(i)):p,q}[W]^{{(Q,\tau(i))}}_{p,q}\right)\cdot M^{(\tau(i))}_{k,l},$ $$

or equivalently,

$$ $\displaystyle\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,\tau(i)):j,k}_{(Q,\tau(i)):p,q}\left[[W]^{{(Q,i)}}\cdot\left(M^{(i)}\right)^{\top}\right]_{p,q}$ $\displaystyle\qquad\qquad\qquad=\sum\limits_{l=1}^{D_{k}}\left(\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(Q,i):j,l}_{(Q,i):p,q}[W]^{{(Q,i)}}_{p,q}\right)\cdot M^{(i)}_{k,l},$ (158) $$

We view the left hand side of Equation ([D.3.3](#A4.Ex339)) as a sum of linear functions on the $q$-th rows of $M^{(i)}$ with $q=1,\ldots,D_{k}$, while the right hand side is just a linear function on the $k$-th row of $M^{(i)}$.
The by applying Corollary [D.3](#A4.Thmtheorem3), we see that all of the coefficients $\Phi^{(Q,\tau(i)):j,k}_{(Q,\tau(i)):p,q}$ of the left side are equal to zero, except those with the indices $k=q$.
Therefore, we can reduce the formula for $[E(W)]^{(Q,i)}$ and the constraint further as follows:

$$ $\displaystyle[E(W)]^{(Q,i)}_{j,k}=\sum\limits_{p=1}^{D}\Phi^{(Q,i):j,k}_{(Q,i):p,k}[W]^{{(Q,i)}}_{p,k}.$ (159) $$

To find the constraints for the coefficients, we observe that, the left hand side of Equation ([D.3.3](#A4.Ex339)) depends on $\tau$, while the right hand side is not.
Therefore, by varying $\tau$, the left hand side does not change.
As a consequence, we must have

$$ $\displaystyle\Phi^{(Q,\tau(i)):j,k}_{(Q,\tau(i)):p,k}=\Phi^{(Q,i):j,k}_{(Q,i):p,k},$ (160) $$

for all $i,j,p,k$ and $\tau$.

Next, by substituting Equation ([159](#A4.E159)) and Equation ([160](#A4.E160)) to Equation ([156](#A4.E156)), we obtain:

$$ $\displaystyle\sum\limits_{p=1}^{D}\Phi^{(Q,i):j,k}_{(Q,i):p,k}\left(\sum\limits_{l=1}^{D_{k}}[W]^{{(Q,i)}}_{p,l}\cdot M^{(i)}_{k,l}\right)=\sum\limits_{l=1}^{D_{k}}\left(\sum\limits_{p=1}^{D}\Phi^{(Q,i):j,l}_{(Q,i):p,l}[W]^{{(Q,i)}}_{p,l}\right)\cdot M^{(i)}_{k,l}.$ (161) $$

By comparing the corresponding coefficients both sides, we finally obtain the constraints

$$ $\displaystyle\Phi^{(Q,\tau(i)):j,k}_{(Q,\tau(i)):p,k}=\Phi^{(Q,i):j,l}_{(Q,i):p,l},$ (162) $$

for all $i,j,p,k,l$ and $\tau$.

One can verify directly that $[E(W)]^{(K,i)}_{j,k}$ defined in the form

$$ $\displaystyle[E(W)]^{(Q,i)}_{j,k}=\sum\limits_{p=1}^{D}\Phi^{(Q,i):j,k}_{(Q,i):p,k}[W]^{{(Q,i)}}_{p,k},$ (163) $$

with the constraints

$$ $\displaystyle\Phi^{(Q,i):j,k}_{(Q,i):p,k}=\Phi^{(Q,\tau(i)):j,k^{\prime}}_{(Q,\tau(i)):p,k^{\prime}},$ (164) $$

for all $i,j,p,k,k^{\prime},\tau$, satisfies the constraint in Equation ([156](#A4.E156)).

##### Case 2. Solve the equation

$[E(gW)]^{(K,i)}=[gE(W)]^{(K,i)}$.

By using the same argument, we also obtain $[E(W)]^{(K,i)}_{j,k}$ in the form

$$ $\displaystyle[E(W)]^{(K,i)}_{j,k}=\sum\limits_{p=1}^{D}\Phi^{(K,i):j,k}_{(K,i):p,k}[W]^{(K,i)}_{p,k},$ (165) $$

with the constraints

$$ $\displaystyle\Phi^{(K,i):j,k}_{(K,i):p,k}=\Phi^{(K,\tau(i)):j,k^{\prime}}_{(K,\tau(i)):p,k^{\prime}},$ (166) $$

for all $i,j,p,k,k^{\prime},\tau$.

##### Case 3. Solve the equation

$[E(gW)]^{(V,i)}=[gE(W)]^{(V,i)}$.

By using the same argument, we also obtain $[E(W)]^{(V,i)}_{j,k}$ in the form

$$ $\displaystyle[E(W)]^{(V,i)}_{j,k}=\sum\limits_{p=1}^{D}\Phi^{(V,i):j,k}_{(V,i):p,k}[W]^{(V,i)}_{p,k},$ (167) $$

with the constraints

$$ $\displaystyle\Phi^{(V,i):j,k}_{(V,i):p,k}=\Phi^{(V,\tau(i)):j,k^{\prime}}_{(V,\tau(i)):p,k^{\prime}},$ (168) $$

for all $i,j,p,k,k^{\prime},\tau$.

##### Case 4. Solve the equation

$[E(gW)]^{(O,i)}=[gE(W)]^{(O,i)}$.

Since

$$ $[E(gW)]^{(O,i)}_{j,k}=[gE(W)]^{(O,i)}_{j,k},$ $$

it follows from Equation ([105](#A4.E105)) that

$$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,i):j,k}$ $\displaystyle\quad\quad\quad\quad=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(\tau(i))}\right)^{-1}\right)_{j,l}\cdot[E(W)]^{(O,\tau(i))}_{l,\pi_{O}(k)}.$ (169) $$

The right hand side of the above equation is just a linear function on $\left(N^{(\tau(i))}\right)^{-1}$.
Therefore, it follows from Corollary [D.3](#A4.Thmtheorem3) that all terms of the left hand side are equal to zero, except those containing $\left(N^{(\tau(i))}\right)^{-1}$ will be identically equal to the right hand side, i.e.

$$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(O,i):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(O,i):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{\begin{subarray}{c}s=1\\ s\neq\tau(i)\end{subarray}}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(O,i):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(O,i):j,k}$ $\displaystyle\quad\quad\quad\quad=0.$ (170) $$

By choosing $g$ to be the identity element of $\mathcal{G}_{\mathcal{U}}$ in Equation ([D.3.3](#A4.Ex349)), and substituting the obtained result to Equation ([D.4](#A4.Ex438)), we obtain a shorter formula for $[E(W)]^{(O,i)}_{j,k}$ as follow:

$$ $\displaystyle[E(W)]^{(O,i)}_{j,k}=\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,i):p,q}[W]^{(O,i)}_{p,q}.$ (171) $$

Thus, the constraint becomes

$$ $\displaystyle\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,i):p,\pi_{O}^{-1}(q)}\left[\left(N^{(\tau(i))}\right)^{-1}\cdot[W]^{(O,\tau(i))}\right]_{p,q}$ $\displaystyle\qquad\qquad\qquad=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(\tau(i))}\right)^{-1}\right)_{j,l}\cdot[E(W)]^{(O,\tau(i))}_{l,\pi_{O}(k)},$ $$

or equivalently,

$$ $\displaystyle\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):j,\pi_{O}(k)}_{(O,\tau(i)):p,\pi_{O}(q)}\left[\left(N^{(i)}\right)^{-1}\cdot[W]^{(O,i)}\right]_{p,q}$ $\displaystyle\qquad\qquad\qquad=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(i)}\right)^{-1}\right)_{j,l}\cdot[E(W)]^{(O,i)}_{l,k}.$ (172) $$

Observe that, the left hand side of Equation ([D.3.3](#A4.Ex359)) is just a linear function on the $j$-th row of $\left(N^{(i)}\right)^{-1}$, while the left hand side is a sum of linear combination of $p$-th rows of $\left(N^{(i)}\right)^{-1}$ with $p=1,\ldots,D_{v}$.
Therefore, only coefficients of $[E(W)]^{(O,i)}_{j,k}$ with $p=j$ are nonzero, i.e

$$ $\displaystyle[E(W)]^{(O,i)}_{j,k}=\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,i):j,q}[W]^{(O,i)}_{j,q}.$ (173) $$

In addition, since the right hand side of Equation ([D.3.3](#A4.Ex359)) is independent of $\tau$ and $\pi_{O}$, the left hand side must remain unchange if we vary $\tau$ and $\pi_{O}$.
As a consequence, we have

$$ $\displaystyle\Phi^{(O,\tau(i)):j,\pi_{O}(k)}_{(O,\tau(i)):j,\pi_{O}(q)}=\Phi^{(O,i):j,k}_{(O,i):j,q}$ (174) $$

for all $i,j,k,q,\tau$ and $\pi_{O}$.
Next, by substituting Equation ([173](#A4.E173)) and Equation ([174](#A4.E174)) to Equation ([D.3.3](#A4.Ex359)), we obtain

$$ $\displaystyle\sum\limits_{q=1}^{D}\Phi^{(O,\tau(i)):j,\pi_{O}(k)}_{(O,\tau(i)):j,\pi_{O}(q)}\left[\left(N^{(i)}\right)^{-1}\cdot[W]^{(O,i)}\right]_{j,q}$ $\displaystyle\qquad\qquad\qquad=\sum\limits_{l=1}^{D_{v}}\left(\left(N^{(i)}\right)^{-1}\right)_{j,l}\cdot\left(\sum\limits_{q=1}^{D}\Phi^{(O,i):l,k}_{(O,i):l,q}[W]^{(O,i)}_{l,q}\right),$ $$

or equivalently,

$$ $\displaystyle\sum\limits_{q=1}^{D}\sum\limits_{l=1}^{D_{v}}$ $\displaystyle\Phi^{(O,\tau(i)):j,\pi_{O}(k)}_{(O,\tau(i)):j,\pi_{O}(q)}\left(\left(N^{(i)}\right)^{-1}\right)_{j,l}\cdot[W]^{(O,i)}_{l,q}$ $\displaystyle=\sum\limits_{q=1}^{D}\sum\limits_{l=1}^{D_{v}}\Phi^{(O,i):l,k}_{(O,i):l,q}\left(\left(N^{(i)}\right)^{-1}\right)_{j,l}\cdot[W]^{(O,i)}_{l,q}.$ (175) $$

By comparing corresponding coefficients both sides, we have

$$ $\displaystyle\Phi^{(O,\tau(i)):j,\pi_{O}(k)}_{(O,\tau(i)):j,\pi_{O}(q)}=\Phi^{(O,i):l,k}_{(O,i):l,q}.$ (176) $$

Finally, one can verify directly that $[E(W)]^{(O,i)}_{j,k}$ defined as
$[E(W)]^{(O,i)}_{j,k}$ with $p=j$ are nonzero, i.e

$$ $\displaystyle[E(W)]^{(O,i)}_{j,k}=\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,i):j,q}[W]^{(O,i)}_{j,q},$ (177) $$

with the constraints

$$ $\displaystyle\Phi^{(O,i):j,k}_{(O,i):j,q}=\Phi^{(O,\tau(i)):j^{\prime},\pi_{O}(k)}_{(O,\tau(i)):j^{\prime},\pi_{O}(q)}$ (178) $$

satisfy the constraint in Equation ([D.3.3](#A4.Ex359)) for all $i,j,j^{\prime},k,q$ and $\tau,\pi_{O}$.

##### Case 5. Solve the equation

$[E(gW)]^{(A)}=[gE(W)]^{(A)}$.

Since $[E(gW)]^{(A)}_{j,k}=[gE(W)]^{(A)}_{j,k}$, we have

$$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):j,k}_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):j,k}_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k}$ $\displaystyle=$ $\displaystyle\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(Q,s):p,q}[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(K,s):p,q}[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(V,s):p,q}[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(O,s):p,q}[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$ (179) $$

We observe that the left hand side is a sum of linear functions on the matrices $M^{(s)}$, $\left(M^{(s)}\right)^{-1}$, $N^{(s)}$, and $\left(N^{(s)}\right)^{-1}$, while the right hand side is independent of these matrices.
Therefore, according to Corollary [D.3](#A4.Thmtheorem3), the terms containing these matrices in the left hand side must be equal to zero.
For the remaining terms, we can use Lemma [D.1](#A4.Thmtheorem1) to compare the corresponding ones with those in the right hand side and obtain the constraints:

$$ $\displaystyle\Phi^{(A):j,k}_{(QK,\tau^{-1}(s)):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,s):p,q},$ $\displaystyle\Phi^{(A):j,k}_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,s):p,q},$ $\displaystyle\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(Q,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(K,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(V,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(O,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):j,k}_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):p,q},$ $\displaystyle\Phi^{(A):j,k}_{(B):\pi_{A}^{-1}(p),q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):p,q},$ $\displaystyle\Phi^{(A):j,k}_{(A):\pi_{A}^{-1}(q)}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):q},$ $\displaystyle\Phi^{(A):j,k}_{(B):q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q},$ $\displaystyle\Phi^{(A):j,k}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$ $$

Therefore,

$$ $\displaystyle\Phi^{(A):j,k}_{(QK,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,\tau(s)):p,q},$ $\displaystyle\Phi^{(A):j,k}_{(VO,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,\tau(s)):p,\pi_{O}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(Q,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):j,k}_{(K,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):j,k}_{(V,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):j,k}_{(O,s):p,q}$ $\displaystyle=0,$ $\displaystyle\Phi^{(A):j,k}_{(A):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{O}(p),\pi_{A}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(B):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):\pi_{A}(p),q},$ $\displaystyle\Phi^{(A):j,k}_{(A):q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{A}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(B):q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q},$ $\displaystyle\Phi^{(A):j,k}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$ $$

One can verify directly that $[E(W)]^{(A)}_{j,k}$ defined by

$$ $\displaystyle[E(W)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k},$ (180) $$

with the constraints

$$ $\displaystyle\Phi^{(A):j,k}_{(QK,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,\tau(s)):p,q},$ $\displaystyle\Phi^{(A):j,k}_{(VO,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,\tau(s)):p,\pi_{O}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(A):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{O}(p),\pi_{A}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(B):p,q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):\pi_{A}(p),q},$ $\displaystyle\Phi^{(A):j,k}_{(A):q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{A}(q)},$ $\displaystyle\Phi^{(A):j,k}_{(B):q}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q},$ $\displaystyle\Phi^{(A):j,k}$ $\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$ $$

for all $j,j^{\prime},k,k^{\prime},s,s^{\prime},p,p^{\prime},q,q^{\prime},\pi_{O},\pi_{A}$ satisfies the condition $[E(gW)]^{(A)}_{j,k}=[gE(W)]^{(A)}_{j,k}$.

##### Case 6: Solve the equation

$[E(gW)]^{(B)}=[gE(W)]^{(B)}$.

By using the similar argument as Case 5, we also see that:

$$ $\displaystyle[E(W)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):j,k},$ (181) $$

with the constraints

$$ $\displaystyle\Phi^{(B):j,k}_{(QK,s):p,q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(QK,\tau(s)):p,q}$ $\displaystyle\Phi^{(B):j,k}_{(VO,s):p,q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(VO,\tau(s)):p,\pi_{O}(q)}$ $\displaystyle\Phi^{(B):j,k}_{(A):p,q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(A):\pi_{O}(p),\pi_{A}(q)}$ $\displaystyle\Phi^{(B):j,k}_{(B):p,q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(B):\pi_{A}(p),q}$ $\displaystyle\Phi^{(B):j,k}_{(A):q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(A):\pi_{A}(q)}$ $\displaystyle\Phi^{(B):j,k}_{(B):q}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(B):q}$ $\displaystyle\Phi^{(B):j,k}$ $\displaystyle=\Phi^{(B):\pi_{A}(j),k}.$ $$

for all $j,k,s,p,q,\tau,\pi_{O},\pi_{A}$,
satisfies the constraint $[E(gW)]^{(B)}_{j,k}=[gE(W)]^{(B)}_{j,k}$.

##### Case 7. Solve the equation

$[E(gb)]^{(A)}=[gE(b)]^{(A)}$.

By using the same argument as Case 5, we also see that $[E(b)]^{(A)}_{k}$ defined by

$$ $\displaystyle[E(b)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(A):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):k},$ (182) $$

with the constraints

$$ $\displaystyle\Phi^{(A):k}_{(QK,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(QK,\tau(s)):p,q}$ $\displaystyle\Phi^{(A):k}_{(VO,s):p,q}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(VO,\tau(s)):p,\pi_{O}(q)}$ $\displaystyle\Phi^{(A):k}_{(A):p,q}$ $\displaystyle=\phi^{(A):\pi_{A}(k)}_{(A):\pi_{O}(p),\pi_{A}(q)}$ $\displaystyle\Phi^{(A):k}_{(B):p,q}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(B):\pi_{A}(p),q}$ $\displaystyle\Phi^{(A):k}_{(A):q}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(A):\pi_{A}(q)}$ $\displaystyle\Phi^{(A):k}_{(B):q}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(B):q}$ $\displaystyle\Phi^{(A):k}$ $\displaystyle=\Phi^{(A):\pi_{A}(k)}.$ $$

for all $j,k,s,p,q,\tau,\pi_{O},\pi_{A}$,
satisfies the constraint $[E(gW)]^{(A)}_{j,k}=[gE(W)]^{(A)}_{j,k}$.

##### Case 8. Solve the equation

$[E(gb)]^{(B)}=[gE(b)]^{(B)}$.

By using the same argument as in Case 5, we see that $[E(b)]^{(B)}_{q}$ defined by

$$ $\displaystyle[E(b)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k},$ (183) $$

with the constraints

$$ $\displaystyle\Phi^{(B):k}_{(QK,s):p,q}$ $\displaystyle=\Phi^{(B):k}_{(QK,\tau(s)):p,q}$ $\displaystyle\Phi^{(B):k}_{(VO,s):p,q}$ $\displaystyle=\Phi^{(B):k}_{(VO,\tau(s)):p,\pi_{O}(q)}$ $\displaystyle\Phi^{(B):k}_{(A):p,q}$ $\displaystyle=\Phi^{(B):k}_{(A):\pi_{O}(p),\pi_{A}(q)}$ $\displaystyle\Phi^{(B):k}_{(B):p,q}$ $\displaystyle=\Phi^{(B):k}_{(B):\pi_{A}(p),q}$ $\displaystyle\Phi^{(B):k}_{(A):q}$ $\displaystyle=\Phi^{(B):k}_{(A):\pi_{A}(q)}$ $\displaystyle\Phi^{(B):k}_{(B):q}$ $\displaystyle=\Phi^{(B):k}_{(B):q}$ $\displaystyle\Phi^{(B):k}$ $\displaystyle=\Phi^{(B):k},$ $$

for all $j,k,s,p,q,\tau,\pi_{A},\pi_{O}$,
satisfies the constraint $[E(gW)]^{(A)}_{j,k}=[gE(W)]^{(A)}_{j,k}$.

### D.4 Final formula for the equivariant polynomial layer

We gather the results of the computations of the equivariant layer in the previous section here for further implementation.

###### Theorem D.4 .

The map $E\colon\mathcal{U}\to\mathcal{U}$ with

$$ $\displaystyle E(U)$ $\displaystyle=\biggl{(}\left([E(W)]^{(Q,i)},[E(W)]^{(K,i)},[E(W)]^{(V,i)},[E(W)]^{(O,i)}\right)_{i=1,\ldots,h},$ $\displaystyle\qquad\qquad\left([E(W)]^{(A)},[E(W)]^{(B)}\right),\left([E(b)]^{(A)},[E(b)]^{(B)}\right)\biggr{)},$ (184) $$

is $\mathcal{G}_{\mathcal{U}}$-equivariant.

Here, the components of $E(U)$ are given below.

- 1.
$[E(W)]^{(Q,i)}$ is defined as
$\displaystyle[E(W)]^{(Q,i)}_{j,k}$
$\displaystyle=\sum\limits_{p=1}^{D}\Phi^{(Q,i):j,k}_{(Q,i):p,k}[W]^{{(Q,i)}}_{p,k},$
with the constraints
$\displaystyle\Phi^{(Q,i):j,k}_{(Q,i):p,k}=\Phi^{(Q,\tau(i)):j,k^{\prime}}_{(Q,\tau(i)):p,k^{\prime}},$
- 2.
$[E(W)]^{(K,i)}$ is defined as
$\displaystyle[E(W)]^{(K,i)}_{j,k}$
$\displaystyle=\sum\limits_{p=1}^{D}\Phi^{(K,i):j,k}_{(K,i):p,k}[W]^{(K,i)}_{p,k},$
with the constraints
$\displaystyle\Phi^{(K,i):j,k}_{(K,i):p,k}=\Phi^{(K,\tau(i)):j,k^{\prime}}_{(K,\tau(i)):p,k^{\prime}},$
- 3.
$[E(W)]^{(V,i)}$ is defined by
$\displaystyle[E(W)]^{(V,i)}_{j,k}$
$\displaystyle=\sum\limits_{p=1}^{D}\Phi^{(V,i):j,k}_{(V,i):p,k}[W]^{(V,i)}_{p,k},$
with the constraints
$\displaystyle\Phi^{(V,i):j,k}_{(V,i):p,k}=\Phi^{(V,\tau(i)):j,k^{\prime}}_{(V,\tau(i)):p,k^{\prime}},$
- 4.
$[E(W)]^{(O,i)}$ is defined by
$\displaystyle[E(W)]^{(O,i)}_{j,k}$
$\displaystyle=\sum\limits_{q=1}^{D}\Phi^{(O,i):j,k}_{(O,i):j,q}[W]^{(O,i)}_{j,q},$
with the constraints
$\displaystyle\Phi^{(O,i):j,k}_{(O,i):j,q}=\Phi^{(O,\tau(i)):j^{\prime},\pi_{O}(k)}_{(O,\tau(i)):j^{\prime},\pi_{O}(q)}\par$
- 5.
$[E(W)]^{(A)}$ is defined by
$\displaystyle[E(W)]^{(A)}_{j,k}$
$\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$
$\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$
$\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$
$\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):j,k},$
with the constraints
$\displaystyle\Phi^{(A):j,k}_{(QK,s):p,q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(QK,\tau(s)):p,q},$
$\displaystyle\Phi^{(A):j,k}_{(VO,s):p,q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(VO,\tau(s)):p,\pi_{O}(q)},$
$\displaystyle\Phi^{(A):j,k}_{(A):p,q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{O}(p),\pi_{A}(q)},$
$\displaystyle\Phi^{(A):j,k}_{(B):p,q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):\pi_{A}(p),q},$
$\displaystyle\Phi^{(A):j,k}_{(A):q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(A):\pi_{A}(q)},$
$\displaystyle\Phi^{(A):j,k}_{(B):q}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}_{(B):q},$
$\displaystyle\Phi^{(A):j,k}$
$\displaystyle=\Phi^{(A):\pi_{O}(j),\pi_{A}(k)}.$
- 6.
$[E(W)]^{(B)}$ is defined by
$\displaystyle[E(W)]^{(B)}_{j,k}$
$\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$
$\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$
$\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):p,q}[W]^{(B)}_{p,q}$
$\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):j,k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):j,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):j,k},$
with the constraints
$\displaystyle\Phi^{(B):j,k}_{(QK,s):p,q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(QK,\tau(s)):p,q}$
$\displaystyle\Phi^{(B):j,k}_{(VO,s):p,q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(VO,\tau(s)):p,\pi_{O}(q)}$
$\displaystyle\Phi^{(B):j,k}_{(A):p,q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(A):\pi_{O}(p),\pi_{A}(q)}$
$\displaystyle\Phi^{(B):j,k}_{(B):p,q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(B):\pi_{A}(p),q}$
$\displaystyle\Phi^{(B):j,k}_{(A):q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(A):\pi_{A}(q)}$
$\displaystyle\Phi^{(B):j,k}_{(B):q}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}_{(B):q}$
$\displaystyle\Phi^{(B):j,k}$
$\displaystyle=\Phi^{(B):\pi_{A}(j),k}.$
- 7.
$[E(b)]^{(A)}$ is defined by
$\displaystyle[E(b)]^{(A)}_{k}$
$\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$
$\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$
$\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\phi^{(A):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):p,q}[W]^{(B)}_{p,q}$
$\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(A):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(A):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):k},$
with the constraints
$\displaystyle\Phi^{(A):k}_{(QK,s):p,q}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(QK,\tau(s)):p,q}$
$\displaystyle\Phi^{(A):k}_{(VO,s):p,q}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(VO,\tau(s)):p,\pi_{O}(q)}$
$\displaystyle\Phi^{(A):k}_{(A):p,q}$
$\displaystyle=\phi^{(A):\pi_{A}(k)}_{(A):\pi_{O}(p),\pi_{A}(q)}$
$\displaystyle\Phi^{(A):k}_{(B):p,q}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(B):\pi_{A}(p),q}$
$\displaystyle\Phi^{(A):k}_{(A):q}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(A):\pi_{A}(q)}$
$\displaystyle\Phi^{(A):k}_{(B):q}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}_{(B):q}$
$\displaystyle\Phi^{(A):k}$
$\displaystyle=\Phi^{(A):\pi_{A}(k)}.$
- 8.
$[E(b)]^{(B)}$ is defined by
$\displaystyle[E(b)]^{(B)}_{k}$
$\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(QK,s):p,q}[WW]^{{(QK,s)}}_{p,q}$
$\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(VO,s):p,q}[WW]^{(VO,s)}_{p,q}$
$\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):p,q}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):p,q}[W]^{(B)}_{p,q}$
$\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):q}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k},$
with the constraints
$\displaystyle\Phi^{(B):k}_{(QK,s):p,q}$
$\displaystyle=\Phi^{(B):k}_{(QK,\tau(s)):p,q}$
$\displaystyle\Phi^{(B):k}_{(VO,s):p,q}$
$\displaystyle=\Phi^{(B):k}_{(VO,\tau(s)):p,\pi_{O}(q)}$
$\displaystyle\Phi^{(B):k}_{(A):p,q}$
$\displaystyle=\Phi^{(B):k}_{(A):\pi_{O}(p),\pi_{A}(q)}$
$\displaystyle\Phi^{(B):k}_{(B):p,q}$
$\displaystyle=\Phi^{(B):k}_{(B):\pi_{A}(p),q}$
$\displaystyle\Phi^{(B):k}_{(A):q}$
$\displaystyle=\Phi^{(B):k}_{(A):\pi_{A}(q)}$
$\displaystyle\Phi^{(B):k}_{(B):q}$
$\displaystyle=\Phi^{(B):k}_{(B):q}$
$\displaystyle\Phi^{(B):k}$
$\displaystyle=\Phi^{(B):k},$
for all $j,j^{\prime},k,k^{\prime},s,s^{\prime},p,p^{\prime},q,q^{\prime}$ and $\tau,\pi_{O},\pi_{A}$.

###### Theorem D.4 .

## Appendix E An Invariant Polynomial Layer

In this section, we will construct a map $I\colon\mathcal{U}\to\mathbb{R}^{D^{\prime}}$ such that $I$ is equivariant with respect to $\mathcal{G}_{\mathcal{U}}$, i.e.

$$ $\displaystyle I(gU)=I(U),$ (185) $$

for all $U\in\mathcal{U}$ and $g\in\mathcal{G}_{\mathcal{U}}$.

### E.1 A general form with unknown coefficients

We select $I(U)$ to be a linear combination of entries of the matrices $[W]^{(Q,s)}$, $[W]^{(K,s)}$, $[W]^{(V,s)}$, $[W]^{(O,s)}$, $[W]^{(A)}$, $[W]^{(B)}$, $[b]^{(A)}$, $[b]^{(B)}$ as well as the matrices $[WW]^{(QK,s)}$ and $[WW]^{(VO,s)}$, i.e

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,s):p,q}\cdot[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,s):p,q}\cdot[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(Q,s):p,q}\cdot[W]^{{(Q,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(K,s):p,q}\cdot[W]^{(K,s)}_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi_{(V,s):p,q}\cdot[W]^{{(V,s)}}_{p,q}+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi_{(O,s):p,q}\cdot[W]^{(O,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):p,q}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):p,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):q}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1},$ (186) $$

where the coefficients $\Phi_{-}$s are matrices of size $D^{\prime}\times 1$.

In the following, we will determine constraints on the coefficients such that $I$ will be $\mathcal{G}_{\mathcal{U}}$-invariant under these constraints.

### E.2 Compute I ​ ( g ​ U ) 𝐼 𝑔 𝑈 I(gU)

By using the same arguments in the computation of $E(gU)$, we have

$$ $\displaystyle I(gW)$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,\tau^{-1}(s)):p,q}[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(Q,\tau^{-1}(s)):p,q}\left[[W]^{{(Q,s)}}\cdot\left(M^{(s)}\right)^{\top}\right]_{p,q}$ $\displaystyle\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{k}}\Phi_{(K,\tau^{-1}(s)):p,q}\left[[W]^{(K,s)}\cdot\left(M^{(s)}\right)^{-1}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{v}}\Phi_{(V,\tau^{-1}(s)):p,q}\left[[W]^{{(V,s)}}\cdot N^{(s)}\right]_{p,q}$ $\displaystyle\quad\quad+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D_{v}}\sum\limits_{q=1}^{D}\Phi_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}\left[\left(N^{(s)}\right)^{-1}\cdot[W]^{(O,s)}\right]_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):\pi_{A}^{-1}(p),q}[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\pi_{A}^{-1}(q)}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}[b]^{(B)}_{q}+\Phi.$ (187) $$

### E.3 Compare I ​ ( g ​ U ) 𝐼 𝑔 𝑈 I(gU) and I ​ ( U ) 𝐼 𝑈 I(U)

According to Lemma [D.1](#A4.Thmtheorem1), in order to have $I(gU)=I(U)$, the corresponding coefficients of $I(gU)$ and $I(U)$ must be equal.
As a consequence, we have

$$ $\displaystyle\Phi_{(QK,\tau^{-1}(s)):p,q}$ $\displaystyle=\Phi_{(QK,s):p,q}$ $\displaystyle\Phi_{(VO,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}$ $\displaystyle=\Phi_{(VO,s):p,q}$ $\displaystyle\Phi_{(Q,\tau^{-1}(s)):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(K,\tau^{-1}(s)):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(V,\tau^{-1}(s)):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(O,\tau^{-1}(s)):p,\pi_{O}^{-1}(q)}$ $\displaystyle=0$ $\displaystyle\Phi_{(A):\pi_{O}^{-1}(p),\pi_{A}^{-1}(q)}$ $\displaystyle=\Phi_{(A):p,q}$ $\displaystyle\Phi_{(B):\pi_{A}^{-1}(p),q}$ $\displaystyle=\Phi_{(B):p,q}$ $\displaystyle\Phi_{(A):\pi_{A}^{-1}(q)}$ $\displaystyle=\Phi_{(A):q}$ $\displaystyle\Phi_{(B):q}$ $\displaystyle=\Phi_{(B):q}$ $\displaystyle\Phi_{1}$ $\displaystyle=\Phi_{1},$ $$

or equivalently,

$$ $\displaystyle\Phi_{(QK,s):p,q}$ $\displaystyle=\Phi_{(QK,\tau(s)):p,q}$ $\displaystyle\Phi_{(VO,s):p,q}$ $\displaystyle=\Phi_{(VO,\tau(s)):p,\pi_{O}(q)}$ $\displaystyle\Phi_{(Q,s):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(K,s):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(V,s):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(O,s):p,q}$ $\displaystyle=0$ $\displaystyle\Phi_{(A):p,q}$ $\displaystyle=\Phi_{(A):\pi_{O}(p),\pi_{A}(q)}$ $\displaystyle\Phi_{(B):p,q}$ $\displaystyle=\Phi_{(B):\pi_{A}(p),q}$ $\displaystyle\Phi_{(A):q}$ $\displaystyle=\Phi_{(A):\pi_{A}(q)}$ $\displaystyle\Phi_{(B):q}$ $\displaystyle=\Phi_{(B):q}$ $\displaystyle\Phi_{1}$ $\displaystyle=\Phi_{1}.$ $$

### E.4 Final formula for the invariant polynomial layer

We summarize the above computation here for implementation.

###### Theorem E.1 .

The map $I\colon\mathcal{U}\to\mathbb{R}^{D^{\prime}}$ defined by

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,s):p,q}\cdot[WW]^{{(QK,s)}}_{p,q}$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,s):p,q}\cdot[WW]^{(VO,s)}_{p,q}$ $\displaystyle\quad\quad\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):p,q}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):p,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle\quad\quad\quad\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):q}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1},$ (188) $$

with the constraints

$$ $\displaystyle\Phi_{(QK,s):p,q}$ $\displaystyle=\Phi_{(QK,\tau(s)):p,q}$ $\displaystyle\Phi_{(VO,s):p,q}$ $\displaystyle=\Phi_{(VO,\tau(s)):p,\pi_{O}(q)}$ $\displaystyle\Phi_{(A):p,q}$ $\displaystyle=\Phi_{(A):\pi_{O}(p),\pi_{A}(q)}$ $\displaystyle\Phi_{(B):p,q}$ $\displaystyle=\Phi_{(B):\pi_{A}(p),q}$ $\displaystyle\Phi_{(A):q}$ $\displaystyle=\Phi_{(A):\pi_{A}(q)}$ $\displaystyle\Phi_{(B):q}$ $\displaystyle=\Phi_{(B):q}$ $\displaystyle\Phi_{1}$ $\displaystyle=\Phi_{1}.$ $$

is $\mathcal{G}_{\mathcal{U}}$-equivariant.

###### Theorem E.1 .

## Appendix F Additional Dataset Details

Figure: Figure 1: Accuracy histogram of MNIST task and AGNews task in the Small Transformer Zoo. The number of samples is showed in log scale for improved visibility.
Refer to caption: /html/2410.04209/assets/x1.png

To create a wide range of transformer model, we opt to vary six hyperparameters in our experiments: train fraction, optimizer (SGD, SGDm, Adam, or RMSprop), learning rate, L2 regularization coefficient, weight initialization standard deviation, and dropout probability. The train fraction determines the proportion of the original training dataset used, while the optimizer dictates the algorithm for parameter updates. Learning rate, L2 regularization, and weight initialization standard deviation control various aspects of the training process, and dropout probability helps in preventing overfitting.

We select a range of typical values for each hyperparameter independently, then generate all possible combinations to form our set of hyperparameter configurations. During our preliminary experiments, we observed that the optimal range of hyperparameters varies significantly between optimizer types. Consequently, we divided our settings into two categories: Adam-RMSprop and SGD-SGDm. Table [4](#A6.T4) provides a detailed overview of our hyperparameter configurations. Overall, there are $8000$ configurations for each category, resulting in $16000$ configurations in total. These configurations are consistently applied across both tasks to ensure comparability. All models are trained for $100$ epochs, with checkpoints and accuracy measurements recorded at epochs $50$, $75$, $100$, and at the epoch with the best accuracy. During the process, we eliminate any runs that crash.

**Table 4: Hyperparameter configurations of the Small Transformer Zoo Dataset**
| Hyperparameter | SGD-SGDm | Adam-RMSprop |
| --- | --- | --- |
| Train Fraction | [1.0, 0.9, 0.8, 0.7] | [1.0, 0.9, 0.8, 0.7] |
| Dropout | [0.2, 0.15, 0.1, 0.05, 0] | [0.2, 0.15, 0.1, 0.05, 0] |
| Learning Rate | [1e-3, 3e-3, 5e-3, 7e-3,<br>1e-2, 3e-2, 5e-2, 7e-2] | [3e-4, 5e-4, 1e-3, 3e-3,<br>5e-3, 1e-2, 3e-2, 5e-2] |
| Weight Init Standard Deviation | [0.1, 0.15, 0.2, 0.25, 0.3] | [0.1, 0.2, 0.3, 0.4, 0.5] |
| L2 Regularization | [1e-8, 1e-7, 1e-6, 1e-4, 1e-2] | [1e-8, 1e-7, 1e-6, 1e-4, 1e-2] |

##### MNIST-Transformers.

The MNIST dataset [] is a widely-used benchmark in computer vision, consisting of $28\times 28$ pixel grayscale images of handwritten digits (0 0-$9$). For this vision task, the model objective is to perform digit classification. The embedding component first applies a 2D convolution to encode image patches and then adds a fixed positional encoding to provide spatial information. The encoder, comprising two stacked transformer blocks, processes these embeddings to capture complex relationships between different parts of the image. The classifier applies global average pooling to the encoder’s output and then passes it through two fully connected layers with a ReLU activation in between, finally outputting probabilities for ten classes corresponding to the digits 0 0-$9$. Using our hyperparameter configurations, we generate a total of $62756$ model samples for the MNIST task, including $15689$ checkpoints at epochs $50$, $75$, $100$, and the best-performing epoch, each trained with a distinct combination of hyperparameters. The accuracy distribution for MNIST in Figure [1](#A6.F1) displays a strong concentration in the $80\%$ to $100\%$ range while remaining models distribute almost uniformly across lower accuracies, with a slight increase around $10\%$.

##### AGNews-Transformers.

The AG’s News Topic Classification Dataset [] is a collection of news articles from the AG’s corpus of news articles on the web, categorized into four classes: World, Sports, Business, and Sci/Tech. We take the description of the articles input and train models to predict its corresponding topic. Our transformer-based model is adapted for this task as follows: The embedding component uses a pre-trained Word2Vec model to map each token to an embedding vector. These embeddings are combined with fixed positional encodings to retain sequential information. The encoder, consisting of two stacked transformer blocks, processes these embeddings to capture contextual relationships within the text. The classifier applies global average pooling to the encoder’s output, then passes it through two fully connected layers with a ReLU activation in between, finally outputting probabilities for the four categories. Our experiments on this task yield a diverse set of $63796$ model checkpoints, derived from $15949$ unique model configurations. These checkpoints are collected at four key points during training: epochs $50$, $75$, $100$, and at the epoch of peak performance. The accuracy distribution for AGNews in Figure [1](#A6.F1) shows a notable concentration in the $50\%$ to $90\%$ range, with a peak around $80\%$ while also exhibiting a smaller cluster of models around $25\%$.

## Appendix G Additional Experimental Details

Table: Table 5: Ablation study on varying the hidden dimension and number of layers in Transformer-NFN, trained on the AGNews-Transformers dataset. Dimensions of all equivariant layers are indicated inside square brackets.

### G.1 General details

Training details The models were trained for a total of 50 epochs, using a batch size of 16. We employed the Adam optimizer with a maximum learning rate of ${10}^{-3}$. A linear warmup strategy was applied to the learning rate, spanning the initial 10 epochs for gradual warmup. We utilize Binary Cross Entropy for the loss function.

Number of parameters
Table [6](#A7.T6) summarizes the total parameter count for all models. The architectural details and hyperparameter configurations for each model are provided in  [G.2](#A7.SS2) and  [G.3](#A7.SS3). Importantly, we optimized the baseline models to their best configurations, and further increasing the number of parameters would likely result in overfitting.

**Table 6: Number of parameters for all models**
| Model | MNIST | AGNews |
| --- | --- | --- |
| Transformer-NFN | 1.812M | 1.804M |
| MLP | 0.933M | 0.896M |
| STATNN | 0.203M | 0.168M |

### G.2 Archiecture and hyperparameters of Transformer-NFN

The Transformer-NFN model is composed of three main components designed to handle the input weights of a transformer network. For the embedding and classifier components, both of which are MLP-based, we employ a standard MLP with ReLU activation to process individual component. The transformer block itself is handled by an invariant architecture, incorporating several equivariant polynomial layers of Transformer-NFN. These layers utilize ReLU activation, which specifically operates on the two MLP components within the transformer block. Following this, the output is passed through an invariant polynomial layer of Transformer-NFN. The outputs of each of these components are represented as vectors, which are concatenated and passed through a final MLP head with Sigmoid activation for prediction.

In our experimental setup, the embedding component is modeled using a single-layer MLP with 10 hidden neurons, while the classifier component is a two-layer MLP, each layer containing 10 hidden neurons. For the invariant architecture of Transformer-NFN, we apply an equivariant polynomial layer of Transformer-NFN with 10 hidden channels to process the weights, followed by an invariant polynomial layer of Transformer-NFN and an MLP that outputs a 10-dimensional vector. The resulting vectors from these components are concatenated and passed through a single classification head to generate the final prediction.

### G.3 Architecture and hyperparameters for other baselines

Here we describe the architecture of all baselines:

- •
MLP In the MLP model, the weights of all components are first flattened and processed through a separate MLP for each component. Specifically, the MLP handling the transformer block and embedding components consists of a single layer with 50 hidden neurons, while the classifier component is modeled using a two-layer MLP, each layer containing 50 neurons. The outputs from all components are concatenated and passed through a final MLP to produce the prediction.
- •
STATNN [] For the STATNN model, we adapt the original approach to work with the transformer block. In this case, we compute statistical features from the weights of the query, key, value, output, as well as the weights and biases of the two linear layers. These features are concatenated and passed through a single-layer MLP with 256 hidden neurons. The classifier component uses the original STATNN model to extract features, which are then processed through an MLP with 256 hidden neurons. For the embedding component, a single-layer MLP with 64 hidden neurons is employed. The outputs from all components are concatenated and passed through a single-layer MLP for the final prediction.
- •
XGBoost [], LightGBM [], Random Forest []: We flattened the weights of all components and directly employed popular gradient boosting libraries for regression. The hyperparameters for all three tree-based models were set to a maximum depth of 10, a minimum child weight of 50, and a maximum of 256 leaves.

Figure: Figure 2: Visualization of all models on test set of AGNews-Transformers dataset.
Refer to caption: /html/2410.04209/assets/x2.png

## Appendix H Implementation of Equivariant and Invariant Layers

We present the multi-channel implementations of the $\mathcal{G}_{\mathcal{U}}$-equivariant map $E\colon\mathcal{U}^{d}\to\mathcal{U}^{e}$ and the $\mathcal{G}_{\mathcal{U}}$-invariant map $I\colon\mathcal{U}^{d}\to\mathbb{R}^{e\times D^{\prime}}$. We summarize the Equivariant and Invariant Layers with the bullet notation $\bullet$ and adopt einops-like pseudocode to maintain consistency and standardization in transformer weight space manipulations.

To facilitate understanding of the implementation, we summarize the key dimensions involved in Table [7](#A8.T7) and define the shapes of the input terms in Table [8](#A8.T8).

**Table 7: Summary of key dimensions involved in the implementation**
| Symbol | Description |
| --- | --- |
| $d$ | Number of input channels for the equivariant and invariant layer |
| $e$ | Number of output channels for the equivariant and invariant layer |
| $D$ | Embedding dimension of the input and output sequences of the transformer block |
| $D_{k}=D_{q}$ | Embedding dimension for key and query vectors in the transformer block |
| $D_{v}$ | Embedding dimension for value vectors in the transformer block |
| $D_{A}$ | Embedding dimension for the linear projection step in the transformer block |
| $h$ | Number of attention heads in the transformer block |
| $b$ | Batch size |
| $D^{\prime}$ | Embedding dimension of the invariant layer’s output |

**Table 8: Shapes of input terms used in the implementation**
| Term | Shape |
| --- | --- |
| $[W]^{(Q,i)}_{p,q}$ | $[b,d,h,D,D_{q}]$ |
| $[W]^{(K,i)}_{p,q}$ | $[b,d,h,D,D_{k}]$ |
| $[W]^{(V,i)}_{p,q}$ | $[b,d,h,D,D_{v}]$ |
| $[W]^{(O,i)}_{p,q}$ | $[b,d,h,D_{v},D]$ |
| $[WW]^{(QK,i)}_{p,q}$ | $[b,d,h,D,D]$ |
| $[WW]^{(VO,i)}_{p,q}$ | $[b,d,h,D,D]$ |
| $[W]^{A}_{p,q}$ | $[b,d,D,D_{A}]$ |
| $[b]^{A}_{q}$ | $[b,d,D_{A}]$ |
| $[W]^{B}_{p,q}$ | $[b,d,D_{A},D]$ |
| $[b]^{B}_{q}$ | $[b,d,D]$ |

### H.1 Summary of Equivariant and Invariant Layers

We summarize our derived Equivariant and Invariant Layers from Appendix [D](#A4) and Appendix [E](#A5). We use the bullet notation $\bullet$ to simlify the notation, making it easier to implementation. Roughly speaking, the bullet stands for ”the parameter is unchanged when the bullet varies across all possible index values”.

#### H.1.1 Equivariant Layers with bullet notation

$$ $\displaystyle[E(W)]^{(Q:i)}_{j,k}$ $\displaystyle=\Phi^{(Q:\bullet):j,\bullet}_{(Q:\bullet):p,\bullet}\cdot[W]^{(Q:i)}_{p,q},$ $\displaystyle[E(W)]^{(K:i)}_{j,k}$ $\displaystyle=\Phi^{(K:\bullet):j,\bullet}_{(K:\bullet):p,\bullet}\cdot[W]^{(K:i)}_{p,q},$ $\displaystyle[E(W)]^{(V:i)}_{j,k}$ $\displaystyle=\Phi^{(V:\bullet):j,\bullet}_{(V:\bullet):p,\bullet}\cdot[W]^{(V:i)}_{p,q},$ $\displaystyle[E(W)]^{(O:i)}_{j,k}$ $\displaystyle=\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{1}\cdot\sum\limits_{k=1}^{D}[W]^{(O:i)}_{j,k}+\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{2}\cdot[W]^{(O:i)}_{j,k},$ $\displaystyle[E(W)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet,\bullet}_{(QK,\bullet):p,q}\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{1}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{2}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,j}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{j,q}$ $\displaystyle+\sum\limits_{p=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{3}[W]^{(A)}_{p,k}+\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{4}[W]^{(A)}_{j,k}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{k,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{2}[b]^{(A)}_{k}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(A):\bullet,\bullet}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\bullet,\bullet},$ $\displaystyle[E(W)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):\bullet,k}_{(QK,\bullet):p,q}\left([WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):\bullet,k}_{(VO,\bullet):p,\bullet}\left([WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D}\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{p,j}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{j,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{2}[b]^{(A)}_{j}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(B):\bullet,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):\bullet,k},$ $\displaystyle[E(b)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(QK,\bullet):p,q}\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(VO,\bullet):p,\bullet}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D}\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{p,k}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{k,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{2}[b]^{(A)}_{k}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\bullet},$ $\displaystyle[E(b)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):k}_{(QK,\bullet):p,q}\left([WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):k}_{(VO,\bullet):p,\bullet}\left([WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\bullet,\bullet}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):\bullet,q}[W]^{(B)}_{p,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\bullet}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k}.$ $$

#### H.1.2 Invariant Layers with bullet notation

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,s):p,q}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{s=1}^{h}\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,s):p,\bullet}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet,\bullet}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):\bullet,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1}$ $$

### H.2 Equivariant Layers Pseudocode

#### H.2.1 [ E ​ ( W ) ] j , k ( Q : i ) subscript superscript delimited-[] 𝐸 𝑊 : 𝑄 𝑖 𝑗 𝑘 [E(W)]^{(Q:i)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(Q:i)}_{j,k}$ $\displaystyle=\Phi^{(Q:\bullet):j,\bullet}_{(Q:\bullet):p,\bullet}\cdot[W]^{(Q:i)}_{p,q}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For}\ \Phi^{(Q:\bullet):j,\bullet}_{(Q:\bullet):p,\bullet}\cdot[W]^{(Q:i)}_{p,q},\text{ with }[W]^{(Q:i)}_{p,q}\ \text{of shape }[b,d,h,D,D_{q}]\ \text{and}\ \Phi^{(Q:\bullet):j,\bullet}_{(Q:\bullet):p,\bullet}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpk,edjp\rightarrow behjk)$ $$

#### H.2.2 [ E ​ ( W ) ] j , k ( K : i ) subscript superscript delimited-[] 𝐸 𝑊 : 𝐾 𝑖 𝑗 𝑘 [E(W)]^{(K:i)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(K:i)}_{j,k}$ $\displaystyle=\Phi^{(K:\bullet):j,\bullet}_{(K:\bullet):p,\bullet}\cdot[W]^{(K:i)}_{p,q}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For}\ \Phi^{(K:\bullet):j,\bullet}_{(K:\bullet):p,\bullet}\cdot[W]^{(K:i)}_{p,q},\text{ with }[W]^{(K:i)}_{p,q}\ \text{of shape }[b,d,h,D,D_{k}]\ \text{and}\ \Phi^{(K:\bullet):j,\bullet}_{(K:\bullet):p,\bullet}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpk,edjp\rightarrow behjk)$ $$

#### H.2.3 [ E ​ ( W ) ] j , k ( V : i ) subscript superscript delimited-[] 𝐸 𝑊 : 𝑉 𝑖 𝑗 𝑘 [E(W)]^{(V:i)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(V:i)}_{j,k}$ $\displaystyle=\Phi^{(V:\bullet):j,\bullet}_{(V:\bullet):p,\bullet}\cdot[W]^{(V:i)}_{p,q}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For}\ \Phi^{(V:\bullet):j,\bullet}_{(V:\bullet):p,\bullet}\cdot[W]^{(V:i)}_{p,q},\text{ with }[W]^{(V:i)}_{p,q}\ \text{of shape }[b,d,h,D,D_{v}]\ \text{and}\ \Phi^{(V:\bullet):j,\bullet}_{(V:\bullet):p,\bullet}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpk,edjp\rightarrow behjk)$ $$

#### H.2.4 [ E ​ ( W ) ] j , k ( O : i ) subscript superscript delimited-[] 𝐸 𝑊 : 𝑂 𝑖 𝑗 𝑘 [E(W)]^{(O:i)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(O:i)}_{j,k}$ $\displaystyle=\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{1}\cdot\sum\limits_{k=1}^{D}[W]^{(O:i)}_{j,k}+\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{2}\cdot[W]^{(O:i)}_{j,k}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For}\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{1}\cdot\sum\limits_{k=1}^{D}[W]^{(O:i)}_{j,k},\text{ with }[W]^{(O:i)}_{j,k}\ \text{of shape }[b,d,h,D_{v},D]\ \text{and}\ \left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{1}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhjk,ed\rightarrow behj)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For}\left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{2}\cdot[W]^{(O:i)}_{j,k},\text{ with }[W]^{(O:i)}_{j,k}\ \text{of shape }[b,d,h,D_{v},D]\ \text{and}\ \left(\Phi^{(O:\bullet):\bullet,\bullet}_{(O:\bullet):\bullet,\bullet}\right)_{2}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhjk,ed\rightarrow behjk)$ $$

#### H.2.5 [ E ​ ( W ) ] j , k ( A ) subscript superscript delimited-[] 𝐸 𝑊 𝐴 𝑗 𝑘 [E(W)]^{(A)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(A)}_{j,k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet,\bullet}_{(QK,\bullet):p,q}\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{1}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{2}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,j}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{j,q}$ $\displaystyle+\sum\limits_{p=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{3}[W]^{(A)}_{p,k}+\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{4}[W]^{(A)}_{j,k}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{k,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{2}[b]^{(A)}_{k}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(A):\bullet,\bullet}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\bullet,\bullet}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(A):\bullet,\bullet}_{(QK,\bullet):p,q}\cdot\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q},\text{ with }[WW]^{(QK,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(A):\bullet,\bullet}_{(QK,\bullet):p,q}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edpq\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{1}\cdot\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q},\text{ with }[WW]^{(VO,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\$ $\displaystyle\qquad\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{1}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edp\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{2}\cdot\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,j},\text{ with }[WW]^{(VO,s)}_{p,j}\ \text{of shape }[b,d,h,D,D]\ \text{and}\$ $\displaystyle\qquad\left(\Phi^{(A):\bullet,\bullet}_{(VO,\bullet):p,\bullet}\right)_{2}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpj,edp\rightarrow bej).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{1}\cdot[W]^{(A)}_{p,q},\text{ with }[W]^{(A)}_{p,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{1}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,ed\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{2}\cdot[W]^{(A)}_{j,q},\text{ with }[W]^{(A)}_{j,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{2}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,ed\rightarrow bej).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{3}\cdot[W]^{(A)}_{p,k},\text{ with }[W]^{(A)}_{p,k}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{3}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,ed\rightarrow bek).unsqueeze(-2)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{4}\cdot[W]^{(A)}_{j,k},\text{ with }[W]^{(A)}_{j,k}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet,\bullet}\right)_{4}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjk,ed\rightarrow bejk)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{1}\cdot[W]^{(B)}_{p,q},\text{ with }[W]^{(B)}_{p,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{1}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edq\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{2}\cdot[W]^{(B)}_{k,q},\text{ with }[W]^{(B)}_{k,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(B):\bullet,q}\right)_{2}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdkq,edq\rightarrow bek).unsqueeze(-2)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{1}\cdot[b]^{(A)}_{q},\text{ with }[b]^{(A)}_{q}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{1}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,ed\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{2}\cdot[b]^{(A)}_{k},\text{ with }[b]^{(A)}_{k}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet,\bullet}_{(A):\bullet}\right)_{2}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdk,ed\rightarrow bek).unsqueeze(-2)$ $\displaystyle\text{For }\Phi^{(A):\bullet,\bullet}_{(B):q}\cdot[b]^{(B)}_{q},\text{ with }[b]^{(B)}_{q}\ \text{of shape }[b,d,D]\ \text{and}\ \Phi^{(A):\bullet,\bullet}_{(B):q}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edq\rightarrow be).unsqueeze(-1).unsqueeze(-1)$ $\displaystyle\text{For }\Phi^{(A):\bullet,\bullet}\text{ with shape }[e]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(e\rightarrow e).unsqueeze(0).unsqueeze(-1).unsqueeze(-1)$ $$

#### H.2.6 [ E ​ ( b ) ] k ( A ) subscript superscript delimited-[] 𝐸 𝑏 𝐴 𝑘 [E(b)]^{(A)}_{k} Pseudocode

From the formula:

$$ $\displaystyle[E(b)]^{(A)}_{k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(QK,\bullet):p,q}\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(VO,\bullet):p,\bullet}\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D}\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{p,k}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{k,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{2}[b]^{(A)}_{k}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(A):\bullet}_{(B):q}[b]^{(B)}_{q}+\Phi^{(A):\bullet}.$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(A):\bullet}_{(QK,\bullet):p,q}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right),\text{ with }[WW]^{(QK,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(A):\bullet}_{(QK,\bullet):p,q}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edpq\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\Phi^{(A):\bullet}_{(VO,\bullet):p,\bullet}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right),\text{ with }[WW]^{(VO,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(A):\bullet}_{(VO,\bullet):p,\bullet}\$ $\displaystyle\qquad\text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edp\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{1}\cdot[W]^{(A)}_{p,q},\text{ with }[W]^{(A)}_{p,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{1}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,ed\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{2}\cdot[W]^{(A)}_{p,k},\text{ with }[W]^{(A)}_{p,k}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\phi^{(A):\bullet}_{(A):\bullet,\bullet}\right)_{2}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpk,ed\rightarrow bek)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{1}\cdot[W]^{(B)}_{p,q},\text{ with }[W]^{(B)}_{p,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{1}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edq\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{2}\cdot[W]^{(B)}_{k,q},\text{ with }[W]^{(B)}_{k,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(A):\bullet}_{(B):\bullet,q}\right)_{2}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdkq,edq\rightarrow bek)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{1}\cdot[b]^{(A)}_{q},\text{ with }[b]^{(A)}_{q}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{1}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,ed\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\left(\Phi^{(A):\bullet}_{(A):\bullet}\right)_{2}\cdot[b]^{(A)}_{k},\text{ with }[b]^{(A)}_{k}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(A):k}_{(A):\bullet}\right)_{2}\ \text{of shape }[e,d]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdk,ed\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(A):\bullet}_{(B):q}\cdot[b]^{(B)}_{q},\text{ with }[b]^{(B)}_{q}\ \text{of shape }[b,d,D]\ \text{and}\ \Phi^{(A):\bullet}_{(B):q}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edq\rightarrow be)\ \texttt{.unsqueeze}(-1)$ $\displaystyle\text{For }\Phi^{(A):\bullet}\text{of shape}\ [e],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(e\rightarrow e)\ \texttt{.unsqueeze}(0)\texttt{.unsqueeze}(-1)$ $$

#### H.2.7 [ E ​ ( W ) ] j , k ( B ) subscript superscript delimited-[] 𝐸 𝑊 𝐵 𝑗 𝑘 [E(W)]^{(B)}_{j,k} Pseudocode

From the formula:

$$ $\displaystyle[E(W)]^{(B)}_{j,k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):\bullet,k}_{(QK,\bullet):p,q}\left([WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):\bullet,k}_{(VO,\bullet):p,\bullet}\left([WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{1}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D}\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{2}[W]^{(A)}_{p,j}$ $\displaystyle+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{1}[W]^{(B)}_{p,q}+\sum\limits_{q=1}^{D}\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{2}[W]^{(B)}_{j,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{1}[b]^{(A)}_{q}+\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{2}[b]^{(A)}_{j}$ $\displaystyle+\sum\limits_{q=1}^{D}\Phi^{(B):\bullet,k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):\bullet,k}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(B):\bullet,k}_{(QK,\bullet):p,q}\cdot[WW]^{{(QK,s)}}_{p,q},\text{ with }[WW]^{(QK,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(B):\bullet,k}_{(QK,\bullet):p,q}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edkpq\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\Phi^{(B):\bullet,k}_{(VO,\bullet):p,\bullet}\cdot[WW]^{(VO,s)}_{p,q},\text{ with }[WW]^{(VO,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(B):\bullet,k}_{(VO,\bullet):p,\bullet}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edkp\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{1}\cdot[W]^{(A)}_{p,q},\text{ with }[W]^{(A)}_{p,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{1}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edk\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{2}\cdot[W]^{(A)}_{p,j},\text{ with }[W]^{(A)}_{p,j}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(A):\bullet,\bullet}\right)_{2}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpj,edk\rightarrow bejk)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{1}\cdot[W]^{(B)}_{p,q},\text{ with }[W]^{(B)}_{p,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{1}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edkq\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{2}\cdot[W]^{(B)}_{j,q},\text{ with }[W]^{(B)}_{j,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(B):\bullet,q}\right)_{2}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdjq,edkq\rightarrow bejk)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{1}\cdot[b]^{(A)}_{q},\text{ with }[b]^{(A)}_{q}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{1}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edk\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{2}\cdot[b]^{(A)}_{j},\text{ with }[b]^{(A)}_{j}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \left(\Phi^{(B):\bullet,k}_{(A):\bullet}\right)_{2}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdj,edk\rightarrow bejk)$ $\displaystyle\text{For }\Phi^{(B):\bullet,k}_{(B):q}\cdot[b]^{(B)}_{q},\text{ with }[b]^{(B)}_{q}\ \text{of shape }[b,d,D]\ \text{and}\ \Phi^{(B):\bullet,k}_{(B):q}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edkq\rightarrow bek)\ \texttt{.unsqueeze}(-2)$ $\displaystyle\text{For }\Phi^{(B):\bullet,k}\text{of shape}\ [e,D],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(ek\rightarrow ek)\ \texttt{.unsqueeze}(0)$ $$

#### H.2.8 [ E ​ ( b ) ] k ( B ) subscript superscript delimited-[] 𝐸 𝑏 𝐵 𝑘 [E(b)]^{(B)}_{k} Pseudocode

From the formula:

$$ $\displaystyle[E(b)]^{(B)}_{k}$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):k}_{(QK,\bullet):p,q}\left([WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\sum\limits_{s=1}^{h}\Phi^{(B):k}_{(VO,\bullet):p,\bullet}\left([WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\bullet,\bullet}[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):\bullet,q}[W]^{(B)}_{p,q}$ $\displaystyle+\sum\limits_{q=1}^{D_{A}}\Phi^{(B):k}_{(A):\bullet}[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi^{(B):k}_{(B):q}[b]^{(B)}_{q}+\Phi^{(B):k}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi^{(B):k}_{(QK,\bullet):p,q}\cdot[WW]^{{(QK,s)}}_{p,q},\text{ with }[WW]^{{(QK,s)}}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(B):k}_{(QK,\bullet):p,q}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edkpq\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}_{(VO,\bullet):p,\bullet}\cdot[WW]^{{(VO,s)}}_{p,q},\text{ with }[WW]^{{(VO,s)}}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi^{(B):k}_{(VO,\bullet):p,\bullet}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edkp\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}_{(A):\bullet,\bullet}\cdot[W]^{(A)}_{p,q},\text{ with }[W]^{(A)}_{p,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \Phi^{(B):k}_{(A):\bullet,\bullet}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edk\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}_{(B):\bullet,q}\cdot[W]^{(B)}_{p,q},\text{ with }[W]^{(B)}_{p,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \Phi^{(B):k}_{(B):\bullet,q}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edkq\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}_{(A):\bullet}\cdot[b]^{(A)}_{q},\text{ with }[b]^{(A)}_{q}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \Phi^{(B):k}_{(A):\bullet}\ \text{of shape }[e,d,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edk\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}_{(B):q}\cdot[b]^{(B)}_{q},\text{with }[b]^{(B)}_{q}\ \text{of shape }[b,d,D]\ \text{and}\ \Phi^{(B):k}_{(B):q}\ \text{of shape }[e,d,D,D]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edkq\rightarrow bek)$ $\displaystyle\text{For }\Phi^{(B):k}\text{of shape }[e,D],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(ek\rightarrow ek)\ \texttt{.unsqueeze}(0)$ $$

### H.3 Invariant Layers Pseudocode

From the formula:

$$ $\displaystyle I(U)$ $\displaystyle=\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(QK,\bullet):p,q}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{{(QK,s)}}_{p,q}\right)$ $\displaystyle+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D}\Phi_{(VO,\bullet):p,\bullet}\cdot\left(\sum\limits_{s=1}^{h}[WW]^{(VO,s)}_{p,q}\right)$ $\displaystyle\quad+\sum\limits_{p=1}^{D}\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet,\bullet}\cdot[W]^{(A)}_{p,q}+\sum\limits_{p=1}^{D_{A}}\sum\limits_{q=1}^{D}\Phi_{(B):\bullet,q}\cdot[W]^{(B)}_{p,q}$ $\displaystyle\quad+\sum\limits_{q=1}^{D_{A}}\Phi_{(A):\bullet}\cdot[b]^{(A)}_{q}+\sum\limits_{q=1}^{D}\Phi_{(B):q}\cdot[b]^{(B)}_{q}+\Phi_{1}$ $$

We define the pseudocode for each term:

$$ $\displaystyle\text{For }\Phi_{(QK,\bullet):p,q}\cdot[WW]^{(QK,s)}_{p,q},\text{ with }[WW]^{(QK,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi_{(QK,\bullet):p,q}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edpqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(VO,\bullet):p,\bullet}\cdot[WW]^{(VO,s)}_{p,q},\text{ with }[WW]^{(VO,s)}_{p,q}\ \text{of shape }[b,d,h,D,D]\ \text{and}\ \Phi_{(VO,\bullet):p,\bullet}\$ $\displaystyle\qquad\text{of shape }[e,d,D,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdhpq,edpk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(A):\bullet,\bullet}\cdot[W]^{(A)}_{p,q},\text{ with }[W]^{(A)}_{p,q}\ \text{of shape }[b,d,D,D_{A}]\ \text{and}\ \Phi_{(A):\bullet,\bullet}\ \text{of shape }[e,d,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(B):\bullet,q}\cdot[W]^{(B)}_{p,q},\text{ with }[W]^{(B)}_{p,q}\ \text{of shape }[b,d,D_{A},D]\ \text{and}\ \Phi_{(B):\bullet,q}\ \text{of shape }[e,d,D,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdpq,edqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(A):\bullet}\cdot[b]^{(A)}_{q},\text{ with }[b]^{(A)}_{q}\ \text{of shape }[b,d,D_{A}]\ \text{and}\ \Phi_{(A):\bullet}\ \text{of shape }[e,d,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{(B):q}\cdot[b]^{(B)}_{q},\text{with }[b]^{(B)}_{q}\ \text{of shape }[b,d,D]\ \text{and}\ \Phi_{(B):q}\ \text{of shape }[e,d,D,D^{\prime}]$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(bdq,edqk\rightarrow bek)$ $\displaystyle\text{For }\Phi_{1}\text{of shape }[e,D^{\prime}],$ $\displaystyle\quad\text{Corresponding pseudocode:}\ \texttt{einsum}(ek\rightarrow ek)\ \texttt{.unsqueeze}(0)$ $$

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I'm curious how sensitive is the maximal symmetric group of the weights in a MHA module, would affect the peformance of final peformance of NFN.
soundness: 3
strengths: - The is the first paper to my knowledge expore neural functional networks (NFN) in the context of transformer, where NFN treat the weights, gradients, or sparsity patterns of a deep neural network (DNN) as input data.
- The proposed Small Transformer Zoo dataset is a valuable resource for benchmarking and studying transformer-based networks, promoting reproducibility and further research.
summary: The paper introduces a novel concept of Transformer Neural Functional Networks (Transformer-NFN), designed as equivariant neural functional networks specifically for transformer architectures.The proposed Transformer-NFN addresses the challenge of applying neural functional networks (NFNs) to transformers by systematically defining weight spaces, symmetry groups, and group actions for multi-head attention and ReLU-MLP components. Experimental results demonstrate the effectiveness of Transformer-NFN in predicting transformer generalization and performance, surpassing traditional models across multiple metrics.
weaknesses: - The paper did not discuss the computational overhead associated with equivariant polynomial layers and whether this could affect scalability for larger transformer models.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: How does this generalize to models with more parameters and trained on larger datasets
soundness: 3
strengths: The paper provides a very structured approach to a the solution of defining NFNs for transformers and is rigorous in it's approach.Some of the mathematical formulation around weight spaces of a multihead attention and the solution for parameter sharing of the NFN is good.
summary: The paper presents a formulation of NFNs for transformers beyond MLPs and CNNs. The paper provides thorough analysis of how to establish a weight space around multihead attention and a way to formalize NFNs through a parameter sharing strategy. The paper also shows the application on multiple transformer models across a basic vision and language task. The paper is very detailed in it's approach and provides for a good approach to understanding how the weights of transformer blocks are affected by hyperparameters and data
weaknesses: The paper is a bit dense to read and understand. Though this can be considered fine given the topic and detailed approach. There could've been a better balance at analyzing the different theorems presented vs covering the mathematical proof.

The experimentation is basic and doesn't necessarily capture the strengths of the method or even transformers. Transformers are known to generalize over large model sizes and large data and it is the biggest weakness of the paper on how this method would hold for larger models. Even if a small scale experiment was included, it would bolster the through mathematical approach much better.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could the authors clarify the expected computational cost of Transformer-NFN compared to traditional NFNs, considering the added design complexity?
2. Will the paper include code to facilitate reproducibility?
soundness: 3
strengths: 1. The paper presents a novel theoretical framework for designing Neural Functional Networks that respects the inherent symmetries of Transformer architectures, representing an innovative approach previously unexplored in this domain.
2. The release of a dataset with 125,000 Transformer model checkpoints offers a valuable resource for future research on Transformer performance prediction and NFNs.
3. The paper provides a detailed methodological development, including thorough derivations and design principles, establishing the theoretical foundation for NFNs tailored to Transformer architectures, which may inspire future studies in this area.
summary: This paper investigates Equivariant Neural Functional Networks for Transformer architectures by developing a specialized model, Transformer-NFN, designed to process the weights, gradients, or sparsity patterns of Transformers as input data. The paper defines a maximal symmetric group for multi-head attention modules and establishes a group action on the weight space of Transformer architectures, creating a systematic framework for NFNs with equivariance properties. Besides, Transformer-NFN, an NFN specifically tailored to Transformer structures, is evaluated on a new dataset containing 125,000 Transformer checkpoints. Experimental results indicate that Transformer-NFN outperforms other models in predicting Transformer model performance.
weaknesses: 1. While the use of equivariant layers in Transformer-NFN is theoretically compelling, empirical results suggest limited practical impact. Specifically, the marginal gains in predictive accuracy on the benchmark dataset raise questions about the method’s utility in real-world applications.
2. The architecture's intricate structure, requiring a specialized understanding of group actions and equivariant layers, could hinder adoption. The dense theoretical foundation, coupled with limited practical resources, may present obstacles to reproducibility and broader implementation in practical settings.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: - How about the performance on language modeling? Is Transformer-NFN able to predict?
- How about the performance of the larger transformer 
- From my perspective, the performance of a checkpoint is related to the training data, the architecture of the network, the training algorithm, hyperparameter configuration, etc. So why does NFN only take the weights and the configuration into account? Is it a sufficient condition for performance prediction?
- The evaluated dataset is far away from the present scale of transformer. The weight of modern transformers is huge (bert-base 110M parameters), is Transformer-NFN able to predict the performance of it?
soundness: 3
strengths: - This topic is quite interesting and do have some foreground. In the foreseeable future, a plenty of pre-trained transformer checkpoints would be released and there is no golden rule to find out which checkpoint is appropriate for a specific task. And the NFN may be useful and draw some inspiration for future research.
- For all I know, this may be one of the first works to develop NFN for transformers. 
- The theoretical analysis is detailed enough.
summary: This submission provides a theoretical study of neural functional networks (NFNs) for transformers. By formally defining the weight space of the standard transformer block and the corresponding group action of such a space, the authors propose several design principles for NFNs and introduce Transformer-NFN to predict the test performance with a transformer checkpoint and the corresponding training configurations. Additionally, a dataset with 125k transformer checkpoints trained on MINST and AGNews will be released for reproduction and further analysis.
weaknesses: - According to the main presentation of this submission, no obvious weaknesses are found.

Meta-review: The paper proposes an algorithm for equivariant transformer architectures of neural fields. Reviewers are unanimously in favor of accepting the paper, and initially had questions about the computational cost, as well as the significance of improvements. Authors answered with additional experiments, showing that the computational cost is reasonable, and improvements are also non-trivial. All in all, this is a good paper.

Decision: Accept (Poster)