Title: Monomial Matrix Group Equivariant Neural Functional Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: neural functional networks

equivariant networks

monomial matrices

symmetry
Abstract: Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of $\operatorname{ReLU}$ networks, and the weight sign flipping symmetries of $\operatorname{sin}$ or $\operatorname{Tanh}$ networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidences to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency. The code is publicly available at https://github.com/MathematicalAI-NUS/Monomial-NFN.

Monomial Matrix Group Equivariant Neural Functional Networks

Abstract

Neural functional networks (NFNs) have recently gained significant attention due to their diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representation. Previous NFN designs often depend on permutation symmetries in neural networks' weights, which traditionally arise from the unordered arrangement of neurons in hidden layers. However, these designs do not take into account the weight scaling symmetries of ReLU networks, and the weight sign flipping symmetries of sin or Tanh networks. In this paper, we extend the study of the group action on the network weights from the group of permutation matrices to the group of monomial matrices by incorporating scaling/sign-flipping symmetries. Particularly, we encode these scaling/sign-flipping symmetries by designing our corresponding equivariant and invariant layers. We name our new family of NFNs the Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFN). Because of the expansion of the symmetries, Monomial-NFN has much fewer independent trainable parameters compared to the baseline NFNs in the literature, thus enhancing the model's efficiency. Moreover, for fully connected and convolutional neural networks, we theoretically prove that all groups that leave these networks invariant while acting on their weight spaces are some subgroups of the monomial matrix group. We provide empirical evidences to demonstrate the advantages of our model over existing baselines, achieving competitive performance and efficiency. The code is publicly available at https://github.com/MathematicalAI-NUS/MonomialNFN.

1 Introduction

Deep neural networks (DNNs) have become highly versatile modeling tools, finding applications across a broad spectrum of fields such as Natural Language Processing [15, 29, 51, 66], Computer

Vision [27, 37, 61], and the Natural Sciences [31, 49]. There has been growing interest in developing specialized neural networks to process the weights, gradients, or sparsity masks of DNNs as data. These specialized neural networks are called neural functional networks (NFNs) [71]. NFNs have found diverse applications, ranging from predicting network generalization and network editing to classifying implicit neural representations. For instance, NFNs have been employed to create learnable optimizers for neural network training [5, 11, 42, 52], extract information from implicit neural representations of data [43, 52, 60], perform corrective editing of network weights [13, 44, 56], evaluate policies [26], and conduct Bayesian inference using networks as evidence [59].

Developing NFNs is inherently challenging due to their high-dimensional nature. Some early methods to address this challenge assume a restricted training process that effectively reduced the weight space [9, 19, 41]. More recent efforts have focused on building permutation equivariant NFNs that can process neural network weights without such restrictions [35, 45, 71, 72]. These works construct NFNs that are equivariant to permutations of weights, corresponded to the rearrangement of neurons in hidden layers. Such permutations, known as neuron permutation symmetries, preserve the network's behavior. However, these approaches often overlook other significant symmetries in weight spaces [12, 24]. Notable examples are weight scaling transformations for ReLU networks [7, 12, 46] and sign flipping transformations for sin and tanh networks [14, 22, 38]. Consequently, two weight spaces of a ReLU networks, that differ by a scaling transformation, two weight spaces of a sin , or tanh networks that differ by a sign flipping transformation, can produce different results when processed by existing permutation equivariant NFNs, despite representing the same functions. This highlights a fundamental limitation of the current permutation equivariant NFNs.

Contribution. In this paper, we extend the study of symmetries in weight spaces of Fully Connected Neural Networks (FCNNs) and Convolution Neural Networks (CNNs) by formally establishing a group of symmetries that includes both neuron permutations and scaling/sign-flipping transformations. These symmetries are represented by monomial matrices, which share the nonzero pattern of permutation matrices but allow nonzero entries to be any value rather than just 1. We then introduce a novel family of NFNs that are equivariant to groups of monomial matrices, thus incorporating both permutation and scaling/sign-flipping symmetries into the NFN design. We name this new family Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFNs). Due to the expanded set of symmetries, Monomial-NFN requires significantly fewer independent trainable parameters compared to baseline NFNs, enhancing the model's efficiency. By incorporating equivariance to neuron permutations and weight scaling/sign-flipping, our NFNs demonstrate competitive generalization performance compared to existing models. Our contribution is three-fold:

We formally describe a group of monomial matrices satisfying the condition that the transformation of weight spaces of FCNNs and CNNs using these group elements does not change the function defined by the networks. For ReLU networks, this group covers permutation and scaling symmetries of the weight spaces, while for sin or tanh networks, this group covers permutation and sign-flipping symmetries. The group is proved to be maximal in certain cases.

We design Monomial-NFNs, the first family of NFNs that incorporate scaling and sign-flipping symmetries of weight spaces as far as we are aware. The main building blocks of MonomialNFNs are the equivariant and invariant linear layers for processing weight spaces.

We show that the number of parameters in our equivariant linear layer is much lower than in recent permutation equivariant NFNs. In particular, our method is linear in the number of layers and dimensions of weights and biases, compared to quadratic as in [71]. This demonstrates that Monomial-NFNs have the ability to process weight spaces of large-scale networks.

Weevaluate Monomial-NFNs on three tasks: predicting CNN generalization from weights using Small CNN Zoo [64], weight space style editing, and classifying INRs using INRs data [71]. Experimental results show that our model achieves competitive performance and efficiency compared to existing baselines.

Organization. We structure this paper as follows: After summarizing some related work in Section 2, we recall the notions of monomial matrix group and describe their maximal subgroups preserved by some nonlinear activations in Section 3. In Section 4, we formalize the general weight space of FCNNs and CNNs, then discuss the symmetries of these weight spaces using the monomial matrices. In Section 5, we construct monomial matrix group equivariant and invariant layers, which are building blocks for our Monomial-NFNs. In Section 6, we present our experimental results to justify the advantages of Monomial-NFNs over the existing permutation equivariant NFN baselines. The paper ends with concluding remarks. More experimental details are provided in the Appendix.

2 Related Work

Symmetries of Weight Spaces. The challenge of identifying the symmetries in the weight spaces of neural networks, or equivalently, determining the functional equivalence of neural networks, is a well-explored area in academic research [3, 10, 16, 23, 47]. This problem was initially posed by Hecht-Nielsen in [28]. Results for various types of networks have been established as in [1, 2, 12, 14, 22, 38, 63].

Neural Functional Networks. Recent research has focused on learning representations for trained classifiers to predict their generalization performance and other insights into neural networks [8, 20, 64, 54, 53, 55]. In particular, low-dimensional encodings for Implicit Neural Representations (INRs) have been developed for downstream tasks [18, 41]. Other studies have encoded and decoded neural network parameters mainly for reconstruction and generation purposes [6, 21, 34, 48].

Equivariant Neural Functional Networks. Permutations and scaling, for ReLU networks, as well as sign-flipping, for sine or tanh networks, symmetries, are fundamental symmetries of weight networks. Permutation-equivariant NFNs are successfully built in [4, 35, 40, 45, 70, 71, 72]. In particular, the authors in [35, 40] carefully construct computational graphs representing the input neural networks' parameters and process the graphs using graph neural networks. In [4], neural network parameters are efficiently encoded by carefully choosing appropriate set-to-set and set-to-vector functions. The authors in [70] view network parameters as a special case of a collection of tensors and then construct maximally expressive equivariant linear layers for processing any collection of tensors given a description of their permutation symmetries. These methods are applicable to several types of networks, including those with branches or transformers. However, these models were not necessarily equivariant to scaling nor sign-flipping transformations, which are important symmetries of the input neural networks.

Our method makes the first step toward incorporating both permutation and non-permutation symmetries into NFNs. In particular, the model proposed in our paper is equivariant to permutations and scaling, for ReLU networks, or sign-flipping, for sine and tanh networks. This leads to a significant reduction in the number of parameters, a property that is particularly useful for large neural networks in modern deep learning, while achieving comparable or better results than those in the literature. The authors in [32, 67] have also developed NFNs that incorporates scaling symmetries.

3 Monomial Matrices Perserved by a Nonlinear Activation

Given two sets X,Y , and a group G acts on them, a function ϕ : X /shortrightarrow Y is called G -equivariant if ϕ ( g · x ) = g · ϕ ( x ) for all x ∈ X and g ∈ G . If G acts trivially on Y , then we say ϕ is G -invariant. In this paper, we consider NFNs which are equivariant with respect to certain symmetries of deep weight spaces. These symmetries will be represented by monomial matrices. In Subsection 3.1, we recall the notion of monomial matrices, as well as their actions on space of matrices. We then formalize the maximal group of matrices preserved by the activations ReLU , sin and Tanh in Subsection 3.2.

3.1 Monomial Matrices and Monomial Matrix Group Actions

All matrices considered in this paper have real entries and n is a positive integer.

Definition 3.1 (See [50, page 46]) . Amatrix of size n × n is called a monomial matrix (or generalized permutation matrix ) if it has exactly one non-zero entry in each row and each column, and zeros elsewhere. We will denote by G n the set of such all matrices.

Permutation matrices and invertible diagonal matrices are special cases of monomial matrices. In particular, a permutation matrix is a monomial matrix in which the non-zero entries are all equal to 1 . In case the nonzero entries of a monomial matrix are in the diagonal, it becomes an invertible diagonal matrix. We will denote by P n and ∆ n the sets of permutation matrices and invertible diagonal matrices of size n × n , respectively. It is well-known that the groups G n , P n , and ∆ n are subgroups of the general linear group GL ( n ) .

Each monomial matrix in G n is a product of an invertible diagonal matrix in ∆ n and a permutation matrix in P n , i.e.

In general, we have PD = DP . However, for D = diag( d 1 , d 2 , . . . , d n ) and P = P π , we have PD = ( PDP -1 ) P which is again a product of the invertible diagonal matrix

̸

and the permutation matrix P . As an implication of Eq. (2), there is a group homomorphism φ : P n /shortrightarrow Aut(∆ n ) , defined by the conjugation, i.e. φ ( P )( D ) = PDP -1 for all P ∈ P n and D ∈ ∆ n . The map φ defines the group G n as the semidirect product G n = ∆ n ⋊ φ P n (see [17]). For convenience, we sometimes denote element DP of G n as a pair ( D,P ) .

The groups G n , P n and ∆ n act on the left and the right of R n and R n × m in a canonical way (by matrix-vector or matrix-matrix multiplications). More precisely, we have:

Proposition 3.2. Let x ∈ R n and A = ( A ij ) ∈ R n × m . Then for D = diag( d 1 , . . . , d n ) ∈ ∆ n , D = diag( d 1 , . . . , d m ) ∈ ∆ m , P π ∈ P n , and P σ ∈ P m , we have:

The above proposition can be verified by a direct computation, and is used in subsequent sections.

3.2 Monomial Matrices Preserved by a Nonlinear Activation

We characterize the maximal matrix groups preserved by the activations σ = ReLU , sin or tanh . Here, ReLU is the rectified linear unit activation function which has been used in most of modern neural networks, sin is the sine function which is often used as an activation function in implicit neural representations [57], and tanh is the hyperbolic tangent activation function. Different variants of the results in this subsection can also be found in [24, 68]. We refine them using the terms of monomial matrices and state explicitly here for the completeness of the paper.

Definition 3.3. A matrix A ∈ GL( n ) is said to be preserved by an activation σ if and only if σ ( A · x ) = A · σ ( x ) for all x ∈ R n .

We adopt the term matrix group preserved by an activation from [68]. This term is then referred to as the intertwiner group of an activation in [24].

Proposition 3.4. For every matrix A ∈ GL( n ) , we have:

(i) A is preserved by the activation ReLU if and only if A ∈ G > 0 n . Here, G > 0 n is the subgroup of G n containing only monomial matrices whose nonzero entries are positive numbers.

(ii) A is preserved by the activation σ = sin or tanh if and only if A ∈ G ± 1 n . Here, G ± 1 n is the subgroup of G n containing only monomial matrices whose nonzero entries are ± 1 .

A detailed proof of Proposition 3.4 can be found in Appendix C.1. As a consequence of the above theorem, G > 0 n (respectively, G ± 1 n ) is the maximal matrix subgroup of the general linear group GL ( n ) that is preserved by the activation ReLU (respectively, sin and tanh ).

Remark 3.5. Intuitively, G > 0 n is generated by permuting and positive scaling the coordinates of vectors in R n , while G ± 1 n is generated by permuting and sign flipping. Formally, these groups can be written as the semidirect products:

are two subgroups of ∆ n .

where

4 Weight Spaces and Monomial Matrix Group Actions on Weight Spaces

In this section, we formulate the general structure of the weight spaces of FCNNs and CNNs. We then determine the group action on these weight spaces using monomial matrices. The activation function σ using on the considered FCNNs and CNNs are assumed to be ReLU or sin or Tanh .

4.1 Weight Spaces of FCNNs and CNNs

Weight Spaces of FCNNs. Consider an FCNN with L layers, n i neurons at the i -th layer, and n 0 and n L be the input and output dimensions, together with the activation σ , as follows:

Here, U = ( W,b ) is the parameters with the weights W = { W ( i ) ∈ R n i × n i -1 } L i =1 and the biases b = { b ( i ) ∈ R n i × 1 } L i =1 . The pair U = ( W,b ) belongs to the weight space U = W×B , where:

Weight Spaces of CNNs. Consider a CNN with L convolutional layers, ending with an average pooling layer then fully connected layers, together with activation σ . Let n i and w i be the number of channels and the size of the convolutional kernel at the i th convolutional layer. We will only take account of the L convolutional layers, since the weight space of the fully connected layers are already considered above, and the pooling layer has no learnable parameters:

Here, U = ( W,b ) is the learnable parameters with the weights W = { W ( i ) ∈ R w i × n i × n i -1 } L i =1 and the biases b = { b ( i ) ∈ R 1 × n i × 1 } L i =1 . The convolutional operator ∗ is defined depending on the purpose of the model, and adding b means adding b ( i ) j to all entries of j -th channel at i th layer. The pair U = ( W,b ) belongs to the weight space U = W×B , where:

Remark 4.1. See in Appendix. C.2 for concrete descriptions of weight spaces of FCNNs and CNNs.

4.2 Monomial Matrix Group Action on Weight Spaces

The weight space U of an FCNN or CNN with L layers and n i channels at i th layer has the general form U = W×B , where:

Here, n i is the number of channels at the i th layer, in particular, n 0 and n L are the number of channels of input and output; w i is the dimension of weights and b i is the dimension of the biases in each channel at the i -th layer. The dimension of the weight space U is:

Notation. When working with weight matrices in W , the space R w i × n i × n i -1 = ( R w i ) n i × n i -1 at the i th layer will be considered as the space of n i × n i -1 matrices, whose entries are real vectors in R w i . s In particular, the symbol W ( i ) denotes a matrix in R w i × n i × n i -1 = ( R w i ) n i × n i -1 , while W ( i ) jk ∈ R w i denotes the entry at row j and column k of W ( i ) . Similarly, the notion b ( i ) denotes a bias column vector in R b i × n i × 1 = ( R b i ) n i × 1 , while b ( i ) j ∈ R b i denotes the entry at row j of b ( i ) .

To define the group action of U using monomial matrices, denote G U as the group:

Ideally, each monomial matrix group G n i will act on the weights and the biases at the i th layer of the network. Each element of G U will be of the form g = ( g ( L ) , . . . , g (0) ) , where:

for some invertible diagonal matrix D ( i ) and permutation matrix P π i . The action of G U on U is defined formally as follows.

Definition 4.2 (Group action on weight spaces) . With the notation as above, the group action of G U on U is defined to be the map G U ×U /shortrightarrow U with ( g, U ) ↦/shortrightarrow gU = ( gW,gb ) , where:

In concrete:

Remark 4.3. The group G U is determined only by the number of layers L and the numbers of channels n i , not by the dimensions of weights w i and biases b i at each channel.

The group G U has nice behaviors when acting on the weight spaces of FCNNs given in Eq. (5) and CNNs given in Eq. (8). In particular, depending on the specific choice of the activation σ , the function f built by the given FCNN or CNN is invariant under the action of a subgroup G of G U , as we will see in the following proposition.

Proposition 4.4 ( G -equivariance of neural functionals) . Let f = f ( · ; U, σ ) be an FCNN given in Eq. (5) or CNN given in Eq. (8) with the weight space U ∈ U and an activation σ ∈ { ReLU , Tanh , sin } . Let us defined a subgroup G of G U as follows:

Then f is G -invariant under the action of G on its weight space, i.e.

for all g ∈ G, U ∈ U and x ∈ R n 0 .

Remark 4.5 (Maximality of G ) . The proof of Proposition 4.4 can be found in Appendix C.2. The group G defined above is even proved to be the maximal choice in the case:

σ = ReLU and n L ⩾ . . . ⩾ n 2 ⩾ n 1 > n 0 = 1 (see [12, 25]), or

σ = tanh (see [14, 22]).

Here, G is maximal in the sense that: if U ′ is another element in U with f ( · ; U, σ ) = f ( · ; U ′ , σ ) , then there exists an element g ∈ G such that U ′ = gU . It is natural to ask whether the group G is still maximal in the other case. This question still remains open and we leave it for future exploration.

According to Proposition 4.4, the symmetries of the weight space of an FCNN or CNN must include not only permutation matrices but also other types of monomial matrices resulting from scaling (for ReLU networks) or sign flipping (for sin and tanh networks) the weights. Recent works on NFN design only take into account the permutation symmetries of the weight space. Therefore, it is necessary to design a new class of NFNs that incorporates these missing symmetries. We will introduce such a class in the next section.

5 Monomial Matrix Group Equivariant and Invariant NFNs

In this section, we introduce a new family of NFNs, called Monomial-NFNs, by incorporating symmetries arising from monomial matrix groups which have been clarified in Proposition 4.4. The main components of Monomial-NFNs are the monomial matrix group equivariant and invariant linear layers between two weight spaces which will be presented in Subsections 5.1 and 5.2, respectively. We will only consider the case of ReLU activation. Network architectures with other activations will be considered in detail in Appendices A and B.

In the following, U = ( W , B ) is the weight space with L layers, n i channels at i th layer, and the dimensions of weight and bias are w i and b i , respectively (see Eqs. (11) and (12)). Since we consider ReLU network architectures, according to Proposition 4.4, the symmetries of the weight space is given by the subgroup G = { id G n L } × G > 0 n L -1 × . . . ×G > 0 n 1 ×{ id G n 0 } of G U .

5.1 Equivariant Layers

We now construct a linear G -equivariant layer between weight spaces. These layers form the fundamental building blocks for our Monomimal-NFNs. Let U and U ′ be two weight spaces of the same network architecture described in Eqs. (11) and (12), i.e. they have the same number of layers as well as the same number of channels at each layer. Denote the dimension of weights and biases in each channel at the i -th layer of U ′ as w ′ i and b ′ i , respectively. Note that, in this case, we have G U = G U ′ . We construct G -equivariant afine maps E : U /shortrightarrow U ′ with x ↦/shortrightarrow a x + b , where a ∈ R dim U ′ × dim U and b ∈ R dim U ′ × 1 are learnable parameters.

To make E to be G -equivarient, a and b have to satisfy a system of constraints (usually called parameter sharing ), which are induced from the condition E ( gU ) = gE ( U ) for all g ∈ G and U ∈ U . We show in details what are these constraints and how to derive the concrete formula of E in Appendix A. The formula of E is presented as follows: For U = ( W,b ) ∈ U , the image E ( U ) = ( W ′ , b ′ ) ∈ U ′ is computed by:

Here, ( p , q , r , s , t ) is the hyperparameter of E . We discuss in detail the dimensions and sharing information between these parameters in Appendix A.1. Note that, we also show that all linear G -equivariant functional are in this form in Appendix A. To conclude, we have:

Theorem 5.1. With notation as above, the linear functional map E : U /shortrightarrow U ′ defined by Eq. (18) is G -equivariant. Moreover, every G -equivariant linear functional map from U to U ′ are in that form.

Number of parameters and comparison to previous works. The number of parameters in our layer is linear in L, n 0 , n L , which is significantly smaller than the number of parameters in layers described in [71], where it is quadratic in L, n 0 , n L (see Table 1). This reduction in parameter count means that our model is suitable for weight spaces of large-scale networks and deep NFNs. Intuitively, the advantage of our layer arises because the group G acting on the weight spaces in our setting is much larger, resulting in a significantly smaller number of orbits in the quotient space U /G . Since the number of orbits is equal to the number of parameters, this leads to a more compact representation. Additionally, the presence of the group ∆ > 0 ∗ forces many coefficients of the linear layer E to be zero, further contributing to the efficiency of our model.

5.2 Invariant Layers

We will construct an G -invariant layer I : U /shortrightarrow R d for a fixed integer d > 0 . In order to do that, we will seek a map I in the form:

where I ∆ > 0 : U /shortrightarrow U is an ∆ > 0 ∗ -invariance and P ∗ -equivariance map, I P : U /shortrightarrow R dim U is an P ∗ -invariant map, and MLP : R dim U /shortrightarrow R d is an arbitrary multilayer perceptron to adjust the output dimension. Since G = G > 0 ∗ = ∆ > 0 ∗ ⋊ φ P ∗ (see Remark 3.5), the composition I = MLP ◦ I P ◦ I ∆ > 0 is clearly G -invariant as expected. The construction of I ∆ > 0 and I P will be presented below.

Construct I ∆ > 0 . To capture ∆ > 0 ∗ -invariance, we recall the notion of positively homogeneous of degree zero maps. For n > 0 , a map α from R n is called positively homogeneous of degree zero if for all λ > 0 and ( x 1 , . . . , x n ) ∈ R n , we have α ( λx 1 , . . . , λx n ) = α ( x 1 , . . . , x n ) . We construct I ∆ > 0 : U /shortrightarrow U by taking collections of positively homogeneous of degree zero functions { α ( i ) jk : R w i /shortrightarrow R w i } and { α ( i ) j : R b i /shortrightarrow R b i } , each one corresponds to weight and bias of U . The maps I ∆ > 0 : U /shortrightarrow U that ( W,b ) ↦/shortrightarrow ( W ′ , b ′ ) is defined by simply applying these functions on each weight and bias entries as follows:

I ∆ > 0 is ∆ > 0 ∗ -invariant by homogeneity of the α functions. To make it become P ∗ -equivariant, some α functions have to be shared arross any axis that have permutation symmetry. We derive this relation in Appendix B. Some candidates for positively homogeneous of degree zero functions are also presented in Appendix B. They can be fixed or learnable.

Construct I P . To capture P ∗ -invariance, we simply take summing or averaging the weight and bias across any axis that have permutation symmetry as in [71]. In concrete, we have I P : U /shortrightarrow R dim U is computed as follows:

Here, ⋆ denotes summation or averaging over the rows or columns of the weight and bias.

Remark 5.2. In our experiments, we use averaging operator since it is empirically more stable.

Finally we compose an MLP before I P and I ∆ > 0 to obtain an G -invariant map. We summarize the above construction as follows.

Theorem 5.3. The functional map I : U /shortrightarrow R d defined by Eq. (19) is G -invariant.

5.3 Monomial Matrix Group Equivariant Neural Functionals (Monomial-NFNs)

We build Monomial-NFNs by the constructed equivariant and invariant functional layers, with activations and additional layers discussed below. The equivariant NFN is built by simply stacking G -equivariant layers. For the invariant counterpart, we follow the construction in [71]. In particular, we first stack some G -equivariant layers, then a ∆ > 0 ∗ -invariant and P ∗ -equivariant layer. This makes our NFN to be ∆ > 0 ∗ -invariant and P ∗ -equivariant. Then we finish the construction by stacking a P ∗ -invariant layer and the end. This process makes the whole NFN to be G -invariant as expected.

Activations of G -equivariant functionals. Dealing with equivariance under action of P ∗ only requires activation of the NFN is enough, since P ∗ acts on only the order of channels in each channel of the weight space. For our G -equivariant NFNs, between each layer that is ∆ > 0 ∗ -equivariant, we have to use the same type of activations as the activation in the network input (i.e. either ReLU , sin or tanh in our consideration) to maintain the equivariance of the NFN.

Fourier Features and Positional Embedding. As mentioned in [35, 71, 72], Fourier Features [30, 62] and (sinusoidal) position embedding play a significant role in the performance of their functionals. Also, in [71], position embedding breaks the symmetry at input and output neurons, and allows us to use equivariant layers that act on input and output neurons. In our G -equivariant layers, we do not consider action on input and output neurons as mentioned. Also, using Fourier Features does not maintain ∆ > 0 ∗ , so we can not use this Fourier layer for our equivariant Monomial-NFNs, and in our invariant Monomial-NFNs, we only can use Fourier layer after the ∆ > 0 ∗ -invariant layer. This can be considered as a limitation of Monomial-NFNs.

6 Experimental Results

In this session, we empirically demonstrate the performance of our Monomial Matrix Group Equivariant Neural Functional Networks (Monomial-NFNs) on various tasks that are either invariant

Monomial-NFN

NP

HNP

STATNN

(predicting CNN generalization from weights and classifying INR representations of images) or equivariant (weight space style editing). We aim to establish two key points. First, our model exhibits more stable behavior when the input undergoes transformations from the monomial matrix groups. Second, our model, Monomial-NFN, achieves competitive performance compared to other baseline models. Our results are averaged over 5 runs. Hyperparameter settings and the number of parameters can be found in Appendix D.

6.1 Predicting CNN Generalization from Weights

Experiment Setup. In this experiment, we evaluate how our Monomial-NFN predicts the generalization of pretrained CNN networks. We employ the Small CNN Zoo [64], which consists of multiple network weights trained with different initialization and hyperparameter settings, together with activations Tanh or ReLU . Since Monomial-NFNs depend on activations of network inputs, we divide the Small CNN Zoo into two smaller datasets based on their activations. The ReLU dataset considers the group G > 0 n , while the Tanh dataset considers the group G ± 1 n .

We construct the dataset with additional weights that undergo random hidden vector permutation and scaling based on their monomial matrix group. For the ReLU dataset with the group G > 0 n , we uniformly sample the diagonal indices of D (see Eq. 14) for various ranges: [1 , 10] , [1 , 1 × 10 2 ] , . . . , [1 , 1 × 10 6 ] , while belonging to {-1 , 1 } in the case of Tanh dataset with the group G ± 1 n . For both datasets, we compare our model with STATNN [65], and with two permutation equivariant neural functional networks from [71], referred to as HNP and NP. To compare the performance of all models, we use Kendall's τ rank correlation metric [33].

Results. We demonstrate the results of all models on the ReLU subset in Figure 1, showing that our model attains stable Kendall's τ when the scale operators are sampled from different ranges. Specifically, when the log of augmentation upper scale is 0, i.e. the data remains unaltered, our model performs as well as the HNP model. However, as the weights undergo more extensive scaling and permutation, the performance of the HNP and STATNN models drops significantly, indicating their lack of scaling symmetry. The NP model exhibits a similar trend, albeit to a lesser extent. In contrast, our model maintains stable performance throughout.

Table 2 illustrates the performance of all models on both the original and augmented Tanh subsets of CNN Zoo. Our model achieves the highest performance among all models and shows the greatest improvement after training with the augmented dataset. The gap between our model and the secondbest model (HNP) increases from 0.006 to 0.008. Additionally, in both experiments, our model utilizes significantly fewer parameters than the baseline models, using only up to 50% of the parameters compared to HNP.

6.2 Classifying implicit neural representations of images

Experiment Setup. In this experiment, our focus is on extracting the original data information encoded within the weights of implicit neural representations (INRs). We utilize the dataset from [71],

which comprises pretrained INR networks [58] that encode images from the CIFAR-10 [36], MNIST [39], and FashionMNIST [69] datasets. Each pretrained INR network is designed to map image coordinates ( x, y ) to color pixel values - 3-dimensional RGB values for CIFAR-10 and 1-dimensional grayscale values for MNIST and FashionMNIST.

Results. Wecompare our model with NP, HNP, and MLP baselines. The results in Table 3 demonstrate that our model outperforms the second-best baseline, NP, for the FashionMNIST and CIFAR10 datasets by 2 . 94% and 0 . 49% , respectively. For the MNIST dataset, our model also obtains comparable performance.

6.3 Weight space style editing.

Experiment setup. In this experiment, we explore altering the weights of the pretrained SIREN model [58] to change the information encoded within the network. We use the network weights provided in the HNP paper for the pretrained SIREN networks on MNIST and CIFAR-10 images. Our focus is on two tasks: the first involves modifying the network to dilate digits from the MNIST dataset, and the second involves altering the SIREN network weights to enhance the contrast of CIFAR-10 images. The objective is to minimize the mean squared error (MSE) training loss between the generated image from the edited SIREN network and the dilated/enhanced contrast image.

Results. Table 4 shows that our model performs on par with the best-performing model for increasing the contrast of CIFAR-10 images. For the MNIST digit dilation task, our model also achieves competitive performance compared to the NP baseline. Additionally, Figure 2 presents random samples of the digits that each model encodes for the dilation and contrast tasks, demonstrating that our model's results are visually comparable to those of HNP and NP in both tasks.

7 Conclusion

In this paper, we formally describe a group of monomial matrices that preserves FCNNs and CNNs while acting on their weight spaces. For ReLU networks, this group includes permutation and scaling symmetries, while for networks with sin or Tanh activations, it encompasses permutation and sign-flipping symmetries. We introduce Monomial-NFNs, a first-of-a-kind class of NFNs that incorporates these scaling or sign-flipping symmetries in weight spaces. We demonstrate that the low number of trainable parameters in our equivariant linear layer of Monomial-NFNs compared to previous works on NFNs, highlighting their capability to efficiently process weight spaces of deep networks. Our NFNs exhibit competitive generalization performance and efficiency compared to existing models across several benchmarks.

One limitation of our model is that, due to the large size of the group considered, the resulting linear layers can be limited in terms of expressivity. For example, a weight corresponding to an edge between two neurons will be updated based only on its previous value, ignoring other edges across the same or other layers. To resolve this issue, it is necessary to construct an equivariant nonlinear layer to encode further relations between these weights, thus enhancing the expressivity. Another limitation is that we are uncertain about the maximality of the group G acting on the weight space of the ReLU network. Therefore, other types of symmetries may exist in the weight space beyond neuron permutation and weight scaling, and our model is not equivariant with respect to these symmetries. We leave the problem of identifying such a maximal group for future research.

Acknowledgments and Disclosure of Funding

This research / project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-012-SGIL). This research / project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2023) (A-8002040-00-00, A-8002039-00-00). This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00).

References

Francesca Albertini and Eduardo D Sontag. For neural networks, function determines form. Neural networks , 6(7):975-990, 1993.

Francesca Albertini and Eduardo D Sontag. Identifiability of discrete-time neural networks. In Proc. European Control Conference , pages 460-465. Springer Berlin, 1993.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning , pages 242-252. PMLR, 2019.

Bruno Andreis, Bedionita Soro, and Sung Ju Hwang. Set-based neural network encoding. CoRR , abs/2305.16625, 2023.

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems , 29, 2016.

Maor Ashkenazi, Zohar Rimon, Ron Vainshtein, Shir Levi, Elad Richardson, Pinchas Mintz, and Eran Treister. Nern: Learning neural representations for neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Understanding symmetries in deep networks. CoRR , abs/1511.01029, 2015.

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings . OpenReview.net, 2018.

Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. CoRR , abs/2302.03130, 2023.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks? , pages 265-287. Routledge, 2013.

Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu networks. In 8th International Conference on Learning Representations , 2020.

Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 6491-6506. Association for Computational Linguistics, 2021.

An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural computation , 5(6):910-927, 1993.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

David Steven Dummit and Richard M Foote. Abstract algebra , volume 3. Wiley Hoboken, 2004.

Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 5694-5725. PMLR, 2022.

Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 2989-3015. PMLR, 2022.

Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: Dissecting the weight space of neural networks. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang, editors, ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020) , volume 325 of Frontiers in Artificial Intelligence and Applications , pages 1119-1126. IOS Press, 2020.

Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14300-14310, 2023.

Charles Fefferman and Scott Markel. Recovering a feed-forward net from its output. Advances in neural information processing systems , 6, 1993.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems , 35:11893-11905, 2022.

Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In International Conference on Machine Learning , pages 11734-11760. PMLR, 2023.

Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint arXiv:2002.11833 , 2020.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015.

Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers , pages 129-135. Elsevier, 1990.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):17351780, 1997.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

John M. Jumper, Richard O. Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russell Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, R. D. Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David L. Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, July 2021. https://europepmc.org/article/MED/34265844 ; https://www.nature. com/articles/s41586-021-03819-2 ; https://www.mendeley.com/catalogue/ bde88f33-525c-3af0-823a-3bb305a93020/ ; https://facultyopinions.com/ prime/740477161 ; https://pubmed.ncbi.nlm.nih.gov/34265844/ ; https: //www.ncbi.nlm.nih.gov/pmc/articles/PMC8371605/ ; https://www.nature.com/ articles/s41586-021-03819-2.pdf ; https://europepmc.org/abstract/MED/ 34265844 ; https://econpapers.repec.org/RePEc:nat:nature:v:596:y:2021:i: 7873:d:10.1038_s41586-021-03819-2 ; https://www.scienceopen.com/document? vid=b1f93136-f0f0-4c78-8e45-27bd037e9bb9 ; http://rucweb.tsg211.com/http/ 77726476706e69737468656265737421e7e056d229317c456c0dc7af9758/articles/ s41586-021-03819-2 ; http://pubmed02.keyan123.cn/34265844/ .

Ioannis Kalogeropoulos, Giorgos Bouritsas, and Yannis Panagakis. Scale equivariant graph metanetworks. arXiv preprint arXiv:2406.10685 , 2024.

M. G. KENDALL. A NEW MEASURE OF RANK CORRELATION. Biometrika , 30(1-2):8193, 06 1938.

Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems , 34:29433-29448, 2021.

Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J. Burghouts, Efstratios Gavves, Cees G. M. Snoek, and David W. Zhang. Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , pages 1106-1114, 2012.

Vera Kurkova and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Computation , 6(3):543-558, 1994.

Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005.

Derek Lim, Haggai Maron, Marc T. Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024.

Luca De Luigi, Adriano Cardace, Riccardo Spezialetti, Pierluigi Zama Ramirez, Samuele Salti, and Luigi Di Stefano. Deep learning on implicit neural representations of shapes. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09760 , 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing at scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning , pages 25790-25816. PMLR, 2023.

Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems , 28, 2015.

Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and generalization in neural networks: an empirical study. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209.12892 , 2022.

M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

Joseph J Rotman. An introduction to the theory of groups , volume 148. Springer Science & Business Media, 2012.

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. 1986.

Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00 , pages 59-63. IEEE, 2000.

Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyperrepresentations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems , 35:27906-27920, 2022.

Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems , 34:16481-16493, 2021.

Konstantin Schürholt, Diyar Taskiran, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Model zoos: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems , 35:38134-38148, 2022.

Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems , 33:7462-7473, 2020.

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Samuel Sokota, Hengyuan Hu, David J Wu, J Zico Kolter, Jakob Nicolaus Foerster, and Noam Brown. A fine-tuning approach to belief state modeling. In International Conference on Learning Representations , 2021.

Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines , 8:131-162, 2007.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 1-9. IEEE Computer Society, 2015.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems , 33:7537-7547, 2020.

Viet-Hoang Tran, Thieu N Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Tran, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. arXiv preprint arXiv:2410.04209 , 2024.

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448 , 2020.

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017.

Thieu N Vo, Viet-Hoang Tran, Tho Tran Huu, An Nguyen The, Thanh Tran, Minh-Khoi NguyenNhat, Duy-Tung Pham, and Tan Minh Nguyen. Equivariant polynomial functional networks. arXiv preprint arXiv:2410.04213 , 2024.

Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. Discrete applied mathematics , 69(1-2):33-60, 1996.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR , abs/1708.07747, 2017.

Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. CoRR , abs/2402.05232, 2024.

Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems , 36, 2024.

Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems , 36, 2024.

Supplement to 'Monomial Matrix Group Equivariant Neural Functional Networks'

Table of Contents

A Construction of Monomial Matrix Group Equivariant Layers

In this appendix, we present how we constructed Monomial Matrix Group Equivariant Layers. We adopt the idea of notation in [71] to derive the formula of linear functional layers. For two weight spaces U and U ′ with the same number of layers L as well as the same number of channels at i -th layer n i :

and

our equivariant layer E : U /shortrightarrow U ′ will has the form as follows:

Here, the map E is parameterized by hyperparameter θ = ( p , q , s , r , t ) with dimensions of each component as follows:

p ijk spq ∈ R w ′ i × w s represents the contribution of W ( s ) pq to W ′ ( i ) jk ,

where

Recall the definition of the group action gU = ( gW,gb ) where:

or in term of entries:

gE ( U ) = gU ′ = ( gW ′ , gb ′ ) is computed as follows:

E ( gU ) = ( gU ) ′ = (( gW ) ′ , ( gU ) ′ ) is computed as follows:

q ijk sp ∈ R w ′ i × b s represents the contribution of b ( s ) p to W ′ ( i ) jk ,

t ijk ∈ R w ′ i is the bias of the layer for W ′ ( i ) jk ;

r ij spq ∈ R b ′ i × w s represents the contribution of W ( s ) pq to b ′ ( i ) j ,

s ij sp ∈ R b ′ i × b s represents the contribution of b ( s ) p to b ′ ( i ) j ,

t ij ∈ R b ′ i is the bias of the layer for b ′ ( i ) j .

We want to see how an element of the group G U acts on input and output of layer E . Let

We need E is G -equivariant under the action of subgroups of G U as in Theorem 4.4. From the above computation, if gE ( U ) = E ( gU ) , the hyperparameter θ = ( p , q , r , s , t ) have to satisfy the system of constraints as follows:

for all possible tuples (( i, j, k ) , ( s, p, q )) and all g ∈ G . Since the two subgroups G considered in Theorem 4.4 satisfy that: G ∩ P i is trivial (for i = 0 or i = L ) or the whole P i (for 0 < i < L ), so we can simplify the above system of constraints by moving all the permutation π 's to LHS, then replacing π -1 by π . The system, denoted as (*), now is written as follows:

We treat each case of activation separately.

̸

̸

A.1 ReLU activation

Recall that, in this case:

̸

̸

̸

So the system of constraints (*) holds for:

all possible tuples (( i, j, k ) , ( s, p, q )) ,

all π i ∈ P i for 0 < i < L , all d ( i ) j > 0 for 0 < i < L , 1 ⩽ j ⩽ n i ,

π i = id G n i and d ( i ) j = 1 for i = 0 or i = L .

By treat each case of tuples (( i, j, k ) , ( s, p, q )) , we solve Eq. *1, Eq. *2, Eq. *3, Eq. *4 in the system (*) for hyperparameter ( p , q , r , s ) as in Table 5. For t ijk and t ij , by Eq. *5, Eq. *6, we have t ijk = 0 for all ( i, j, k ) , t ij = 0 if i < L , and t Lj is arbitrary for all 1 ⩽ j ⩽ n L . In conclusion, the formula of equivariant layers E in case of activation ReLU is presented as in Table 6.

Example A.1. Let us consider a two-hidden-layers MLP with activation σ = ReLU. Assume that n 0 = n 1 = n 2 = n 3 = 2 , i.e., all layers have two neurons. This MLP defines a function f : R 2 /shortrightarrow R 2 given by

the weight space U consists of the tuples

and it has dimension 18.

According to Eq. (27), an equivariant layer E over U has the form

where

These equations can be written in a friendly matrix form as follows.

A.2 Sin or Tanh activation

Recall that, in this case:

So the system of constraints (*) holds for:

all possible tuples (( i, j, k ) , ( s, p, q )) ,

all π i ∈ P i for 0 < i < L , all d ( i ) j ∈ {± 1 } for 0 < i < L , 1 ⩽ j ⩽ n i ,

We assume L ⩾ 3 , the case L ⩽ 2 can be solved similarly. By treat each case of tuples (( i, j, k ) , ( s, p, q )) , we solve Eq. *1, Eq. *2, Eq. *3, Eq. *4 in the system (*) for hyperparameter ( p , q , r , s ) as in Table 7. For t ijk and t ij , by Eq. *5, Eq. *6, we have t ijk = 0 for all ( i, j, k ) , t ij = 0 if i < L , and t Lj is arbitrary for all 1 ⩽ j ⩽ n L . In conclusion, the formula of equivariant layers E in case of sin or Tanh activation is presented as in Table 8.

̸

̸

B Construction of Monomial Matrix Group Invariant Layers

In this appendix, we present how we constructed Monomial Matrix Group Invariant Layers. Let U be a weight spaces with the number of layers L as well as the number of channels at i -th layer n i . We want to construct G -invariant layers I : U /shortrightarrow R d for some d > 0 . We treat each case of activations separately.

B.1 ReLU activation

Recall that, in this case:

̸

̸

Since G > 0 ∗ is the semidirect product of ∆ > 0 ∗ and P ∗ with ∆ > 0 ∗ is the normal subgroup, we will treat these two actions consecutively, ∆ > 0 ∗ first then P ∗ . We denote these layers by I ∆ > 0 and I P . Note that, since I ∆ > 0 comes before I P , I ∆ > 0 is required to be ∆ > 0 ∗ -invariant and P ∗ -equivariant, and I P is required to be P ∗ -invariant.

∆ > 0 ∗ -invariance and P ∗ -equivariance. To capture ∆ > 0 ∗ -invariance, we recall the notion of positively homogeneous of degree zero maps. For n > 0 , a map α from R n is called positively

homogeneous of degree zero if

B.2 Sin or Tanh activation

Recall that, in this case:

Since G ± 1 ∗ is the semidirect product of ∆ ± 1 ∗ and P ∗ with ∆ ± 1 ∗ is the normal subgroup, we will treat these two actions consecutively, ∆ ± 1 ∗ first then P ∗ . We denote these layers by I ∆ ± 1 and I P . Note that, since I ∆ ± 1 comes before I P , I ∆ ± 1 is required to be ∆ ± 1 ∗ -invariant and P ∗ -equivariant, and I P is required to be P ∗ -invariant.

for all λ > 0 and ( x 1 , . . . , x n ) ∈ R n . We construct I ∆ > 0 : U /shortrightarrow U by taking collections of positively homogeneous of degree zero functions { α ( i ) jk : R w i /shortrightarrow R w i } and { α ( i ) j : R b i /shortrightarrow R b i } , each one corresponds to weight and bias of U . The maps I ∆ > 0 : U /shortrightarrow U that ( W,b ) ↦/shortrightarrow ( W ′ , b ′ ) is defined by simply applying these functions on each weight and bias entries as follows:

I ∆ > 0 is ∆ > 0 ∗ -invariant by homogeneity of the α functions. To make it become P ∗ -equivariant, some α functions have to be shared arross any axis that have permutation symmetry, presented in Table 9.

Candidates of function α . We simply choose positively homogeneous of degree zero function α : R n /shortrightarrow R n by taking α (0) = 0 and:

where β : R n /shortrightarrow R n is an arbitrary function. The function β can be fixed or parameterized to make α to be fixed or learnable.

P ∗ -invariance. To capture P ∗ -invariance, we simply take summing or averaging the weight and bias across any axis that have permutation symmetry as in [71]. In concrete, some d > 0 , we have I P : U /shortrightarrow R d is computed as follows:

Here, ⋆ denotes summation or averaging over the rows or columns of the weight and bias.

G -invariance. Now we simply compose I P ◦ I ∆ > 0 to get an G -invariant map. We use an MLP to complete constructing an G -invariant layer with output dimension d as desired:

∆ ± 1 ∗ -invariance and P ∗ -equivariance. To capture ∆ ± 1 ∗ -invariance, we use even functions, i.e. α ( x ) = α ( -x ) for all x . We construct I ∆ ± 1 : U /shortrightarrow U by taking collections of even functions { α ( i ) jk : R w i /shortrightarrow R w i } and { α ( i ) j : R b i /shortrightarrow R b i } , each one corresponds to weight and bias of U . The maps I ∆ ± 1 : U /shortrightarrow U that ( W,b ) ↦/shortrightarrow ( W ′ , b ′ ) is defined by simply applying these functions on each weight and bias entries as follows:

I ∆ ± 1 is ∆ ± 1 ∗ -invariant by design. To make it become P ∗ -equivariant, some α functions have to be shared arross any axis that have permutation symmetry, presented in Table 9.

Candidates of function α . We simply choose even function α : R n /shortrightarrow R n by:

where β : R n /shortrightarrow R n is an arbitrary function. The function β can be fixed or parameterized to make α to be fixed or learnable.

P ∗ -invariance. To capture P ∗ -invariance, we simply take summing or averaging the weight and bias across any axis that have permutation symmetry as in [71]. In concrete, some d > 0 , we have I P : U /shortrightarrow R d is computed as follows:

Here, ⋆ denotes summation or averaging over the rows or columns of the weight and bias.

G -invariance. Now we simply compose I P ◦ I ∆ ± 1 to get an G -invariant map. We use an MLP to complete constructing an G -invariant layer with output dimension d as desired:

C Proofs of Theoretical Results

C.1 Proof of Proposition 3.4

Proof. We simply denote the activation ReLU or sin or tanh by σ . Let A ∈ GL( n )

for all x ∈ R n . This means:

for all x 1 , . . . , x n ∈ R . We rewrite this equation as:

or equivalently:

Thus, that satisfies: ,

for all x 1 , . . . , x n ∈ R and i = 1 , . . . , n . We will consider the case i = 1 , i.e.

and treat the case i > 1 similarly. Now we consider the activation σ case by case as follows.

(i) Case 1. σ = ReLU . We have some observations:

Let x 1 = 1 , and x 2 = . . . = x n = 0 . Then from Eq. (60), we have:

which implies that a 11 ⩾ 0 . Similarly, we also have a 12 , . . . , a 1 n ⩾ 0 .

Since A is an invertible matrix, the entries a 11 , . . . , a 1 n in the first row of A can not be simultaneously equal to 0 .

There is at most only one nonzero number among the entries a 11 , . . . , a 1 n . Indeed, assume by the contrary that a 11 , a 12 > 0 . Let x 3 = . . . = x n = 0 , from Eq. (60), we have:

Let x 2 = -1 , we have:

Now, let x 1 > 0 be a sufficiently large number such that a 11 x 1 -a 12 > 0 . (Note that this number exists since a 11 > 0 ). Then we have:

which implies a 12 = 0 , a contradiction.

It follows from these three observations that there is exactly one non-zero element among the entries a 11 , . . . , a 1 n . In other words, matrix A has exactly one nonzero entry in the first row. This applies for every row, so A has exactly one non-zero entry in each row. Since A is invertible, each column of A has at least one non-zero entry. Thus A also has exactly one non-zero entry in each column. Hence, A is in G n . Moreover, all entries of A are non-negative, so A is in G > 0 n .

It is straight forward to check that for all A in G > 0 n we have σ ( A · x ) = A · σ ( x ) .

(ii) Case 2. σ = Tanh or σ = sin . We have some observations:

Since A is an invertible matrix, the entries a 11 , . . . , a 1 n in the first row of A can not be simultaneously equal to 0 .

̸

There is at most only one nonzero number among the entries a 11 , . . . , a 1 n . Indeed, assume by the contrary that a 11 , a 12 = 0 . Let x 3 = . . . = x n = 0 , from Eq. (60), we have:

Note that a 11 , a 12 ∈ {-1 , 1 } , so by consider all the cases, we will lead to a contradiction.

It follows from the above three observations that there is exactly one non-zero element among the entries a 11 , . . . , a 1 n . In other words, matrix A has exactly one nonzero entry in the first row. This applies for every row, so A has exactly one non-zero entry in each row. Note that, since A is invertible, each column of A has at least one non-zero entry. Therefore, A also has exactly one non-zero entry in each column. Hence, A is in G n . Moreover, all entries of A are in {-1 , 0 , 1 } , so A is in G ± 1 n .

It is straight forward to check that for all A in G ± 1 n we have σ ( A · x ) = A · σ ( x ) .

The proposition is then proved completely.

C.2 Proof of Proposition 4.4

Proof. For both Fully Connected Neural Networks case and Convolutional Neural Networks case, we consider a network f with three layers, with n 0 , n 1 , n 2 , n 3 are number of channels at each layer, and its weight space U . We will show the proof for part ( i ) where activation σ is ReLU , and part ( ii ) can be proved similarly. For part ( i ) , we prove f to be G -invariant on its weight space U , for the group G that is defined by:

Case 1. f is a Fully Connected Neural Network with three layers, with n 0 , n 1 , n 2 , n 3 are number of channels at each layer as in Eq. 5:

Case 2. f is a Convolutional Neural Network with three layers, with n 0 , n 1 , n 2 , n 3 are number of channels at each layer as in Eq. 8:

We have some observations:

For case 1. For W ∈ R m × n , x ∈ R n and a > 0 , we have:

For case 2. For simplicity, we consider ∗ as one-dimentional convolutional operator, and other types of convolutions can be treated similarly. For W = ( w 1 , . . . , w m ) ∈ R m , b ∈ R and x = ( x 1 , . . . , x n ) ∈ R n , we have:

With these two observations, we can see the proofs for both cases are similar to each other. We will show the proof for case 2, when f is a convolutional neural network since it is not trivial as case 1. Now we have U = ( W,b ) with:

where:

So for a > 0 , we have:

Let g be an element of G :

where:

We compute gU :

where:

and,

Now we show that f ( x ; U, σ ) = f ( x ; gU, σ ) for all x = ( x 1 , . . . , x n 0 ) ∈ R n 0 . For 1 ⩽ i ⩽ n 3 , we compute the i -th entry of f ( x ; gU, σ ) as follows:

End of proof.

D Additional experimental details

D.1 Runtime and Memory Consumption

We provide the runtime and memory consumption of Monomial-NFNs and the previous NFNs in Tables 10 and 11 to compare the computational and memory costs in the task of predicting CNN generalization (see Section 6.1). It is observable that our model runs faster and consumes significantly less memory than NP/HNP in [71] and GNN-based method in [35]. This highlights the benefits of parameter savings in Monomial-NFN.

D.2 Comparison of Monomial-NFNs and GNN-based NFNs

We provide experimental result to compare the efficiency of our model and a permutation equivariant GNN-based NFN [35] in two scenarios below.

Training the model on augmented train data and testing with the augmented test data (see Tables 12 and 13).

Here, we present the experimental results on the original dataset and the results on the augmented dataset. The augmentation levels for the ReLU subset are 1, 2, 3, and 4,

corresponding to augmentation ranges of [1 , 10] , [1 , 10 2 ] , [1 , 10 3 ] , [1 , 10 4 ] . The augmented dataset for the Tanh subset corresponds to the augmentation range of [ -1 , 1]

The results for GNN exhibit a similar trend as other baselines that do not incorporate the scaling symmetry into their architectures. In contrast, our model has stable performance. A notable observation is that the GNN model uses 5.5M parameters (4 times more than our model), occupies 6000MB of memory, and takes 4 hours to train.

Training the model on original train data and testing with the augmented test data (see Tables 14 and 15).

In these more challenging scenario, GNN's performance drops significantly, which highlights the lack of scaling symmetry in the model. Our model maintains consistent performance, matching the case in which we train with the augmented data.

D.3 Predicting generalization from weights

Dataset. The original ReLU subset of the CNN Zoo dataset includes 6050 instances for training and 1513 instances for testing. For the Tanh dataset, it includes 5949 training and 1488 testing instances. For the augmented data, we set the augmentation factor to 2, which means that we augment the original data once, resulting in a new dataset of double the size. The complete size of all datasets is presented in Table 16

Implementation details. Our model follows the same architecture as in [71], comprising three equivariant Monomial-NFN layers with 16, 16, and 5 channels, respectively, each followed by ReLU activation ( ReLU dataset) or Tanh activation ( Tanh dataset). The resulting weight space features are input into an invariant Monomial-NFN layer with Monomial-NFN pooling (Equation 19) with learnable parameters ( ReLU case) or mean pooling ( Tanh case). Specifically, the Monomial-NFN pooling layer normalizes the weights across the hidden dimension and takes the average for rows (first layer), columns (last layer), or both (other layers). The output of this invariant Monomial-NFN layer is flattened and projected to R 200 ( ReLU case) or R 1000 ( Tanh case). This resulting vector is then passed through an MLP with two hidden layers with ReLU activations. The output is linearly projected to a scalar and then passed through a sigmoid function. We use the Binary Cross Entropy (BCE) loss function and train the model for 50 epochs, with early stopping based on τ on the validation set, which takes 35 minutes to train on an A100 GPU. The hyperparameters for our model are presented in Table 18.

For the baseline models, we follow the original implementations described in [71], using the official code (available at: https://github.com/AllanYangZhou/nfn). For the HNP and NP models, there are 3 equivariant layers with 16, 16, and 5 channels, respectively. The features go through an average pooling layer and 3 MLP layers with 1000 hidden neurons. The hyperparameters of our model and the number of parameters for all models in this task can be found in Table 17.

D.4 Classifying implicit neural representations of images

Dataset. We utilize the original INRs dataset provided by [71], with no augmentation. The data is obtained by implementing a single SIREN model for each image in each dataset: CIFAR-10, MNIST, and Fashion-MNIST. The size of training, validation, and test samples for each dataset is provided in Table 19.

Implementation details. In these experiments, our general architecture includes 2 MonomialNFN layers with sine activation, followed by 1 Monomial-NFN layer with absolute activation. The choice of hidden dimension in the Monomial-NFN layer depends on each dataset and is described in Table 20. The architecture then follows the same design as the NP and HNP models in [71], where a Gaussian Fourier Transformation is applied to encode the input with sine and cosine components, mapping from 1 dimension to 256 dimensions. If the base layer is NP, the features will go through IOSinusoidalEncoding, a positional encoding designed for the NP layer, with a maximum frequency of 10 and 6 frequency bands. After that, the features go through 3 HNP or NP layers with ReLU activation functions. Then, an average pooling is applied, and the output is flattened, and the resulting vector is passed through an MLP with two hidden layers, each containing 1000 units and ReLU activations. Finally, the output is linearly projected to a scalar. For the MNIST dataset, there is an additional Channel Dropout layer after the ReLU activation of each HNP layer and a Dropout layer after the ReLU activation of each MLP layer, both with a dropout rate of 0.1. We use the Binary Cross Entropy (BCE) loss function and train the model for 200,000 steps, which takes 1 hour and 35

minutes on an A100 GPU. For the baseline models, we follow the same architecture in [71], with minor modifications to the model hidden dimension, reducing it from 512 to 256 to avoid overfitting. We use a hidden dimension of 256 for all baseline models and our base model. The number of parameters of all models can be found in Table 21

D.5 Weight space style editing

Dataset. We use the same INRs dataset as used for classification task, which has the size of train, validation and test set described in Table 19.

Implementation details. In these experiments, our general architecture includes 2 Monomial-NFN layers with 16 hidden dimensions. The architecture then follows the same design as the NP model in [71], where a Gaussian Fourier Transformation with a mapping size of 256 is applied. After that, the features go through IOSinusoidalEncoding and then through 3 NP layers, each with 128 hidden dimensions and ReLU activation. Finally, the output goes through an NP layer to project into a scalar and a LearnedScale layer described in the Appendix of [71]. We use the Binary Cross Entropy (BCE) loss function and train the model for 50,000 steps, which takes 35 minutes on an A100 GPU. For the baseline models, we keep the same settings as the official implementation. Specifically, the HNP or NP model will have 3 layers, each with 128 hidden dimensions, followed by a ReLU activation. An NFN of the same type will be applied to map the output to 1 dimension and pass it through a LearnedScale layer. The number of parameters of all models can be found in Table 22. The detailed hyperparameters for our model can be found in Table 23.

Dilate (MNIST)

Contrast (CIFAR-10)

q2835466

Ground truth

2

3

Original INR

q2

8

MLP

9283

HNP

C

928354

NP

92835l

Monomial-NF

D.6 Ablation Regarding Design Choices

We provide the ablation study on the choice of architecture for the task Predict CNN Generalization on ReLU subset in Table 24. We denote:

Monomial Equivariant Functional Layer (Ours): MNF

Activation: ReLU

Scaling Invariant and Permutation Equivariant Layer (Ours): Norm

Hidden Neuron Permutation Invariant Layer (in [71]): HNP

Permutation Invariant Layer: Avg

Multilayer Perceptron: MLP

Among these designs, the architecture incorporating three layers of Monomial-NFN with ReLU activation achieves the best performance.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction are clearly stated in the Contribution in the Section 1. These claims accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results in the paper are given together with the full set of assumptions and complete/correct proofs (See Appendix C.2 and Appendix C.1).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the experiment details in the Implementation details section in the Appendix D of our manuscript. We also provide the source code so that the results in the paper can be easily reproduced.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the source code in the supplementary resources with detailed guide to run so that the results in the paper can be easily reproduced. We verify our proposed methods using public benchmarks (See the Section 6 in our manuscript)

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details necessary to understand the results in the Implementation details section in the Appendix D of our manuscript.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars suitably and correctly defined of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources for all experiments in our Implementation details in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impacts in Appendix ?? .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the githubs we use and the baselines we compare with in the Implementation details part in Appendix D of our manuscript.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the important field of learning over weight spaces, where neural networks process other neural networks. Previous research has highlighted the importance of designing equivariant architectures that account for the symmetries of the input neural network, with a primary focus on permutation symmetries. However, previous literature did not account for all symmetries of the input NNs, particularly the weight scaling symmetries of ReLU networks and the weight sign flipping symmetries of sin or Tanh networks. This paper naturally extends previous approaches to account for these activation-based symmetries. \nThis paper first formalizes the group of symmetries that includes both neuron permutations and scaling or sign-flipping transformations using monomial matrices. Next, it proposes a novel architecture for weight space networks that are equivariant to groups of monomial matrices. The new architecture is more efficient in terms of the number of trainable parameters compared to baseline weight space networks (which are only permutation equivariant).\n\nSTRENGTHS:\n1. The paper is mostly well-written and well-structured.\n2. The paper addresses the important and timely problem of learning in deep weight spaces, presenting a novel architecture that extends previous permutation equivariant networks to also account for scale/sign-flipping symmetries.\n3. The proposed architecture is parameter efficient compared to baseline weight space networks.\n\nWEAKNESSES:\nMy main concern is the limited empirical evaluation and missing natural baselines. Also, the presented empirical results, mostly show marginal to no improvement over the limited baseline methods evaluated. The insufficient empirical study of this paper significantly damages the paper’s contribution. Given the current state of the learning over weight spaces literature, I would expect a more diverse, challenging, and comprehensive empirical evaluation.\n\n1. The main text provides very few details on the construction of G-equivariant layers. I suggest the authors to provide at least one concrete example for mapping between a subspace of U, for example, mapping between some bias $b_i$ to a bias $b’_j$. \n2. Since the method is built over NFN, it is limited in the sense that each monomial-NFN can process a specific input architecture. Building over or extending the work to GNN-based weight space networks will allow [1,2] the processing of diverse input architectures.\n3. Missing weight-space baselines, like GNN-based models [1,2], DWSNets [3] (which, while mathematically equiv. to NFNs, obtains better empirical performance, see e.g., [6]) and NFT [4]. \n4. Another important missing natural baseline is to use a permutation equivariant baseline like DWS/NFN or the GNN-based models together with scaling/sign-flipping data augmentations as in [5].\n5. Some evaluation and comparison of runtime and memory consumption w.r.t. baselines would be beneficial.\n6. Also, adding some ablation regarding design choices would be beneficial.\n\nReferences\n\n[1] Graph neural networks for learning equivariant representations of neural networks, ICLR 2024.\n\n[2] Graph metanetworks for processing diverse neural architectures, ICLR 2024.\n\n[3] Equivariant Architectures for Learning in Deep Weight Spaces, ICML 2023.\n\n[4] Neural Functional Transformers, NeurIPS 2024.\n\n[5] Improved Generalization of Weight Space Networks via Augmentations, ICML 2024.\n\n[6] Neural Processing of Tri-Plane Hybrid Neural Fields, ICLR 2024.\n\nQUESTIONS:\n1. Is it possible to use the proposed method together with normalization layers?\n2. How easy would it be to extend the method to other activation functions?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present manuscript concerns the design of Neural Networks capable of processing the weights and biases of other Neural Networks, particularly Fully Connected and Convolutional NNs (known in the literature as *weight space networks, neural functional networks or metanetworks*). Previous works considered only hidden-neuron-permutation symmetries of weights/biases that preserve the function of the NN. In contrast, the authors highlight the existence of symmetries induced by certain activation functions (ReLU – positive scaling symmetries and sine/tanh – sign symmetries – Eq. (20)), following the works of Godfrey et al., NeurIPS’22 and Wood and Shawe-Taylor, Discr. Appl. Math.’96, where in certain cases are mentioned to be maximal (i.e. the only symmetries that preserve the NN function) as proved in the past. Combining these symmetries with permutations leads to the so-called “monomial-matrix groups”. Based on this background, the authors design monomial-matrix equivariant NNs, following the classical NN design strategy: linear layers interleaved with non-linearities. To characterise the former, they identify the weight-sharing pattern that equivariance imposes by solving a system of weight constraints for both the symmetries above (thus fully characterising linear layers), while for the former they use the same non-linearities as the activation functions at hand. Finally, they propose a method for monomial-matrix *invariant* layer design, which is combined with equivariant layers to yield an end-to-end invariant NN. The method is experimentally tested in various tasks: CNN generalisation prediction and Implicit Neural Representation (INR) classification and editing, showing competitive performance against permutation-symmetry-only baselines and a significant reduction in the number of learnable parameters.\n\nSTRENGTHS:\n**Significance/Impact.** The topic of NN processing has been steadily gaining traction in the last year and has the potential to provide significant advantages in various applications such as meta-learning and processing signals of arbitrary nature (encoded into INRs) under a unifying framework. Therefore, improving computational efficiency and incorporating new inductive biases, as done in this work, is an important step towards advancing and popularising the field. \n\n**Presentation**. The paper is in general well-presented, with appropriately chosen notations and clear descriptions of the background concepts involved and the innovations proposed.\n\n**Novelty**. \n   - *Studied problem*. The symmetries discussed in this work, although mentioned in the literature, have not been approached so far in the context of weight space networks. \n   - *Methodology*: This work introduces novel layers that are equivariant or invariant to a group that has been underexplored (monomial matrices).\n   - *Theory*. Additionally, following traditional weight-sharing proof strategies, the layers are characterized as the only linear layers that have the equivariance property, while Remark 4.5. borrows results from relevant papers (that might be not widely known to the community) to highlight the cases where the studied symmetries are the maximally function-preserving ones (however some results are missing – see weaknesses). \n\n**Quality/Execution**. The paper is well-motivated, provides a comprehensive background discussion, follows a rigorous and well-established methodology to design monomial-matrix group equivariant/invariant layers and provides adequate experimental comparisons, including in regimes where previous works fail.\n\n**Computational efficiency**. The proposed method leads to a significant reduction in the number of parameters, a property which is particularly useful for large NNs (a typical use case in modern deep learning).\n\nWEAKNESSES:\n**Limited expressivity (possibly reflected in the experimental results?)**\n- Although characterising the linear layers by solving the weight sharing constraints is a fairly general and rigorous technique for equivariant layer design, I have the impression that the resulting weight sharing, in this case, is severe. I.e. due to the large size of the group considered, the resulting layers seem weak in terms of expressivity. For example, in Eq. (22) all hidden layer weights and bias updates are calculated by linearly processing each element individually, i.e. a weight corresponding to an edge between two neurons will be updated based only on its previous value, ignoring other edges across the same or other layers.\n- Additionally, the activation functions that can be used are quite limited to preserve equivariance (the authors mention that they use the same activation as in the NN that is being processed).\n- Although these design choices are necessary for the current construction, this probably will not be the case for a different one, i.e. by directly designing non-linear layers. In other words, I am concerned that working with the standard NN pattern (linear layers interleaved with non-linearities) might be too limiting for this family of symmetries.\n- I am also wondering if limitations in expressivity are induced by the choice of the invariant layers. Could the authors elaborate on this?\n- I do not consider the above as grounds for rejection, since I believe that even incorporating these symmetries into NFNs and the linear layer characterisation are sufficient contributions. However, they seem like important limitations, which I suspect are reflected in the experimental results, and therefore should be highlighted by the authors.\n\n**Related work and existing results**.  \n- Given that the field is relatively new, I think that the authors should devote more space to a more detailed literature review, e.g. describing the weight symmetries that have been discovered more thoroughly and discussing/comparing the weight space networks that have been proposed so far. Currently, most methods are cited, but not adequately explained. I understand that space might not allow this, but at least adding an extended related work section in the appendix would help.\n- Additionally, the following very related works are missing:  (1) In the topic of weight space networks:\n   - Universal Neural Functionals, Zhou et al., arxiv’24\n   - Graph metanetworks for processing diverse neural architectures, Lim et al., ICLR’23\n- (2) In the topic of weight space symmetries (conditions for maximality):\n   - Reverse-engineering deep ReLU networks, Rolnick et al., ICML’20\n   - Hidden Symmetries of ReLU Networks, Grigsby et al., ICML’23\n- As far as I understand, the works of Chen et al., Neur. Comp.’93, Fefferman et al., NIPS’93 characterise maximal weight space symmetries only for the tanh activation and not for sine, as the authors mention L208 – in case this is true, the statement in Remark 4.5. should be rectified.\n\n**Extensibility**. Judging by the derivation of the weight-sharing patterns, it appears that it is not straightforward to extend this approach to symmetries induced by other activation functions (e.g. some are discussed in Godfrey et al.). Could the authors discuss this (and if true include it in their discussion about limitations?).\n\nQUESTIONS:\n- As far as I know, Proposition 3.4. and Proposition 4.4 follow from the characterisation of intertwiner groups done by Godfrey et al., NeurIPS’22 (e.g. see Theorem E.14 and Proposition 3.4), but the authors provide alternative proofs. In case these proofs are of independent technical interest, I would recommend that the authors briefly mention a proof sketch in the main paper and the differences with Godfrey et al. (this would also help in properly accrediting this prior work). \n- L210: The authors mention “It is natural to ask whether the group $G$ is still maximal in the other case”. What do they mean by the “other case”?\n- Perhaps an illustrative example of the resulting weight-sharing would help. Or maybe give some intuition on the resulting layers?\n- Maybe adding the exact number of parameters along with the performance in the tables would help in grasping the actual reduction. \n\n**Minor**.\n- L126-L128 are unclear. Perhaps describe in more detail the notation $\\text{Aut}(\\Delta_n)$ and the terms conjugation, semi-direct product etc. (depending on the importance of these statements).\n- L242: hyperparams --> Perhaps you mean learnable params?\n\nLIMITATIONS:\nSome of the limitations of this work are mentioned in the text (last paragraph of Section 5 and conclusion section). However, I think that some have not been adequately discussed (see weaknesses). I would recommend adding a separate section for this purpose.\n\nI do not foresee any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the extension of permutation equivariant neural functionals to accommodate the monomial group, which is a generalization of the permutation group. This extension leads to a new class of NFN called monomial NFN that can also handle the scaling symmetry of positively homogenous activation (RELU) and sign-flipping symmetry of activations such as tanh. The paper then characterizes the subset of all monomial matrices that are either *preserved* by either ReLU or sin/tanh ($\\sigma A(x) = A(\\sigma x)$). The paper then proceeds to construct group $G_\\mathcal{U}$ which is the product of all monomial groups that act on a weight space object $U$, and two subgroups of $G_\\mathcal{U}$ under which the models with ReLU or sin / tanh are invariant. Finally, the paper constructs an affine weight space layer that is $G$-equivariant and also $G$-invariant weight space layers, which are more parameter efficient than prior works. Generalization prediction experiments show that the proposed layer performs better than prior works under standard conditions and significantly outperforms when the models are perturbed by scale, corroborating the statement made by the paper. On INR classification and editing tasks, monomial-NFNs are competitive or better than prior works.\n\nSTRENGTHS:\nOverall, this is a well-written and technically solid paper with a novel contribution to the neural functional literature. The theoretical results seem sound. While I did not thoroughly check every proof detail, Figure 1 demonstrates that the constructed layer is indeed equivariant to the scale of the weights. The other experimental results are also quite compelling. The parameter saving could also potentially be a big upside of the monomial-NFN which can bring better generalization.\n\nWEAKNESSES:\nI don't see major weaknesses in the paper. Perhaps one shortcoming is that the majority of performance improvement comes when the weights are scaled, which is not a very natural perturbation in reality. On the other hand, many tasks that cannot be \"solved\" by previous NFNs remain difficult for monomial NFNs (e.g., CIFAR10 classification), which might indicate some fundamental limitation of this line of work.\n\nQUESTIONS:\n1. How do the computation cost and memory compare to NFNs? In my opinion, the issue with NFNs' big parameter count is that it's very memory-intensive. While monomial-NFN saves parameters, the benefit is perhaps less significant if it doesn't come with a computational advantage.\n\nLIMITATIONS:\nAuthors have adequately discussed the limitation though it would be good to include some computation cost analysis.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Paper improves neural functional networks by taking into account weight scaling properties of ReLU networks and weight sign flipping symmetries of sin or Tanh networks.\n\n- Monomial matrices are used to represent these symmetries, both permutation and scale/sign-flipping transformations.\n\n- Proposed model has fewer trainable (independent) parameters compared to the original NFN family of models.\n\nSTRENGTHS:\n- Proposes a principled way to incorporate activation functions in neural network representations for MLPs and CNNs.\n\nWEAKNESSES:\n- No discussion/comparison with previous works on the subject [1,2,3,5].\n\n- No discussion of extensions to architectures with branches/ transformers?\n\n- Not the first to consider activation functions in representing neural network weights as claimed. In [1] and [2], activation functions are encoded as nodes in a graph.\n\n- The performance gain over the baselines is minimal. Perhaps this suggests that the role of activation function encoding for this task is minimal as shown in the original neural functional works and those of [1, 2, 3, 4].\n\n- There is a focus on very specific activation functions (relu, sin, tanh), admittedly, these are common in many architectures, but the authors do not provide any discussion of how the proposed method can be applied to other activation functions.\n\nQUESTIONS:\n- Minimal equivariance is the goal for this task as equivariance is generally easy to achieve. How does the proposed model satisfy minimal equivariance while ignoring permutations that are not functionally equivarant?\n\n- Can the model handle a mixed modelzoo with different activation functions? Note that [1] can handle this case and [3] already demonstrates that this is possible. Per my understanding, the proposed model will need different instantiations of the model for each modelzoo with a different activation function.\n\n- How are the input/output layers of the considered networks handled in the proposed framework? Note that these layers do not follow the permutation symmetries of MLPs(only the hidden layers do).\n\n## References\n\n[1] Lim, Derek, et al. \"Graph metanetworks for processing diverse neural architectures.\" arXiv preprint arXiv:2312.04501 (2023).\n\n[2] Kofinas, Miltiadis, et al. \"Graph neural networks for learning equivariant representations of neural networks.\" arXiv preprint arXiv:2403.12143 (2024).\n\n[3] Andreis, Bruno, Soro Bedionita, and Sung Ju Hwang. \"Set-based neural network encoding.\" arXiv preprint arXiv:2305.16625 (2023).\n\n[4] Unterthiner, Thomas, et al. \"Predicting neural network accuracy from weights.\" arXiv preprint arXiv:2002.11448 (2020).\n\n[5] Zhou, Allan, Chelsea Finn, and James Harrison. \"Universal neural functionals.\" arXiv preprint arXiv:2402.05232 (2024).\n\nLIMITATIONS:\nWhile limitations are not discussed, i do not see any immediate negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}