SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
z (U) _ {j} ^ {(i)} := \sum_ {s = 1} ^ {L} \sum_ {p = 1} ^ {n _ {s}} \sum_ {q = 1} ^ {n _ {s - 1}} \varphi_ {s p q} ^ {i j} W _ {p q} ^ {(s)} + \sum_ {s = 1} ^ {L} \sum_ {p = 1} ^ {n _ {s}} \psi_ {s p} ^ {i j} v _ {p} ^ {(s)}. \tag {6}
$$
We introduce the notation  $U_{\alpha}$  to identify individual entries of  $U$ . Here  $\alpha$  is a tuple of length two or three, for indexing into either a weight or bias. We denote the space of valid index tuples of  $\mathcal{W}$  and  $\mathcal{V}$  by  $\mathbb{W}$  and  $\mathbb{V}$ , respectively, and define  $\mathbb{U} := \mathbb{W} \cup \mathbb{V}$  as the combined index space of  $\mathcal{U}$ . For example, if  $\alpha = (i,j,k) \in \mathbb{W}$ , then  $U_{\alpha} = W_{jk}^{(i)}$ .
We can then define the index space  $\mathbb{I} \coloneqq \mathbb{U} \times \mathbb{U}$  for parameters  $\theta \in \Theta$ . We use  $\theta_{\beta}^{\alpha}$  to index an entry of  $\theta$  with upper and lower indices  $[\alpha, \beta] \in \mathbb{I}$ . For example, if  $\alpha = (i,j,k) \in \mathbb{W}$  and  $\beta = (s,p,q) \in \mathbb{W}$ , we have  $\theta_{\beta}^{\alpha} = \vartheta_{spq}^{ijk}$ .
By flattening  $U$  into a vector, we can think of  $\theta$  as a matrix with indices  $\alpha$  and  $\beta$ . Eq. 4 can be rewritten in the flattened form:
$$
T (U; \theta) _ {\alpha} = \sum_ {\beta} \theta_ {\beta} ^ {\alpha} U _ {\beta}. \tag {7}
$$
In the flattened representation, we can write the action of  $\sigma \in S$  on  $\operatorname{vec}(U)$  by a permutation matrix  $P_{\sigma}$ . Equivariance requires that  $P_{\sigma} \theta \operatorname{vec}(U) = \theta P_{\sigma} \operatorname{vec}(U)$  for any  $\sigma \in S$ . Since the input  $U$  can be anything, we get the following constraint on  $\theta$ :
$$
P _ {\sigma} \theta = \theta P _ {\sigma}, \quad \forall \sigma \in \mathcal {S}. \tag {8}
$$
We can re-express the action of  $S$  on  $\mathcal{U}$  (Eq. 1) as an action on the index space  $\mathbb{U}$ . For  $\alpha \in \mathbb{U}$ , we write  $\sigma(\alpha)$  which is defined:
$$
\sigma (i, j, k) = (i, \sigma_ {i} (j), \sigma_ {i - 1} (k)) \quad (i, j, k) \in \mathbb {W} \tag {9}
$$
$$
\sigma (i, j) = (i, \sigma_ {i} (j)) \quad (i, j) \in \mathbb {V}, \tag {10}
$$
for any  $\sigma \in S$ . We extend this definition into an action of  $S$  on  $\mathbb{I}$ :
$$
\sigma [ \alpha , \beta ] := [ \sigma \alpha , \sigma \beta ], \quad [ \alpha , \beta ] \in \mathbb {U} \times \mathbb {U}. \tag {11}
$$
We can now rewrite the equivariance constraint in Eq. 8 using indices  $\alpha, \beta$ :
$$
\left[ P _ {\sigma} \theta \right] _ {\beta} ^ {\alpha} = \theta_ {\beta} ^ {\sigma^ {- 1} (\alpha)} = \theta_ {\sigma (\beta)} ^ {\alpha} = \left[ \theta P _ {\sigma} \right] _ {\beta} ^ {\alpha}. \tag {12}
$$
By relabeling  $\alpha \gets \sigma^{-1}(\alpha)$ , we can rewrite this condition  $\theta_{\beta}^{\alpha} = \theta_{\sigma (\beta)}^{\sigma (\alpha)}$ . Hence, equivariance is equivalent to requiring that  $\theta$  share parameters within orbits under the action of  $S$  on its indices  $\alpha ,\beta$  [54, Prop 3.1]. We characterize these orbits in Sec. B.2.1, yielding Table 7 which gives the parameter sharing as a system of constraints on the indices of  $\theta$ . These constraints characterize the space of  $S$ -equivariant  $T(\cdot ,\theta)$ . In Sec. B.2.2, we conclude the proof by showing that under parameter sharing,  $T(\cdot ,\theta)$  is equivalent to Eq. 3.
B.2.1 Parameter orbits and sharing
We now partition the parameters of  $\theta$  into orbits under the  $S$ -action on its index space, in order to generate the parameter sharing constraints given in Table 7. The index space of  $\theta$  is  $\mathbb{I} = \mathbb{U} \times \mathbb{U}$ . There are four subsets of  $\mathbb{I}$ :
1.  $\mathbb{I}^{WW} := \mathbb{W} \times \mathbb{W}$ : Contains  $[\alpha, \beta] = [(i,j,k),(s,p,q)]$ , indexing parameters  $\vartheta_{spq}^{ijk}$ .
2.  $\mathbb{I}^{WV} \coloneqq \mathbb{W} \times \mathbb{V}$ : Contains  $[\alpha, \beta] = [(i,j,k),(s,p)]$ , indexing parameters  $\phi_{sp}^{ijk}$ .
3.  $\mathbb{I}^{VW} \coloneqq \mathbb{V} \times \mathbb{W}$ : Contains  $[\alpha, \beta] = [(i,j), (s,p,q)]$ , indexing parameters  $\varphi_{spq}^{ij}$ .
4.  $\mathbb{I}^{VV} \coloneqq \mathbb{V} \times \mathbb{V}$ : Contains  $[\alpha, \beta] = [(i,j),(s,p)]$ , indexing parameters  $\psi_{sp}^{ij}$ .
Consider the block of indices  $\mathbb{I}^{WW} = \mathbb{W}\times \mathbb{W}$ , containing  $[\alpha ,\beta ] = [(i,j,k),(s,p,q)]$  indexing parameters  $\vartheta_{\beta}^{\alpha}$ . Since the  $S$ -action never changes the layer indices  $(i,s)$ , we can independently consider orbits within sub-blocks of indices  $\mathbb{I}_{i,s}^{WW} = \{[(i,j,k),(s,p,q)]\mid \forall j,k,p,q\}$ . The number of orbits within each sub-block  $\mathbb{I}_{i,s}^{WW}$  depends on the relationship between the layer indices  $i$  and  $s$ : they are either the same layer ( $s = i$ ), they are adjacent ( $s = i - 1$  or  $s = i + 1$ ), or they are non-adjacent ( $s\notin \{i - 1,i,i + 1\}$ ). We now analyze the orbits of sub-blocks for a few cases.
If  $s = i - 1$ , then choose any two indices  $[\alpha^{(1)},\beta^{(1)}],[\alpha^{(2)},\beta^{(2)}]\in \mathbb{I}_{i,s}^{WW}$  where the first satisfies  $p\neq k$  and the second satisfies  $p = k$ . Then the orbits of each index are:
$$
\operatorname {O r b i t} \left(\left[ \alpha^ {(1)}, \beta^ {(1)} \right]\right) = \left\{\left[ (i, j, k), (s, p, q) \right] \mid \forall j, k, p, q: p \neq k \right\} \tag {13}
$$
$$
\operatorname {O r b i t} \left(\left[ \alpha^ {(2)}, \beta^ {(2)} \right]\right) = \left\{\left[ (i, j, k), (s, p, q) \right] \mid \forall j, k, p, q: p = k \right\}. \tag {14}
$$
We see that these two orbits actually partition the entire sub-block of indices  $\mathbb{I}_{i,s}^{WW}$ , with each orbit characterized by whether or not  $p = k$ . We introduce the parameters  $a_{\vartheta}^{i,i - 1}$  (for the first orbit) and  $b_{\vartheta}^{i,i - 1}$  (for the second orbit). Under equivariant parameter sharing, all parameters of  $\vartheta$  corresponding  $\mathbb{I}_{i,s}^{WW}$  are equal to either  $a_{\vartheta}^{i,i - 1}$  or  $b_{\vartheta}^{i,i - 1}$ , depending on whether  $p = k$  or  $p \neq k$ .
If  $s = i + 1$ , we instead choose any two indices where the first satisfies  $j \neq q$  and the second satisfies  $j = q$ . Then the sub-block of indices  $\mathbb{I}_{i,s}^{WW}$  is again partitioned into two orbits:
$$
\{[ (i, j, k), (s, p, q) ] \mid \forall j, k, p, q: j \neq q \}, \text {a n d} \{[ (i, j, k), (s, p, q) ] \mid \forall j, k, p, q: j = q \} \tag {15}
$$
depending on the condition  $j = q$ . We name two parameters  $a_{\vartheta}^{i,i + 1}$  and  $c_{\vartheta}^{i,i + 1}$  for this sub-block, with one for each orbit.
We can repeat this process for sub-blocks of  $\mathbb{I}^{WW}$  where  $i = s$  and  $s \notin \{i - 1, i, i + 1\}$ , as well as for the other three blocks of  $\mathbb{I}$ . Table 7 shows the complete parameter sharing constraints on  $\theta$  resulting from partitioning all possible sub-blocks into orbits.
Number of parameters. We also note that every layer pair  $(i,s)$  introduces only a constant number of parameters: the number of parameters in each cell of Table 7 has no dependence on the input, output, or hidden dimensions of  $\mathcal{U}$ . Hence the number of distinct parameters after parameter sharing simply grows with the number of layer pairs, i.e.  $O(L^2)$ .
B.2.2 Equivalence to equivariant NF-Layer definition
All that remains is to show that the map  $T(\cdot; \theta): \mathcal{U} \to \mathcal{U}$  with  $S$ -equivariant parameter sharing (Table 7) is equivalent to the NF-Layer  $H$  we defined in Eq. 3.
Consider a single term from Eq. 4 where  $s = i - 1$ . Substituting using the constraints of Table 7, we simplify:
$$
\begin{array}{l} \sum_ {p, q} \vartheta_ {i - 1, p, q} ^ {i, j, k} W _ {p q} ^ {(i - 1)} = a ^ {i, i - 1} \sum_ {q} \sum_ {k \neq p} W _ {p, q} ^ {(i - 1)} + b ^ {i, i - 1} \sum_ {q} W _ {k, q} ^ {(i - 1)} \tag {16} \\ = a ^ {i, i - 1} W _ {\star , \star} ^ {(i - 1)} + (b ^ {i, i - 1} - a ^ {i, i - 1}) W _ {k, \star} ^ {(i - 1)}. \\ \end{array}
$$
We can then reparameterize  $b^{i,i-1} \gets b^{i,i-1} - a^{i,i-1}$ , resulting in two terms that appear in Eq. 3. We can simplify every term of Eq. 4 in a similar manner using the parameter sharing of Table 7, reducing the general layer to the  $S$ -equivariant NF-Layer.
B.3 Proof of Proposition 2
Analogous to Sec. B.2, we begin with general parameterized linear maps  $T(\cdot; \theta): \mathcal{U} \to \mathbb{R}^d$  and characterize the parameter sharing implied by  $S$ -invariance. We then show that  $T(\cdot, \theta)$  under these constraints is equivalent to  $f \circ P$ , where  $f: \mathbb{R}^{2L} \to \mathbb{R}^d$  is a parameterized linear map (i.e., a fully connected layer) and  $P(U) = \left(W_{\star, \star}^{(1)}, \dots, W_{\star, \star}^{(L)}, v_{\star}^{(1)}, \dots, v_{\star}^{(L)}\right)$ . We will assume  $d = 1$  for the rest of this section, but it is straightforward to generalize our arguments to arbitrary  $d$ .
In general, we can parameterize linear maps  $\mathcal{U}\rightarrow \mathbb{R}$  by  $\theta \in \mathbb{R}^{\dim (\mathcal{U})}$ , where:
$$
T (U, \theta) = \sum \theta_ {\alpha} U _ {\alpha}, \tag {17}
$$
where  $\alpha$  is an index for either weights or biases as in Sec. B.2. The invariance constraint  $T(U) = T(\sigma U)$  implies:
$$
\theta_ {\alpha} = \theta_ {\sigma (\alpha)}, \tag {18}
$$
where  $\sigma (\alpha)$  is defined by Eq. 9.
We can now characterize the orbits of the indices of  $\theta$  under  $\mathcal{S}$ . In this case, the orbits (and resulting parameter sharing) are relatively simple. For a fixed  $i$ , all  $(i,j,k)\in \mathbb{W}$  form an orbit, and we introduce a shared parameter  $a^i$ . Similarly, for each  $i$  all  $(i,j)\in \mathbb{V}$  form an orbit, and we introduce a shared parameter  $b^{i}$  for each. We can then rewrite the layer:
$$
T (U; \theta) = \sum_ {i} \left(a ^ {i} \sum_ {j k} W _ {j k} ^ {(i)} + b ^ {i} \sum_ {j} v _ {j} ^ {(i)}\right) = \sum_ {i} a ^ {i} W _ {\star , \star} ^ {(i)} + b ^ {i} v _ {\star} ^ {(i)}. \tag {19}
$$
This is simply a linear combination of the output of  $P(U)$  parameterized by  $\{(a^i, b^i)\}$ , and can be written as the composition  $f \circ P$  where  $f$  is a fully connected layer.
C NF-Layers for the HNP setting
C.1 Equivariant NF-Layer
Because an expression for the  $\tilde{S}$ -equivariant NF-Layer analogous to Eq. 3 would be unwieldy, we instead define the layer in terms of its parameter sharing (Tables 8-11) on  $\theta$ .
We can derive HNP-equivariant parameter sharing of  $\theta$  using a similar strategy to Sec. B.2.1: we partition the index spaces  $\mathbb{I}^{WW},\mathbb{I}^{WV},\mathbb{I}^{VW},\mathbb{I}^{VV}$  into orbits under the action of  $\tilde{S}$ , and share parameters within each corresponding orbit of  $\vartheta, \phi, \varphi, \psi$ . The resulting parameter sharing is different from the NP-setting because while the action of  $\mathcal{S}$  on  $\mathcal{U}$  could permute the rows and columns of every weight and bias, the action of  $\tilde{S}$  on  $\mathcal{U}$  does not affect the columns of  $W^{(1)}$  or the rows of  $W^{(L)}, v^{(L)}$ , which correspond to input and output dimensions (respectively).
The orbits are again analyzed within sub-blocks defined by the values of the layer indices  $(i,s)$ . As with the NP setting, there are broadly four types of sub-blocks based on whether  $i = s$ ,  $s = i - 1$ ,  $s = i + 1$ , or  $s \notin \{i - 1,i,i + 1\}$ . However, there are now additional considerations based on whether  $i$  or  $s$  is an input or output layer. For example, consider the sub-block of  $\mathbb{I}^{WW}$  where  $i = s = 1$ , which we denote  $\mathbb{I}_{1,1}^{WW}$ . The action on the indices in this sub-block can be written  $\sigma[\alpha,\beta] = [(1,\sigma_1(j),k),(1,\sigma_1(p),q)]$ . Importantly, the column indices  $k,q$  are never permuted since they correspond to the input layer. We see that  $\mathbb{I}_{1,1}^{WW}$  contains two orbits for each  $k \in [[1..n_0]]$  and  $q \in [[1..n_0]]$ , with the two orbits characterized by whether or not  $j = p$ . Hence we have  $2n_0^2$  orbits and Table 8 introduces  $2n_0^2$  parameters  $\left\{a_{\vartheta}^{1,1,k,q}, b_{\vartheta}^{1,1,k,q} \mid k,q \in [[1..n_0]]\right\}$  for this sub-block of parameters.
Now consider another sub-block of  $\mathbb{I}^{WW}$  where  $1 < i = s < L$ . Now the action of  $\tilde{S}$  on indices in this sub-block can be written  $\sigma[\alpha, \beta] = [(i, \sigma_i(j), \sigma_{i-1}(k)), (i, \sigma_i(p), \sigma_{i-1}(q))]$ . Then we have a total of two orbits characterized by whether or not  $k = p$ , rather than  $2n_0^2$  orbits for the  $i = 1$  case. Tables 8-11 present the complete parameter sharing for each of  $\vartheta, \phi, \varphi, \psi$ , resulting from analyzing every possible orbit within any sub-block of  $\mathbb{I}^{WW}, \mathbb{I}^{WV}, \mathbb{I}^{VW}, \mathbb{I}^{VV}$ .
C.2 Invariant NF-Layer
While the NP-invariant NF-Layer sums over the rows and columns of every weight and bias, under HNP assumptions there is no need to sum over the columns of  $W^{(1)}$  (inputs) or the rows of  $W^{(L)}$ ,  $v^{(L)}$  (outputs). So the HNP invariant NF-Layer  $\tilde{P}:\mathcal{U}\to \mathbb{R}^{2L + n_0 + 2n_L}$  is defined:
$$
\tilde {P} (U) = \left(P (U), W _ {\star ,:} ^ {(1)}, W _ {:, \star} ^ {(L)}, v ^ {(L)}\right), \tag {20}
$$
where  $W_{\star,:}^{(1)}$  and  $W_{:, \star}^{(L)}$  denote summing over only the rows or only the columns of the matrix, respectively. Note that  $\tilde{P}$  satisfies  $\tilde{S}$ -invariance without satisfying  $S$ -invariance.
D.1 Predicting generalization
The model we use consists of three equivariant NF-Layers with 16, 16, and 5 channels respectively. We apply ReLU activations after each linear NF-Layer. The resulting weight-space features are passed
Table 8: HNP-equivariant parameter sharing on  $\vartheta \subseteq \theta$ , corresponding to the NF-Layer  $\tilde{H}:\mathcal{U}\to \mathcal{U}$ .
s=i-1	i=2	2<i<L	i=L
{a2,1,qk≠p
b2,1,qk=p	{ai,i-1k≠p
bi,i-1k=p	{aL,L-1,jk≠p
bL,L-1,jk=p
s=i	i=1	1<i<L	i=L
{a1,1,k,qj≠p
b1,1,k,qj=p	{ai,j≠p,k≠q
bj,i j=p,k≠q
cj,i j≠p,k=q
di,i j=p,k=q	{aL,L,j,pk≠q
cL,L,j,pk=q
s=i+1	i=1	1<i<L-1	i=L-1
{a1,2,kj≠q
c1,2,kj=q	{ai,2j≠q
cj,2j=q	{aL-1,L,pj≠q
cL-1,L,pj=q
other s	i=1,1<s<L	i=1,s=L	1<i<L,s=L
a1,s,k	a1,L,k,p	ai,L,p
1<i<L,s=1	i=L,s=1	i=L,1<s<L
ai,1,q	aL,1,j,q	aL,s,j
1<i<L,1<s<L
ai,s
Table 9: HNP-equivariant parameter sharing on  $\phi  \subset  \theta$  ,corresponding to the NF-Layer  $\widetilde{H} : \mathcal{U} \rightarrow  \mathcal{U}$  .
φijk
φsp
s=i-1	1<i<L	i=L
{aφi,i-1k≠p
bφi,i-1k=p	{aφL,L-1,jk≠p
bφL,L-1,jk=p
s=i	i=1	1<i<L	i=L
{aφ1,1,kj≠p
bφ1,1,kj=p	{aφi,ij≠p
bφi,ij=p	bφL,L,j,p
other s	i=1,1<s<L	i=1,s=L	1<i<L,s=L
aφ1,s,k	aφ1,L,k,p	aφi,L,p
1<i<L,1≤s<L	i=L,1≤s<L
aφi,s	aφL,s,j
Table 10: HNP-equivariant parameter sharing on  $\varphi \subset \theta$ , corresponding to the NF-Layer  $\tilde{H}:\mathcal{U}\to \mathcal{U}$ .
φijspq
s=i	i=1	1<i<L	i=L
{a1,1,q j≠p
b1,1,k j=p	{ai,i j≠p
bj,i j=p	bL,L,j,p
s=i+1	1≤i<L-1	i=L-1
{ai,i+1 j≠p
bj,i+1 j=p	{AL-1,L,p j≠p
bL-1,L,p j=p
other s	1≤i<L,s=1	1≤i<L,1<s<L	1≤i<L,s=L
ai,s,q	ai,s	ai,L,p
i=L,s=1	i=L,1<s<L
aL,1,j,q	aL,s,j
Table 11: HNP-equivariant parameter sharing on  $\psi \subset \theta$ , corresponding to the NF-Layer  $\tilde{H}:\mathcal{U}\to \mathcal{U}$ .
ψspij
s=i	1≤i<L	i=L
{ai,i j≠p
bji j=p	bL,L,j,p
other s	1≤i<L,s=L	i=L,1≤s<L	1≤i<L,1≤s<L
ai,L,p	aL,s,j	a,i,s
into an invariant NF-Layer with mean pooling. The output of the invariant NF-Layer is flattened and projected to  $\mathbb{R}^{1,000}$ . The resulting vector is then passed through an MLP with two hidden layers, each with 1,000 units and ReLU activations. The output is linearly projected to a scalar and passed through a sigmoid function. Since the output of the model can be interpreted as a probability, we train the model with binary cross-entropy with hyperparameters outlined in Table 12. The model is trained for 50 epochs with early stopping based on  $\tau$  on the validation set, which takes 1 hour on a Titan RTX GPU.
Table 12: Hyperparameters for predicting generalization on Small CNN Zoo.
Name	Values
Optimizer	Adam
Learning rate	0.001
Batch size	8
Loss	Binary cross-entropy
Epoch	50
D.2 Predicting "winning ticket" masks from initialization
Concretely, the encoder learns the posterior distribution  $q_{\theta}(Z \mid U_0, M)$  where  $Z \in \mathbb{R}^{\dim(\mathcal{U}) \times C}$  is the latent variable for the winning tickets and  $C$  is the number of latent channels. The decoder learns  $p_{\theta}(M \mid U_0, Z)$ , and both encoder and decoder are implemented using our equivariant NF-Layers. For the prior  $p(Z)$  we choose the isometric Gaussian distribution, and train using the evidence lower bound (ELBO):
$$
\mathcal {L} _ {\theta} (M, U _ {0}) = \mathbb {E} _ {z \sim q _ {\theta} (\cdot | U _ {0}, M)} \Big [ \ln p _ {\theta} (M | U _ {0}, z) \Big ] - \mathrm {D} _ {\mathrm {K L}} \Big (q _ {\theta} (\cdot | U _ {0}, M) | | p (\cdot) \Big).
$$
The initialization and sparsity mask are concatenated so the input to the encoder  $q_{\theta}$  is  $(U, M) \in \mathbb{R}^{\dim(\mathcal{U}) \times 2}$ . After the bottleneck, we concatenate the latent variables and the original mask along the channels, i.e. the decoder input is  $(U_0, Z) \in \mathbb{R}^{\dim(\mathcal{U}) \times (C + 1)}$ .
The first dataset uses three-layer MLPs with 128 hidden units trained on MNIST and the second uses CNNs with three convolution layers (128 channels) and 2 fully-connected layers trained on CIFAR-10. In each dataset, we include 400 pairs for training and hold out 50 for evaluation. The hyperparameter details are in Table 13. The encoder and decoder models contain 4 equivariant NF-Layers with 64 hidden channels within each layer. The latent variable is 5 dimensions. Training takes 5H on a Titan RTX GPU.
Table 13: Hyperparameters for predicting LTH on MNIST and CIFAR-10.
Name	Values
Optimizer	Adam
Learning rate	1 × 10-3
Batch size	[4, 8]
Epoch	200
D.3 Classifying INRs
We use SIREN [58] for our INRs of CIFAR-10, FashionMNIST, and MNIST. For the SIREN models, we used a three-layer architecture with 32 hidden neurons in each layer. We trained the SIRENs for 5,000 steps using Adam optimizer with a learning rate of  $5 \times 10^{-5}$ . Datasets were split into 45,000 training images, 5,000 validation images, and 10,000 (MNIST, CIFAR-10) or 20,000 (FashionMNIST) test images. We trained 10 copies (MNIST, FashionMNIST) or 20 copies (CIFAR-10) of SIRENs on each training image with different initializations, and a single SIREN on each validation and test image. No additional data augmentation was applied. For 3D shape classification, we adopt the same protocol introduced in [12], and we train each SIREN to fit the Unsigned Distance Function (UDF) value of points sampled around a shape. Each SIREN is composed of a single hidden layer with 128 neurons. We use Adam as an optimizer and we train for 1,000 steps.
We also trained neural functionals with three equivariant NF-Layers + ReLU activations, each with 512 channels, followed by invariant NF-Layers (mean pooling) and a three-layer MLP head with 1,000 hidden units and ReLU activation. Dropout was applied to the MLP head only. For the NFN IO-encoding, we used sinusoidal position encoding with a maximum frequency of 10 and 6 frequency bands (dimension 13). The training hyperparameters are shown in Table 14, and training took  $\sim 4\mathrm{H}$  on a Titan RTX GPU.
Table 14: Hyperparameters for classifying INRs on MNIST and CIFAR-10 using neural functionals.
Name	Values
Optimizer	Adam
Learning rate	1 × 10-4
Batch size	32
Training steps	2 × 105
MLP dropout	0.5
We also experimented with larger MLPs (4,000 and 8,000 hidden units per layer) that have parameter counts comparable to those of the NFNs, but found that it did not significantly increase test accuracy, as shown in Table 16.
D.4 Weight-space style editing
For weight-space editing, we use the same INRs as the ones used for classification but we do not augment the dataset with additional INRs. Let  $U_{i}$  be the INR weights for the  $i^{\text{th}}$  image and SIREN(x,y;U) be the output of the INR parameterized by  $U$  at coordinates  $(x,y)$ . We edit the INR
weights  $U_{i}^{\prime} = U_{i} + \gamma \cdot \mathrm{NFN}(U_{i})$ , and  $\gamma$  is a learned scalar initialized to 0.01. Letting  $f_{i}(x,y)$  be the pixel values of the ground truth edited image (obtained from image-space processing), the objective is to minimize mean squared error:
$$
\mathcal {L} (\mathrm {N F N}) = \frac {1}{N \cdot d ^ {2}} \sum_ {i = 1} ^ {N} \sum_ {x, y} ^ {d} \| \text {S I R E N} (x, y; U ^ {\prime}) - f _ {i} (x, y) \| _ {2} ^ {2}. \tag {21}
$$
Note that since the SIREN itself is differentiable, the loss can be directly backpropagated through  $U^{\prime}$  to the parameters of the NFN.
The neural functionals contain 3 equivariant NF-Layers with 128 channels, one invariant NF-Layer (mean pooling) followed by 4 linear layers with 1,000 hidden neurons. Every layer uses ReLU activation. The training hyperparameters can be found in Table 15, and training takes  $\sim 1$  hour on a Titan RTX GPU.
Table 15: Hyperparameters for weight-space style editing using neural functionals.
Name	Values
Optimizer	Adam
Learning rate	1 × 10-3
Batch size	32
Training steps	5 × 104
E Additional experiments and analysis
E.1 Analyzing train vs test accuracies in INR classification
Table 16 shows the training and test accuracies of our NFNs and other weight space methods across the benchmarks we consider in this paper. An interesting observation is that in addition to better test performance, NFN typically achieve much higher training accuracies compared to the MLPs.  $\mathrm{MLP}_{\mathrm{Aug}}$ , the MLP variant trained with permutation augmentations, especially struggles to achieve low training error. This indicates that training architectures to respect neuron permutation symmetries is extremely challenging, perhaps because these symmetry groups are extremely large [1]. Therefore, encoding equivariance or invariance via architecture design, as we do with NFNs, is much more effective.
E.2 Non-weight-space baselines for image classification
For the image and 3D shape classification tasks discussed in Sec. 3.2, we compared the neural functionals to non-weight-space baselines, namely residual networks [26] for MNIST, FashionMNIST, and CIFAR-10, and PointNet [52] for ShapeNet-10 and ScanNet-10. The results, shown in Table 16, indicate that applying state of the art architectures to pixels or point clouds is more effective than any weight space method. This may be because these architectures can leverage geometric properties of image and shape classification tasks. While neural functionals handle weight space symmetries, it remains an open problem to incorporate the geometric symmetries of common machine learning problems into weight space methods.
E.3 Ablating positional encoding
We ablate the effect of IO-encoding on the  $\mathrm{NFN_{NP}}$  architectures, in the 2D-INR classification and style editing tasks. The results, reported in Tables 17 and 18, show that IO-encoding adds a very small (sometimes negligible) boost to  $\mathrm{NFN_{NP}}$  performance, though it never hurts. Since  $\mathrm{NFN_{NP}}$  often performs as well as or better than  $\mathrm{NFN_{HNP}}$ , this indicates that even the base NP variant can solve many weight-space tasks without needing to break that symmetry.
E.4 Interpreting learned lottery ticket masks
We further analyze the behavior of  $\mathrm{NFN_{PT}}$  on lottery ticket mask prediction by plotting the mask score predicted for a given initialization value at each layer. To make the visualization clear we train