SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 16: Classification train and test accuracies (\%) on datasets of 2D INRS (MNIST, Fashion-MNIST, and CIFAR-10) and 3D INRs (ShapeNet-10 and ScanNet-10). Our equivariant NFNs outperform the MLP baselines, even when the MLPs have permutation augmentations to encourage invariance, and even when the MLPs are significantly larger to match NFN's parameter counts (MLP-4000/8000). NFNs also outperform recent non-equivariant methods developed for 3D INRs such as inr2vec [12]. ResNet-18 and PointNet operate on gridded or pointcloud signals, instead of weight space representations, and their performance is given for reference. PointNet results were obtained directly from Qi et al. [52]. Where given, uncertainties indicate standard error over three runs.
MNIST	FashionMNIST	CIFAR-10	ShapeNet-10	ScanNet-10
\(\mathbf{NFN_{HNP}}\)	Train	94.9 ± 0.579	82.3 ± 2.78	75.5 ± 0.810	100 ± 0.0	100.0 ± 0.0
Test	92.5 ± 0.071	72.7 ± 1.53	44.1 ± 0.471	86.9 ± 0.860	64.1 ± 0.572
\(\mathbf{NFN_{NP}}\)	Train	95.0 ± 0.115	81.8 ± 0.868	66.0 ± 0.694	100.0 ± 0.0	100.0 ± 0.0
Test	92.9 ± 0.218	75.6 ± 1.07	46.6 ± 0.072	88.7 ± 0.461	65.9 ± 1.10
MLP	Train	42.4 ± 2.44	44.5 ± 2.17	23.7 ± 2.39	100.0 ± 0.0	100.0 ± 0.0
Test	14.5 ± 0.035	12.5 ± 0.111	16.9 ± 0.250	25.4 ± 0.121	32.9 ± 0.351
\(\mathbf{MLP_{Aug}}\)	Train	20.5 ± 0.401	14.9 ± 1.45	19.1 ± 1.75	34.0 ± 0.0	42.7 ± 0.012
Test	21.0 ± 0.172	15.9 ± 0.181	18.9 ± 0.432	33.8 ± 0.126	45.5 ± 0.126
MLP-4000	Train	72.6 ± 1.39	-	30.4 ± 0.521	-	-
Test	15.5 ± 0.090	-	17.1 ± 0.120	-	-
\(\mathbf{MLP-4000_{Aug}}\)	Train	19.4 ± 1.39	-	20.5 ± 0.333	-	-
Test	21.1 ± 0.010	-	19.3 ± 0.325	-	-
MLP-8000	Train	77.8 ± 1.74	-	35.9 ± 0.868	-	-
Test	15.8 ± 0.014	-	17.3 ± 0.280	-	-
\(\mathbf{MLP-8000_{Aug}}\)	Train	20.3 ± 3.30	-	18.1 ± 0.347	-	-
Test	21.3 ± 0.075	-	19.6 ± 0.060	-	-
inr2vec[12]	Train	-	-	-	99.0 ± 0.0	93.8 ± 0.090
Test	-	-	-	39.1 ± 0.385	38.2 ± 0.409
ResNet-18	Test	99.21 ± 0.28	95.48 ± 0.13	93.95 ± 0.08	-	-
PointNet	Test	-	-	-	94.3	72.7
Table 17: Classification test accuracies (%) when ablating the positional encoding for implicit neural representations (INRs) of MNIST, FashionMNIST, and CIFAR-10. Uncertainties indicate standard error over three runs.
NFNNP	NFNNP (no positional encoding)
MNIST	92.9 ± 0.218	92.9 ± 0.077
CIFAR-10	46.6 ± 0.072	46.5 ± 0.160
FashionMNIST	75.6 ± 1.07	73.4 ± 0.701
Table 18: Test mean squared error when ablating the positional encoding on the Dilate (MNIST) and Contrast (CIFAR-10) implicit neural representation (INR) editing task. Uncertainties indicate standard error over three runs.
NFNNP	NFNNP (no positional encoding)
Contrast (CIFAR-10)	0.020 ± 0.000	0.020 ± 0.000
Dilate (MNIST)	0.068 ± 0.000	0.070 ± 0.001
Input Weight Value
Figure 5: Mask scores vs weight magnitude for modified  $\mathrm{NFN}_{\mathrm{PT}}$
$\mathrm{NFN_{PT}}$  on MLP mask prediction without layer norm, which can be viewed as a scalar function of the initialization  $f^{(i)}:\mathbb{R}\to \mathbb{R}$  for each layer  $i$ . Figure 5 plots, for a fixed latent value, the predicted mask score as a function of the initialization value (low mask scores are pruned, while high mask scores are not). These plots suggest that, in the MLP setting, neural functionals are learning something similar to magnitude pruning of the initialization. In our setting, this turns out to be a strong baseline for lottery ticket mask prediction: the test accuracy of models pruned with the modified network is  $95.0\%$ .
Table 19: Kendall's  $\tau$  coefficient and  $R^2$  between predicted and actual test accuracies of three- and five-layer MLPs trained on MNIST. Our equivariant neural functionals outperform the baseline from [64] which predicts generalization using only simple weight statistics as features. Uncertainties indicate standard error over five runs.
\( \mathrm{NFN}_{\mathrm{HNP}} \)	\( \mathrm{NFN}_{\mathrm{NP}} \)	STATNN
\( \tau \)	3-Layer	0.876 ± 0.003	0.859 ± 0.002	0.854 ± 0.002
5-Layer	0.871 ± 0.001	0.855 ± 0.001	0.860 ± 0.001
\( R^2 \)	3-Layer	0.957 ± 0.003	0.9424 ± 0.003	0.937 ± 0.002
5-Layer	0.956 ± 0.002	0.947 ± 0.001	0.950 ± 0.001
E.5 Predicting MLP generalization from weights
In addition to predicting generalization on the Small CNN Zoo benchmark (Section 3.1), we also construct our own datasets to evaluate predicting generalization on MLPs. Specifically, we study three- and five-layer MLPs with 128 units in each hidden layer. For each of the two architectures, we train 2,000 MLPs on MNIST with varying optimization hyperparameters, and save 10 randomly-selected checkpoints from each run to construct a dataset of 20,000 (weight, test accuracy) pairs. Runs are partitioned according to a  $90\% / 10\%$  split for training and testing.
We evaluate  $\mathrm{NFN_{HNP}}$  and  $\mathrm{NFN_{NP}}$  on this task and compare them to the STATNN baseline [64] which predicts test accuracy from hand-crafted features extracted from the weights. Table 19 shows that  $\mathrm{NFN_{NP}}$  and STATNN are broadly comparable, while  $\mathrm{NFN_{HNP}}$  consistently outperform other methods across both datasets in two measures of correlation: Kendall's tau and  $R^2$ . These results confirm that processing the raw weights with permutation equivariant neural functionals can lead to greater predictive power when assessing generalization from weights.