SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A.4.4 Additional Comparison to Group Averaging
In this section, we provide additional analysis of our approach in comparison to sampling-based group averaging in terms of sample variance and convergence. We use the experimental setup of EXP-classify (Section 3.1;  $S_{n}$  invariance) and experiment with MLP-PS and MLP-GA.
We first analyze whether using the equivariant distribution  $p_{\omega}(g|\mathbf{x})$  for symmetrization offers a lower variance estimation, i.e., a better sample efficiency, compared to group averaging with  $\mathrm{Unif}(G)$ . This supplements the results in Figure 2 that  $p_{\omega}(g|\mathbf{x})$  learns to produce low-variance permutations compared to  $\mathrm{Unif}(G)$ . Specifically, we fix a randomly initialized MLP  $f_{\theta}$  and symmetrize it using our approach and group averaging. We then measure (1) the sample variance of the unbiased estimator
Figure 7: Sample variance of output  $g \cdot f_{\theta}(g^{-1} \cdot \mathbf{x})$  (left) and loss  $\mathcal{L}(\mathbf{y}, g \cdot f_{\theta}(g^{-1} \cdot \mathbf{x}))$  (right) of an identical MLP  $f_{\theta}$  symmetrized by equivariant distribution (PS) and uniform distribution (GA).
Figure 8: Norm of full gradient over training epochs with respect to the parameters of an identically initialized MLP symmetrized by equivariant distribution (PS) and uniform distribution (GA).
$g\cdot f_{\theta}(g^{-1}\cdot \mathbf{x})$  of the symmetrized function (Eq. (1) and Eq. (4)), and (2) the sample variance of the estimated task loss  $\mathcal{L}(\mathbf{y},g\cdot f_{\theta}(g^{-1}\cdot \mathbf{x}))$  where  $\mathcal{L}$  is binary cross entropy. All measurements are repeated 100 times and averaged over the inputs and labels  $(\mathbf{x},\mathbf{y})$  of the validation dataset. The results are in Figure 7, showing that symmetrization with equivariant distribution  $p_{\omega}(g|\mathbf{x})$  consistently offers a lower variance estimation than group averaging across inference sample sizes.
In addition, we analyze whether the equivariant distribution  $p_{\omega}(g|\mathbf{x})$  for symmetrization offers more stable gradients for the base function  $f_{\theta}$  during training compared to group averaging, as conjectured in Section 2. For this, we fix a randomly initialized MLP  $f_{\theta}$  and symmetrize it using our approach and group averaging. For every few training epochs, we measure the full gradient of the task loss over the entire training dataset with respect to the parameters of the base MLP  $f_{\theta}$ . This averages out the variance from individual data points and provides the net direction of the gradient on the base function offered by  $p_{\omega}(g|\mathbf{x})$  or  $\mathrm{Unif}(G)$ . The results are in Figure 8, showing that that symmetrization with equivariant distribution  $p_{\omega}(g|\mathbf{x})$  offers a consistently larger magnitude of the net gradient, while group averaging with  $\mathrm{Unif}(G)$  leads to near-zero net gradients. This indicates, for  $\mathrm{Unif}(G)$ , the gradients from each training data instances are oriented in a largely divergent manner and therefore the training signal is collectively not very informative, while using  $p_{\omega}(g|\mathbf{x})$  for symmetrization leads to more consistent gradient across training data instances, i.e., it offers a more stable training signal.
A.4.5 Additional Comparison to Canonicalization
In this section, we provide additional analysis of our approach in comparison to canonicalization [41] that uses a single group element  $g$  from an equivariant canonicalizer  $C_{\omega} : \mathbf{x} \mapsto \rho(g)$ . The main claim is that there always exist certain inputs that canonicalization fails to guarantee exact  $G$  equivariance, while our approach guarantees equivariance for all inputs in expectation as in Theorem 1.
(a) Inner-symmetric nodes
(b) MLP-PS (Ours)
(c) MLP-Canonical.
Figure 9: A graph  $\mathbf{x}$  with stabilizer subgroup  $G_{\mathbf{x}}\cong \mathrm{S}_6$
(a) Inner-symmetric nodes
(a) Inner-symmetric nodes
Figure 11: A graph  $\mathbf{x}$  with stabilizer subgroup  $G_{\mathbf{x}}\cong \mathrm{D}_4$
Figure 10: A graph  $\mathbf{x}$  with stabilizer subgroup  $G_{\mathbf{x}} \cong \mathrm{S}_3 \times \mathrm{S}_2$ .
(c) MLP-Canonical.
More specifically, let us recall the definition of  $G$  equivariant canonicalizer from [41]. A canonicalizer  $C_{\omega}$  is  $G$  equivariant if  $C_{\omega}(g \cdot \mathbf{x}) = \rho(g)C_{\omega}(\mathbf{x})$  for all  $g \in G$  and  $\mathbf{x} \in \mathcal{X}$ . Consider an input  $\mathbf{x}$  which has a non-trivial stabilizer  $G_{\mathbf{x}} = h \in G|h \cdot \mathbf{x} = \mathbf{x}$ , i.e., has inner symmetries. It can be shown that equivariant canonicalizers are ill-defined for these inputs. To see this, let  $g_1 = gh_1$  and  $g_2 = gh_2$  for some  $g \in G$  and any  $h_1, h_2 \in G_{\mathbf{x}}$  where  $h_1 \neq h_2$ . Then we have  $C_{\omega}(g_1 \cdot \mathbf{x}) = C_{\omega}(gh_1 \cdot \mathbf{x}) = C_{\omega}(g \cdot \mathbf{x}) = C_{\omega}(gh_2 \cdot \mathbf{x}) = C_{\omega}(g_2 \cdot \mathbf{x})$ , implying that  $\rho(g_1)C_{\omega}(\mathbf{x}) = \rho(g_2)C_{\omega}(\mathbf{x})$ . Since  $g_1 \neq g_2$ , this contradicts the group axiom, and thus an equivariant canonicalizer cannot exist for inputs with non-trivial inner-symmetries. To handle all inputs, canonicalization [41] adopts relaxed equivariance: a canonicalizer  $C_{\omega}$  satisfies relaxed equivariance if  $C_{\omega}(g \cdot \mathbf{x}) = \rho(gh)C_{\omega}(\mathbf{x})$  up to arbitrary action from the stabilizer  $h \in G_{\mathbf{x}}$ . As a result, the symmetrization  $\phi_{\theta,\omega}(\mathbf{x}) = g \cdot f_{\theta}(g^{-1} \cdot \mathbf{x})$  performed using a relaxed canonicalizer  $C_{\omega}$  only guarantees relaxed equivariance  $\phi_{\theta,\omega}(g \cdot \mathbf{x}) = gh \cdot \phi_{\theta,\omega}(\mathbf{x})$  up to arbitrary action from the stabilizer  $h \in G_{\mathbf{x}}$  (Theorem A.2 of [41]). In other words, canonicalization does not guarantee equivariance for inputs with inner symmetries.
To visually demonstrate this, we perform a minimal experiment using several graphs  $\mathbf{x}$  with nontrivial stabilizers  $G_{\mathbf{x}}$ , i.e., inner symmetries, taken from [90]. We fix a randomly initialized MLP
$f_{\theta}:\mathbb{R}^{n\times n}\to \mathbb{R}^n$  and symmetrize it using our approach and canonicalization. When symmetrized, the MLP is expected to provide scalar embedding of each node, which we color-code for visualization. The results are in Figures 9, 10, and 11. For each graph, we illustrate three panels: the leftmost one illustrates the color-coding of the inner symmetry of nodes (automorphism), the middle one illustrates node embedding from MLP-PS, and the rightmost one illustrates embedding from MLP-Canonical. If a method is  $G$  equivariant, it is expected to give identical embeddings for automorphic nodes, since an equivariant model cannot distinguish them in principle [88]. As in the Figures 9, 10, and 11, in the presence of inner symmetry (left panels), MLP with probabilistic symmetrization (middle panels) achieves  $G$  equivariance and produces close embeddings for automorphic nodes. However, the same MLP with canonicalization produces relatively unstructured embeddings (right panels). The result illustrates a potential advantage of probabilistic symmetrization over canonicalization when learning data with inner symmetries, which is often found in applications such as molecular graphs [60].
A.5 Limitations and Broader Impacts (Continued from Section 4)
While the equivariance, universality, simplicity, and scalability of our approach offers a potential for positive impact for deep learning for chemistry, biology, physics, and mathematics, it also has limitations and potential negative impacts. The main limitation of our work is that it trades off certain desirable traits in equivariant deep learning in favor of achieving architecture agnostic equivariance. For example, (1) our approach is less interpretable compared to equivariant architectures due to less structured computations in the base model, (2) our approach is presumably less parameter and data efficient compared to equivariant architectures due to less imposed prior knowledge on parameterization, and (3) our approach is expected to be challenged when input size generalization is required, partially because the maximum input size has to be specified in advance. Another genuine weakness compared to canonicalization is that, our method is stochastic and therefore incurs  $\mathcal{O}(N)$  cost when using  $N$  samples for estimation. These limitations might lead to potential negative environmental impacts, since less interpretability and lower efficiency implies higher reliance on larger models with more computation cost. We acknowledge the aforementioned limitations and impacts of our work, and will make effort to address them in follow-up research. For example, we believe data efficiency of our approach could improve with pretrained knowledge transfer from other domains, as it would impose a strong prior on the hypothesis space and may work similarly to architectural priors that benefit data efficiency. Also, for the sampling cost, since the sampling is completely parallelizable and analogous to using a larger batch size, we believe it can be overcome to some degree by leveraging parallel computing techniques developed for scaling batch size.