SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data. The METR-LA traffic dataset, [76], contains traffic information collected from 207 sensors in the highway of Los Angeles County from Mar 1st 2012 to Jun 30th 2012 [87]. We use the same traffic data normalization and  $70/10/20$  train/validation/test data split as [76]. We consider two different traffic graphs constructed from the pairwise road network distance matrix: (1) the sensor graph  $G$  introduced in [76] based on applying a thresholded Gaussian kernel (degree distribution in Figure 8e); (2) the sparser graph  $G_{s}$  based on applying the binary mask where the  $(i,j)$  entry is nonzero if and only if nodes  $i,j$  lie on the same highway (degree distribution in Figure 8d). We construct the second variant to more faithfully model the geometry of the highway, illustrated in Figure 8a.
Graph coarsening. We choose 2 clusters based on highway intersection and flow direction, indicated by colors (Figure 8b (b)), and 9 clusters based on highway labels (Figure 8c (c)).
(a) Our faithful traffic graph
(b) Graph clustering (2 clusters)
(c) Graph clustering (9 clusters)
(d) Our faithful graph degree distribution
Figure 8: METR-LA traffic graph: visualization, clustering, and degree distribution
(e) Original sensor graph degree distribution
Model. We use a standard baseline, DCRNN proposed in [76]. DCRNN is built on a core recurrent module, DCGRU cell, which iterates as follows: Let  $x_{i,t}, h_{i,t}$  denote the  $i$ -th node feature and hidden state vector at time  $t$ ; Let  $X_{t}, R_{t}, H_{t-1}$  be the matrices of stacking feature vectors  $x_{i,t}, r_{i,t}, h_{i,t-1}$  as rows.
$$
z _ {i, t} = \sigma_ {g} \left(W _ {z} x _ {i, t} + U _ {z} h _ {i, t - 1} + b _ {z}\right) \tag {61}
$$
$$
r _ {i, t} = \sigma_ {g} \left(W _ {r} x _ {t} + U _ {r} h _ {t - 1} + b _ {r}\right) \tag {62}
$$
$$
\hat {h} _ {i, t} = \phi_ {h} \left([ A X W _ {h} ] _ {[ i,: ]} ^ {\top} + [ A (R _ {t} \odot H _ {t - 1}) U _ {h} ] _ {[ i,: ]} ^ {\top} + b _ {h}\right) \tag {63}
$$
$$
h _ {i, t} = z _ {t} \odot h _ {t - 1} + (1 - z _ {t}) \odot \hat {h} _ {t}, \tag {64}
$$
where  $W_{z}, U_{z}, b_{z}, U_{r}, W_{r}, b_{r}, W_{h}, U_{h}, b_{h}$  are learnable weights and biases,  $\sigma_{g}$  is the sigmoid function,  $\phi_{g}$  is the hyperbolic tangent, and  $h_{i,0} = 0$  for all  $i$  at initialization. The crucial difference from a vanilla GRU lies in eqn (63) where graph convolution replaces matrix multiplication.
We then modify the graph convolution in (63) from global weight sharing to tying weights among clusters of nodes, similar to the implementation in Appendix D.4 for Relax- $S_{16}$ . For example, in the
case of two clusters (orbits), we change  $XW_{h}$  to
$$
\operatorname {s w a p} \left(\operatorname {c o n c a t} \left[ X _ {c _ {1}} W _ {h, c _ {1}}; X _ {c _ {2}} W _ {h, c _ {2}} \right]\right), \tag {65}
$$
where  $X_{c_i}$  denotes the submatrix of  $X$  including the rows of nodes from cluster  $i$  only, and  $W_{h,c_1}, W_{h,c_2}$  are two learnable matrices. In words, we perform cluster-specific linear transformation, combine the transformed features, and reorder the rows (i.e., swap) to ensure compatibility with the graph convolution.
Experiment Set-up. For our experiments, we use DCRNN model with 1 RNN layer and 1 diffusion step. We choose  $T' = 3$  (i.e., 3 historical graph signals) and  $T = 3$  (i.e., predict the next 3 period graph signals). We train all variants for 30 epochs using ADAM optimizer with learning rate 0.01. We report the test set performance selected by the best validation set performance.
D.3.1 Assumption Validation: Approximate Equivariant Map
Before applying our construction of approximate symmetries, we validate the assumption of the target function  $f^{*}$  being an approximately equivariant mapping using a trained DCRNN model as a proxy. We proceed as follows:
Data. We use the validation set of METR-LA (traffic graph signals in LA), which has 207 nodes and consists of 14,040 input and output signals. Each input  $X \in \mathbb{R}^{207 \times 2}$  represents the traffic volume and speed in the past at the 207 stations, and output  $Y \in \mathbb{R}^{207}$  representing future traffic volume.
Model. We use a trained DCRNN model on our faithful graph, with input being 3 historical signals  $\mathbf{X} = (X_{T - 3}, X_{T - 2}, X_{T - 1}) \in \mathbb{R}^{3 \times 207 \times 2}$  to predict the future signals  $\mathbf{Y} = (X_T, X_{T + 1}, X_{T + 2}) \in \mathbb{R}^{3 \times 207}$ . We denote this model as  $f$ . It gives reasonable performance with Mean Absolute Error  $\approx 3$ , and serves as a good proxy for the target (unknown) function  $f^*$ .
Neighbors. We take our faithful traffic graph that originally has 397 non-loop edges, and only consider a subset of 260 edges by thresholding the distance values to eliminate geometrically far-away nodes. This defines our 260 neighboring node pairs.
Equivalence error. For each node pair  $(i,j)$ , we swap their input signals by interchanging the  $(i,j)$ -th slices in the node dimension of the tensor  $X$ , denoted as  $X_{(i,j)}$ , and check if the swapped output  $\hat{Y}_{(i,j)} = f(X_{(i,j)})$  is close to the original output  $\hat{Y} = f(X)$  with  $(i,j)$ -th slices swapped. We measure "closeness" via the relative equivariant error at the node pair. Concretely, let  $X[i,j]$  denote the tensor slices at the  $(i,j)$  node pair, and  $X[j,i]$  being the swapped version by interchanging  $(i,j)$ -th slices. The relative different is computed as
$$
\left| \hat {\mathbf {Y}} _ {(i, j)} [ j, i ] - \hat {\mathbf {Y}} [ i, j ] \right| / \hat {\mathbf {Y}} [ i, j ],
$$
where / denotes element-wise division. We then compute the mean relative equivariance error over all instances in the validation set, which equals to  $5.17\%$ . This gives concrete justification to enforce approximate equivariance in the traffic flow prediction problems.
D.4 Application: Human Pose Estimation
D.4.1 Equivariant Layer for Human Skeleton Graph
We apply the constructions in Section A.1 to our human skeleton graph. We first show how to parameterize all linear  $\mathcal{A}_G$ -equivariant functions. Observe that  $\mathcal{A}_G \cong (\mathcal{S}_2)^2 = \{e, a, l, al\}$ , where the nontrivial actions correspond to the arm flip with respect to the spine, the leg flip with respect to the spine, and their composition. To fix ideas, we first treat both input and output graph signals as vectors, and construct  $\mathcal{A}_G$ -equivariant linear maps  $f: \mathbb{R}^{16} \to \mathbb{R}^{16}$ .
Step 1: Obtain the character table for  $(S_2)^2$
e	a	l	al
χe	1	1	1	1
χ2	1	1	-1	-1
χ3	1	-1	1	-1
χ4	1	-1	-1	1
Table 2: Character table for  ${\left( {\mathcal{S}}_{2}\right) }^{2}$
Step 2: Construct the basis for isotropic decomposition. Here we choose to index the leg joint pairs as  $(1,4)$ ,  $(2,5)$ ,  $(3,6)$ , arm joint pairs as  $(10,13)$ ,  $(11,14)$ ,  $(12,15)$ , and spline joints  $0,7,8,9$ .
$$
\begin{array}{l} B = \left[ \mathcal {B} \left(P _ {\chi_ {e}}\right); \mathcal {B} \left(P _ {\chi_ {2}}\right); \mathcal {B} \left(P _ {\chi_ {3}}\right); \mathcal {B} \left(P _ {\chi_ {4}}\right) \right] \text {w h e r e} \\ \mathcal {B} \left(P _ {\chi_ {e}}\right) = \left[ \left(e _ {1} + e _ {4}\right) / \sqrt {2}; \dots ; \left(e _ {1 2} + e _ {1 5}\right) / \sqrt {2}; e _ {0}; e _ {7}; e _ {8}; e _ {9} \right] \in \mathbb {R} ^ {1 6 \times 1 0}. \\ \mathcal {B} \left(P _ {\chi_ {2}}\right) = \left[ \left(e _ {1} - e _ {4}\right) / \sqrt {2}; \left(e _ {2} - e _ {5}\right) / \sqrt {2}; \left(e _ {3} - e _ {6}\right) / \sqrt {2} \right] \in \mathbb {R} ^ {1 6 \times 3}; \\ \mathcal {B} \left(P _ {\chi_ {3}}\right) = \left[ \left(e _ {1 0} - e _ {1 3}\right) / \sqrt {2}; \left(e _ {1 1} - e _ {1 4}\right) / \sqrt {2}; \left(e _ {1 2} - e _ {1 5}\right) / \sqrt {2} \right] \in \mathbb {R} ^ {1 6 \times 3}; \\ \mathcal {B} \left(P _ {\chi_ {4}}\right) = \emptyset \tag {66} \\ \end{array}
$$
Step 3: Parameterize  $f:\mathbb{R}^{16}\to \mathbb{R}^{16}$  by  $f:\mathcal{B}(P_{\chi_e})\to \mathcal{B}(P_{\chi_e})$  and  $f:\mathcal{B}(P_{\chi_2})\to \mathcal{B}(P_{\chi_2})$ , i.e. for all  $v\in \mathbb{R}^{16}$ , let  $v = \mathcal{B}(P_{\chi_e})\pmb {c}_e + \mathcal{B}(P_{\chi_2})\pmb {c_2} + \mathcal{B}(P_{\chi_3})\pmb {c_3}$ , then
$$
f (v) = W _ {e} \boldsymbol {c} _ {\boldsymbol {e}} + W _ {2} \boldsymbol {c} _ {\boldsymbol {2}} + W _ {3} \boldsymbol {c} _ {\boldsymbol {3}}, \tag {67}
$$
where  $W_{e} \in \mathbb{R}^{10 \times 10}$ ,  $W_{2} \in \mathbb{R}^{3 \times 3}$ ,  $W_{3} \in \mathbb{R}^{3 \times 3}$  are (learnable) weight matrices. Now  $f$  expresses all linear, equivariant maps w.r.t  $(S_{2})^{2}$ .
The following calculation based on  $f: \mathbb{R}^{16} \to \mathbb{R}^{16}$  shows how much degree of freedom (measured by learnable parameters) is gained by relaxing the symmetry from global (group  $S_{16}$ ), exact  $\mathcal{A}_G \cong (\mathcal{S}_2)^2$ , to trivial group (i.e., no symmetry).
$$
f _ {S _ {1 6}} = w \mathbf {I} _ {1 6} + w ^ {\prime} \left(\mathbf {1} - \mathbf {I} _ {1 6}\right), \quad (2 \text {p a r a m e t e r s}); \tag {68}
$$
$$
f _ {\mathcal {A} _ {G}} = W _ {e} \oplus W _ {2} \oplus W _ {3}, \quad (1 1 8 \text {p a r a m e t e r s o n t h e i s o t y p i c c o m p o n e n t s}); \tag {69}
$$
$$
f _ {\text {t r i v .}} = W, \quad (2 5 6 \text {p a r a m e t e r s}). \tag {70}
$$
To parameterize equivariant linear function  $f: \mathbb{R}^{16 \times d} \to \mathbb{R}^{16 \times d'}$ , we proceed by decoupling the input space into  $\mathbb{R}^{10 \times d}$ ,  $\mathbb{R}^{3 \times d}$ ,  $\mathbb{R}^{3 \times d}$  and the output space into  $\mathbb{R}^{10 \times d'}$ ,  $\mathbb{R}^{3 \times d'}$ ,  $\mathbb{R}^{3 \times d'}$ . Now the learnable weight matrices for multidimensional input/output become  $W_{e} \in \mathbb{R}^{10d \times 10d'}$ ,  $W_{2} \in \mathbb{R}^{3d \times 3d'}$ ,  $W_{3} \in \mathbb{R}^{3d \times 3d'}$ . The construction is summarized in Algorithm 2.
Algorithm 2 Equivariant layer  $f_{\mathcal{A}_G}:\mathbb{R}^{16\times d}\to \mathbb{R}^{16\times d'}$  for  $\mathcal{A}_G\cong (S_2)^2$
Require: The basis  $B \in \mathbb{R}^{16 \times 16}$  in (66) for isotypic decomposition of  $\mathcal{A}_G = (\mathcal{S}_2)^2$ , input  $h^{(l)} \in \mathbb{R}^{16 \times d}$ .
Initialize: The learnable weights  $W_{e}^{(l)} \in \mathbb{R}^{10d' \times 10d}; W_{2}^{(l)}, W_{3}^{(l)} \in \mathbb{R}^{3d' \times 3d}; M^{(l)} \in \mathbb{R}^{16 \times 16}$ .
1. Project  $h^{(l)}$  to the isotypic component:  $z^{(l)} = B^{\top}h^{(l)}$ ;
2. Perform block-wise linear transformation:
$z_{e} = W_{e}$  flatten  $(z_{[:.;10]}^{(l)})$
$z_{2} = W_{2}$  flatten  $(z_{[:,10:13]}^{(l)})$
$z_{3} = W_{3}$  flatten  $(z_{[:13:]}^{(l)})$
$z^{(l + 1)} = \mathrm{concat}[z_e;z_2;z_3]\in \mathbb{R}^{16\times d'}$
3. Project back to the standard basis:  $\bar{h}^{(l + 1)} = Bz^{(l + 1)}$ .
4. Perform pointwise nonlinearity:  $h^{(l + 1)} = \sigma (\bar{h}^{(l + 1)})$
return  $h^{(l + 1)}$
Figure 9: Human skeleton graph  $G$ , its coarsened graph  $G'$  (clustering leg joints), and blow-up of  $G'$
D.4.2 Experiment Details
Data. We use the standard benchmark dataset, Human3.6M [77], with the same protocol as in [78]: We train the models on 1.56M poses (from human subjects  $S1$ ,  $S5$ ,  $S6$ ,  $S7$ ,  $S8$ ) and evaluate them on 0.54M poses (from human subjects  $S9$ ,  $S11$ ). We use the method described in [88] to normalize the inputs (2D joint poses) to  $[-1, 1]$  and align the targets (3d joint poses) with the root joint.
Model. We give a detailed description of  $\mathcal{G}$ -Net and its variants used in the experiments. Figure inset illustrates the architecture of  $\mathcal{G}$ -Net.
For the human skeleton graph with  $N = 16$ , we have  $f_{\mathcal{G}}: \mathbb{R}^{16 \times d} \to \mathbb{R}^{16 \times k}$  where  $d, k$  represent the input dimension and output dimension (for each layer). Let  $f_{\mathcal{G}}[i,j]: \mathbb{R}^{16} \to \mathbb{R}^{16}$  denote its  $(i,j)$ -th slice.
1.  $\mathcal{G}$ -Net with strict equivariance using equivariant linear map  $f_{\mathcal{G}}$  (see Table 3):
-  $S_{16}$ :  $f_{S_{16}}[i,j] \in \mathbb{R}^{16 \times 16}$  is a diagonal matrix, with one learnable scalar  $a$  on diagonal and another learnable scalar  $b$  off diagonal.
- Relax- $S_{16}$ : We relax  $f_{S_{16}}[i,j]$  by having 16 different pairs of scalars  $(a_i,b_i), i \in [16]$ , such that each node  $i$  can map to itself and communicate to its neighbors in a different way (controlled by  $(a_i,b_i)$ ), while still treat all neighbors equally (by using the same  $b_i$  for nodes  $j \neq i$ ).
-  $\mathcal{A}_G = \mathcal{S}_2^2$ : We use Algorithm 2.
- Trivial: We allow  $f[i,j] \in \mathbb{R}^{16 \times 16}$  to be arbitrary, i.e., it has  $16 \times 16$  learnable scalars.
We remark that for  $S_{16}$  and Relax- $S_{16}$ , we implement them by tying weights; for  $\mathcal{A}_G$ , we implement them by projecting to isotypic component as shown in Algorithm 2.
2.  $\mathcal{G}$ -Net augmented with graph convolution  $A f_{\mathcal{G}}(x)$ , denoted as  $\mathcal{G}$ -Net(gc) (see Table 3): We apply the equivariant linear map  $f_{\mathcal{G}}$  in 1. and obtain the output  $f_{\mathcal{G}}(x) \in \mathbb{R}^{16 \times k}$ ; We then apply graph convolution by multiplication from the left, i.e.,  $A f_{\mathcal{G}}(x) \in \mathbb{R}^{16 \times k}$ .
3.  $\mathcal{G}$ -Net augmented with graph convolution and learnable edge weights, denoted as  $\mathcal{G}$ -Net  $(\mathrm{gc + ew})$  (see Table 4): We further learn the edge weights for the adjacency matrix  $A$ , by  $\text{softmax}(M \odot A)$  where  $M \in \mathbb{R}^{16}$  represents the learnable edge weights, and  $M_{i,j}$  is nonzero when  $A_{i,j} \neq 0$  and 0 elsewhere. This is inspired from SemGCN [78]. Besides the groups discussed above, we also implemented  $\text{Relax} - (\mathcal{S}_6)^2$  which corresponds to tying weights among the coarsened graph orbits, consists of 4 spline nodes (singleton orbits) and 2 orbits for the left/right arm and leg nodes.
4.  $\mathcal{G}$ -Net augmented with graph locality constraints  $(A \odot f_{\mathcal{G}})(x)$  and learnable edge weights, denoted as  $\mathcal{G}\text{-Net}(\mathrm{pt + ew})$  (see Table 3): We perform pointwise multiplication  $A \odot f_{\mathcal{G}}[i,j]$  at each  $(i,j)$ -th slice of  $f_{\mathcal{G}}$ . In practice, we also allow learnable edge weights as done in 3.
Experimental Set-up. We design  $\mathcal{G}$ -Net to have 4 layers (with batch normalization and residual connections in between the hidden layers), 128 hidden units, and use ReLU nonlinearity. This allows  $\mathcal{G}$ -Net(gc+ew) to recover SemGCN [78] when choosing  $\mathcal{G} = S_{16}$ . We train our models for at most 30
epochs with early stopping. For comparison purpose, we use the same optimization routines as in SemGCN [78] and perform the hyper-parameter search of learning rates  $\{0.001, 0.002\}$ .
Evaluation. Table 3 shows results of  $\mathcal{G}$ -Net and its variants when varying the choice of  $\mathcal{G}$ . We observe that using the automorphism group  $\mathcal{A}_G$  does not give the best performance, while imposing no symmetries (Trivial) or a relaxed version of  $S_{16}$  yields better results. Here, enforcing no symmetry achieves better performance since the human skeleton graph is very small with 16 nodes only. As shown in other experiments with larger graphs (e.g. image inpainting), enforcing symmetries indeed yields better performance.
Table 3: 3D human pose prediction using  $\mathcal{G}$ -Net and its variants. Error (± std) measured by Mean Per-Joint Position Error (MPJPE) and MPJPE after rigid alignment (P-MPJPE) across 3 runs. All methods use the same hidden dimension  $d = 128$ . Bold type indicates the top-2 performance among each variant. "NA" indicates the loss fails to converge.
MPJPE ↓	S16	Relax-S16	AG=(S2)2	Trivial
G-Net	NA	47.97 ± 0.47	48.30 ± 0.69	42.86 ± 0.64
G-Net(gc)	NA	54.50 ± 4.33	49.40 ± 1.37	43.24 ± 0.82
G-Net(pt+ew)	41.54 ± 0.47	40.44 ± 0.61	40.63 ± 0.26	38.41 ± 0.31
P-MPJPE ↓	S16	Relax-S16	AG=(S2)2	Trivial
G-Net	NA	36.45 ± 0.56	37.17 ± 0.59	32.59 ± 0.62
G-Net(gc)	NA	40.61 ± 0.99	37.62 ± 1.32	33.05 ± 0.81
G-Net(pt+ew)	32.31 ± 0.03	31.11 ± 0.68	31.35 ± 0.14	29.68 ± 0.22
Additional Evaluation. Table 4 shows the experiments when we keep the number of parameters roughly the same across different choices of  $\mathcal{G}$ .
Table 4: 3D human pose prediction using  $\mathcal{G}$ -Net(gc+ew), where the models induced from each choice of  $\mathcal{G}$  are set to have roughly the same number of parameters.  $d$  denotes the number of hidden units.
G-Net	Number of Parameters	Number of Epochs	MPJPE	P-MPJPE
S16	0.27M (d=128)	50	43.48	34.96
Relax-S16	0.27M (d=32)	20	40.08	32.08
AG=(S2)2	0.22M (d=16)	30	44.10	34.12
Trivial	0.22M (d=10)	30	45.05	34.79