SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the equations above,  $\kappa_{k}$  and  $\kappa_{v}$  are the kernels derived in Ex. 9 Eq. 16,  $C$  is the equivariant weight matrix satisfying Eq. 31.
The expression above indicates that the feature types of both key and value correspond to the regular representation of translation, as well as the feature type of the query. Moreover, the transformer operates on each point along the ray independently. It should be noted that the features  $(f_{k})_{t}$ ,  $(f_{q})_{t}$ ,  $(f_{v})_{t}$  and  $(f_{2}^{in})_{t}$  may have multiple channels and may consist of different types of features corresponding to various representations of  $SO(2)$ . The inner product  $\langle \cdot ,\cdot \rangle$  can only happen in the field type of the same representation of  $SO(2)$ . This allows for the implementation of a multi-head attention module, where each head can attend to a specific type of feature and multiple channels.
I.3 Self-attention over Points Along the Ray
After the cross-attention over rays, we get the features of the points along the ray, i.e., the feature attached to the ray corresponding to the regular representation of translation.  $SE(3)$  acts on the feature  $f^{\prime}$  attached to the point along the ray as mentioned in Eq.15 :
$$
\left(\mathcal {L} _ {g} f ^ {\prime}\right) (\boldsymbol {x}, \boldsymbol {d}) = \rho_ {1} \left(R _ {Z} \left(R _ {g ^ {- 1}}, \boldsymbol {d}\right)\right) ^ {- 1} f ^ {\prime} \left(g ^ {- 1} \boldsymbol {x}, R _ {g ^ {- 1}} \boldsymbol {d}\right),
$$
where  $\rho_{1}$  is the group representation of  $SO(2)$ .
We will apply the self-attention model to these points along the same ray. For two points  $\pmb{x}_1$  and  $\pmb{x}_2$  on the same ray  $(\pmb{d}, \pmb{x}_1 \times \pmb{d})$ , one can observe that for the same type of feature,  $\langle (\mathcal{L}_g f')(\pmb{x}_1, \pmb{d}), (\mathcal{L}_g f')(\pmb{x}_1, \pmb{d}) \rangle = \langle f'(g^{-1} \pmb{x}_1, R_{g^{-1}} \pmb{d}), f'(g^{-1} \pmb{x}_1, R_{g^{-1}} \pmb{d}) \rangle$ , which makes attention weight invariant, the transformer could be formulated as:
$$
f ^ {\text {o u t}} (x) = \sum_ {\mathrm {y} \text {o n} \text {t h e} \text {s a m e r a y} \text {a s} \mathrm {x}} \frac {\exp \left(\left\langle f _ {q} \left(f ^ {i n} , x\right) , f _ {k} \left(f ^ {i n} , x , y\right) \right\rangle\right)}{\sum_ {\mathrm {y} \text {o n} \text {t h e} \text {s a m e r a y} \text {a s} \mathrm {x}} \exp \left(\left\langle f _ {q} \left(f ^ {i n} , x\right) , f _ {k} \left(f ^ {i n} , x , y\right) \right\rangle\right)} f _ {v} (x, y, f ^ {i n}), \tag {35}
$$
$$
f _ {k} ^ {l} (x, y, f ^ {i n}) = c _ {k} (d (x, y)) I \left(f ^ {i n}\right) ^ {l} (y)
$$
$$
f _ {v} ^ {l} (x, y, f ^ {i n}) = c _ {v} (d (x, y)) I \left(f ^ {i n}\right) ^ {l} (y)
$$
$$
f _ {q} ^ {l} (x, f ^ {i n}) = c _ {q} I (f ^ {i n}) ^ {l} (x),
$$
and  $x$  and  $y$  are the points along the same ray with direction  $\mathbf{d}$ , we can denote  $x$  as  $(\mathbf{x}, \mathbf{d})$  and  $y$  as  $(\mathbf{y}, \mathbf{d})$ ,  $d(x, y)$  is the signed distance  $\langle \mathbf{d}, \mathbf{y} - \mathbf{x} \rangle$ ,  $c_k, c_v$  are arbitrary functions that take signed distance as the input and output complex values and  $c_q$  is an arbitrary constant complex. It should be noted that the features  $f_k, f_q, f_v$ , and  $f^{in}$  may have multiple channels and consist of different types of features corresponding to various representations of  $SO(2)$ , the inner product  $\langle \cdot, \cdot \rangle$  can only happen in the same type of field. This allows for implementing a multi-head attention module, where each head can attend to a specific feature type and multiple channels. Here,  $f_k^l$  denotes the type-  $l$  feature in feature  $f_k, f_v^l$  represents the type-  $l$  feature in feature  $f_v, f_q^l$  denotes the type-  $l$  feature in feature  $f_l,$  and  $(f^{in})^l$  represents the type-  $l$  feature in feature  $f^{in}$ .
Note that this transformer architecture also follows the general format of the transformer in Eq. 4. We only simplify the kernel  $\kappa_{k},\kappa_{v}$  to be trivial equivariant kernels.
To obtain a scalar feature density for each point, the feature output of each point can be fed through an equivariant MLP, which includes equivariant linear layers and gated/norm nonlinear layers. These layers are similar to the ones used in [62] and [63].
J 3D Reconstruction Experiment
J.1 Generation of the Dataset
The I dataset is obtained by fixing the orientation of the object as well as the eight camera orientations. With the object orientation fixed, we can independently rotate each camera around its optical axis by a random angle in a uniform distribution of  $(- \pi, \pi]$  to obtain the Z dataset. For the R dataset, we rotate every camera randomly by any rotation in  $SO(3)$  while fixing the object. The equivariance stands with the content unchanged. Therefore, in practice, we require that the object projection after the rotation does not have new parts of the object. We satisfy this assumption by forcing the camera to fixate on a new random point inside a small neighborhood and subsequently rotate each camera around its optical axis with the uniformly random angle in  $(- \pi, \pi]$ . We generate the  $Y$  dataset by rotating the object only with azimuthal rotations while keeping the camera orientations the same. The  $SO(3)$  dataset is generated by rotating the object with random rotation in  $SO(3)$  with the orientations of cameras unchanged, which will potentially result in new image content. Equivalence is not theoretically guaranteed in this setup, but we still want to test the performance of our method.
Figure 27: The number of parameters and FLOPs of  $SE(2)$  equivariant CNNs. We set batch size as one to calculate number of FLOPs.
J.2 Implementation Details
We use  $SE(2)$  equivariant CNNs to approximate the equivariant convolution over the rays. We use the same ResNet backbone as implemented in [30] that is equivariant to the finite group  $C_8$ , which we find achieves the best result compared with other  $SE(2)$  equivariant CNNs. We use a similar pyramid structure as [69] that concatenates the output feature of every block. Since every hidden feature is the regular representation, in the final layer we use  $1 \times 1 SE(2)$ -equivariant convolutional layers to transfer the hidden representation to scalar type.
For the fusion from the ray space to the point space model, we use one layer of convolution and three combined blocks of updating ray features and  $SE(3)$  transformers. For the equivariant  $SE(3)$  multi-head-attention, we only use the scalar feature and the vector (type-1) feature in the hidden layer. The kernel matrix includes the spherical harmonics of degrees 0 and 1. We also concatenate every output point feature of every block as in the  $2D$  backbone. Since the output feature of every block includes the vector feature, we transfer it to the scalar feature through one vector neuron layer and the inner vector product. We use the same weighted SDF loss as in [69] during training, which applies both uniform and near-surface sampling. We report the number of parameters and floating-point operations (FLOPs) of our  $2D$  backbone and light fusion networks in Fig. 27 and Fig. 28 respectively.
Figure 28: The number of parameters and FLOPs of the ray fusion model, which is composed of convolution from rays to points and transformer from rays to points. We set batch size as one to calculate the number of FLOPs.
J.3 Discussion of Results
There is still a performance gap between  $I / I$  and  $I / Z$ . Although  $SE(2)$  equivariant networks are theoretically strictly equivariant, the error in practice is introduced by the finite sampling of the image and the pooling layers. Additionally, we use the ResNet that is equivariant to  $C_8$  approximation of  $SO(2)$ , which causes this gap but increases the whole pipeline performance in the other tasks. There is no significant difference between  $I / Z$  and  $I / R$ , which shows that approximating the spherical field convolution by  $SE(2)$  equivariant convolution is reasonable in practice.
J.4 Qualitative Results
Figure 29 shows a qualitative result for the chair category. There are more qualitative results shown in Fig. 37, Fig. 38, and Fig. 39.
J.5 Ablation Study
First, we replace the  $SE(2)$  CNNs backbone with the conventional CNNs to test the effectiveness of  $SE(2)$  CNNs. Secondly, we remove the equivariant convolution/transformer part and use trivial aggregation (max-pooling) combined with MLP. Finally, we run an equivariant convolution and transformer without using the type-1 (vector) feature while keeping the number of parameters similar to our model.
Figure 29: Qualitative results for equivariant reconstruction. Left: input views; Right: reconstruction meshes of different models and ground truth meshes. The captions below the meshes show how the model is trained and tested, explained in the text.
Table 3 summarizes the result on the chair category, which illustrates that in the  $I / I$  and  $Y / Y$  trials,  $SE(2)$  CNN is less expressive than traditional CNN, but it contributes to the equivariance of our model looking at the results of  $I / Z, I / R$ , and  $Y / SO(3)$ . Equivariant ray convolution and transformer improve both the reconstruction performance and the equivariance outcome. We also compare the ray convolution and transformer with the models operating only on scalar features without vector features, and again, we see a drop in performance in every setting, proving the value of taking ray directions into account.
We also compare to a baseline where the ray difference information is encoded in the feature explicitly. Most models that encode ray directions aim at rendering, like IBRnet. Here, we modified IBRnet (Fig.2 of IBRnet paper) to query 3D points only for their SDF value instead of querying all densities along the ray that would be necessary for rendering. We replaced the ray direction differences with the ray directions themselves because we use a query point and not a query ray. We report in table 4 IoU result for Y/Y and Y/SO(3) (where Y is augmentation only along the vertical axis) for two models - IBRNet with conventional CNNs as 2D backbone and IBRNet with SE(2)-equivariant CNNs as 2D backbone. For the  $SO(3)$  setting, we rotate the whole 8 cameras with the same rotation, which is equivalent to rotating the object with the inverse rotation, and we use the object canonical frame to encode the ray information.
The baseline is not equivariant: It explicitly uses the ray directions as inputs to MLPs. Ray directions or their differences change when the coordinate system is transformed, breaking, thus, equivariance. Table 4 demonstrates that our model is more resilient to object rotations. We can enhance equivariance
Method	w/o SE(2)	w/o conv& trans	w/o type-1	Full model
I/I	0.767/0.079	0.695/0.105	0.722/0.093	0.731/0.090
I/Z	0.430/0.234	0.533/0.175	0.553/0.158	0.631/0.130
I/R	0.417/0.249	0.442/0.241	0.466/0.203	0.592/0.137
R/R	0.672/0.112	0.658/0.122	0.682/0.109	0.689/0.105
Y/Y	0.731/0.090	0.644/0.124	0.677/0.111	0.698/0.102
Y/SO(3)	0.467/0.0.217	0.534/0.170	0.569/0.163	0.589/0.142
SO(3)/SO(3)	0.655/0.120	0.616/0.142	0.636/0.130	0.674/0.113
Table 3: Ablation: w/o  $SE(2)$  means replacing  $SE(2)$  equivariant network with conventional; w/o ray conv& trans denotes the model where we replace the light field convolution and the light field equivariant transformer with max-pooling; w/o type-1 means using only scalar features in convolution and transformers.
Method	Y/Y	Y/SO(3)	SO(3)/SO(3)
IBRNet [61] w/o SE(2)	0.689	0.432	0.611
IBRNet [61] w/SE(2)	0.652	0.501	0.619
Ours	0.698	0.598	0.674
Table 4: Comparison of our model and a baseline which encodes the ray information explicitly. IBRNet w/o SE(2) is the modified IBRNet with conventional CNN backbone, IBRNet w/SE(2) is the model where we replace the conventional CNN backbone with the SE(2) equivariant CNN.
by using SE(2) equivariant modeling, and our model outperforms the baseline in the Y/Y setting. We believe that the transformer in our model is responsible for the performance improvement.
K Neural Rendering Experiment
K.1 Experiment Settings Discussion
Two experiment settings illustrate our model's equivariance:  $I / I$  and  $I / SO(3)$ .  $I / I$  is the canonical setting, where we train and test the model in the same canonical frame defined in the dataset.  $I / SO(3)$  is that we test the model trained in the canonical frame under arbitrary rotated coordinate frames, which means that all the camera poses in one scene are transformed by the same rotation without changing their relative camera poses and relative poses between the camera and the scene, which doesn't change the content of the multiple views. The reason we don't apply translation to the cameras is that there exists a depth range for points sampling in the model and the comparing baseline [61], which effectively mitigates the impact of translation.
We should note that the  $SO(3)$  setting in this experiment setting differs from  $R$  and  $SO(3)$  settings in reconstruction.  $R$  changes the relative pose of the cameras, and each image is transformed due to the rotation of each camera without altering the content, i.e., the sampling of the light field is nearly unchanged. The  $R$  setting aims to demonstrate that replacing the conventional method with ray-based convolution can get rid of the canonical frame for each view.
$SO(3)$  in reconstruction is to rotate the object pose randomly without changing the pose of the camera, which is equivalent to transforming the cameras by the inverse rotation but fixing the object, resulting in changes in the relative poses between the camera and the object, the content of the image and, therefore, the sampling of the light field. This setting shows that even for non-theoretically equivariant cases, our model in reconstruction still demonstrates robustness.
In the rendering experiment using the  $SO(3)$  setting, each image itself is not transformed, unlike the  $R$  setting in the reconstruction. The content of the images remains unchanged, including the light field sampling, unlike the  $SO(3)$  setting in the reconstruction. Since each image is not transformed, even if the conventional  $2D$  convolution is applied to the image, the scalar feature attached to the ray is not altered, and the light feature field sampling remains the same up to the transform of the coordinate frame. This setting was used to demonstrate that our model is  $SE(3)$ -equivariant when the input is the scalar light feature field.
Figure 30: The number of parameters and FLOPs of the model, which takes the scalar feature attached to rays as input and predicts the color and density for points along the target ray. The calculation of FLOPs is performed for single-pixel rendering with 10 source views.
Figure 31: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
K.2 Implementation Details
As described in the paper, we use a similar architecture as [61], where we replace the aggregation of view features by equivariant convolution and equivariant transformer over rays. In equivariant convolution, the input is scalar feature field over rays, which means that  $\omega_{in}^{1} = 0$  and  $\omega_{in}^{2} = 0$ ; for the output field, we use regular representation of translation as described in Sec. 3.3, and we use  $\omega_{out}^{1} = 0, 2^{1}, \dots, 2^{7}$  for group representation of  $SO(2)$ , each field type has 4 channels. In equivariant transformer over rays, we update the key and value before going to the attention module in the experiment; the specific operation is that we concatenate key  $f_{k}$  and query  $f_{q}$ , we concatenate
Figure 32: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Figure 33: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
$f_{v}$  and query  $f_{q}$ , and then we feed the concatenated key and value into two equivariant MLPs (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in [62]) to get the newly updated key and updated value, which will be fed into attention module. In line with [61], our approach does not involve generating features for the color of every point. In our implementation, we directly multiply the attention weights obtained from the softmax operator in the transformer with the corresponding colors in each view to perform color regression.
We replace the ray transformer with the equivariant transformer over the points along the ray; the input features comprise the feature types corresponding to the group representations  $\omega_{in} = 0,2^{1},\dots ,2^{7}$  for  $SO(2)$ . Each feature type has 4 channels; the output comprises the same feature type, and each type has 2 channels. We will first convert the feature into a scalar feature by an equivariant MLP (equivariant linear layers and gated/norm nonlinear layers, similar to the ones used in [62].) and then feed it into a conventional MLP to get the density. We report in Fig. 30 the number of parameters and floating-point operations (FLOPs) of the model composed of the convolution and transformers.
K.3 Qualitative Results
Fig. 32, Fig. 33, Fig. 34, Fig. 31, Fig. 35 and Fig. 36 show the qualitative results on Real-Forward-Facing [41] and Realistic Synthetic  $360^{\circ}$  [49] data. Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Figure 34: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
IBRNet(I)
IBRNet(SO(3))
Ours(I)
Ours(SO(3))
Figure 35: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Figure 36: In terms of qualitative results for rendering, we compare the performance of IBRNet and our model in both the given canonical frame (denoted as "IBRNet(I)" and "Ours(I)" respectively) and a rotated frame (denoted as "IBRNet(SO(3))" and "Ours(SO(3))" respectively). Our model performs comparably to IBRNet in the canonical setting. However, IBRNet experiences a performance drop in the rotated frame, while our model remains robust to the rotation.
Figure 37: Qualitative Result for the chair. Left: input views; Right: reconstruction meshes of different models. The captions below the meshes show how the model is trained and tested.
Figure 38: Qualitative Result for the car. Left: input views; Right: reconstruction meshes of different models. The captions below the meshes show how the model is trained and tested.
Figure 39: Qualitative Result for the car. Left: input views; Right: reconstruction meshes of different models. The captions below the meshes show how the model is trained and tested.