SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f ^ {l _ {\text {o u t}}} (x) = \int_ {y \in \mathcal {N} (x)} \kappa (s (x) ^ {- 1} y) \rho_ {i n} (\mathrm {h} (s (x) ^ {- 1} s (y))) f ^ {l _ {i n}} (y) d y \\ = \int_ {y \in \mathcal {N} (x)} \kappa (s (x) ^ {- 1} y) \rho_ {i n} \left(h _ {s (x) ^ {- 1} y \rightarrow \eta}\right) \\ \rho_ {i n} \left(h _ {s (x) ^ {- 1} y \rightarrow \eta}\right) ^ {- 1} \rho_ {i n} \left(\mathrm {h} (s (x) ^ {- 1} s (y)) f ^ {l _ {i n}} (y) d y \right. \\ = \int_ {y \in \mathcal {N} (x)} \kappa (s (x) ^ {- 1} y) \rho_ {i n} \left(h _ {s (x) ^ {- 1} y \rightarrow \eta}\right) \\ \rho_ {i n} \left(h _ {y \rightarrow x}\right) ^ {- 1} f ^ {l _ {i n}} (y) d y \\ = \int_ {y \in \mathcal {N} (x)} \kappa^ {\prime} (s (x) ^ {- 1} y) \rho_ {i n} \left(h _ {y \rightarrow x}\right) ^ {- 1} f ^ {l _ {i n}} (y) d y, \\ \end{array}
$$
where  $\eta = [0,0,1]^T$ , the fixed origin point in  $\mathbb{S}^2$ , and  $\kappa'(x) = \kappa(x)\rho_{in}(h_{x \to \eta})^{-1}$  for any  $x \in \mathcal{N}(\eta)$ . We can derive the equivariant condition that  $\kappa'$  should satisfy:
$$
\begin{array}{l} \kappa^ {\prime} (h x) = \kappa (h x) \rho_ {i n} \left(h _ {h x \rightarrow \eta}\right) ^ {- 1} \\ = \rho_ {o u t} (h) \kappa (x) \rho_ {i n} (\mathrm {h} (h, x)) ^ {- 1} \rho_ {i n} (h _ {h x \rightarrow \eta}) \\ = \rho_ {o u t} (h) \kappa (x) \rho_ {i n} \left(h _ {x \rightarrow \eta}\right) ^ {- 1} \rho_ {i n} \left(h ^ {- 1}\right) \\ = \rho_ {o u t} (h) \kappa^ {\prime} (x) \rho_ {i n} ^ {- 1} (h). \\ \end{array}
$$
Therefore, the spherical convolution can be expressed as the gauge equivariant convolution format:
E Converting Spherical Convolution to  $SE(2)$  Equivariant Convolution
As stated in Sec. D, spherical convolution is gauge equivariant with respect to the choice of section map  $s_a$ , and the spherical convolution can be written as gauge equivariant convolution. In this section, we use the gauge equivariant convolution to analyze the  $SE(2)$  equivariant convolution's approximation of spherical convolution.
Since each view performs spherical convolution on its own, we only analyze the convolution for one view for the sake of simplicity. We use  $V$  to denote the space of the rays in the same view, where  $V \subset \mathbb{S}^2$ . For any  $x \in V$ , we can choose the section map  $s_a$  such that  $h_{x \to o} = e$ , where  $o \in \mathbb{S}^2$  that  $o$  aligns with the optical axis as shown in the figure 20. Again, we use  $s(x)$  to denote  $s_a(x)$  for any  $x \in \mathbb{S}^2$  in this section.
When  $FOV$  is small, for any  $x, y \in V$ , we can have such approximation:  $h_{x \to y} = e$ . Then the above gauge equivariant convolution in Sec. D can be approximated as
$$
\begin{array}{l} f ^ {l _ {o u t}} (x) = \int_ {y \in \mathcal {N} (x)} \kappa^ {\prime} (s (x) ^ {- 1} y) f ^ {l _ {i n}} (y) d y \\ \underline {{\underline {{t = s (x) ^ {- 1} y}}}} \int_ {t \in \mathcal {N} (\eta)} \kappa^ {\prime} (t) f ^ {l _ {i n}} (s (x) t) d t, \\ \end{array}
$$
where  $\eta = [0,0,1]^T$ , the fixed origin in  $\mathbb{S}^2$ , and  $\kappa'(hx) = \rho_{out}(h)\kappa'(x)\rho_{in}^{-1}(h)$  for any  $h \in SO(2)$ .
Additionally, as illustrated in figure 21, we have a map from  $V$  to the projection points on the picture plane represented as  $\omega : V \to \mathbb{R}^2$ , where  $\omega(o)$  is defined as  $[0,0]^T$ . When  $FOV$  is small, we have such approximation that for any  $h \in SO(2)$ ,  $t \in \mathcal{N}(\eta)$ , and  $x \in V$ ,
$$
\omega (s (x) t) \approx \omega (x) + \omega (s (o) t).
$$
Figure 20: Section choice for every view
It is because
$$
\begin{array}{l} \omega (s (x) t) = \omega (x) + \omega (s (o) t) \\ + r (\frac {s i n \beta_ {t}}{c o s \beta_ {t}} - \frac {s i n \beta_ {t}}{c o s \beta_ {x} c o s (\beta_ {x} + \beta_ {t})}), \\ \end{array}
$$
$$
\begin{array}{l} \lim  _ {t \rightarrow \eta} r \left(\frac {\sin \beta_ {t}}{\cos \beta_ {t}} - \frac {\sin \beta_ {t}}{\cos \beta_ {x} \cos \left(\beta_ {x} + \beta_ {t}\right)}\right) \\ = r \left(\tan \beta_ {x}\right) ^ {2} \beta_ {t} + o \left(\beta_ {t} ^ {2}\right), \\ \end{array}
$$
when  $\beta_{x}$  is small (FOV is small), the approximation stands.
Then  $f^{l_{out}}(x) = \kappa'(t)f^{l_{in}}(s(x)t)dt$  can be approximately conducted in the image plane:
$$
\begin{array}{l} f ^ {\prime} l _ {o u t} (\omega (x)) \\ = \int_ {\omega (s (o) t) \in \mathcal {N} ([ 0, 0 ] ^ {T})} \kappa^ {\prime \prime} (\omega (s (o) t)) f ^ {\prime l _ {i n}} (\omega (x) + \omega (s (o) t)) \\ d (\omega (s (o) t)), \tag {27} \\ \end{array}
$$
where for any  $x\in \mathbb{S}^2$ $f^{\prime}(\omega (x)) = f(x)$  , and for any  $t\in \mathcal{N}(\eta)$ $\kappa ''(\omega (s(o)t)) = \kappa '(t)$
Since for any  $h \in SO(2)$  and any  $t \in \mathcal{N}(\eta)$ ,  $\omega(s(o)ht) = h\omega(s(o)t)$ , we have for any  $h \in SO(2)$  and any  $t \in \mathcal{N}(\eta)$ ,
$$
\begin{array}{l} \kappa^ {\prime \prime} (h \omega (s (o) t)) = \kappa^ {\prime \prime} (\omega (s (o) h t)) = \kappa^ {\prime} (h t) \\ = \rho_ {o u t} (h) \kappa^ {\prime} (t) \rho_ {i n} ^ {- 1} (h) = \rho_ {o u t} (h) \kappa^ {\prime \prime} (s (o) t) \rho_ {i n} ^ {- 1} (h) \\ \frac {p = \omega (s (o) t) \in \mathbb {R} ^ {2}}{k ^ {\prime \prime} (h p)} \\ = \rho_ {o u t} (h) \kappa^ {\prime \prime} (p) \rho_ {i n} ^ {- 1} (h). \\ \end{array}
$$
Figure 21: Illustration of projection map  $\omega$
Therefore, convolution 27 is exactly  $SE(2)$  equivariant convolution and it can be used to approximate the spherical convolution.
In other words, we can intuitively approximate the equivariant convolution over the partial sphere using the  $SE(2)$  equivariant network when the distortion of the sphere and the tangent plane of the optical axis is modest.
F Construction of Features in Equivariant Light Field Transformer
Noted that  $f_{2}^{out}$ ,  $f_{2}^{in}$  and  $f_{1}^{in}$  are features that are composed of fields of different types, denoted as  $f_{2}^{out} = \oplus_{i}f_{2}^{l_{out i}}$ ,  $f_{2}^{in} = \oplus_{i}f_{2}^{l_{ini}}$ , and  $f_{1}^{in} = \oplus_{i}f_{1}^{l_{ini3}}$ .  $f_{k}$ ,  $f_{q}$ , and  $f_{v}$  are constructed equivariant key features, query features, and value features, respectively, which are composed of fields of different types as well.
We use  $f_{k} = \oplus_{i}f_{k}^{l_{ki}}$ ,  $f_{q} = \oplus_{i}f_{q}^{l_{ki}}$ , and  $f_{v} = \oplus_{i}f_{v}^{l_{vi}}$  to denote  $f_{k}, f_{q}$  and  $f_{v}$ , respectively. We construct the features  $f_{k}, f_{q}$  and  $f_{v}$  through the equivariant kernels  $\kappa_{k} = \oplus_{j,i}\kappa_{k}^{l_{kj},l_{in_i}^{\prime}}$ ,  $\kappa_{v} = \oplus_{j,i}\kappa_{v}^{l_{vj},l_{in_i}^{\prime}}$  and equivariant matrix  $W_{q} = \oplus_{j,i}W_{q}^{l_{kj},l_{in_i}}$ :
$$
\begin{array}{l} f _ {k} ^ {l _ {k j}} (x, y, f _ {1} ^ {i n}) \\ = \sum_ {i} \kappa_ {k} ^ {l _ {k _ {j}}, l _ {i n _ {i}} ^ {\prime}} \left(s _ {2} (x) ^ {- 1} y\right) \rho_ {1} ^ {l _ {i n _ {i}} ^ {\prime}} \left(\mathrm {h} _ {1} \left(s _ {2} (x) ^ {- 1} s _ {1} (y)\right)\right) f _ {1} ^ {l _ {i n _ {i}} ^ {\prime}} (y); \tag {28} \\ \end{array}
$$
$$
\begin{array}{l} f _ {v} ^ {l _ {v j}} (x, y, f _ {1} ^ {i n}) \\ = \sum_ {i} \kappa_ {v} ^ {l _ {v j}, l _ {i n i} ^ {\prime}} \left(s _ {2} (x) ^ {- 1} y\right) \rho_ {1} ^ {l _ {i n i} ^ {\prime}} \left(\mathrm {h} _ {1} \left(s _ {2} (x) ^ {- 1} s _ {1} (y)\right)\right) f _ {1} ^ {l _ {i n i} ^ {\prime}} (y); \tag {29} \\ \end{array}
$$
$$
f _ {q} ^ {l _ {k j}} \left(x, f _ {2} ^ {i n}\right) = \sum_ {i} W _ {q} ^ {l _ {k j}, l _ {i n _ {i}}} f _ {2} ^ {l _ {i n _ {i}}} (x), \tag {30}
$$
where for any  $i,j$ , any  $h_2 \in SO(3)$ , and any  $x \in \mathcal{R} \kappa_k^{l_{k_j}, l_{ini_i}'}$  and  $\kappa_v^{l_{v_j}, l_{ini_i}'}$  should satisfy that:
$$
\kappa_ {k} ^ {l _ {k _ {j}}, l _ {i n i} ^ {\prime}} (h _ {2} x) = \rho_ {2} ^ {l _ {k _ {j}}} (h _ {2}) \kappa_ {k} ^ {l _ {k _ {j}}, l _ {i n i} ^ {\prime}} (x) \rho_ {1} ^ {l _ {i n i} ^ {\prime}} (\mathsf {h} _ {1} ^ {- 1} (h _ {2}, x));
$$
$$
\kappa_ {v} ^ {l _ {v _ {j}}, l _ {i n _ {i}} ^ {\prime}} (h _ {2} x) = \rho_ {2} ^ {l _ {v _ {j}}} (h _ {2}) \kappa_ {v} ^ {l _ {v _ {j}}, l _ {i n _ {i}} ^ {\prime}} (x) \rho_ {1} ^ {l _ {i n _ {i}} ^ {\prime}} (\mathsf {h} _ {1} ^ {- 1} (h _ {2}, x)),
$$
<sup>3</sup>Since here the homogeneous spaces of input and output might be different, so as the stabilizer groups, we use  $l$  and  $l'$  to denote the representations of different stabilizer groups.
where  $\mathrm{h}_1(h_2,x) = s_1(h_2x)^{-1}h_2s_1(x)$  is the twist function, and for any  $i,j$  and any  $h_2\in SO(3)$ ,  $W_{q}^{l_{k_{j}},l_{ini}}$  satisfies that:
$$
\rho_ {2} ^ {l _ {k _ {j}}} \left(h _ {2}\right) W _ {q} ^ {l _ {k _ {j}}, l _ {i n _ {i}}} = W _ {q} ^ {l _ {k _ {j}}, l _ {i n _ {i}}} \rho_ {1} ^ {l _ {i n _ {i}}} \left(h _ {2}\right). \tag {31}
$$
When the group representation is irreducible representation, due to Schur's Lemma, we have  $W_{q}^{l_{k_{j}},l_{in_{i}}} = cI$  when  $l_{k_j} = l_{ini}$ , where  $c$  is an arbitrary real number, otherwise  $W_{q}^{l_{k_{j}},l_{in_{i}}} = 0$ .
G Proof for Equivariance of Light Field Transformer
The equivariant light field transformer defined in the paper reads:
$$
\begin{array}{l} f _ {2} ^ {\text {o u t}} (x) \\ = \sum_ {y \in \mathcal {N} (x)} \frac {\exp \left(\left\langle f _ {q} \left(x , f _ {2} ^ {i n}\right) , f _ {k} \left(x , y , f _ {1} ^ {i n}\right) \right\rangle\right)}{\sum_ {y \in \mathcal {N} (x)} \exp \left(\left\langle f _ {q} \left(x , f _ {2} ^ {i n}\right) f _ {k} \left(x , y , f _ {1} ^ {i n}\right) \right\rangle\right)} \\ f _ {v} (x, y, f _ {1} ^ {i n})) \tag {32} \\ \end{array}
$$
is in a general form.
According to [18], one can prove that  $f_{q}$ ,  $f_{k}$  and  $f_{v}$  are equivariant, that is, for any  $g \in SE(3)$ ,  $x \in \mathbb{R}^3$  and  $y \in \mathcal{R}$ ,
$$
\begin{array}{l} f _ {q} ^ {l _ {k _ {j}}} (g \cdot x, \mathcal {L} _ {g} ^ {i n} (f _ {2} ^ {i n})) = \rho_ {2} ^ {l _ {k _ {j}}} \left(\mathrm {h} _ {2} (g ^ {- 1}, g \cdot x) ^ {- 1}\right) f _ {q} ^ {l _ {k _ {j}}} (x, f _ {2} ^ {i n}); \\ f _ {k} ^ {l _ {k _ {j}}} (g \cdot x, g \cdot y, \mathcal {L} _ {g} ^ {\prime i n} (f _ {1} ^ {i n})) = \rho_ {2} ^ {l _ {k _ {j}}} \left(\mathrm {h} _ {2} (g ^ {- 1}, g \cdot x) ^ {- 1}\right) f _ {k} ^ {l _ {k _ {j}}} (x, y, f _ {1} ^ {i n}); \\ f _ {v} ^ {l _ {v _ {j}}} (g \cdot x, g \cdot y, \mathcal {L} _ {g} ^ {\prime i n} (f _ {1} ^ {i n})) = \rho_ {2} ^ {l _ {v _ {j}}} \left(\mathrm {h} _ {2} (g ^ {- 1}, g \cdot x) ^ {- 1}\right) f _ {v} ^ {l _ {v _ {j}}} (x, y, f _ {1} ^ {i n}), \\ \end{array}
$$
where  $\mathcal{L}^{in}$  and  $\mathcal{L}^{\prime in}$  are group action of  $SE(3)$  on  $f_{2}^{in}$  and  $f_{1}^{in}$ , respectively.
The inner product  $\langle f_q, f_k \rangle = \sum_i (\overline{f_q^{l_{k_i}}})^T f_k^{l_{k_i}}$  is invariant due to the property of unitary representation, which results in the equivariance of the transformer.
H From  $SE(3)$  Equivariant Transformer in Ray Space to  $SE(3)$  Equivariant Transformer in Euclidean Space
In our implementation for the reconstruction task, the attention model is always only applied over the rays going through the points. We can continue to use the interpretation in the convolution from ray space to  $\mathbb{R}^3$  in Ex. 10 that treats any ray  $y$  passing through the point  $x$  as a point  $y'$  such that  $y' - x = d_{s_2(x)^{-1}y}$  as shown in the figure 17.
After we get the initial feature of query points through equivariant convolution from  $\mathcal{R}$  to  $\mathbb{R}^3$ , we update the neighboring ray feature by directly concatenating the query point feature to every ray feature before through a  $SO(3)$  equivariant MLP as shown in the figure 22.  $SO(3)$  equivariant MLP is composed of an equivariant nonlinear layer and self-interaction layer as in the tensor field networks [55].
Since  $y$  becomes point  $y'$ , and  $f_1^{in}$  is the feature over  $R^3$  attached to "points"  $y'$ , it becomes  $\oplus_i f_1^{lin_i4}$ . Then transformer 32 would be converted to the transformer in [27] over  $\mathbb{R}^3$ :
<sup>4</sup>Since here  $f_1^{in}$  is the fields over  $\mathbb{R}^3$ , we use  $l$  instead of  $l'$  as the denotation
Figure 22: The structure of ray updating and  $SE(3)$  transformer. We treat any ray  $y$  going through point  $x$  as a point  $y' \in \mathbb{R}^3$  such that  $y' - x = d_{s_2(x)^{-1}y}$ . The blue block indicates the ray feature update, and the pink block is the equivariant attention model. For the ray feature updating, the point feature (lavender) is concatenated to every ray feature (light yellow, light blue, and light red) and goes through an equivariant MLP. For the transformer, we get the equivariant query, key, and value feature through the designed linear matrix  $W_q$ , designed kernels  $\kappa_k$  and  $\kappa_v$ , then apply multi-head attention to obtain the output point feature, which can subsequently be fed into the next ray feature updating and  $SE(3)$  transformer block.
$$
f _ {2} ^ {o u t} (x)
$$
$$
= \sum_ {y ^ {\prime} \in \mathcal {N} (x)} \frac {e x p (\langle f _ {q} (x , f _ {2} ^ {i n}) , f _ {k} (x , y ^ {\prime} , f _ {1} ^ {i n}) \rangle)}{\sum_ {y ^ {\prime} \in \mathcal {N} (x)} e x p (\langle f _ {q} (x , f _ {2} ^ {i n}) f _ {k} (x , y ^ {\prime} , f _ {1} ^ {i n}) \rangle}
$$
$$
f _ {v} \left(x, y ^ {\prime}, f _ {1} ^ {i n}\right)), \tag {33}
$$
where the subscript denotes the points to which the feature is attached, i.e.,  $x$  and  $y'$ .
The features  $f_{k}, f_{v}$  are constructed by the equivariant kernels  $\kappa_{k} = \oplus_{j,i}\kappa_{k}^{l_{k_{j}},l_{ini}}$ ,  $\kappa_{v} = \oplus_{j,i}\kappa_{v}^{l_{v_{j}},l_{ini}}$ :
$$
f _ {k} ^ {l _ {k _ {j}}} (x, y, f _ {1} ^ {i n}) = \sum_ {i} \kappa_ {k} ^ {l _ {k _ {j}}, l _ {i n _ {i}}} (y ^ {\prime} - x) f _ {1} ^ {l _ {i n _ {i}}} (y);
$$
$$
f _ {v} ^ {l _ {k j}} (x, f _ {2} ^ {i n}) = \sum_ {i} \kappa_ {v} ^ {l _ {v _ {j}}, l _ {i n _ {i}}} (y ^ {\prime} - x) f _ {2} ^ {l _ {i n _ {i}}} (y),
$$
where for any  $i,j$ , any  $h_2 \in SO(3)$ , and any  $x \in \mathbb{R}^3$ $\kappa_k^{l_{k_j},l_{ini}}$  and  $\kappa_v^{l_{v_j},l_{ini}}$  should satisfy that:
$$
\kappa_ {k} ^ {l _ {k j}, l _ {i n _ {i}}} (h _ {2} x) = \rho_ {2} ^ {l _ {k j}} (h _ {2}) \kappa_ {k} ^ {l _ {k j}, l _ {i n _ {i}}} (x) \rho_ {2} ^ {l _ {i n _ {i}}} (h _ {2} ^ {- 1});
$$
$$
\kappa_ {v} ^ {l _ {v _ {j}}, l _ {i n _ {i}}} (h _ {2} x) = \rho_ {2} ^ {l _ {v _ {j}}} (h _ {2}) \kappa_ {v} ^ {l _ {v _ {j}}, l _ {i n _ {i}}} (x) \rho_ {1} ^ {l _ {i n _ {i}}} (h _ {2} ^ {- 1})
$$
as stated in [27].
The feature  $f_{q}$  is constructed in the same way as Equation 30.
Figure 23: The comparison of the equivariant light field transformer and the conventional transformer. The left is the equivariant light field transformer, and the right is the conventional transformer. In our light field transformer, the position encoding is not directly concatenated to the features because this is not equivariant. We first obtain the equivariant feature attached to the point by equivariant convolution over the rays. We then construct features  $f_{k}$ , and  $f_{v}$  with derived designed kernels  $\kappa_{k}$  and  $\kappa_{v}$  to keep them equivariant; we construct  $f_{q}$  by the designed equivariant linear layer  $W_{q}$ . Since  $f_{k}, f_{q}$ , and  $f_{v}$  are all equivariant, the inner product of  $f_{k}$  and  $f_{q}$  is invariant, which results in invariant attention weight. Therefore, the whole transformer is equivariant. In contrast, the conventional transformer concatenates the ray position encoding with the feature attached to the ray, uses the point position encoding for the query feature for the point, and applies multi-head attention using  $f_{k}, f_{q}$ , and  $f_{v}$ , which are obtained by the Linear layer. We should note that  $W_{q}$  in the light field transformer is designed to be equivariant, satisfying equation 31, which differs from the conventional linear map  $W_{q}$  in the conventional transformer. For the attention blocks after the first block, the query features of the point in our model and the conventional model are both the output of the last attention block. The difference is that our query feature keeps equivariant while the feature in the conventional transformer is not.
Figure 22 shows the structures of ray feature update and  $SE(3)$  equivariant transformer.
In figure 23, we compare the  $SE(3)$  equivariant transformer and the conventional transformer to illustrate how the equivariance is guaranteed in the equivariant transformer. In figure 24, we present the types of futures in  $SE(3)$  equivariant attention head and conventional attention head, respectively. It indicates that geometric information is aggregated equivariantly in multi-head attention in the equivariant transformer.
I Equivariant Neural Rendering
Equivariant rendering relates to equivariant  $3D$  reconstruction, where we focus on multiple views instead of the entire light field. The equivariance property is maintained when the ray sampling is invariant up to a coordinate change.
I.1 Convolution from Rays to Rays
For neural rendering tasks, we query one ray and apply the convolution over the neighboring rays to obtain the feature attached to the target query ray. Similar to the reconstruction, we utilize a kernel with local support. However, there is a distinction in that for neural rendering, the kernel  $\kappa$  is constrained to be nonzero only when  $d(x,\eta) = 0$ , while there are no constraints on  $\angle (d_x,[0,0,1]^T)$ .
Figure 24: The comparison of multi-head attention modules in the equivariant light field transformer and in the conventional transformer. The figure above is the multi-head attention module in an equivariant light transformer, and the figure below is the conventional transformer. In the light field transformer, the query, key, and value features are composed of different types of features; they can be scalars, vectors, or higher-order tensors. The inner product should apply to the same type of features, and the type of feature determines the way of applying the inner product. In contrast, the feature in a conventional transformer doesn't contain vectors and tensors, and the inner product is conventional.
As a result, the neighboring rays exclusively encompass the rays on the epipolar line for the target ray in each source view, as depicted in Figure 25.
The scalar field over rays serves as the input to the convolution. The output field type corresponds to the regular representation of translation. This is because this field type serves as the input for the cross-attention module later on. If this field type were not utilized, the transformer would reach the entire neighboring set, leading to inferior performance compared to applying the transformer individually for each point and then applying it over the points along the ray. A similar observation is made in [58], which states that the two-stage transformer outperforms the one-stage transformer. Using the field type corresponding to the regular representation of the translation as the input, the transformer from rays to rays is equivalent to performing a transformer for each point, respectively, as explained in the following section.
In Eq. 18, we already provide the solution of the kernel. We give a detailed explanation in this case and show that it is equivalent to performing convolution from rays to rays with output field types corresponding to irreducible representations, followed by applying Inverse Fourier Transform. Given that the input field is a scalar field, we have  $\omega_{in}^{1} = 0$  and  $\omega_{in}^{2} = 0$ . When considering an output field type of  $(\omega_{out}^{1},reg)$ , where  $reg$  represents the regular representation of translation, the convolution can be expressed as follows:
$$
\begin{array}{l} (f _ {o u t} ^ {(\omega_ {o u t} ^ {1}, r e g)}) _ {t} = \int_ {y \in \mathcal {N} (x)} \kappa_ {1} (s (x) ^ {- 1} y) (\kappa_ {2} (s (x) ^ {- 1} y)) _ {t} f _ {i n} (y) d y \\ = \int_ {y \in \mathcal {N} (x)} \kappa_ {1} (s (x) ^ {- 1} y) f (d (\eta , s (x) ^ {- 1} y), \angle ([ 0, 0, 1 ] ^ {T}, \boldsymbol {d} _ {s (x) ^ {- 1} y})) \delta (t - g (s (x) ^ {- 1} y)) f _ {i n} (y) d y \\ = \int_ {g (s (x) ^ {- 1} y) = t} \kappa_ {1} (s (x) ^ {- 1} y) f (d (\eta , s (x) ^ {- 1} y), \angle ([ 0, 0, 1 ] ^ {T}, \pmb {d} _ {s (x) ^ {- 1} y})) f _ {i n} (y) d y. \\ \end{array}
$$
From the above equation, we can intuitively find that when the output field corresponds to the regular representation of the translation, the convolution happens at every point along the ray, respectively. We can treat  $f_{out}^{(\omega_{out}^{1},reg)}$  as a function over  $\mathbb{R}$ , and for any  $\omega \in \mathbb{R}$  we apply the Fourier Transform to  $f_{out}^{(\omega_{out}^{1},reg)}$ :
Figure 25: For simplification, we show two source views. For a target query ray  $x$ , the neighboring rays (denoted by red rays) are on the epipolar lines (denoted as yellow dotted dashes) for the target ray in each source view. For any ray  $y \in \mathcal{N}(x)$ ,  $d(x,y) = 0$ .
$$
\begin{array}{l} \mathcal {F} (\omega) = \int_ {t} f _ {o u t} ^ {\left(\omega_ {o u t} ^ {1}, r e g\right)} (t) e ^ {- i \omega t} d t \\ = \int_ {t} \int_ {g (s (x) ^ {- 1} y) = t} \kappa_ {1} (s (x) ^ {- 1} y) f (d (\eta , s (x) ^ {- 1} y), \angle ([ 0, 0, 1 ] ^ {T}, \boldsymbol {d} _ {s (x) ^ {- 1} y})) f _ {i n} (y) d y e ^ {- i \omega t} d t \\ = \int_ {y} \kappa_ {1} (s (x) ^ {- 1} y) f (d (\eta , s (x) ^ {- 1} y), \angle ([ 0, 0, 1 ] ^ {T}, \boldsymbol {d} _ {s (x) ^ {- 1} y})) e ^ {- i \omega g (s (x) ^ {- 1} y)} f _ {i n} (y) d y \\ = \int_ {y} \kappa_ {1} (s (x) ^ {- 1} y) \kappa_ {2} ^ {\prime} (s (x) ^ {- 1} y) f _ {i n} (y) d y, \\ \end{array}
$$
where  $\kappa_2' = f(d(\eta, s(x)^{-1}y), \angle([0, 0, 1]^T, \pmb{d}_{s(x)^{-1}y}))e^{-i\omega g(s(x)^{-1}y)}$ , which is exactly the kernel corresponding to  $\omega_{out}^2 = \omega$  and  $\omega_{in}^2 = 0$  as stated in Eq. 14. Therefore, we know that the field corresponding to the irreducible representation of the translation can be treated as the Fourier coefficients of the field corresponding to the regular representation. We can first obtain the features of different irreducible representations attached to the ray and subsequently apply the Inverse Fourier Transform to get the features for points along the ray, as shown in figure 26.
I.2 Cross-attention over Rays
The feature that generates the query in the transformer is the feature attached to the target ray, whose feature type corresponds to the regular representation of the translation. The feature that generates the key and value in the transformer is attached to the neighboring rays in the source view, whose
Figure 26: The features for points along the ray (the field type corresponds to the regular representation) can be obtained by the Inverse Fourier Transform of features attached to the ray, where the types of feature fields correspond to the irreducible representation of the translation.
feature type corresponds to the scalar field. The output is the feature attached to the target ray, whose feature type corresponds to the regular representation. Therefore, the transformer becomes:
$$
\left(f _ {2} ^ {\text {o u t}} (x)\right) _ {t} = \sum_ {y \in \mathcal {N} (x)} \frac {\exp \left(\left\langle \left(f _ {q} \left(x , f _ {2} ^ {\text {i n}}\right)\right) _ {t} , \left(f _ {k} \left(x , y , f _ {1} ^ {\text {i n}}\right)\right) _ {t} \right\rangle\right)}{\sum_ {y \in \mathcal {N} (x)} \exp \left(\left\langle \left(f _ {q} \left(x , f _ {2} ^ {\text {i n}}\right)\right) _ {t} \left(f _ {k} \left(x , y , f _ {1} ^ {\text {i n}}\right)\right) _ {t} \right\rangle}\right)} \left(f _ {v} (x, y, f _ {1} ^ {\text {i n}})\right) _ {t}, \tag {34}
$$
$$
\left(f _ {k} (x, y, f _ {1} ^ {i n})\right) _ {t} = \left(\kappa_ {k} \left(s _ {2} (x) ^ {- 1} y\right)\right) _ {t} f _ {1} ^ {i n} (y)
$$
$$
\left(f _ {v} (x, y, f _ {1} ^ {i n})\right) _ {t} = \left(\kappa_ {v} \left(s _ {2} (x) ^ {- 1} y\right)\right) _ {t} f _ {1} ^ {i n} (y)
$$
$$
\left(f _ {q} \left(x, f _ {2} ^ {i n}\right)\right) _ {t} = C \left(f _ {2} ^ {i n} (x)\right) _ {t}.
$$