GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers
Authors: Takeru Miyato
Bernhard Jaeger
Max Welling
Andreas Geiger
Venue: ICLR 2024 poster (2024)
Keywords: transformer
attention
geometry-aware attention
multi-view understanding
novel view synthesis
group theory
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a principled way to encode geometric structure of tokens into vision transformers.

Abstract: As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional learned parameters and only minor computational overhead.

Published as a conference paper at ICLR 2024
GTA: A GEOMETRY-AWARE ATTENTION MECHANISM FOR MULTI-VIL TRANSFORMERS
Takeru Miyato', Bernhard Jaeger',,, Max Welling', Andreas Geiger' University of Tübingen, Tübingen AI Center University of Amsterdam
ABSTRACT
As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since isting positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural prop- erties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their under- lying 3D geometric structure. Based on this hypothesis, we propose a geometry- aware attention mechanism that encodes the geometric structure of tokens rela- tive transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and perfor- mance of oate--ofttheart transfrmeerbased NVS models without any additional learned parameters and only minor computational overhead.
INTRODUCTION
The transformer model (Vaswani al., 2017), which is composed of stack of permutation sym- metric layers, processes input tokens as set and lacks direct awareness of the tokens' structural information. Consequently, transformer models are not solely perceptible to the structures of input tokens, such as word order in NLP or 2D positions of image pixels or patches in image processing.
A common way make transformers position-aware is through vector embeddings: in NLP, typical way is to transform the position values of the word tokens into embedding vectors to be added to input tokens or attention weights (Vaswani al., 2017: Shaw al., 2018). While initially designed for NLP, these positional encoding techniques are widely used for 2D and 3D vision tasks today (Wang et al., 2018; Dosovitskiy et al., 2021; Sajjadi et al., 2022b; Du et al., 2023).
Here, a natural question arises: "Are existing encoding schemes suitable for tasks with very different geometric structures?". Consider for example 3D vision tasks using multi-view images paired with camera transformations. The 3D Euclidean symmetry behind multi-view images is a more intricate structure than the ID sequence of words. With the typical vector embedding approach, the model is tasked with uncovering useful camera poses embedded in the tokens and consequently struggles to understand the effect of non-commutative Euclidean transformations.
Our aim is seek principled way incorporate the geometrical structure of the tokens into the transformer. To this end, we introduce a method that encodes the token relationships as transfor- mations directly within the attention mechanism. More specifically, formation determined by the geometric relation between the query and the key-value tokens. We then apply those transformations the key-value pairs, which allows the model to compute QKV attention in an aligned coordinate space.
exploit the relative trans-
We evaluate the proposed attention mechanism on several novel view synthesis (NVS) tasks with sparse and wide-baseline multi-view settings, which are particularly hard tasks where a model needs learn strong 3D geometric priors from multiple training scenes. We show that existing positional encoding schemes are suboptimal and that our geometric- aware attention, named geometric trans- form attention (GTA), significantly improves learning efficiency and performance of state-of-the-art transformer-basee NVS models, just by replacing the existing positional encodings with GTA.
Correspondence takeru. mi yatoogmm 11 com. Code: http:://ghhhh.cmmmmmmmmmssssssss

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
Given token features X E Radd the attention layer's outputs O E Rand are computed as follows:
O == Attn(Q,, K,V) = (oftmax(QK"))k
(1)
where = = = XW* XWW E B ((Q...V) E iRdxd, and (n, d) is the number of tokens and channel dimensions. We omit the scale factor inside the softmax function for simplicity. The output in Eq. (1) is invariant to the permutation of the key-value vector indices. To break this permutation symmetry, we explicitly encode positional information into the transformer, which is called positional encoding (PE). The original transformer (Vaswani et al., 2017) incorporates positional information by adding embeddings to all input tokens. This absolute positional encoding (APH) scheme has the following form:
(2) where P denotes the positional attributes of the tokens X and y is a PE function. From here, a bold symbol signifies that the corresponding variable consists of list of elements. y is typically the si- nusoidal function, which transforms position values into Fourier features with multiple frequencies. Shaw al. (2018) proposes an alternative PE method, encoding the relative distance between each pair of query and key-value tokens as biases added to each component of the attention operation:
softmax ((Q + (P) + (P))W*)) (V + +P))W
softmax (QKT + (P)) (V + +(al(())
where Trel (P) E I and Y(u (P) Ruxd are the bias terms that depend on the distance between tokens. This encoding scheme is called relative positional encoding (RPE) and ensures that the embeddings do not rely on the sequence length, with the aim of improving length generalization.
Following the success in NLP, transformers have demonstrated their efficacy on various image- based computer vision tasks (Wang et al., 2018; Ramachandran et al., 2019; Carion et al., 2020; Dosovitskiy et al.. 2021; Ranftl ct al., 2021; Romero et al., 2020; Wu et al., 2021; Chitta et al., 2022). Those works use variants of APE or RPE applied to 2D positional information to make the model aware of 2D image structure. Implementation details vary across studies. Besides Diivis,on, there has been surge of application of transformer-basee
models to 3D-vision (Wang et al., 2021a; Liu et al., 2022; Kulhánek et al., 2022; Sajadii et al., 2022b; Watson et al., 2023; Varma et al., 2023; Xu et al., 2023; Shao al., 2023; Venkat al., 2023; Du et al., 2023; Liu al., 2023a).
Various PE schemes have been proposed in 3D vision, mostly relying on APE- or RPEE-based en- codings. In NVS Kulhánek et al. (2022); Watson al. (2023); Du et al. (2023) embed the camera extrinsic information by adding linearly transformed, flattened camera extrinsic matrices to the to- kens. In Sajjadi et al. (2022b); Sañn et al. (2023), camera extrinsic and intrinsic information is encoded through ray embeddings that are added or concatenated to tokens. Venkat et al. (2023) also uses ray information and biases the attention matrix by the ray distance computed from ray information linked to each pair of query and key tokens. An additional challenge in 3D detection and segmentation is that the output is typically in an orthographic camera grid, differing from the perspective camera inputs. Additionally, sparse attention (Zhu et al., 2021) is often required because high resolution feature grids (Lin et al., 2017) are used. Wang et al. (2021b); Li et al. (2022) use learnable PE for the queries and PE for keys and values. Peng et al. (2023) find that using stan- dard learnable PE for each camera does not improve performance when using deformable attention. Liu et al. (2022; 2023b) do add PE keys and values by generating 3D points at multiple depths for each pixel and adding the points to the image features after encoding them with an MLP. Zhou & Krähenbühl (2022) learn positional embeddings using camera parameters and apply them the queries and keys in a way that mimics the relationship between camera and target world coordi- nates. Shu et al. (2023) improves performance by using available depths link image tokens with their 3D positions. Besides APE and RPE approaches, Hong et al. (2023); Zou et al. (2023); Wang et al. (2023) modulate tokens by MiL-M- approach (Perez et al., 2018), where they element-wise multiply tokens with features computed from camera transformation.
In point cloud transformers, Yu et al. (2021a) uses APE to encode 3D positions of point clouds Qin et al. (2022) uses an RPE-based attention mechanism, using the distance or angular difference between tokens as geometric information. Epipolar-base sampling techniques are used to sample geometrically relevant tokens of input views in attention layers (He et al., 2020; Suhail et al., 2022; Saha et al., 2022; Varma et al., 2023; Du et al., 2023), where key and value tokens are sampled along an epipolar line determined by the camera parameters between target view and an input view.
2

---

Published as a conference paper at ICLR 2024
3 GEOMETRIC ENCODING BY RELATIVE TRANSFORMATION
In this work, we focus on novel view synthesis (NVS), which is a fundamental task in 3D-vision. The NVS task is to predict an image from novel viewpoint, given set of context views of a scene and their viewpoint information represented as 4 x 4 extrinsic matrices, each of which maps 3D points in world coordinates to the respective points in camera coordinates. NVS tasks require the model to understand the scene geometry directly from raw image inputs.
The main problem in existing encoding schemes of the camera transformation is that they do not respect the geometric structure of the Fuclidean transformations. In Eq. (2) and Eq. (3), the embed- ding is added each token or to the attention matrix. However, the geometry behind multi-view images is governed by Euclidean symmetry. When the viewpoint changes, the change of the object's s pose in the camera coordinates is computed based on the corresponding camera transformation.
Our proposed method incorporates geometric transformations directly into the transformer's atten- tion mechanism through a relative transformation of the QKV features. Specifically, each key-value token is transformed by a relative transformation that is determined by the geometric attributes be- tween query and key-value tokens. This can be viewed as coordinate system alignment, which has an analogy in geometric processing in computer vision: when comparing two sets of points each represented in a different camera coordinate space, we move one of the sets using a relative trans- formation cc to obtain all points represented in the same coordinate space. Here, and c' are the extrinsics of the respective point sets. Our attention performs this coordinate alignment within the attention feature space. This alignment allows the model not only compare query and key vectors in the same reference coordinate space, but also to perform the addition of the attention output at the residual path in the aligned local coordinates of each token due the value vector's 's transformation.
This direct application of the transformations to the attention features shares its philosophy with the classic transforming autoencoder (Hinton al., 2011; Cohen & Welling, 2014; Worrall et al., 2017; Rhodin et al., 2018; Falorsi et al., 2018; Chen et al., 2019; Dupont et al., 2020), capsule neural networks (Sabour et al., 2017; Hinton et al., 2018), and equivariant representation learning models (Park et al., 2022; Miyato et al., 2022; Koyama et al., 2023). In these works, geometric infor- transformation applied to latent variables of neural networks. Suppose (xx)
mation is provided is an encoded feature, where 4 is a neural network, x is an input feature, and M is an associated transformation (e.g. rotation). Then the pair (o))) M) is identified with (x) We integrate this feature transformation into the attention to break its permutation symmetry.
We briefly introduce the notion of group and a representation be- cause we describe our proposed attention through the language of group theory, which handles different geometric structures in a unified manner, such as camera transformations and image posi- tions. In short, group G with its element g. is associative set that is closed under multiplication, has the identity element and each element has an inverse. E.g. the set of camera transformations satisfies the axiom of a group and is called special Euclidean group: (F). A (real) representation is function p G + (TR) such that p(g)p(g') = p(gg') for any g, g E G. The property p(g))(g') p(gg') ) is called homomorphism. Here, (T.)((R)) denotes the set of d x d invertible real-valued matrices. We denote by Pg == p(g) E iRdxd a representation of g. A simple choice for
Group and representation:
the representation Py for E SE(3) is a 4 X 4 rigid transformation matrix [ 1'1 E R-4x4 where R E RR33 a 3D rotation and T E R3x1 a 3D translation. A block concatenation of multiple group representations is also representation. What representation to use is the user's choice. We will present different design choices of p for several NVS applications in Section 3. 3.2 and A.3.2.
3.1 GEOMETRIC TRANSFORM ATTENTION
Suppose that we have token features X E Rxd and list of geometric attributes g [9 ........ Inl where 9 is an i-th token's geometric attribute represented as a group element. For example, each patch feature, and 9: corresponds to camera transformation and an image patch position. Given a representation p and Q K, K,V== =WQ, XW XWK,XWV E R we define our geometry-aware attention given query Qi Rdd by:
X, E Rd corresponds
O - n
exp(Q. 99.07" (x -p==1 explQ. (p ( Kgy)) (P 9:9, vV)),
K2))
(4)
3

---

Published as a conference paper at ICLR 2024
Using the homomorphism property Po.g," Ps.Po the above equation can be transformed into exp Q1)² (Ps, kK)))
O, Ps:
E5 exp c (p Qi) (po7' Ky)) (Po, vV))
(5)
Note that the latter expression is computationally and memory-wise more efficient, requiring com- putation and storage of n²² values of each (P 99007 Kj. Pa.o, V2) in Eq. (4) versus only n values
for (p Qi Pag Kj Pas Vi) and Pg: Os in Eq. (5), where Ō is the output of the leftmost sum.
Eq. (5), given all queries Q, can be compactly rewritten in an implementatioon-fienndy form:
0 O= Pg @ Attn (P&" Q C K,P の (6)
VV)
p
where Pg denotes list of representations for different tokens: Pg [Ps................ ........P....................... (. and "@" denotes token-wise matrix multiplication: Px @
Attention
K- = [0 Ki Ps.. Kn|*= E Rnxdd Also, the transpose and the in- verse -1 operate element-wisst on Pg g (e.g., Pg T := 10 :............................... ...... We
p
p
p
call the attention mechanism in Eq. (6) geometric transform attention (GTA) and show the diagram of (6) in Fig. 1. Note that the additional computation of GTA is smaller than the QKV attention and the MLP in the transformer when constructing Pg from a set of small matrices, which we will detail in Section 3.2 and in Appendix A.
Q
K
V
A simple NVS experiment: We first demonstrate that GTA improves learning as compared to APE and RPE in simplified NVS experiment. We construct setting where only camera rotations are relevant to show that the complexity Py can be adapted to the problem complexity. A single empty scene surrounded by an enclosing sphere whose texture is shown in Fig. left considered. All cameras are placed in the center of the scene where they can be rotated but not translated. Each scene consists of 8 context images with 32x32 pixel resolution rendered with pinhole camera model. The camera poses are chosen by randomly sampling camera rotations. We randomize the global coordinate system by setting it to the first input image. This increases the difficulty of the task and is similar to standard NVS tasks, where the global origin may be placed anywhere in the scene. The goal is to render a target view given its camera extrinsic and a set of context images.
Fig. 1: GTA mechanism. p= and p together take Q, K and V a shared coordinate space, and the P gets the attention output back to each token's 's coor- dinate space.
We employ a transformer-based encoder-decoder architecture shown on the right of Fig. 2. Camera extrinsics in this experiment form the 3D rotation group: SO(3). We choose Pg to be block concatenation of the camera rotation matrix:
Pg.
Ra $ ... R.
d/3 times
where R; is the 3 x 3 matrix representation of the extrinsic 9: E SO(() linked to the i-th token. A @ denotes conk-concatenation: A B I BJ. Because here each Pg. orthogonal, the transpose of Pg. becomes the inverse, thus the same transformation is applied across query, key, and value vector for each patch.
We compare this model to APE- and RPE-based transformers as baselines. For the APE-based transformer, we add each flattened rotation matrix associated with each token to each attention layer's input. Since we could not find an RPE-baaed method that is directly applicable to our setting with rotation matrices, we use an RPE-version of our attention where instead of multiplying the matrices with the QKV features, we apply the matrices to biases. More specifically, for each head, we prepare learned bias vectors 89, 6², 6V E IRD concatenated with each of the QKV vectors of each head and apply the representation matrix defined by p(g) := R e R @ R Rx9 only to the bias vectors. We describe this RPe-version of GTA in more detail in Appendix C.I.
Fig. 3 the left shows that the Gas-based transformer outperforms both the APE and RPE-based transformers in terms of both training and test performance. In Fig. 3 on the right, the GTA-based transformer reconstructs the image structure better than the other PE schemes.
4

---

Published as a conference paper at ICLR 2024
Encoder
Decoder
Context
.t.
GTA sall
FF
GTA
FF
GTA cross
GTA cross
Output
images
.NN
Target query
Fig. 2: Synthetic experiment. Left: Texture of the surrounding sphere. Right: Model architecture.
The query pair consists of learned constant value and target extrinsic g"
0.02
Ground truth
GTA
RPE
APE
GTA
RPE
0.01
MSE
APE
0.005
0.002
Steps
Fig. 3: Results on the synthetic dataset. Left: The solid and dashed lines indicate and train errors. Right: Patches predicted with different PE schemes.
3.2 TOKEN STRUCTURE AND DESIGN OF REPRESENTATION FOR NVS
In the previous experiment, tokens were simplified to comprise an entire image feature and an asso- ciated camera extrinsic. This differs from typical NVS model token structures where patched image tokens are used, and each of the tokens can be linked not only to a camera transformation but also 2D location within image. To adapt GTA such NVS models, now describe how associate each feature with geometric attribute and outline one specific design choice for
Token structure: We follow common way to compose the input tokens for the transformer as in (Sajjadi et al., 2022b; Du et al., 2023). We assume that for each view, we have image patches or pixels of the size of H x W, and each patch or pixel token consists pair of feature value a camera extrinsic E SE(() and a 2D image position. For image PE, it would be natural encode each position as an element of the 2D translation group T(2). However, we found, similarly to the Fourier feature embeddings used APE and RPE and rotary PE (Su et al., 2021), encoding the image positions as elements of the 2D rotation group SO( exhibits better performance than using element of the direct product of the two SO(2)
(c, On, 0w) 0
0n
Rd and geometric attributes that are
view
Fig. 4: Geometric attributes.
T(2). Thus, represent each image position groups: (fn, fw) E (0(2) x 50(2) where 0n, 0n E [0, 2T). Here, we identify the S0(2) element with the 2D rotation angle. We associate the top left patch (or pixel) with the value (0, 0), while the bottom right patch corresponds to (2 -1) 2 1)/W). For the intermediate patches, we compute their values using linear interpolation of the angle values between the top left and bottom right patches. Overall, we represent the geometric attribute of each token of the i-th view by
g = (c,, On, O E SE(3) SO(2) x SO(2) =: G.
Fig. 4 illustrates how represent each geometric attribute of each token.
Design of p: What representation to use a design choice similar to the design choice of the embedding in APE and RPE. As specific design choice for the representation for NVS tasks, we propose to compose Pg by the direct sum of multiple irreducible representation matrices, each responding to a specific component of the group G. Specifically, Pg composed of four different types of representations and is expressed in block-diagonal form as follows:
Pg = σ camm (c) @ o fot (r) @ oh @u (0,) @ 0
$0 (0"(w)
where "Ø" denotes clock-concatenation
+ B = 0 B
0 and t0 indicates repeating the block
concatenation of A a total of a times. We introduce additional representation Frot (r) that captures
5

---

Published as a conference paper at ICLR 2024
Table Components of Pg-
J Jeam (c)
rot (r)
on (0p)
(ww)
D!!I)
(M (h)
TM
R T 0 1
matrix form
Dr Nrord
M
(fNa)
M
(
multiplicity
b
v
Table Test metrics. Left: CLEVR-TR, Right: MSN-Hard. +Models are trained and evaluated on MultiShapeNet, not MSN-Hard. They are different but generated from the same distribution.
PSNR
PSNRT
LPIPS
SSIM
APE
33.66
LFN' (Sitzmann al., 2021)
14.77
0.582
0.328
RPE
36.08
PixeINeRFi (Yu et al., 2021b)
21.97
0.332
0.689
SRT
33.51
SRT (Sajjadi al., 2022b)
24.27
0.368
0.741
RePAST
37.27
RePAST (Safin et al., 2023)
24.48
0.348
0.751
GTA (Ours)
39.63
SRT+GTA (Ours)
25.72
0.289
0.798
Context images
26
PSNR
24
SRT
RePAST
GTA (Ours)
Ground truth
23
GTA
RePAST
21
2 Steps
Fig. 6: Validation PSNR curves on MSN-Hard.
Fig. 5: Qualitative results on MSN-Hard.
only the rotational information of c, with which we find moderate improvements in performance. Table summarizes the matrix form we use for each representation. Specifically, M( is 2D rotation matrix with frequency f that is analogous to the frequency parameter used in Fourier feature embeddings in APE and RPE. D can be thought of as the 3D version of ( Please refer to Appendix A.2 for more detailed descriptions these matrices. Fig. in the Appendix displays the actual representation matrices used in our experiments. The use of the Kronecker product is also typical way compose representations, which describe in Appendix A.3.2.
EXPERIMENTAL EVALUATION
We conducted experiments on several sparse NVS tasks to evaluate GTA and compare the recon- struction quality with different PE schemes well as existing NVS methods.
Datasets: We evaluate our method on two synthetic 360 datasets with sparse and wide base- line views (CLEVR-TR and MSN-Ilard) and on two datasets of real scenes with distant views (RealEstatelOk and ACID). We train a separate model for each dataset and describe the properties of each dataset below. CLEVR with translation and rotation (CLEVR-TR. is multi-view version of CLEVR (Johnson et al., 2017) that we propose. It features scenes with randomly arranged basic objects captured by cameras with azimuth, elevation, and translation transformations. We this

---

Published as a conference paper at ICLR 2024
Table Results on RealEstatel0K and ACID, Top: NeRF methods. Bottom: transformer methods.
RealEsttate 10k
ACID LPIPS
PSNR1
LPIPS
SSIM
PSNR
SSIM
PixeINeRF (Yu et al., 2021b)
13.91
0.591
0.460
16.48
0.628
0.464
StereoNeRF (Chibane et al., 2021)
15.40
0.604
0.486
IBRNet (Wang et al., 2021a)
15.99
0.532
0,484
19.24
0.385
0.513
GeoNeRF (Johari et al., 2022)
16.65
0.541
0.511
MatchNeRF (Chen et al., 2023)
23.06
0.258
0.830
GPNR (Suhail et al., 2022)
18.55
0.459
0.748
17.57
0.558
0.719
Du et al. (2023)
21.65
0.285
0.822
23.35
0.334
0.801
Du al. (2023) GTA (Ours)
22.85
0.255
0.850
24.10
0.291
0.824
Context images
Du et al. (2023)
GTA (Ours)
Ground truth
Fig. 7: Qualitative results. Top: ACID, Bottom: RealEstate 10k.
dataset measure the ability of models to understand the underlying geometry of scenes. We the number of context views to for this dataset. Generating 360" images from 2 context views is challenging because parts of the scene will be unobserved. The task is solvable because all rendered objects have simple shapes and textures. This allows models to infer unobserved regions if they have a good understanding of the scene geometry. MultiShapeNet-Hare (MSN-Hard) is a challeng- ing dataset introduced in Sajjadi et al. (2022a;b). Up to 32 objects appear in each scene and are drawn from 51K ShapeNet objects (Chang al., 2015), each of which can have intricate textures and shapes. Each view is captured from a camera pose randomly sampled from 360 viewpoints. Objects test scenes are withheld during training. MSN-Hard assesses both the understanding of complex scene geometry and the capability to learn strong 3D object priors. Each scene has 10 views, and following Sajjadi et al. (2022a;b), we use views as context views and the remaining views as target views. RealEstate 10k (Zhou et al., 2018) consists of real indoor and outdoor scenes with estimated camera parameters. ACID (Liu et al., 2021) is similar to RealEstate 10k, but solely includes outdoor scenes. Following Du al. (2023), during training, we randomly select con- text views and one intermediate target view per scene. At test time, we sample distant context views with 128 time-step intervals and evaluate the reconstruction quality of intermediate views.
Scene representation transformer (SRT) (Sajjadi et al., 2022b), a transformer-based NVS method, serves as our baseline model on CLEVR-TR and MSN-Hard. SRT is a similar ar- chitecture to the one we describe in Fig. 2, but instead of the extrinsic matrices, SRT encodes the ray information into the architecture by concatenating Fourier feature embeddings of rays to the input pixels of the encoder. SRT is an APE-based model. Details of the SRT rendering process are provided in Appendix C.2.1 and Fig. 15. We also train another more recent transformer-based NVS model called RePAST (Safin et al., 2023). This model is variant of SRT and encodes ray informa- tion via an RPE scheme, where, in each attention layer, the ray embeddings are added to the query and key vectors. The rays linked the queries and keys are transformed with the extrinsic matrix associated with a key-value token pair, before feeding them into the Fourier embedding functions, to represent both rays in the same coordinate system. RePAST is the current state-of-the-art method
Baselines:
7

---

Published as a conference paper at ICLR 2024
Table 4: PE schemes. MLN: Modulated layer normalization (Hong et al., 2023; Liu et al., 2023a).
ElemMul: Element-wise Multiplication. GBT: reometry-biasee transformers (Venkat et al., 2023).
FM: Frustum Embedding (Liu al., 2022). RoPE+FTL: (Su et al., 2021; Worrall et al., 2017).
MLN
SRT
ElemMul
GBT
FM
RoPE+FTL
GTA
PSNR
32.48
33.21
34.74
35.63
37.23
38.18
38.99
Table 5: Effect of the transformation V. Left: Test PSNRs on CIEVR-TR and MSN-Hard.
Right: Inception scores (IS) and FIDs of DiT-B/2 (Peebles & Xie, 2023) on 256x256 ImageNet.
ISt FID-50K
CLEVR-TR MSN-Hard
DiT (Peebles & Xic, 2023)
145.3
7.02
No Pg on V GTA
36.54
23.77
DiT + 2D-RoPE
151.8
6.26
38.99
24.58
DiT + GTA
158.2
5.87
on MSN-Hard. The key difference between GTA and RePAST is that the relative transformation is applicd directly to QKV features in GTA, while is applied to rays in RePAST.
For RealEstate 10k and ACID, we use the model proposed in Du et al. (2023), which is the state-of- the-art model on those datasets, as our baseline. Their model similar to SRT, but has architectural improvements and uses an epipolar-based token sampling strategy. The model encodes extrinsic matrices and 2D image positions to the encoder via APE, and also encodes rays associated with each query and context image patch token in the decoder via APE.
We implement our model by extending those baselines. Specifically, we replace all attention layers in both the encoder and decoder with GTA and remove any vector embeddings of rays, extrinsic matrices, and image positions from the model. We train our models and baselines with the same settings within each dataset. We train each model for 2M and 4M iterations on CLEVR-TR and MSH-Hard and for 300K iterations on both RealEstate 10k and ACID, respectively. We report the reproduced numbers of baseline models the main tables and show comparisons between the re- ported values and our reproduced results in Table 16 and Table 17 in Appendix C.2. Please also see Appendix C.2 for more details about our experimental settings.
Results:
Tables 2 and 3 show that GTA improves the baselines in all reconstruction metrics on all datasets. Fig. shows that on MSN-Hard, the GTA-based model renders sharper images with more accurate reconstruction of object structures than the baselines. Fig. 7 shows that our GTA- based transformer further improves the geometric understanding the scenes over Du et al. (2023) as evidenced by the sharper results and the better recovered geometric structures. Appendix D provides additional qualitative results. Videos are provided in the supplemental material. We also train models, encoding 2D positions and camera extrinsies via APE and RPE for comparison. See Appendix C.2.3 for details. Fig. 6 shows that GTA-based models improve learning efficiency over SRT and RePAST by significant margin and reach the same performance RePAST using only 1/6 of the training steps on MSN-Hard, GTA also outperforms RePAST in terms of wall-clock time as each gradient update step slightly faster than RePAST, see also Table 14 in Appendix B.8.
Comparison to other PE methods: We compare GTA with other PE methods on CLEVR-TR. All models are trained for IM iterations. See Appendix C.2.4 for the implementation details. Ta- ble 4 shows that GTA outperforms other PE schemes. GTA is better than RoPE+FTL, which uses RoPE (Su et al., 2021) for the encoder-decoder transformer and transforms latent features of the encoder with camera extrinsics (Worrall et al., 2017). This shows the efficacy of the layer-wise geometry-aware interactions in GTA.
Effect of the transformation on V: Rotary positional encoding (RoPE) (Su et al., 2021; Sun et al., 2022) is similar to the 50(2) representations in GTA. An interesting difference to GTA is that the RoPE only applies transformations to query and key vectors, and not to the value vectors. In our setting, this exclusion leads to a discrepancy between the coordinate system of the key and value vectors, both of which interact with the tokens from which the query vectors are derived. Table 5 left shows that removing the transformation on the value vectors leads to a significant drop in performance in our NVS tasks. Additionally, Table 5 right shows the performance the ImageNet
8

---

Published as a conference paper at ICLR 2024
Query
RePAST
GTA
0.6
GTA
RoPAST
PR-AUC
0.5
0.3
0.2
0.1
Layer
Fig. 8: Attention analysis. Given query token (white region), the attention weights on context view are visualized. GTA can identify the shape of the object that corresponds to the given query. Right: Quantitative evaluation alignments between attention matrices and object masks.
Table 6: Representation design. Test PSNRs of models trained for 1M iterations.
(a) (V): representations are not used in the encoder. Camera and image position are needed in the decoder to identify which pixel render.
(b) Image positions encodings. 'Single frequency.
SE(3) SO(2) SO(3) CLEVR-TR MSN-Hard
CLEVR-TR MSN-Hard
<535
37.45
20.33
SE(3) + T(2)
37.20
23.69
38.26
23.82
SE(3) SO(2)*
38.82
23.98
38.99
24.58
SE(3) + SO(()
38.99
24.58
/
39.00
24.80
generative modeling task with diffusion models. Even on this purely 2D task, the GTA mechanism is better compared RoPE as an image positional encoding method (For more details of the diffusion experiment, please refer to Appendix C.3).
Object localization: As demonstrated Fig. 8 on MSN-Hard, the GTA-based transformer not only correctly finds patch-to-patch associations but also recovers patch-to-object associations already in the second attention layer of the encoder. For quantitative evaluation, we compute precision- recall-AUC (PR-AUC) scores based on object masks provided by MSN-Hard. In short, the score represents, given a query token belonging to a certain object instance, how well the attention matrix aligns with the object masks across all context views. Details how compute PR-AUC provided in Appendix B.7. The PR-AUCs for the second attention layer are 0,492 and 0.204 with GTA and RePAST, respectively, which shows that our GTA-based transformer quickly identifies where to focus attention at the object level.
Representation design: Table 6a shows that, without camera encoding (SE(3) or image PE (SO(2)) in the encoder, the reconstruction quality degrades, showing that both representations are helpful in aggregating multi-view features. Using SO(3) representations causes a moderate im- provement on MSN-Hard and no improvement on CLEVR-TR. A reason for this could be that MSN-Hard consists of wide variety of objects. By using the SO(3) representation, which is invariant to camera translations, the model may be able to encode object-centric features more effi- ciently. Table 6b confirms that similar to Fourier feature embeddings used in APE and RPE, multiple frequencies of the (2( representations benefit the reconstruction quality.
CONCLUSION
We have proposed novel scometry-aware attention mechanism for transformers and demonstrated its efficacy by applying it to sparse wide-baselin novel view synthesis tasks. A limitation of GTA is that GTA and general PE schemes rely on known poses or poses estimated by other algorithms, such as COLMAP (Schönberger & Frahm, 2016). An interesting future direction is to simultaneously learn the geometric information together with the forward propagation of features in the transformer. Developing algorithm capable of autonomously acquiring such structural information solely from observations, specifically seeking a universal learner for diverse forms of structure akin to human capacity, represents captivating avenue for future research.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
Takeru Miyato, Bernhard Jaeger, and Andreas Geiger were supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC number 2064/1 project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for sup- porting Bernhard Jaeger. We thank Mehdi Sajjadi and Yilun Du for their comments and guidance on how To reproduce the results and thank Karl Stelzner for his open-source contribution of the SRT models. We thank Haofei Xu and Anpei Chen for conducting the MatchNeRF experiments. We also thank Haoyu He, Gege Gao, Masanori Koyama, Kashyap Chitta, and Naama Pearl for their feedback and comments. Takeru Miyato acknowledges his affiliation with the ELLIS (European Laboratory for Learning and Intelligent Systems) PhD program.
REFERENCES
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling, Geometric and physical quantities improve E(3) equivariant message passing. In Proc. of the International Conf. on Learn- ing Representations (ICLR), 2022. 18
Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformers. In Ad- vances Neural Information Processing Systems (NeurIPS), 2023. 18
Nicolas Carion, Francisco Massa, Gabriel Synnacve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 2
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rice 3D Model Repository. Technical Report arXiv:1512.0301 [cs.GR], Stanford University Princeton University Toyota Technological Institute Chicago. 2015. 7.24
Xu Chen, Jie Song, and Otmar Hilliges. Monocular neural image based rendering with continuous view control. In Proc. IEEE Conf. on Compuder Vision and Pattern Recognition (CVPR), 2019.
Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Explicit correspon- dence matching for generalizable neural radiance fields. arXiv.org, 2304. 2304.12294 12294, 2023. 7
Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (SRF): learning view synthesis for sparse views of novel scenes. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 7
Gregory S Chirikjian. Engineering applications of noncommutative harmonic analysis: with emphasis on rotation and motion groups. CRC press, 2000. 16
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Reaz, and Andreas Geiger. Transfuser: Imitation with transformer-bas sensor fusion for autonomous driving. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 2022. 2
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral enn. In Proc. of the International Conf. on Machine learning (ICML) 2019.
Taco Cohen and Max Welling. Transformation properties learned visual representations. Proc. of the International Conf. on Learning Representations (ICLR), 2014. 3
Pim De Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 18
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Un- terthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 1,2
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide- baseline stereo pairs. In Proc. IEFF Conf. on Computer Vision and Pattern Recognition (CVPR), 2023. 1,2, 19, 20, 21, 22, 24, 25, 26, 31,
10

---

Published as a conference paper at ICLR 2024
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivari- ant neural rendering. In Proc. of the International Conf. on Machine learning (ICML), 2020.
Luca Falorsi, Pim De Haan, Tim R Davidson. Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen. Explorations in homeomorphic variational auto-encoding. arXivoorg, 1807.04689, 2018.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Eti- enne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubrie: scalable dataset generator. In CVPR. 2022. 24
Lingshen He, Yiming Dong, Yisen Wang. Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. In
Advances in Neural Information Processing Systems (NeurlPS), 2021. 18
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-1 Yu. Epipolar transformers. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2020. 2
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang- Transforming auto-encoders. In Proc. of the Interna- tional Conf. Artificial Neural Networks (ICANN), 2011. 3
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In Proc. of the International Conf. on Learning Representations (ICLR), 2018. 3
Yicong Hong, Kai Zhang, Jiuxiang Gu. Sai Bi, Yang Zhou, Difan Liu, Feng Liu. Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXivoorg, 2311.04400, 2023. 2, 8, 27
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. Proc, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 7
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrenee Zitnick, and Ross Gir- shick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
Masanori Koyama, Kenji Fukumizu, Kohei Hayashi, and Takeru Miyato. Neural fourier transform: A general approach to equivariant representation learning. arXiv.org, 2305. 18484, 2023.
Jonáš Kulhánek, Erik Demerr, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. Proc. of the European Conf. on Computer Vision (ECCV), 2022.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bev- former: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2017.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjao Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 7
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharow. and Carl Vondrick. Zero-I-ua 3: Zero-shot one image to 3d object. arXiv.org, 2303. 10.3.11128... 2023a. 2, 8, 27
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi- view 3d object detection. In Proc. of the European Conf. Computer Vision (ECCV), 2022. 2, 8, 26
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A uni- fied framework for 3d perception from multi-camera images. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv.org, 1711.05101, 2017. 22, 24
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Advances Neural Information Processing Systems (NeurIPS), 2022.
11

---

Published as a conference paper at ICLR 2024
Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. In Proc. of the International Conf. on Machine learning (ICML), 2022. 3
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. of the IEEE Interna-
tional Conf. on Computer Vision (ICCV). 2023. 8. 15, 28, 33, 34, 35, 36, 37
Lang Peng, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. Bevsegformer: Bird's eye view
semantic segmentation from arhitrary camera rigs. In Proc. of the IEEE Winter Conference Applications of Computer Vision (WACV), 2023. 2
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proc. of the Conf. Artificial Intelligence (AAAI), 2018. 2
Zheng Qin, Hao Yu, Changjian Wang. Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand- alone self-attention in vision models. Advances in Neural Information Processing Systems (NeurlPS), 2019.
René Ranitl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proc. of the IEEE International Conf. Computer Vision (ICCV), 2021. 2.
Cody Reading, Ali Harakeh, Julia Chae, and Steven L.. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021. 26
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware reppes---tttrrreeeeeetttttttai for 3d human pose estimation. In Proc. of the European Conf. on Computer Vision (ECCV), 2018. 3
David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoomn. Attentive group cquivariant convolu- tional networks. In Proc. of the International Conf. on Machine learning (ICML), 2020. 2
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 11 115(3):211- 1-252, 2015. 28
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in
Neural Information Processing Systems (NewrIPS), 2017. 3
Aleksandr Safin, Daniel Durckworth, and Mehdi SM Sajjadi. RePAST: Relative pose attention scene represen-
tation transformer. arXiv.org. 2304.00947, 2023. 2, 6, 7, 19, 24
Avishkar Saha, Oscar Mendez Maldonado, Chris Russell, and Richard Bowden. Translating Images into Maps. In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2022.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran. Sjoerd van Steenkiste, Filip Pavetić, Mario Lučic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In Advances in Neural Information Processing Systems (NeurIPS). 2022a. 24
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann. Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- free novel view synthesis through set-latent scene representations. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022h. 1, 2, 5, 6, 24
Johannes Lutz Schönberger and Jan Michael Frahm. Structure-fiee coom-mttoon revisited. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
Hao Shao, Letian Wang. Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. In Proc. Conf. on Robot Learning (CoRL), 2023. 2
Peter Shaw, Jakoh Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In An- nual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL- HLT), 2018. 1,2
Changyong Shu, Jiajun Deng. Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for transformer- based multi-camera 3d object detection. In Proc. of the IEFE International Conf. on Computer Vision (ICCV), 2023. 2
12

---

Published as a conference paper at ICLR 2024
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field net- works: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding, arXiv preprint arXiv:2104.00 09864, 2021. 5, 8. 27,
Mohammed Suhail, Carlos Esteves. Leonid Sigal, and Ameesh Makadia. Generalizable patch- match-based neural rendering. In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2,7
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang. Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length- extrapoaatae transformer. In ACL, 2022.
Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In Proc. of the International Conf. Learning Representations (ICLR), 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. 1,2
Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry-biased transformers for novel view synthesis. arXiv.org, 2301.04651 2023, 2, 8,
Shaofei Wang, Andreas Geiger, and Siyu Tang. Locally aware piccewise transformation fields for 3d human mesh registration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021a. 2,
Shihao Wang. Yingfei Liu, Tiancai Wang. Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. arXiv.org, 2303.11926, 2023.
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7794-7803, 2018. 1,2
Yue Wang. Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3d object detection from multi-vion images via 3d queries. Proc. Conf. on Robot Learning (CoRL), 2021b,
Daniel Watson, William Chan, Ricardo Martin-Brualiy Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In Proc. of the International Conf. on Learning Representations (ICLR), 2023. 2
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel Brostow, Interpretable trans- formations with encoder-decnder networks. In Proc. of the IEFE International Conf. on Computer Vision (ICCV), 2017. 3, 8, 27
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 2
Haofei Xu, Jing Zhang. Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Trans, on Pattern Analysis and Machine Intelligence (PAMI), 2023.
2
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021a. 2
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021b. 6,7
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances Neural Information Processing Systems (NeurIPS), volume 32, 2019. 28
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 586-556655 595,5995222 2018. 26
Brady Zhou and Philipp Krähhebüüh. Cross-view Transformers for real-time Map-view Semantic Segmenta- tion. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR). 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Sterco magnification. ACM Transactions on Graphics, 2018.
13

---

Published as a conference paper at ICLR 2024
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In Proc. of the International Conf. Learning Representations (ICLR), 2021.
Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo. Yangguang Li. Ding Liang, Yan-Pei Cao, and Song- Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction transformers. arXiv.org, 2312.09147, 2023.
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Additional details GTA
15
Computational complexity..
A.2 Details the representation matrices
15
A.3 Variants of GTA
17
A.4 Relation to equivariant and gauge equivariant networks
18
B Additional experimental results
19
B.I Training curves CLEVR-TR and MSN-Hard.
19
B.2 Results with higher resolution
19
B.3 Results with 3 context views
19
B.4 Robustness camera noise
19
Performance with different random seeds
20
B.6 Performance dependence on reference coordinates
20
B.7 Analysis of attention patterns
21
B.8 Computational time
21
C Experimental settings
22
C.I Details of the synthetic experiments in Section 3.1
22
C.2 Experimental settings in Section 4
22
C33 2D image generation with DiT (Peebles & Xie, 2023)
28
Rendered images
29
Generated images DiTs
33
A ADDITIONAL DETAILS OF GTA
Algorithm provides algorithmic description based on Eq. (6) for single-head self-attention. For multi-head attention, we simply apply the group representations to all QKV vectors of each head.
A.1 COMPUTATIONAL COMPLEXITY
Since the P & @ operation is an n-times multiplication of x matrix with a d-dimensional vector, the computational complexity of additional computation for our attention over the vanilla attention O(nd)). This be reduced by constructing the representation matrix with block diagonal, with each block being small. we keep the largest block size of the representation constant against and d, then the order of the P operation becomes ((nd). Thus, if dmax is relatively small, or we increase n or d, the computation overhead of the @ operation becomes negligible compared to the computation times of the other components of transformer, which are O(n²d) for attention and (ndd) for feedforward layers. In our experiments, we use a block-structurre representation with maximum block size of (see Section 3.2 and Fig. 9).
A.2 DETAILS OF THE REPRESENTATION MATRICES
Pg is composed of four different types of representations Pe: Pry Pa., Po... with the multiplicities of u, Below, describe the details of each representation.
15

---

Published as a conference paper at ICLR 2024
Algorithm GTA for single head self-attention.
Input: Input tokens: X E RNxd group representations: Pg Wº, w*, WV E Rdxd
Pyg Paz. ... Pexl, and weights:
1. Compute query, key, and value from X:
Q = XwQ, K XW* v = xW
2. Transform each variable with the group representations:
Q P Q, Ps -
-
3. Compute QKV attention in the same way as in the vanilla attention:
QK T
0 = softmax
V
Vd
Apply group representations to O:
0 f- P-P 0
Return
(c):(c): We use a homogenous rigid transformation as the representation of c SE(3):
= R T 1
Ceam (c)
E Rxx
(10)
Troc We compose Tou (r) via block concatenation of Wigner-D-men D-matrices (Chirikjian, 2000).
Trot (r) == θ (1x) O (r), o (r) D E 1122+1))
(11)
where D. 1-th Wigner-D-matee given Here, DaES A(a) == A e.... ((1)) and we omit the index set symbol from the above equation. We use these matrices because Wigner-D-maarrces are the only irreducible representations of SO(3). Any linear representation or, r SO(3) is
equivalent to direct sum of the matrices under similarity transformation (Chirikjian, 2000).
(0,) and an (Ow): Similar to (ron()), we use 2D rotation matrices with different frequencies for on (0n) and a (0w). Specifically, for 0% (0n) given a set of frequencies {fk [ representation follows:
NN, we define the
Th (On) = Onon
(1w) (0n), (O)), o0 (0n)) = M, =
[cos(f0r) sin(ffr)) sin(ffr) cos(ffr)
E 22x2
(12)
Tn (f..) is defined analogously.
We use the following strategy to choose the multiplicities s, t, and frequencies (4), (fr {fu):
1. Given the feature dimension d of the attention layer, we split the dimensions into three compo- nents based on the ratio of 2 : 1.
2. We apply o Ss cam to the first half of the dimensions. As deam does not possess multiple fre- quencies, its multiplicity is set to d/8.
Ør is applied to a quarter of the dimensions. For the frequency parameters (1) of drats consistently use the 1st and 2nd degrees of the Winger-D matrices. Considering the combined sizes these matrices is 8, the multiplicity for orot becomes d/32.
For the remaining 1/4 of the dimensions of each QKV vector, we apply both @t and @u Regarding the frequency parameters {fx.. {fwr), }. we utilize d/16 16 octaves with the maximum frequency set at for both d), and dw- The multiplicities for these are both 1.
Based on this strategy, we use the multiplicities and frequencies shown in Table 7. Also Fig. displays the actual representation matrices used on the MSN-Hard dataset.
16

---

Published as a conference paper at ICLR 2024
0
72
96
Fig. 9: Representation matrices MSN-Hard. Left: with SO(3), Right: without SO(3). Left: Dimensions 1-48 correspond to a (c), (c), dimensions 49-72 correspond to o (r), and dimensions 73-96 correspond to on (fh) and (w Right: Dimensions 1-48 correspond to o #12 (c) and di- mensions 49-96 correspond to 0% (0%) and (0w.)
Table 7: Multiplicity and frequency parameters. Here, d is the dimensions of each attention head. Since the baseline model on RealEsate 10k and ACID uses different feature sizes for query- key vectors and value vectors, we also use different sizes of representation matrices for each feature.
fs,t, u, o}
(4,,... IN..
{fi..... Fi.....AN.oo LN (in
CLEVR-TR
18,3,1,1 1,
[1,2)
(1. 1/23)
64
CLEVR-TR wo/ 5O(3)
1,
(1,1/2,
1/27
MSH-Hard
112,3,1,1
{1,2}
[1, 1/.../.,,111/. 1/25
96
MSH-Hard wo/ SO(3)
(12,0,1,1
11,1/2,1/4,..
1/2111
Realestate 10k and ACID (Encoder)
64
(8,3,1,1
[1,2]
(1,-.., 1/23
Realestate 10k and ACID (Decoder. key)
128
{16, 6,
[1,2]
(1, 1/27
Realestate 10k and ACID (Decoder, value)
256
{32, 1,1
[1,2]
[1. 1/215
A.3 VARIANTS OF GTA
Here we would like to introduce two variants of GTA. The one the Euclidean version of GTA where the Euclidean distance for the attention similarity. The other one is GTA with group representation composed the Kronekcer product of smaller representations. We see Table 8 that the performances of those variants of GTA little degraded but relatively close to the original GTA. We will detail each variant in the following sections.
A.3.1 EUCLIDEAN GTA
The unconventional aspect of Eq. (6) is the presence of the transpose in the transformation of the query vectors. The transpose is necessary for having the reference coordinate invariance, and the need arises from the fact that the dot-product similarity is not invariant under SE(3) transforma- tions when the translation component is non-zero. To ensure both reference coordinate invariance and consistent transformations across the Q, K, V vectors, ity for computing the attention matrix instead the dot-product similarity. The formula for the self-attention layer with squared Euclidean distance is given by:
utilize the Euclidean similar-
Attniuetial (Q, K, V) softmax(E(Q,
(13)
where K) E R N EG, = -111 - -K,12
(14)
Then the Euclidean version of GTA (GTA-Euclid) is written in the following form:
0 P Attmpuelid (P&
Q.P K,PE 1 V).
(15)
17

---

Published as a conference paper at ICLR 2024
Eq. (15) possesses the reference coordinate invariance property since the square distance is preserved under rigid transformations. The numbers Table 8 are produced under the same setting as the original GTA, except that replaced the dot-product attention with the Euclidean attention that describe above.
A.3.2 KRONECKER GTA
Another typical way to compose representation matrix is using the Kronecker product. The Kro- necker product of two square matrices A, B E IR mx R is defined as:
01 B
01m
A B
R
m
(16)
0m1 B
0mm
The important property of the Kronecker product is that the Kronecker product of two representa- tions is also a representation: (p1 @ p2) = (01 @ (g)(() @ (g) where (pr & (g) = (1()) @ p2(g). We implement the Kronecker version of GTA, which denote STA--Konecker, where we use the Kronecker product of the SE(3) representation and SO(2) representations as representation Py:
Pg = Peam (c) @ (pn(0%) @ (ww)), where (c, On, 0w.
(17)
In the results presented in Table 8, the multiplicity of Peam. Ph. are set to 1, and the number of frequencies for both Pa and Pwo is set to 4 on CLEVR-TR and 6 MSN-Hard.
Table 8: Results with different representation forms.
CLEVR-TR MSN-Hard
G"TA-Kronecker
38.32
24.52
GTA-Euclid
38.59
24.75
GTA
38.99
24.80
A.4 RELATION TO EQUIVARIANT AND GAUGE EQUIVARIANT NETWORKS
The gauge transform used in gauge equivariant networks (Cohen et al., 2019; De Haan et al., 2021; He al., 2021; Brandstetter et al., 2022) related the relative transform p(g:9)") used in attention mechanism. However, the equivariant models and ours differ because they are built on different motivations. In short, the gauge equivariant layers are built preserve the feature field structure determined by gauge transformation. In contrast, since image features themselves are 3D-structured, our model applies the relative transform only the query and key-value pair in the attention mechanism but does not impose equivariance on the weight matrices of the attention and the feedforward layers. The relative transformation in GTA can be thought of as form of guidance that helps the model learn structured features within the attention mechanism from the initially unstructured raw multi-view images.
Brehmer et al. (2023) introduce geometric algebra (GA) to construct equivariant transformer net- works. Elements of GA are themselves operators that can act on GA, which may enable us to construct expressive equivariant models by forming bilinear layers that allow interactions between different multi-vector subspaces. In NVS tasks, where the input consists of raw images lacking geometric structure, directly employing such equivariant models may not be straightforward. How- ever, integrating GA into the GTA mechanism could potentially enhance the network's 's expressivity, warranting further investigation.
18

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 TRAINING CURVES ON CLEVR-TR AND MSN-HARD.
25.5
25.0
MSE
10
36
MSF
0.006
23.0
22.0
10
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.6
15
2.0
0.003
Steps
Steps
Steps
Steps
GTA
RePAST
SRT
GTA
RePAST
SRT
Fig. 10: Training and validation curves, Left: CLEVR-TR, Right: MSN-Hard.
B.2 RESULTS WITH HIGHER RESOLUTION
Table 9 shows results on RealEstate 10k with 384x384 resolutions (1.5 times larger height and width than in Table 3). We see that GTA also improves over the baseline model at higher resolution.
Table 9: x84x388 resolution on RealEstate10K
PSNRT
LPIPS 4 SSIM
Du et al. (2023) Du et al. (2023) GTA (Ours)
21.77
0.316
0.848
22.77
0.290
0.864
B.3 RESULTS WITH 3 CONTEXT VIEWS
We train models with 3-context views and show the results in Table 10. We see that GTA is also better with context views more than 2.
Table 10: Results with different numbers of context views on RealEstate10K
2-view PSNR1
3-view PSNR
Du et al. (2023) Du al. (2023) + GTA (Ours)
21.65
21.88 23.22
22.85
B.4 ROBUSTNESS TO CAMERA NOISE
Table 11 shows results on CLEVR-TR with a presence camera noise. We train RePAST and GTA with camera noise added each camera extrinsic of the second view. We perturb camera extrinsics by adding Gaussian noise to the coefficients of the S (E) algebra basis. The mean and variance of the noise is to (m, σ) (0, 0.1) during training. GTA shows better performance than RePAST regardless of the noise level.
Table 11: Test PSNRs with camera noise on CLEVR-TR and MSN-Hard. Ttest indicates the noise strength at test time. No noise injection during training.
CLEVR-TE TR
MSN-Hard
Seest
0.01
0.1
0.01
0.1
RePAST (Safin et al., 2023)
35.26
35.14
22.76
22.60
SRT+GTA (Ours)
36.66
36.57
24.06
24.16
19

---

Published as a conference paper at ICLR 2024
Table 12: Test metries on CLEVR-TR.
PSNRT
LPIPS 4
SSIM
APE
33.66
0.161
0.960
RPE
36.06
0.159
0.971
SRT
33.51
0.158
0.960
RePAST
37.27
0.119
0.977
GTA (Ours)
39.63
0.108
0.984
Table 14: Computational time perform one gradient step, encode a single scene, and render a single entire image on MSN-Hard (top) and RealEstate (bottom). All time values are expressed in milliseconds (ms). As for one gradient step time, we only measure time for forward- backward props and weight updates and exclude data loading time. We measure each time on single A 100 with 6 precision for MSN-Hard and float32 precision for RealEstate 10K.
Method
One gradient step
Encoding
Rendering
SRT RePAST
296
5.88
16.4
394
7.24
21.4
GTA
379
17.7
20.9
Method
One gradient step
Encoding
Rendering
Du et al. (2023)
619
49.8
1.42x10*
GTA
806
74.3
2.05 103
B.5 PERFORMANCE WITH DIFFERENT RANDOM SEEDS
We observe that the performance variance of different random weight initializations is quite small, as shown in Fig. 11, which displays the mean and standard deviation across 4 different seeds. We see that the variance relatively insignificant compared to the performance difference between the compared methods. Consequently, the results reported above are statistically meaningful.
B.6 PERFORMANCE DEPENDENCE ON THE REFERENCE COORDINATES
Table 13 highlights the importance of coordinate-choice invariance. "SRT (global coord)" is trained with camera poses that have their origin set to always enables the model to know how far the ray origin from the center of the scene, therefore enabling the model to easily find the position of the surface of objects that intersect with the ray. We see that SRT's performance heavily depends by construction, invariant to the choice of reference coordinate system of cameras and outperforms even the privileged version of SRT.
the center of all objects. This setting
the choice of reference coordinate system. Our model is,
Table 13: Test PSNRs in setting where global coordinates shared across scenes. All numbers show test PSNRs and are produced with models trained for 1M iterations. Note that GTA invariant the reference coordinates of the extrinsies, and the performance affected by the choice of the reference coordinate system.
Method
CLEVR-TR MSH-Hard
SRT
32.97
23.15
SRT (global coord)
37.93
24.20
GTA wo SO(3)
38.99
24.58
20

---

Published as a conference paper at ICLR 2024
CLEVR-TR
MSN-Hard
23.0
36
GTA RePAST
34
22.5
22.0
PSNR
PSNR
30
21.5
28
21.0
26
GTA
20.5
24
RePAST
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps
Steps
Fig. 11: Mean and standard deviation plots of validation PSNRs on CLEVR-TR and MSN- Hard. Due to the heavy computation requirements for training, we only trained models with 200,000 iterations and measured the validation PSNRs during the course of the training.
B.7 ANALYSIS OF ATTENTION PATTERNS
We conducted an analysis on the attention matrices the encoders trained on MSN-Hard. We found that the GTA based model tends attend features different views more than RePAST, which we show in Fig. 12. Furthermore, we see that GTA not only correctly attends to the respective patches of different views, but also can attend to object level regions (Fig. 8 and 13). Surprisingly, these attention patterns are seen at the very beginning of the course of the encoding process: the visualized attention maps are obtained in the 2nd attention layer. To evaluate how well the attention maps weigh respective object features across views, compute retrieval-based metric with instance segmentation masks of objects provided by MSN-Hard. Specifically, given a certain layer's attention maps
1. We randomly sample the i-th query patch token with 2D position p (1, 16) (1,..., 16).
2. We compute the attention map ã; [0, 1]5" 16"16 averaged over all heads.
3. We then identify which object belongs to that token's 's position by looking at the corresponding
8 x 8 region of the instance masks. Note that multiple objects can belong to the region.
4. For each belonging object, we compute precision and recall values with 1l [ > 1] as prediction and 0-1 masks of the corresponding object as ground truth on all context views, by changing the threshold value E [0, 1].
In the final step, we calculate weighted average of the precision and recall values for each object. To determine the weight of each object, we consider the number of pixels assigned to that object's mask within the 8x8 region. We then normalize these weights so that their sum equals to be one.
We collect multiple precision and recall values by randomly sampling scenes and patch positions 2000 times and then compute the average of the collected precision-recall curves. In Fig. 14, we show averaged precision-recall curves. Table shows the under the precision-recall curves (PR-AUCs) of each layer. We see that the GTA-based model learns well-aligned attention maps with the ground truth object masks for every layer.
B.8 COMPUTATIONAL TIME
We measure the time perform one-step gradient descent, well as encoding and decoding for each method. Table shows that the computational overhead added by the use of GTA com- parable to RePAST MSN-Hard. In contrast to GTA and RePAST-based models which encode positional information into every layer, SRT and Du et al. (2023) add positional embeddings only to each encoder and decoder input. As result, the computational time of SRT for one-step gradient
21

---

Published as a conference paper at ICLR 2024
GTA
RePAST
Fig. 12: Visualization of view-to-view attention maps. The (i. j-th element of each 5x5 matrix represents the average of attention weights between all pairs of each query token of the i-th con- text view and each key token of the j-th context view. The (1, m))t panel shows the weight of the m-th head at the 1-th layer. Yellow and dark purple cells indicate high and low attention weight, respectively. A matrix with high diagonal values means that the corresponding attention head at- tends within each view while with high non-diagonal values means the corresponding attention head attends across views.
descent is around 1.3x faster than RePAST and GTA, and that of Du al. (2023) 1.3x faster than GTA.
C EXPERIMENTAL SETTINGS
C.I DETAILS OF THE SYNTHETIC EXPERIMENTS IN SECTION 3.1
We use 10, 000 training and test scenes. For the intrinsics, both the vertical and horizontal sensor width are set to 1.0, and the focal length is set 4.0, leading to an angle of view of 28"
For optimization, we use AdamW (Loshchilov & Hutter, 2017) with weight decay 0.001. For each PE method, we trained multiple models with different learning rates of (0.0001, 0.0002, 0.0005) and found 0.0002 to work best for all models, and hence show results with this learning rate. We use three attention layers for both 32 32 x 3. This feature is flattened and fed into 2 layer-MLP dimensions as the token dimension d. We also apply a layer-MLP to the output of the decoder to obtain the 072 dimensional predicted image feature. The token dimensions d are APE and RPE. As we mention in the descriptions of the synthetic experiment, Py is composed of block concatenation of 3 rotation matrices, and we d to 510 for GTA, which is divisible by 3. Note that there is no difficulty with the case where is not divisible by 3. In that case, we can apply Pg only certain components of vectors whose dimensions are divisible by and apply no transformation to the other dimensions. This corresponds to applying trivial representation, i.e., the identity matrix, to the remaining vectors.
encoder and the decoder. The image feature dimension is
be transformed into the same
512 for
The RPE-based model we designed is a sensible model. For example. if 5 = bk and the set of three-dimensional vector blocks of 12 forms an orthonormal basis, then the inner product of the transformed query and key bias vectors becomes the trace of the product of the rotation matrices: (p(r)62, p(r")b") = tr( R'). tr(A" B) a natural inner product for matrices, by which we can bias the attention weight based on the imner-product-based similarity of matrices. Hence, initialize each the biases with vectorized identity matrices.
C.2 EXPERIMENTAL SETTINGS IN SECTION
Table 15 shows dataset properties and hyperparameters that we use in our experiments. We train with 4 RTX 2080 Ti GPUs CLEVR-TR and with 4 Nvidia A 100 GPUs the other datasets.
22

---

Published as a conference paper at ICLR 2024
GTA
RePAST
GTA
RePAST
GTA
RePAST
Fig. 13: Additional attention map visualizations.
Layer= 1
2
3
4
5
Prectiion
PreciSion
Pootison
04
Fig. 14: Precision-recall curves of the attention matrices of each encoder layer.
23

---

Published as a conference paper at ICLR 2024
Table 15: Dataset properties and architecture hyperparameters. target pixels indicate how many query pixels are sampled for each scene during training. "We use 12 heads for the attention layers in SRT and 8 heads in RePAST and GTA because 12 head models do not fit into our GPU memory with those methods. The decoder's attention layers only have a single head. Also, the token dimensions in the decoder are set to 128 for query-key vectors and 256 for value vectors.
dataset
CLEVR-TR MSN-Hard RealEstate
ACID
Training scenes
20,000
1,000,000
66,837
10,974
# Test scenes Batch size
1,000
10,000
7,192
1,910
32
64
48
48
Training steps
2,000,000
4,000,000
300,000
200,000
I Learning rate
le-4
le-4
5e-4
# Context views # Target pixels
2
512
2,048
192
# Self-attention layers the encoder
12
# Cross-attentie layers in the decoder
2
Heads in attention layers
12/8*
12
Token dimensions
384
768
768t
MLP dimensions
768
1,536
3,072
Table 16: Performance comparison between numbers reported Safin al. (2023) and our reproduced numbers. Note that Safin et al. (2023) uses 4x larger batch size than available in our experimental setting (4 A100s). The number of iterations for which we train each model is the same as Safin et al. (2023).
PSNR*
LPIPSyGG/Alex SSIM
SRT (Sajaadi et al., 2022b)
24.56
NA/0.223
0.784
RePAST (Safin et al., 2023)
24.89
NA/0.202
0.794
SRT
24.27
0.368/0.279
0.741
RePAST
24.48
0.348/0.24
0.751
SRT+GTA (Ours)
25.72
0.289/0.185
0.798
CLEVR-TR and MSN-Hard CLEVR-TR synthesized by using Kubric (Greff al., 2022).
The resolution of each image is 240 x 320. The camera poses of the dataset include translation, azimuth, and elevation transformations. The camera does not always look at the center of the scene.
MSN-Hard is also a synthetically generated dataset. Up to 32 objects sampled from ShapeNet (Chang et al., 2015) appear in each scene. All 51K ShapeNet objects are used for this dataset, and the training and test sets do not share the same objects with each other. MSN-Hard includes instance masks for each object scene, which we use to compute the attention matrix alignment score described in Section and Appendix B.7. The resolution of each image 128x x 128.
We basically follow the same architecture and hyperparameters of the improved version of SRT AdamW (Loshchilov &
deseribed in the appendix of Sajjadi al. (2022a), except that Hutter, 2017) with the weight decay set to the default parameter and dropout with a ratio 0.01 at every attention output and hidden layers of feedforward MIPs.
Since there is no official code or released models available for SRT and RePAST, we train both baselines ourselves and obtain almost comparable but slightly worse results (Table 16). This is because we train the models with a smaller batch size and target ray samples than in the original setting due to our limited computational resources (4 A100s). Note that our model, which is also trained with smaller batch size, still outperforms the original SRT and RePAST models' scores.
RealEstate 10k and ACID Both datasets are sampled from videos available on YouTube. At the time conducted experiments, some of the scenes used in Du et al. (2023) were longer available on YouTube. We used scenes 66, 837 and 10, 974 training scenes and 7, 192 and 1, .10 test scenes for RealFistate 10k and ACID), respectively. The resolution of each image in the original
24

---

Published as a conference paper at ICLR 2024
Table 17: Comparison between results reported in Du al. (2023) (Top) and our reproduced results (Bottom).
RcalEstate 10k
ACID LPIPS
PSNR
LPIPS
SSIM1
PSNR
SSIM
Du al. (2023)
21.38
0.262
0.839
23.63
0.364
0.781
Du et al. (2023)
21.65
0.284
0.822
23.35
0.334
0.801
Du et al. (2023)
GTA (Ours)
22.85
0.255
0.850
24.10
0.291
0.824
Input rays
Encoder
(self attention)
Transformer
Context images
CNN
Output
Target ray
Decoder
attention)
Transformer
(oross-
MLP
Fig, 15: Scene representation transformer (SRT) rendering process. The encoder E consisting of a stack of convolution layers followed by a transformer encoder translates context images into a set representation S. The decoder D predicts an RGB pixel value given target ray and S. In our model, every attention layer in both the encoder and decoder is replaced with GTA. We also remove the input and target ray embeddings from the input of the encoder and decoder, respectively. We input learned constant vector to the decoder instead of the target ray embeddings.
sequences is 360 X 640. For training, we apply downsampling followed by a random crop and random horizontal flipping each image, and the resulting resolution is 256 x 256. For test time, we apply downsampling followed by center crop to each image. The resolution of each processed image also 256 256. We follow the same architecture and optimizer hyperparameters of Du et al. (2023). Although the authors of Du et al. (2023) released the training code and their model on RealEstate 10k, observed that the model produces worse results than those reported in their work. The results were still subpar even when we trained models with their code. As result, we decided train each model with more iterations (300K) compared the 100K iterations mentioned their paper and achieved comparable scores both datasets. Consequently, we also trained GTA-based models for 300K iterations as well.
C.2.1 SCENE REPRESENTATION TRANSFORMER (SRT)
Encoding views: Let us denote -rintext- of input view images and their associated camera information by I == {[Is, Ct, M,) Neonteve where Neoutext is the number of context views, I, e x the i-th input RGB image, and c E RAx1 M, E R3x3 are a camera extrinsic and camera intrinsic matrix associated of the i-th view. The SRT encoder E' encodes the context of views into scene representation S and composed of a CNN and a transformer Etranstorerr- First, a 6- layer CNN ECNN is applied to ray-concatenated image I' of each view obtain (H/D) resolution features:
F = EC) E R(H/D)x(//)xxd I/ Lito O ((Tatw)
(18)
where d' is the output channel size the CNN, and D the downsampling factor, which is set to 8. y is Fourier embedding function that transforms ray r (o, d) R S into concatenation of the Fourier features with multiple frequencies. Each ray Tiko is computed from given camera's extrinsic and intrinsic parameters (C,, M2). Here, "ø" denotes vector concatenation.
25

---

Published as a conference paper at ICLR 2024
Next, transformer-baase encoder Buransformer processes the flattened CNN features of all views together to output the scene representation:
S = {+t |=l
Neumess (I)
Paransfomer (1i)) Neumiess (M/D)
(19)
where
{fi} is the set of flattened CNN features.
Rendering view: Given the scene representation S and target ray r", the decoder D outputs an RGB pixel:
0r- = D(77("), ERS.
(20)
where y is the same function used in Eq. (18). The architecture of D comprises two stacks a attssnntt block followed by feedforward MLP. The cross-atenntion layers determine which token the set S to attend to, to render a pixel corresponding to the given target ray. The output of the cross-atted wross-attention layers is then processed by 4-layer MI.,, get the final RGB prediction. The number of hidden dimensions of this MLP is set to 1536.
Optimization: The encoder and the decoder are optimized by minimizing the mean squared error between given target pixels 0r and the predictions:
((E
Σ ||a.- - a
(21)
C.2.2 DETAILS OF THE ARCHITECTURE AND LOSS OF DU ET AL. (2023)
Du et al. (2023) proposes an SRT-based transformer NVS model with a sophisticated architec- ture. The major differences between their model and SRT that they use dense vision trans- former (Ranftl et al., 2021) for their encoder. They also use an epipolar-basel sampling technique select context view tokens, process that helps render pixels efficiently in the decoding process.
We use the same optimization losses for training models based on this architecture as Du et al. (2023). Specifically,
the L1 loss between target and predicted pixels RealEstate 10k and ACID, We also use the following combined loss after the 30K-th iterations on ACID.
L (PP, + ALPIPS LePIPs (P, P) Adepta (P, P)
(22)
where P, P' E 322x32x3 are target and predicted patches. LLPIPS is the perceptual similarity metric proposed by Zhang et al. (2018). Ldepth is regularization loss that promotes the smoothness of estimated depths in the model. Please refer to Du et al. (2023) for more details. On RealEstate 10k, found that using the combined loss above deteriorates reconstruction metrics. Therefore, train models on RealEsatate 10k solely with the L1 loss for 300K iterations.
C.2.3 APE- AND RPE-BASED TRANSFORMERS ON CLEVR-TR
For the APE-based model, we replace the ray embeddings in SRT with a linear projection of the combined 2D positional embedding and flattened SF(3) matrix. To build RPE-based model, we follow the same procedure as in Section 3.1 and apply the representations to the bias vectors appended to the QKV vectors. Each bias dimension is set to 16 for the deam and 16 for d and d The multiplicities and frequency parameters are determined as described in Section 3.2. {s, u, v} set {4, 1, 1} and {/} is set to {1,..., 1/28] } for both σn and o Table 12 shows extended version of Table 2, which includes LPIPS (Zhang et al., 2018) and SSIM performance.
C.2.4 IMPLEMENTATION OF OTHER PE METHOOSS
Frustum positional embeddings (Liu et al., 2022): Given an intrinsic K E g3x3 we transform K-1 y, 1)T. We follow Liu et al. (2022) and generate
the 2D image position of each token by K- (zz points at multiple depths with the linear-increasigg discretization (Reading et al., 2021), where each depth value at index 1.... D is computed by
dmin dmar dmin
i(E +
(23)
D(D+1)
26

---

Published as a conference paper at ICLR 2024
100
102h
Label
Spd 2
Labes
Kiabor
Latel
Fig. 16: Frustum points on CLEVR-TR. The black point indicates the origin (0, 0, 0). Each object is sampled with its center in the range of E [-4, 4] x [-4,4] x {t/2] where t is the height of the object.
where [dman: dmax] is the full depth range and D is the number of depth bins. Examples of the generated 3D points are visualized in Fig. 16. The concatenation of 3D points of multiple depth at each pixel is further processed by a learned I-layer MLP, and added to input.
Modulated layer normalization (Hong et al., 2023; Liu et al., 2023a): Modulated layer nor- malization (MLN) modulates and biases each token feature z by using vector features y, B each of which encodes geometric information. In Liu al. (2023a), each token's 's geometric information is triplet of camera transformation, velocity, and time difference of consecutive frames. However, in our NVS tasks, the last two information does not exist. Thus, the vectors simply encode the camera transformation. Each y, 8 is computed by: ? (vec((E )), , B Eg (vec(E= where vec flat- tens the imput matrix and Er.9 are learned linear transformations. Each token æ is transformed with y and 3 as follows:
y LN)))
(24)
where @ denotes wiem multiplication.
Geometry-biased transformers (GBT) (Venkat et al., 2023): GBT biases the attention matrix of each layer by using the ray distance. Specifically, suppose each token associates with a ray r= (o, d) TR3 x S2. GBT first converts into plücker coordinate r' (d, (d,m) m) where
x d. Then the ray distance between two rays r'y = (d",m²) m3) and ni = (d m linked to each query vector q and key vector k is computed by:
[d +d"
if d x d + d sd. 0.
dist(r'9, =
Iid, x
(25)
||d (m 8 -m /a)+a"laa
The GBT's attention matrix is computed by:
softmax(QK3 - -
(26)
learned scaler parameter that controls the magnitude of the distance bias. Following Venkat et al. (2023), in addition to this bias term, we also add Fourier positional embedding computed with the plücker coordinate representation of the ray at each patch in the encoder and at each pixel in the decoder.
where D)Q e RNNN, Diy(Q,
t 7
Element-wise multiplication: In this approach, for each token with a geometric attribute g. we first concatenate the flattened SF(() homogeneous matrix and flattened SO(2) image positional representations with multiple frequencies. The number of frequencies set to the same number as in GTA CLEVR-TR. The concatenated flattened matrices are then linearly transformed to the same dimensional vectors as each Q, K, V. Then these vectors are element-wise multiplied to Q, K, V and the output of Attn in Eq. (6) in similar way to GTA.
RoPE+FTL (Su et al., 2021; Worrall et al., 2017): RoPE (Su et al., 2021) is similar to GTA but does not the SE(3) part (extrinsic matrices) well transformations value vectors. In this approach, we remove SE(3) component from the representations. Also, we remove the transforma- tions on the value vectors from each attention layer. As implementation of FTL (Worrall et al.,
27

---

Published as a conference paper at ICLR 2024
2017), apply SE(3) matrices the encoder output get transformed features to render novel views with the decoder.
C.3 2D IMAGE GENERATION WITH DIT (PEEBLES & XIE, 2023)
RoPE (Su al., 2021) is a method commonly used encode positional information in transformer models. GTA and RoPE are similar but differ in that, in GTA, group transformations are applied to the value vectors in addition the query and key vectors, leading improvements in our NVS tasks compared to models without this transformation. To further investigate the effectiveness of the value transformation, we conduct 2D image generation experiment. We will describe the experimental setting in the following. We also opensource the code for this experiments in the same repository as our NVS experiments, and please refer to for further details.
Following the experimental setup of DiT (Peebles & Xie, 2023), we use transformer-basee denois- ing network for image generation on ImageNet (Russakovsky et al., 2015). The image resolution set to 256x256, and choose the DiT-B/2 model as our baseline. Since the original DiT model does not adopt RoPE encoding, we trained models with both RoPE and GTA positional encodings. We use the same representation matrix Pg for both RoPE and GTA, which is written follows:
Pg == σ% (0%) an (0w.)
(27)
Here, the notation of each symbol is the same as in the main section. The representation design of each σi, and ow follows the original work of RoPE (Su et al., 2021). Training of each model is conducted for 2.5M iterations (approximately 500 epochs) with batch size of 256. We experiment with mixed-precision training (BFloat16), but observed instability when using RoPE and GTA. To address this, we adopt RMSNorm (Zhang & Sennrich, 2019) applied to each Q and K vector, with which we find that no instability made throughout the training. We report Table (Right) inception scores and FIDs with classifier-free guidance and its scale set to 1.5. We show comparisons of generated images in Section E.
28

---

Published as a conference paper at ICLR 2024
D RENDERED IMAGES
Context images
APE
RPE
SRT
RePAST
GTA
Ground truth
Context images
APE
RPE
SRT
RePAST
GTA
Ground truth
Fig. 17: Qualitative results CLEVR-TR.
29

---

Published as a conference paper at ICLR 2024
Context images
U
SRT
RePAST
GTA
Ground truth
Context images
SRT
RePAST
GTA
Ground truth
Context images
SRT
RePAST
GTA
Ground truth
Fig. 18: Qualitative results on MSN-Hard.
30

---

Published as a conference paper at ICLR 2024
Context images
Du et al. (2023)
GTA
Ground truth
Fig. 19: Qualitative results on RealEstatel0k.
31

---

Published as a conference paper at ICLR 2024
Context images
Du et al. (2023)
GTA
Ground truth
Fig. 20: Qualitative results on ACID.
32

---

Published as a conference paper at ICLR 2024
E
GENERATED IMAGES OF DITs
DiT GTA
DiT + RoPE
DiT (Peebles & Xie, 2023)
Fig. 21: Class-conditional generation ImageNet. Labels and noises randomly sampled.
33

---

Published as a conference paper at ICLR 2024
DiT GTA
DiT + RoPE
DiT (Peebles & Xie, 2023)
Fig. 22: Generated images with class label 'Goldfish'
34

---

Published as a conference paper at ICLR 2024
DiT GTA
DiT RoPE
DiT (Peebles & Xie, 2023)
Fig. 23: Generated images with class label 'Tree frog'
35

---

Published as a conference paper at ICLR 2024
DiT GTA
DiT + RoPE
DiT (Peebles & Xie, 2023)
Fig. 24: Generated images with label 'Boston bull'
36

---

Published as a conference paper at ICLR 2024
DiT GTA
DiT + RoPE
DiT (Peebles & Xie, 2023)
Fig. 25: Generated images with label 'Peacock'
37


Reviews:
[{"review_id": "1xaIU3Gml4", "reviewer": "Reviewer_myD9", "summary": "This work proposes an attention mechanism for multi-view transformers which factors in the geometry information, such as camera pose and image position, of the inputs. This differs from most previous work that often condition Transformers using absolute positional encoding of that information. A key intuition given by the authors is that attention from one token to another should happen in a canonical coordinate space. \nSparse-view novel-view synethesis results on a number of datasets show that the proposed method is state-of-the-art compared against other recent methods as well as a number of baselines. These datasets include synthetic yet challenging scenes (like MSN-Hard) as well as real world indoor and outdoor scenes (ACID and RealEstate10k).", "questions": "A number of questions/suggestions:\n\n- As suggested above, I would like to see a variant of GTA that does not involve block concatenation of the geometry representations matrices along with their multiplicites (to mnatch the transformation to the input dimensionality `d`). Have the authors explored a variant such as: concatenate the flattened representation of all groups, followed by a linear projection up to `d`, element-wise multplication with input features (with the rest of the GTA method as proposed)?\n\n- The definition of $\\rho_{g_i g_j}$ in Eq. 4 I believe is missing, which makes it difficult to understand how the authors go from Eq 4. to Eq. 5, as well as their claims regarding the quadratic complexity in Eq 4.\n\n- While some results are provided on real world datasets, these are fairly limited in both size and quality (from looking at the videos). It's therefore difficult to conclude that this method is SoTA more generally in real-world cases. For instance, results would be strengthened if a study of the sensitivity of the model's predictions are to camera errors during training and/or test time. This is relevant given that other methods in the literature have carried out these types of experiments (see e.g. SRT paper, Fig 6).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a very well written and for the most part clear paper, with a clear motivation and intuition behind their method, as well as a proper literature review of relevant prior works.\n\nIt aims to tackle an important and challenging problem of novel view synthesis in the sparse (few input images) and wide-baseline (long-range camera variation) case; as such, strong results in challenging benchmarks imply a significant contribution to the community. \n\nThe method and claims are sound. The method itself does not involve a big departure from the standard Transformer architecture, which means it's easy to implement and widely applicably potentially to other problems.\n\nThe experimental set-up is generally thorough. A number of SoTA method are included for comparison and different synthetic as well as real-world datasets are used for evaluation.", "weaknesses": "While I agree with the intuition behind this method, what the actual implementation does, and how it affects attention, is somewhat unclear to me. It makes sense to transform features of different views to canonical reference space during attention, and that's obvious only when those are actual coordinates (e.g. points in 3D space); it's not so obvious when features are linear projections of other features. \n\n\nIt's also not obvious that block concatenation of multiple group representations, each with a certain multiplicity, is the best way to achieve it, and I believe this work could do with a bit more discussion on these points, and potentially further ablations/baselines of other ways one could implement the same intuition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1xZ16edeK", "reviewer": "Reviewer_ZuJ5", "summary": "The paper proposes a new methodology for encoding positional information and camera transformations in 3D vision tasks. It employs a special Euclidean group to encode the geometric relationship between queries and key-value pairs within the attention mechanism. Experimental results demonstrate the effectiveness and efficiency of the proposed method, achieving top performance on the novel view synthesis dataset.", "questions": "1. Table 4 validates that the method proposed in the paper effectively improves accuracy.  I am curious about the performance of other similar methods when evaluated against the same baseline and the same settings. Are they still inferior to the approach presented in the paper?\n2. Additionally, since geometry-aware attention is a general method, can it also enhance accuracy in other perceptual tasks apart from novel-view synthesis?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The geometric relationship is crucial for multi-view geometry tasks. The proposed method straightforwardly and reasonably encodes transformation and position information into the attention mechanism. The method has demonstrated superior results on several novel view synthesis datasets.\n2. The involvement of the special Euclidean group in encoding transformations is interesting and useful for capturing geometric relationships.\n3. The proposed method requires minimal computation, making it easy to implement and integrate.", "weaknesses": "1. Geometry-aware attention for transformers has been extensively explored, and the idea of encoding position and transformation information to improve 3D vision tasks has been discussed previously. While the detailed implementation of the encoding process (such as applying transformations to value vectors) differs from existing works, they share a similar underlying concept, thereby diminishing the novelty of this paper. \n2. The evaluation and ablation study primarily focus on the novel view synthesis task. However, since the proposed method is easily integrable into existing attention-based 3D vision tasks, I highly recommend validating the approach on other 3D vision tasks, such as multi-view 3D detection. \n3. Unlike the feature warping approach, the encoded transformation information relies on matrix multiplication. To be honest, I don't fully grasp the physical meaning behind this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SfF2zadF9t", "reviewer": "Reviewer_LyNT", "summary": "This paper addresses the limitation of existing positional encoding designs in 3D vision tasks, which do not consider task-relevant 3D geometric structures. The authors propose a geometry-aware attention mechanism that encodes the geometric structure of tokens for the novel view synthesis task. Extensive experiments demonstrate the effectiveness of the proposed method, and qualitative comparisons with previous state-of-the-art techniques highlight significant improvements.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed GTA consistently outperforms existing state-of-the-art methods on multiple datasets, including RealEstate10k, ACID, CLEVR-TR, and MSN-Hard.\n\n2. The authors provide extensive qualitative comparisons and analyses, effectively supporting the superiority of the proposed method over previous state-of-the-art techniques.", "weaknesses": "1. This paper's organization could be improved. For instance, placing the related work (Section 5) after the introduction would help readers understand the difference between conventional positional encoding schemes and the proposed relative transformation encoding scheme. The authors should summarize existing geometric-related positional encoding techniques (not only in novel view synthesis, but also in many 3D vision tasks), and conduct a comprehensive comparison. Since the background information of transformer (the first half of section 2 Background) is already well-known, it can be moved to the appendix. \n\n2. The paper's contribution appears to be incremental, offering limited insights from the proposed relative transformation positional encoding alone.\n\n3. While the paper introduces GTA as a novel positional encoding scheme, it seems to function more like a transformer block (consisting of positional encoding and the attention block) than just a positional encoding scheme. A more detailed ablation study of each component within GTA is needed.\n\nIn general, improving the paper's organization and conducting a more comprehensive comparison with existing techniques would strengthen its contributions. Additionally, more detailed analysis of the components within GTA would enhance the paper's overall quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZhwiZT5UV", "reviewer": "Reviewer_nfN9", "summary": "This paper focuses on enhancing the attention mechanism by incorporating geometric information. The motivation behind this is the observation that existing positional encoding techniques are not optimally suited for vision tasks. In particular, the authors encode the geometric structure of tokens by representing them as relative transformations and introduce a novel geometric-aware attention mechanism. Experiments are conducted on novel view synthesis tasks, where the proposed method achieves state-of-the-art performance among transformer-based neural video synthesis (NVS) methods.", "questions": "The resolution of training images used in the experiment seems to be not high. How does this method perform on high-resolution data, and what is the impact of different numbers of training views?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel attention mechanism grounded in geometric information, and the motivation is reasonable.\n2. The paper is well-written and the experiments yield compelling qualitative results compared to baseline models.", "weaknesses": "- Lack of comparisons with the latest NeRF-based models, except Pixel-NeRF which is proposed in 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)