Published as a conference paper at ICLR 2024

GTA: A G EOMETRY-AWARE ATTENTION M ECHANISMFOR M ULTI -V IEW T RANSFORMERS
Takeru Miyato1 , Bernhard Jaeger1 , Max Welling2 , Andreas Geiger1
University of Tübingen, Tübingen AI Center 2 University of Amsterdam

1

A BSTRACTAs transformers are equivariant to the permutation of input tokens, encoding thepositional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks,
their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding
schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometryaware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries andkey-value pairs. By evaluating on multiple novel view synthesis (NVS) datasetsin the sparse wide-baseline multi-view setting, we show that our attention, called
Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional
learned parameters and only minor computational overhead.

1

I NTRODUCTION

The transformer model (Vaswani et al., 2017), which is composed of a stack of permutation symmetric layers, processes input tokens as a set and lacks direct awareness of the tokens’ structural
information. Consequently, transformer models are not solely perceptible to the structures of input
tokens, such as word order in NLP or 2D positions of image pixels or patches in image processing.
A common way to make transformers position-aware is through vector embeddings: in NLP, atypical way is to transform the position values of the word tokens into embedding vectors to be
added to input tokens or attention weights (Vaswani et al., 2017; Shaw et al., 2018). While initiallydesigned for NLP, these positional encoding techniques are widely used for 2D and 3D vision tasks
today (Wang et al., 2018; Dosovitskiy et al., 2021; Sajjadi et al., 2022b; Du et al., 2023).
Here, a natural question arises: “Are existing encoding schemes suitable for tasks with very different
geometric structures?”. Consider for example 3D vision tasks using multi-view images paired withcamera transformations. The 3D Euclidean symmetry behind multi-view images is a more intricate
structure than the 1D sequence of words. With the typical vector embedding approach, the model istasked with uncovering useful camera poses embedded in the tokens and consequently struggles tounderstand the effect of non-commutative Euclidean transformations.
Our aim is to seek a principled way to incorporate the geometrical structure of the tokens into thetransformer. To this end, we introduce a method that encodes the token relationships as transformations directly within the attention mechanism. More specifically, we exploit the relative transformation determined by the geometric relation between the query and the key-value tokens. We
then apply those transformations to the key-value pairs, which allows the model to compute QKV
attention in an aligned coordinate space.
We evaluate the proposed attention mechanism on several novel view synthesis (NVS) tasks withsparse and wide-baseline multi-view settings, which are particularly hard tasks where a model needs
to learn strong 3D geometric priors from multiple training scenes. We show that existing positional
encoding schemes are suboptimal and that our geometric-aware attention, named geometric transform attention (GTA), significantly improves learning efficiency and performance of state-of-the-art
transformer-based NVS models, just by replacing the existing positional encodings with GTA.
Correspondence to takeru.miyato@gmail.com. Code: https://github.com/autonomousvision/gta.

1

Published as a conference paper at ICLR 2024

2

R ELATED WORK

Given token features X ∈ Rn×d , the attention layer’s outputs O ∈ Rn×d are computed as follows:
O := Attn(Q, K, V ) = softmax(QK T )V,
Q

K

V

n×d

{Q,K,V }(1)

d×d

where Q, K, V = XW , XW , XW ∈ R,W∈R, and (n, d) is the number oftokens and channel dimensions. We omit the scale factor inside the softmax function for simplicity.
The output in Eq. (1) is invariant to the permutation of the key-value vector indices. To breakthis permutation symmetry, we explicitly encode positional information into the transformer, whichis called positional encoding (PE). The original transformer (Vaswani et al., 2017) incorporates
positional information by adding embeddings to all input tokens. This absolute positional encoding(APE) scheme has the following form:
softmax (Q + γ(P)W Q )(K + γ(P)W K )T V + γ(P)W V ,(2)where P denotes the positional attributes of the tokens X and γ is a PE function. From here, a boldsymbol signifies that the corresponding variable consists of a list of elements. γ is typically the sinusoidal function, which transforms position values into Fourier features with multiple frequencies.
Shaw et al. (2018) proposes an alternative PE method, encoding the relative distance between eachpair of query and key-value tokens as biases added to each component of the attention operation:0softmax QK T + γrel (P) (V + γrel
(P)),
(3)
0
where γrel (P) ∈ Rn×n and γrel(P) ∈ Rn×d are the bias terms that depend on the distance between
tokens. This encoding scheme is called relative positional encoding (RPE) and ensures that theembeddings do not rely on the sequence length, with the aim of improving length generalization.

Following the success in NLP, transformers have demonstrated their efficacy on various imagebased computer vision tasks (Wang et al., 2018; Ramachandran et al., 2019; Carion et al., 2020;
Dosovitskiy et al., 2021; Ranftl et al., 2021; Romero et al., 2020; Wu et al., 2021; Chitta et al.,2022). Those works use variants of APE or RPE applied to 2D positional information to make themodel aware of 2D image structure. Implementation details vary across studies. Besides 2D-vision,
there has been a surge of application of transformer-based models to 3D-vision (Wang et al., 2021a;
Liu et al., 2022; Kulhánek et al., 2022; Sajjadi et al., 2022b; Watson et al., 2023; Varma et al., 2023;
Xu et al., 2023; Shao et al., 2023; Venkat et al., 2023; Du et al., 2023; Liu et al., 2023a).
Various PE schemes have been proposed in 3D vision, mostly relying on APE- or RPE-based encodings. In NVS Kulhánek et al. (2022); Watson et al. (2023); Du et al. (2023) embed the camera
extrinsic information by adding linearly transformed, flattened camera extrinsic matrices to the tokens. In Sajjadi et al. (2022b); Safin et al. (2023), camera extrinsic and intrinsic information is
encoded through ray embeddings that are added or concatenated to tokens. Venkat et al. (2023)
also uses ray information and biases the attention matrix by the ray distance computed from ray
information linked to each pair of query and key tokens. An additional challenge in 3D detection
and segmentation is that the output is typically in an orthographic camera grid, differing from theperspective camera inputs. Additionally, sparse attention (Zhu et al., 2021) is often required because
high resolution feature grids (Lin et al., 2017) are used. Wang et al. (2021b); Li et al. (2022) use
learnable PE for the queries and no PE for keys and values. Peng et al. (2023) find that using standard learnable PE for each camera does not improve performance when using deformable attention.
Liu et al. (2022; 2023b) do add PE to keys and values by generating 3D points at multiple depths
for each pixel and adding the points to the image features after encoding them with an MLP. Zhou
& Krähenbühl (2022) learn positional embeddings using camera parameters and apply them to thequeries and keys in a way that mimics the relationship between camera and target world coordinates. Shu et al. (2023) improves performance by using available depths to link image tokens with
their 3D positions. Besides APE and RPE approaches, Hong et al. (2023); Zou et al. (2023); Wanget al. (2023) modulate tokens by FiLM-based approach (Perez et al., 2018), where they element-wise
multiply tokens with features computed from camera transformation.
In point cloud transformers, Yu et al. (2021a) uses APE to encode 3D positions of point clouds.
Qin et al. (2022) uses an RPE-based attention mechanism, using the distance or angular difference
between tokens as geometric information. Epipolar-based sampling techniques are used to sample
geometrically relevant tokens of input views in attention layers (He et al., 2020; Suhail et al., 2022;
Saha et al., 2022; Varma et al., 2023; Du et al., 2023), where key and value tokens are sampled along
an epipolar line determined by the camera parameters between a target view and an input view.
2

Published as a conference paper at ICLR 2024

3

G EOMETRIC ENCODING BY RELATIVE TRANSFORMATIONIn this work, we focus on novel view synthesis (NVS), which is a fundamental task in 3D-vision.
The NVS task is to predict an image from a novel viewpoint, given a set of context views of a scene
and their viewpoint information represented as 4 × 4 extrinsic matrices, each of which maps 3D
points in world coordinates to the respective points in camera coordinates. NVS tasks require themodel to understand the scene geometry directly from raw image inputs.
The main problem in existing encoding schemes of the camera transformation is that they do notrespect the geometric structure of the Euclidean transformations. In Eq. (2) and Eq. (3), the embedding is added to each token or to the attention matrix. However, the geometry behind multi-view
images is governed by Euclidean symmetry. When the viewpoint changes, the change of the object’s
pose in the camera coordinates is computed based on the corresponding camera transformation.
Our proposed method incorporates geometric transformations directly into the transformer’s attention mechanism through a relative transformation of the QKV features. Specifically, each key-value
token is transformed by a relative transformation that is determined by the geometric attributes between query and key-value tokens. This can be viewed as a coordinate system alignment, which has
an analogy in geometric processing in computer vision: when comparing two sets of points each
represented in a different camera coordinate space, we move one of the sets using a relative transformation cc0−1 to obtain all points represented in the same coordinate space. Here, c and c0 are theextrinsics of the respective point sets. Our attention performs this coordinate alignment within theattention feature space. This alignment allows the model not only to compare query and key vectors
in the same reference coordinate space, but also to perform the addition of the attention output at theresidual path in the aligned local coordinates of each token due to the value vector’s transformation.
This direct application of the transformations to the attention features shares its philosophy withthe classic transforming autoencoder (Hinton et al., 2011; Cohen & Welling, 2014; Worrall et al.,2017; Rhodin et al., 2018; Falorsi et al., 2018; Chen et al., 2019; Dupont et al., 2020), capsuleneural networks (Sabour et al., 2017; Hinton et al., 2018), and equivariant representation learningmodels (Park et al., 2022; Miyato et al., 2022; Koyama et al., 2023). In these works, geometric information is provided as a transformation applied to latent variables of neural networks. Suppose Φ(x)is an encoded feature, where Φ is a neural network, x is an input feature, and M is an associatedtransformation (e.g. rotation). Then the pair (Φ(x), M) is identified with MΦ(x). We integrate thisfeature transformation into the attention to break its permutation symmetry.
Group and representation: We briefly introduce the notion of a group and a representation because we describe our proposed attention through the language of group theory, which handles
different geometric structures in a unified manner, such as camera transformations and image positions. In short, a group G with its element g, is an associative set that is closed under multiplication,
has the identity element and each element has an inverse. E.g. the set of camera transformations
satisfies the axiom of a group and is called special Euclidean group: SE(3). A (real) representation
is a function ρ : G → GLd (R) such that ρ(g)ρ(g 0 ) = ρ(gg 0 ) for any g, g 0 ∈ G. The property
ρ(g)ρ(g 0 ) = ρ(gg 0 ) is called homomorphism. Here, GLd (R) denotes the set of d × d invertible
real-valued matrices. We denote by ρg := ρ(g) ∈ Rd×d a representation of g. A simple choice forthe representation ρg for g ∈ SE(3) is a 4 × 4 rigid transformation matrix [ R0 T1 ] ∈ R4×4 whereR ∈ R3×3 is a 3D rotation and T ∈ R3×1 is a 3D translation. A block concatenation of multiple
group representations is also a representation. What representation to use is the user’s choice. We
will present different design choices of ρ for several NVS applications in Section 3.1, 3.2 and A.3.2.
3.1

G EOMETRIC TRANSFORM ATTENTION

Suppose that we have token features X ∈ Rn×d and a list of geometric attributes g = [g1 , . . . , gn ],where gi is an i-th token’s geometric attribute represented as a group element. For example, each
Xi ∈ Rd corresponds to a patch feature, and gi corresponds to a camera transformation and an
image patch position. Given a representation ρ and Q, K, V = XW Q , XW K , XW V ∈ Rn×d , wedefine our geometry-aware attention given query Qi ∈ Rd by:
Oi =

n
X
j

Pn

exp(QTi (ρgi g −1 Kj ))j 0 =1jexp(QT
Kj 0 ))i (ρgi g −10j3

(ρgi g−1 Vj ),j(4)Published as a conference paper at ICLR 2024

Using the homomorphism property ρgi g−1 = ρgi ρg−1 , the above equation can be transformed intojOi =ρgi

n
X
j

Pn

j

T
exp((ρT
gi Qi ) (ρg −1 Kj ))j 0 =1jT
Kj 0 ))
exp((ρT
gi Qi ) (ρg −10(ρg−1 Vj ).

(5)

j

j

Note that the latter expression is computationally and memory-wise more efficient, requiring computation and storage of n2 values of each (ρgi g−1 Kj , ρgi g−1 Vj ) in Eq. (4) versus only n values
j

j

−1
−1
for (ρT
gi Qi , ρgj Kj , ρgj Vj ) and ρgi Ôi in Eq. (5), where Ôi is the output of the leftmost sum.

Eq. (5), given all queries Q, can be compactly rewritten in an
implementation-friendly form:
O = Pg } Attn Pg T } Q, Pg −1 } K, Pg −1 } V ,(6)where Pg denotes a list of representations for different tokens: Pg :=[ρg1 , . . . , ρgn ], and “}” denotes token-wise matrix multiplication: Pg }T
K = [ρg1 K1 · · · ρgn Kn ] ∈ Rn×d . Also, the transpose T and the inT
verse −1 operate element-wise on Pg (e.g., Pg T := [ρT
g1 , . . . , ρgn ]). We
call the attention mechanism in Eq. (6) geometric transform attention(GTA) and show the diagram of (6) in Fig. 1. Note that the additional
computation of GTA is smaller than the QKV attention and the MLP
in the transformer when constructing ρg from a set of small matrices,which we will detail in Section 3.2 and in Appendix A.

+ρAttention
ρT

ρ-1

ρ-1

Q

K

V

A simple NVS experiment: We first demonstrate that GTA improves Fig. 1: GTA mechanism.
learning as compared to APE and RPE in a simplified NVS experiment. ρ−1 and ρT together take
We construct a setting where only camera rotations are relevant to show Q, K and V to a sharedthat the complexity of ρg can be adapted to the problem complexity. A coordinate space, and thesingle empty scene surrounded by an enclosing sphere whose texture is ρ gets the attention output
shown in Fig. 2 left is considered. All cameras are placed in the center back to each token’s coorof the scene where they can be rotated but not translated. Each scene dinate space.
consists of 8 context images with 32x32 pixel resolution rendered witha pinhole camera model. The camera poses are chosen by randomly sampling camera rotations.
We randomize the global coordinate system by setting it to the first input image. This increases thedifficulty of the task and is similar to standard NVS tasks, where the global origin may be placed
anywhere in the scene. The goal is to render a target view given its camera extrinsic and a set ofcontext images.
We employ a transformer-based encoder-decoder architecture shown on the right of Fig. 2. Camera
extrinsics in this experiment form the 3D rotation group: SO(3). We choose ρg to be a block
concatenation of the camera rotation matrix:(7)
ρgi := Ri ⊕ · · · ⊕ Ri ,{z}|
d/3 times

where Ri is the 3 × 3 matrix representation of the extrinsic gi ∈ SO(3) linked to the i-th token.
A ⊕ B denotes block-concatenation: A ⊕ B = [ A0 B0 ]. Because here each ρgi is orthogonal, thetranspose of ρgi becomes the inverse, thus the same transformation is applied across query, key, andvalue vector for each patch.

We compare this model to APE- and RPE-based transformers as baselines. For the APE-basedtransformer, we add each flattened rotation matrix associated with each token to each attention
layer’s input. Since we could not find an RPE-based method that is directly applicable to our settingwith rotation matrices, we use an RPE-version of our attention where instead of multiplying thematrices with the QKV features, we apply the matrices to biases. More specifically, for each head,
we prepare learned bias vectors bQ , bK , bV ∈ R9 concatenated with each of the QKV vectors ofeach head and apply the representation matrix defined by ρ(g) := R ⊕ R ⊕ R ∈ R9×9 , only to thebias vectors. We describe this RPE-version of GTA in more detail in Appendix C.1.
Fig. 3 on the left shows that the GTA-based transformer outperforms both the APE and RPE-basedtransformers in terms of both training and test performance. In Fig. 3 on the right, the GTA-basedtransformer reconstructs the image structure better than the other PE schemes.
4(11) (11)
(10)
(11)

✓

6 6h ✓.h 6 ✓hM7(m6 )7M (m6 ) 7
6✓ . 7 ✓
7
. . 6. . . 7⇢✓ h =⇢✓h6
4 = 4⇢✓h =5h . .5 h 54(m6 ) (m6 )(10)(10)
(11)

(m )

2 M✓2 M✓hh(n1 ) 2024(n)
Published as a conference paper at M
ICLR
M 1

6
M✓3
3
h

(11)(11)(11)(12) (12)
(12)

(12)

Decoder(14)

Output(13)…
(14)(13) (14)FF(13)(13)(12)(13) (13)…

GTAcross

(13)

FF

(c(,
(cg, (,✓,w✓g)h ,),✓wg)⇤i⇤(, g1Target),(, (,g1g),2 ),(, (,g2g),1N),(,),
ggiN,g⇤✓2),h),N
queryi-th i-th
viewview(ci , ✓h , ✓(cwi), ✓h , ✓w )GTAcross(12)(12)FF

GTAself

FF

GTAself…

…

6 ✓w6 .✓w .
7 7
6 6 . .3 . . 3
7 7
2
41 )
5 5
(n
(n1 )4M✓w M✓w(n6 ) (n6 )
6
6
7
M
M
2 26
3 . 37 ✓w 7
2
3
6.
7✓w
(n1 ) (n1 ) 4 . . (n1 ) . .5
5
M✓w4M
M✓w
✓w
6 6 . 6 M7(n6 )7M (n6 ) 7 Encoder
6 6 . . 6. . 7✓w. .7 ✓w 7
. (,gg), ),⇤4 4
5⇤(,4g1.),(,)(,5
gg1 2),),5(,
2 N (, g N ), g
(n6 )
6 (n6 )
Context M✓(nM
M
…
✓
✓w
w
w
images),g(,⇤ gN ), g ⇤(, g1 ), (,(,gg2 ),1 ),(,(,gg
N2),
2

(14)

Fig. 2: Synthetic experiment. Left: Texture of the surrounding sphere. Right: Model architecture.
i-thpairview
i-thquery
view
, ✓(c
(14)(14)g ∗ . (14)✓(c)
h ,i ,✓✓
wh), constantw i), ✓h , ✓w
The
consists of(cailearned
value
and a target extrinsic
i-thi-th
viewview
i-th view
0.02

GTA
RPE
APE

MSE

0.01
0.005
0.002 0Ground truth

2

1

2 3
Steps
2

4 21e4
5

2

2

2

GTA

RPE

APE

2

Fig. 3: Results on the synthetic dataset. Left: The solid and dashed lines indicate test and train
errors. Right: Patches predicted with different PE schemes.
3.2

T OKEN STRUCTURE AND DESIGN OF REPRESENTATION ρ FOR NVS

In the previous experiment, tokens were simplified to comprise an entire image feature and an associated camera extrinsic. This differs from typical NVS model token structures where patched image
tokens are used, and each of the tokens can be linked not only to a camera transformation but also
to a 2D location within an image. To adapt GTA to such NVS models, we now describe how weassociate each feature with a geometric attribute and outline one specific design choice for ρ.
Token structure: We follow a common way to compose the input
tokens for the transformer as in (Sajjadi et al., 2022b; Du et al.,2023). We assume that for each view, we have image patches or
pixels of the size of H × W , and each patch or pixel token consistsof a pair of a feature value x ∈ Rd and geometric attributes that area camera extrinsic c ∈ SE(3) and a 2D image position. For image
PE, it would be natural to encode each position as an element of
the 2D translation group T (2). However, we found, similarly to
the Fourier feature embeddings used in APE and RPE and rotary
Fig. 4: Geometric attributes. PE (Su et al., 2021), encoding the image positions as elements ofthe 2D rotation group SO(2) exhibits better performance than using
T (2). Thus, we represent each image position as an element of the direct product of the two SO(2)groups: (θh , θw ) ∈ SO(2) × SO(2) where θh , θw ∈ [0, 2π). Here, we identify the SO(2) elementwith the 2D rotation angle. We associate the top left patch (or pixel) with the value (0, 0), while thebottom right patch corresponds to (2π(H −1)/H, 2π(W −1)/W ). For the intermediate patches, we
compute their values using linear interpolation of the angle values between the top left and bottom
right patches. Overall, we represent the geometric attribute of each token of the i-th view by
g := (ci , θh , θw ) ∈ SE(3) × SO(2) × SO(2) =: G.
(8)
Fig. 4 illustrates how we represent each geometric attribute of each token.
Design of ρ: What representation to use is a design choice similar to the design choice of theembedding in APE and RPE. As a specific design choice for the representation for NVS tasks,
we propose to compose ρg by the direct sum of multiple irreducible representation matrices, each
responding to a specific component of the group G. Specifically, ρg is composed of four different
types of representations and is expressed in block-diagonal form as follows:⊕t⊕s⊕v
ρg := σcam(c) ⊕ σrot(r) ⊕ σh⊕u (θh ) ⊕ σw(θw ),(9)
where “⊕” denotes block-concatenation A ⊕ B = [ A0 B0 ] and A⊕a indicates repeating the block
concatenation of A a total of a times. We introduce an additional representation σrot (r) that captures5

Published as a conference paper at ICLR 2024

Table 1: Components of ρg .
σcam (c)matrix form



R T
0 1

multiplicitysσrot (r) (l1 )

Dr
..


.
(lNrot )Drtσh (θh ) (f1 )

Mθh


..


.
(fNh )Mθhu

σw (θw ) (f1 )
Mθ w


..


.
(fNw )Mθ w
v

Table 2: Test metrics. Left: CLEVR-TR, Right: MSN-Hard. †Models are trained and evaluated on
MultiShapeNet, not MSN-Hard. They are different but generated from the same distribution.
PSNR↑
†

PSNR↑

LPIPS ↓

SSIM↑

APE
RPE

33.66
36.08

LFN (Sitzmann et al., 2021)
PixelNeRF† (Yu et al., 2021b)

14.77
21.97

0.582
0.332

0.328
0.689

SRT
RePAST
GTA (Ours)

33.51
37.27
39.63

SRT (Sajjadi et al., 2022b)
RePAST (Safin et al., 2023)
SRT+GTA (Ours)

24.27
24.48
25.72

0.368
0.348
0.289

0.741
0.751
0.798

Context images

SRT

RePAST

GTA (Ours)

Ground truth

Fig. 6: Validation PSNR
curves on MSN-Hard.
Fig. 5: Qualitative results on MSN-Hard.
only the rotational information of c, with which we find moderate improvements in performance.
(f )Table 1 summarizes the matrix form we use for each representation. Specifically, Mθ is a 2D
rotation matrix with frequency f that is analogous to the frequency parameter used in Fourier feature(l)(f )embeddings in APE and RPE. Dr can be thought of as the 3D version of Mθ . Please refer toAppendix A.2 for more detailed descriptions of these matrices. Fig. 9 in the Appendix displays theactual representation matrices used in our experiments. The use of the Kronecker product is also atypical way to compose representations, which we describe in Appendix A.3.2.

4

E XPERIMENTAL E VALUATION

We conducted experiments on several sparse NVS tasks to evaluate GTA and compare the reconstruction quality with different PE schemes as well as existing NVS methods.
Datasets: We evaluate our method on two synthetic 360° datasets with sparse and wide baseline views (CLEVR-TR and MSN-Hard) and on two datasets of real scenes with distant views(RealEstate10k and ACID). We train a separate model for each dataset and describe the propertiesof each dataset below. CLEVR with translation and rotation (CLEVR-TR) is a multi-view version
of CLEVR (Johnson et al., 2017) that we propose. It features scenes with randomly arranged basic
objects captured by cameras with azimuth, elevation, and translation transformations. We use this6

Published as a conference paper at ICLR 2024

Table 3: Results on RealEstate10k and ACID. Top: NeRF methods. Bottom: transformer methods.
RealEstate10k
PSNR↑ LPIPS↓ SSIM↑

PSNR↑

ACID
LPIPS↓

SSIM↑

PixelNeRF (Yu et al., 2021b)
StereoNeRF (Chibane et al., 2021)
IBRNet (Wang et al., 2021a)
GeoNeRF (Johari et al., 2022)
MatchNeRF (Chen et al., 2023)

13.91
15.40
15.99
16.65
23.06

0.591
0.604
0.532
0.541
0.258

0.460
0.486
0.484
0.511
0.830

16.48
−
19.24
−
−

0.628
−
0.385
−
−

0.464
−
0.513
−
−

GPNR (Suhail et al., 2022)
Du et al. (2023)
Du et al. (2023) + GTA (Ours)

18.55
21.65
22.85

0.459
0.285
0.255

0.748
0.822
0.850

17.57
23.35
24.10

0.558
0.334
0.291

0.719
0.801
0.824

Context images

Du et al. (2023)

GTA (Ours)

Ground truth

Fig. 7: Qualitative results. Top: ACID, Bottom: RealEstate10k.
dataset to measure the ability of models to understand the underlying geometry of scenes. We set
the number of context views to 2 for this dataset. Generating 360° images from 2 context views ischallenging because parts of the scene will be unobserved. The task is solvable because all renderedobjects have simple shapes and textures. This allows models to infer unobserved regions if they
have a good understanding of the scene geometry. MultiShapeNet-Hard (MSN-Hard) is a challenging dataset introduced in Sajjadi et al. (2022a;b). Up to 32 objects appear in each scene and are
drawn from 51K ShapeNet objects (Chang et al., 2015), each of which can have intricate texturesand shapes. Each view is captured from a camera pose randomly sampled from 360° viewpoints.
Objects in test scenes are withheld during training. MSN-Hard assesses both the understanding ofcomplex scene geometry and the capability to learn strong 3D object priors. Each scene has 10
views, and following Sajjadi et al. (2022a;b), we use 5 views as context views and the remaining
views as target views. RealEstate10k (Zhou et al., 2018) consists of real indoor and outdoor scenes
with estimated camera parameters. ACID (Liu et al., 2021) is similar to RealEstate10k, but solely
includes outdoor scenes. Following Du et al. (2023), during training, we randomly select two context views and one intermediate target view per scene. At test time, we sample distant context views
with 128 time-step intervals and evaluate the reconstruction quality of intermediate views.
Baselines: Scene representation transformer (SRT) (Sajjadi et al., 2022b), a transformer-based
NVS method, serves as our baseline model on CLEVR-TR and MSN-Hard. SRT is a similar architecture to the one we describe in Fig. 2, but instead of the extrinsic matrices, SRT encodes theray information into the architecture by concatenating Fourier feature embeddings of rays to the
input pixels of the encoder. SRT is an APE-based model. Details of the SRT rendering process areprovided in Appendix C.2.1 and Fig. 15. We also train another more recent transformer-based NVS
model called RePAST (Safin et al., 2023). This model is a variant of SRT and encodes ray information via an RPE scheme, where, in each attention layer, the ray embeddings are added to the queryand key vectors. The rays linked to the queries and keys are transformed with the extrinsic matrix
associated with a key-value token pair, before feeding them into the Fourier embedding functions,
to represent both rays in the same coordinate system. RePAST is the current state-of-the-art method7

Published as a conference paper at ICLR 2024

Table 4: PE schemes. MLN: Modulated layer normalization (Hong et al., 2023; Liu et al., 2023a).
ElemMul: Element-wise Multiplication. GBT: geometry-biased transformers (Venkat et al., 2023).
FM: Frustum Embedding (Liu et al., 2022). RoPE+FTL: (Su et al., 2021; Worrall et al., 2017).
MLN

SRT

ElemMul

GBT

FM

RoPE+FTLGTAPSNR↑ 32.48 33.21
34.74
35.63 37.23
38.18
38.99
Table 5: Effect of the transformation on V . Left: Test PSNRs on ClEVR-TR and MSN-Hard.
Right: Inception scores (IS) and FIDs of DiT-B/2 (Peebles & Xie, 2023) on 256x256 ImageNet.
IS↑ FID-50K↓

CLEVR-TR MSN-Hard
No ρg on V
GTA

36.54
38.99

23.77
24.58

DiT (Peebles & Xie, 2023) 145.3
DiT + 2D-RoPE
151.8
DiT + GTA
158.2

7.02
6.26
5.87

on MSN-Hard. The key difference between GTA and RePAST is that the relative transformation isapplied directly to QKV features in GTA, while it is applied to rays in RePAST.
For RealEstate10k and ACID, we use the model proposed in Du et al. (2023), which is the state-ofthe-art model on those datasets, as our baseline. Their model is similar to SRT, but has architectural
improvements and uses an epipolar-based token sampling strategy. The model encodes extrinsic
matrices and 2D image positions to the encoder via APE, and also encodes rays associated witheach query and context image patch token in the decoder via APE.
We implement our model by extending those baselines. Specifically, we replace all attention layers
in both the encoder and decoder with GTA and remove any vector embeddings of rays, extrinsic
matrices, and image positions from the model. We train our models and baselines with the samesettings within each dataset. We train each model for 2M and 4M iterations on CLEVR-TR and
MSH-Hard and for 300K iterations on both RealEstate10k and ACID, respectively. We report the
reproduced numbers of baseline models in the main tables and show comparisons between the reported values and our reproduced results in Table 16 and Table 17 in Appendix C.2. Please also see
Appendix C.2 for more details about our experimental settings.
Results: Tables 2 and 3 show that GTA improves the baselines in all reconstruction metrics on
all datasets. Fig. 5 shows that on MSN-Hard, the GTA-based model renders sharper images withmore accurate reconstruction of object structures than the baselines. Fig. 7 shows that our GTAbased transformer further improves the geometric understanding of the scenes over Du et al. (2023)
as evidenced by the sharper results and the better recovered geometric structures. Appendix D
provides additional qualitative results. Videos are provided in the supplemental material. We also
train models, encoding 2D positions and camera extrinsics via APE and RPE for comparison. See
Appendix C.2.3 for details. Fig. 6 shows that GTA-based models improve learning efficiency over
SRT and RePAST by a significant margin and reach the same performance as RePAST using only
1/6 of the training steps on MSN-Hard. GTA also outperforms RePAST in terms of wall-clock time
as each gradient update step is slightly faster than RePAST, see also Table 14 in Appendix B.8.
Comparison to other PE methods: We compare GTA with other PE methods on CLEVR-TR.
All models are trained for 1M iterations. See Appendix C.2.4 for the implementation details. Table 4 shows that GTA outperforms other PE schemes. GTA is better than RoPE+FTL, which uses
RoPE (Su et al., 2021) for the encoder-decoder transformer and transforms latent features of theencoder with camera extrinsics (Worrall et al., 2017). This shows the efficacy of the layer-wisegeometry-aware interactions in GTA.
Effect of the transformation on V : Rotary positional encoding (RoPE) (Su et al., 2021; Sunet al., 2022) is similar to the SO(2) representations in GTA. An interesting difference to GTA isthat the RoPE only applies transformations to query and key vectors, and not to the value vectors.
In our setting, this exclusion leads to a discrepancy between the coordinate system of the key andvalue vectors, both of which interact with the tokens from which the query vectors are derived.
Table 5 left shows that removing the transformation on the value vectors leads to a significant drop in
performance in our NVS tasks. Additionally, Table 5 right shows the performance on the ImageNet
8

Published as a conference paper at ICLR 2024

RePAST

GTA

PR-AUC

Query

0.6
0.5
0.4
0.3
0.2
0.1

GTA

1

2

RePAST

3

Layer

4

5

Fig. 8: Attention analysis. Given a query token (white region), the attention weights on a context
view are visualized. GTA can identify the shape of the object that corresponds to the given query.
Right: Quantitative evaluation of alignments between attention matrices and object masks.
Table 6: Representation design. Test PSNRs of models trained for 1M iterations.
(a) (X): representations are not used in the encoder.
Camera and image position are needed in the decoder
to identify which pixel to render.

(b) Image positions encodings. ∗ Single frequency.

SE(3) SO(2) SO(3) CLEVR-TR MSN-Hard

CLEVR-TR MSN-Hard

X
(X)
X
X

(X)
X
X
X

X

37.45
38.26
38.99
39.00

20.33
23.82
24.58
24.80

SE(3) + T (2)SE(3) + SO(2)∗SE(3) + SO(2)

37.20
38.82
38.99

23.69
23.98
24.58

generative modeling task with diffusion models. Even on this purely 2D task, the GTA mechanism isbetter compared to RoPE as an image positional encoding method (For more details of the diffusion
experiment, please refer to Appendix C.3).
Object localization: As demonstrated in Fig. 8 on MSN-Hard, the GTA-based transformer not only
correctly finds patch-to-patch associations but also recovers patch-to-object associations alreadyin the second attention layer of the encoder. For quantitative evaluation, we compute precisionrecall-AUC (PR-AUC) scores based on object masks provided by MSN-Hard. In short, the score
represents, given a query token belonging to a certain object instance, how well the attention matrixaligns with the object masks across all context views. Details on how we compute PR-AUC areprovided in Appendix B.7. The PR-AUCs for the second attention layer are 0.492 and 0.204 with
GTA and RePAST, respectively, which shows that our GTA-based transformer quickly identifies
where to focus attention at the object level.
Representation design: Table 6a shows that, without camera encoding (SE(3)) or image PE(SO(2)) in the encoder, the reconstruction quality degrades, showing that both representations arehelpful in aggregating multi-view features. Using SO(3) representations causes a moderate improvement on MSN-Hard and no improvement on CLEVR-TR. A reason for this could be that
MSN-Hard consists of a wide variety of objects. By using the SO(3) representation, which isinvariant to camera translations, the model may be able to encode object-centric features more efficiently. Table 6b confirms that similar to Fourier feature embeddings used in APE and RPE, multiple
frequencies of the SO(2) representations benefit the reconstruction quality.

5

C ONCLUSION

We have proposed a novel geometry-aware attention mechanism for transformers and demonstrated
its efficacy by applying it to sparse wide-baseline novel view synthesis tasks. A limitation of GTA isthat GTA and general PE schemes rely on known poses or poses estimated by other algorithms, suchas COLMAP (Schönberger & Frahm, 2016). An interesting future direction is to simultaneously
learn the geometric information together with the forward propagation of features in the transformer.
Developing an algorithm capable of autonomously acquiring such structural information solely fromobservations, specifically seeking a universal learner for diverse forms of structure akin to humancapacity, represents a captivating avenue for future research.
9

Published as a conference paper at ICLR 2024

6

ACKNOWLEDGEMENT

Takeru Miyato, Bernhard Jaeger, and Andreas Geiger were supported by the ERC Starting Grant
LEGO-3D (850533) and the DFG EXC number 2064/1 - project number 390727645. The authors
thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Bernhard Jaeger. We thank Mehdi Sajjadi and Yilun Du for their comments and guidanceon how to reproduce the results and thank Karl Stelzner for his open-source contribution of the SRT
models. We thank Haofei Xu and Anpei Chen for conducting the MatchNeRF experiments. We also
thank Haoyu He, Gege Gao, Masanori Koyama, Kashyap Chitta, and Naama Pearl for their feedback
and comments. Takeru Miyato acknowledges his affiliation with the ELLIS (European Laboratory
for Learning and Intelligent Systems) PhD program.

R EFERENCES
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling. Geometric andphysical quantities improve E(3) equivariant message passing. In Proc. of the International Conf. on Learning Representations (ICLR), 2022. 18
Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco Cohen. Geometric algebra transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 18
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey
Zagoruyko. End-to-end object detection with transformers. In Proc. of the European Conf. on Computer
Vision (ECCV), 2020. 2
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio
Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An
Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University— Princeton University — Toyota Technological Institute at Chicago, 2015. 7, 24
Xu Chen, Jie Song, and Otmar Hilliges. Monocular neural image based rendering with continuous view control.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 3
Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Explicit correspondence matching for generalizable neural radiance fields. arXiv.org, 2304.12294, 2023. 7
Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (SRF): learningview synthesis for sparse views of novel scenes. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2021. 7
Gregory S Chirikjian. Engineering applications of noncommutative harmonic analysis: with emphasis on
rotation and motion groups. CRC press, 2000. 16
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser:
Imitation with transformer-based sensor fusion for autonomous driving. IEEE Trans. on Pattern Analysis
and Machine Intelligence (PAMI), 2022. 2
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks
and the icosahedral cnn. In Proc. of the International Conf. on Machine learning (ICML), 2019. 18
Taco S Cohen and Max Welling. Transformation properties of learned visual representations. In Proc. of theInternational Conf. on Learning Representations (ICLR), 2014. 3
Pim De Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic
convolutions on geometric graphs. In Proc. of the International Conf. on Learning Representations (ICLR),
2021. 18
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil
Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. of theInternational Conf. on Learning Representations (ICLR), 2021. 1, 2
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from widebaseline stereo pairs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 2,
5, 7, 8, 19, 20, 21, 22, 24, 25, 26, 31, 32

10

Published as a conference paper at ICLR 2024

Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In Proc. of the International Conf. on Machine learning (ICML), 2020. 3
Luca Falorsi, Pim De Haan, Tim R Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S Cohen.
Explorations in homeomorphic variational auto-encoding. arXiv.org, 1807.04689, 2018. 3
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan
Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam
Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann,
Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, andAndrea Tagliasacchi. Kubric: a scalable dataset generator. In CVPR, 2022. 24
Lingshen He, Yiming Dong, Yisen Wang, Dacheng Tao, and Zhouchen Lin. Gauge equivariant transformer. In
Advances in Neural Information Processing Systems (NeurIPS), 2021. 18
Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2020. 2
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In Proc. of the International Conf. on Artificial Neural Networks (ICANN), 2011. 3
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In Proc. of theInternational Conf. on Learning Representations (ICLR), 2018. 3
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui,
and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv.org, 2311.04400, 2023. 2, 8,
27
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometrypriors. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 7
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 6
Masanori Koyama, Kenji Fukumizu, Kohei Hayashi, and Takeru Miyato. Neural fourier transform: A generalapproach to equivariant representation learning. arXiv.org, 2305.18484, 2023. 3
Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural renderingfrom few images using transformers. In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.
In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2
Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature
pyramid networks for object detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2017. 2
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite
nature: Perpetual view generation of natural scenes from a single image. In Proc. of the IEEE International
Conf. on Computer Vision (ICCV), 2021. 7
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to3: Zero-shot one image to 3d object. arXiv.org, 2303.11328, 2023a. 2, 8, 27
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multiview 3d object detection. In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2, 8, 26
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2023b. 2
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv.org, 1711.05101, 2017. 22,
24
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure fromsequences. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3

11

Published as a conference paper at ICLR 2024

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetricembeddings for equivariant world models. In Proc. of the International Conf. on Machine learning (ICML),
2022. 3
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2023. 8, 15, 28, 33, 34, 35, 36, 37
Lang Peng, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. Bevsegformer: Bird’s eye view
semantic segmentation from arbitrary camera rigs. In Proc. of the IEEE Winter Conference on Applicationsof Computer Vision (WACV), 2023. 2
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning
with a general conditioning layer. In Proc. of the Conf. on Artificial Intelligence (AAAI), 2018. 2
Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer forfast and robust point cloud registration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2022. 2
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Standalone self-attention in vision models. Advances in Neural Information Processing Systems (NeurIPS), 2019.
2
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proc. of
the IEEE International Conf. on Computer Vision (ICCV), 2021. 2, 26
Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribution network formonocular 3d object detection. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR),
2021. 26
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3d humanpose estimation. In Proc. of the European Conf. on Computer Vision (ECCV), 2018. 3
David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In Proc. of the International Conf. on Machine learning (ICML), 2020. 2
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej
Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale
Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. 28
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in
Neural Information Processing Systems (NeurIPS), 2017. 3
Aleksandr Safin, Daniel Durckworth, and Mehdi SM Sajjadi. RePAST: Relative pose attention scene representation transformer. arXiv.org, 2304.00947, 2023. 2, 6, 7, 19, 24
Avishkar Saha, Oscar Mendez Maldonado, Chris Russell, and Richard Bowden. Translating Images into Maps.
In Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2022. 2
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić,Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In Advances in
Neural Information Processing Systems (NeurIPS), 2022a. 7, 24
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora,
Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometryfree novel view synthesis through set-latent scene representations. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2022b. 1, 2, 5, 6, 7, 24
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2016. 9
Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous drivingusing interpretable sensor fusion transformer. In Proc. Conf. on Robot Learning (CoRL), 2023. 2
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACLHLT), 2018. 1, 2
Changyong Shu, Jiajun Deng, Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for transformerbased multi-camera 3d object detection. In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2023. 2

12

Published as a conference paper at ICLR 2024

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Information
Processing Systems (NeurIPS), 2021. 6
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. 5, 8, 27, 28
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neuralrendering. In Proc. of the European Conf. on Computer Vision (ECCV), 2022. 2, 7
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song,and Furu Wei. A length-extrapolatable transformer. In ACL, 2022. 8
Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is
attention all that nerf needs? In Proc. of the International Conf. on Learning Representations (ICLR), 2023.
2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1, 2
Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry-biased transformers fornovel view synthesis. arXiv.org, 2301.04650, 2023. 2, 8, 27
Shaofei Wang, Andreas Geiger, and Siyu Tang. Locally aware piecewise transformation fields for 3d humanmesh registration. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021a. 2, 7
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporalmodeling for efficient multi-view 3d object detection. arXiv.org, 2303.11926, 2023. 2
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7794–7803, 2018. 1, 2
Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3d object
detection from multi-view images via 3d-to-2d queries. In Proc. Conf. on Robot Learning (CoRL), 2021b. 2
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad
Norouzi. Novel view synthesis with diffusion models. In Proc. of the International Conf. on Learning
Representations (ICLR), 2023. 2
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Interpretable transformations with encoder-decoder networks. In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 3, 8, 27
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative
position encoding for vision transformer. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2021. 2
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying
flow, stereo and depth estimation. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 2023.
2
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-timerendering of neural radiance fields. In Proc. of the IEEE International Conf. on Computer Vision (ICCV),
2021a. 2
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few
images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021b. 6, 7
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information
Processing Systems (NeurIPS), volume 32, 2019. 28
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectivenessof deep features as a perceptual metric. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), pp. 586–595, 2018. 26
Brady Zhou and Philipp Krähenbühl. Cross-view Transformers for real-time Map-view Semantic Segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 2
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification. ACM
Transactions on Graphics, 2018. 7

13

Published as a conference paper at ICLR 2024

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformabletransformers for end-to-end object detection. In Proc. of the International Conf. on Learning Representations(ICLR), 2021. 2
Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang.
Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers.
arXiv.org, 2312.09147, 2023. 2

14

Published as a conference paper at ICLR 2024

Appendix
Table of Contents

A

A Additional details of GTA
A.1 Computational complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Details of the representation matrices . . . . . . . . . . . . . . . . . . . . . . .
A.3 Variants of GTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.4 Relation to equivariant and gauge equivariant networks . . . . . . . . . . . . . .

15
15
15
17
18

B Additional experimental results
B.1 Training curves on CLEVR-TR and MSN-Hard. . . . . . . . . . . . . . . . . .
B.2 Results with higher resolution . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Results with 3 context views . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Robustness to camera noise . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5 Performance with different random seeds . . . . . . . . . . . . . . . . . . . . .
B.6 Performance dependence on the reference coordinates . . . . . . . . . . . . . .
B.7 Analysis of attention patterns . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.8 Computational time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19
19
19
19
19
20
20
21
21

C Experimental settings
C.1 Details of the synthetic experiments in Section 3.1 . . . . . . . . . . . . . . . .
C.2 Experimental settings in Section 4 . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 2D image generation with DiT (Peebles & Xie, 2023) . . . . . . . . . . . . . .

22
22
22
28

D Rendered images

29

E Generated images of DiTs

33

A DDITIONAL DETAILS OF GTA

Algorithm 1 provides an algorithmic description based on Eq. (6) for single-head self-attention. For
multi-head attention, we simply apply the group representations to all QKV vectors of each head.
A.1

C OMPUTATIONAL COMPLEXITY

Since the Pg }· operation is an n-times multiplication of a d×d matrix with a d-dimensional vector,
the computational complexity of additional computation for our attention over the vanilla attentionis O(nd2 ). This can be reduced by constructing the representation matrix with a block diagonal,
with each block being small. If we keep the largest block size of the representation constant againstn and d, then the order of the Pg } · operation becomes O(nd). Thus, if dmax is relatively small, orif we increase n or d, the computation overhead of the } operation becomes negligible compared tothe computation times of the other components of a transformer, which are O(n2 d) for attention andO(nd2 ) for feedforward layers. In our experiments, we use a block-structured representation with amaximum block size of 5 (see Section 3.2 and Fig. 9).
A.2

D ETAILS OF THE REPRESENTATION MATRICES

ρg is composed of four different types of representations ρc , ρr , ρθh , ρθw with the multiplicities ofs, t, u, v. Below, we describe the details of each representation.
15

Published as a conference paper at ICLR 2024

Algorithm 1 GTA for single head self-attention.
Input: Input tokens: X ∈ RN ×d , group representations: Pg = [ρg1 , ρg2 , ..., ρgN ], and weights:
W Q , W K , W V ∈ Rd×d .
1. Compute query, key, and value from X:
Q = XW Q , K = XW K , V = XW V .
2. Transform each variable with the group representations:
Q ← Pg T } Q, K ← Pg −1 } K, V ← Pg −1 } V
3. Compute QKV attention in the same way as in the vanilla attention:


QK T
√
O = softmax
Vd
4. Apply group representations to O:
O ← Pg } O
5. Return O

σcam (c): We use a homogenous rigid transformation as the representation of c ∈ SE(3):R T
σcam (c) :=∈ R4×4 .
0 1(10)σrot (r): We compose σrot (r) via block concatenation of Wigner-D-matrices (Chirikjian, 2000).
σrot (r) :=

L

(lk )
(l)
(l)
(2l+1)×(2l+1)
k σrot (r), σrot (r) := Dr ∈ R

(11)

L
(l)
where Dr is l-th Wigner-D-matrix given r. Here, a∈S A(a) := A(a1 ) ⊕· · ·⊕A(a|S| ) and we omit
the index set symbol from the above equation. We use these matrices because Wigner-D-matrices
are the only irreducible representations of SO(3). Any linear representation σr , r ∈ SO(3) isequivalent to a direct sum of the matrices under a similarity transformation (Chirikjian, 2000).
σh (θh ) and σw (θw ): Similar to σrot (r), we use 2D rotation matrices with different frequencieshfor σh (θh ) and σw (θw ). Specifically, for σh (θh ) given a set of frequencies {fk }N
k=1 , we define therepresentation as follows:
L (f )
cos(f θh ) − sin(f θh )
(f )
(f )
σh (θh ) := k σh k (θh ), σh (θh ) := Mθh =∈ R2×2 .
(12)
sin(f θh ) cos(f θh )σw (θw ) is defined analogously.

We use the following strategy to choose the multiplicities s, t, u, v and frequencies {l}, {fh }, {fw }:1. Given the feature dimension d of the attention layer, we split the dimensions into three components based on the ratio of 2 : 1 : 1.
⊕s
2. • We apply σcamto the first half of the dimensions. As σcam does not possess multiple frequencies, its multiplicity is set to d/8.
⊕r
• σrotis applied to a quarter of the dimensions. For the frequency parameters {l} of σrot ,we consistently use the 1st and 2nd degrees of the Winger-D matrices. Considering the
combined sizes of these matrices is 8, the multiplicity for σrot becomes d/32.
⊕u• For the remaining 1/4 of the dimensions of each QKV vector, we apply both σh⊕t and σw.
Regarding the frequency parameters {fh }, {fw }, we utilize d/16 octaves with the maximum
frequency set at 1 for both σh and σw . The multiplicities for these are both 1.
Based on this strategy, we use the multiplicities and frequencies shown in Table 7. Also Fig. 9
displays the actual representation matrices used on the MSN-Hard dataset.
16

Published as a conference paper at ICLR 2024

0

0

48

48

72
96 0

48

72

96 0

96

48

96

Fig. 9: Representation matrices on MSN-Hard. Left: with SO(3), Right: without SO(3). Left:⊕3⊕12
Dimensions 1-48 correspond to σcam(c), dimensions 49-72 correspond to σrot(r), and dimensions⊕12
73-96 correspond to σh (θh ) and σw (θw ). Right: Dimensions 1-48 correspond to σcam(c) and dimensions 49-96 correspond to σh (θh ) and σw (θw ).

Table 7: Multiplicity and frequency parameters. Here, d is the dimensions of each attention head.
Since the baseline model on RealEsate10k and ACID uses different feature sizes for query-key
vectors and value vectors, we also use different sizes of representation matrices for each feature.
d{s, t, u, v} {l1 , ..., lNrot }CLEVR-TR{8, 3, 1, 1}
64
CLEVR-TR wo/ SO(3){8, 0, 1, 1}
MSH-Hard{12, 3, 1, 1}
96
MSH-Hard wo/ SO(3){12, 0, 1, 1}
Realestate10k and ACID (Encoder)64 {8, 3, 1, 1}
Realestate10k and ACID (Decoder, key) 128 {16, 6, 1, 1}
Realestate10k and ACID (Decoder, value) 256 {32, 12, 1, 1}

A.3{1, 2}−{1, 2}−{1, 2}{1, 2}{1, 2}{f1 , ..., lN{h,w} }{1, ..., 1/23 }{1, 1/2, 1/4, ..., 1/27 }{1, 1/2, 1/4, ..., 1/25 }{1, 1/2, 1/4, ..., 1/211 }{1, ..., 1/23 }{1, ..., 1/27 }{1, ..., 1/215 }VARIANTS OF GTA

Here we would like to introduce two variants of GTA. The one is the Euclidean version of GTA
where we use the Euclidean distance for the attention similarity. The other one is GTA with a grouprepresentation composed of the Kronekcer product of smaller representations. We see in Table 8 thatthe performances of those variants of GTA are a little degraded but relatively close to the original
GTA. We will detail each variant in the following sections.
A.3.1

E UCLIDEAN GTA

The unconventional aspect of Eq. (6) is the presence of the transpose in the transformation of thequery vectors. The transpose is necessary for having the reference coordinate invariance, and theneed arises from the fact that the dot-product similarity is not invariant under SE(3) transformations when the translation component is non-zero. To ensure both reference coordinate invarianceand consistent transformations across the Q, K, V vectors, one can utilize the Euclidean similarity for computing the attention matrix instead of the dot-product similarity. The formula for the
self-attention layer with squared Euclidean distance is given by:
O := AttnEuclid (Q, K, V ) = softmax(E(Q, K))V,

(13)

N ×N

(14)

where E(Q, K) ∈ R, Eij (Q, K) = −kQi − Kj k22 .

Then the Euclidean version of GTA (GTA-Euclid) is written in the following form:
O = Pg } AttnEuclid Pg −1 } Q, Pg −1 } K, Pg −1 } V .
17

(15)

Published as a conference paper at ICLR 2024

Eq. (15) possesses the reference coordinate invariance property since the square distance is preserved
under rigid transformations. The numbers of Table 8 are produced under the same setting as theoriginal GTA, except that we replaced the dot-product attention with the Euclidean attention that we
describe above.
A.3.2

K RONECKER GTA

Another typical way to compose a representation matrix is using the Kronecker product. The Kronecker product of two square matrices A, B ∈ Rm×m , Rn×n is defined as:


a11 B

A ⊗ B =  ...

am1 B

···
..
.
···


a1m B
..  ∈ Rmn×mn .
. 

(16)

amm B

The important property of the Kronecker product is that the Kronecker product of two representations is also a representation: (ρ1 ⊗ ρ2 )(gg 0 ) = (ρ1 ⊗ ρ2 )(g)(ρ1 ⊗ ρ2 )(g 0 ) where (ρ1 ⊗ ρ2 )(g) :=ρ1 (g) ⊗ ρ2 (g). We implement the Kronecker version of GTA, which we denote GTA-Kronecker,
where we use the Kronecker product of the SE(3) representation and SO(2) representations as arepresentation ρg :
ρg = ρcam (c) ⊗ (ρh (θh ) ⊕ ρw (θw )), where g = (c, θh , θw ).

(17)

In the results presented in Table 8, the multiplicity of ρcam , ρh , ρg are set to 1, and the number of
frequencies for both ρh and ρw is set to 4 on CLEVR-TR and 6 on MSN-Hard.
Table 8: Results with different representation forms.
CLEVR-TR MSN-Hard
GTA-Kronecker
GTA-Euclid
GTA

A.4

38.32
38.59
38.99

24.52
24.75
24.80

R ELATION TO EQUIVARIANT AND GAUGE EQUIVARIANT NETWORKS

The gauge transform used in gauge equivariant networks (Cohen et al., 2019; De Haan et al., 2021;
He et al., 2021; Brandstetter et al., 2022) is related to the relative transform ρ(gi gj−1 ) used in ourattention mechanism. However, the equivariant models and ours differ because they are built on
different motivations. In short, the gauge equivariant layers are built to preserve the feature fieldstructure determined by a gauge transformation. In contrast, since image features themselves are
not 3D-structured, our model applies the relative transform only on the query and key-value pair in
the attention mechanism but does not impose equivariance on the weight matrices of the attention
and the feedforward layers. The relative transformation in GTA can be thought of as a form of
guidance that helps the model learn structured features within the attention mechanism from theinitially unstructured raw multi-view images.
Brehmer et al. (2023) introduce geometric algebra (GA) to construct equivariant transformer networks. Elements of GA are themselves operators that can act on GA, which may enable us toconstruct expressive equivariant models by forming bilinear layers that allow interactions between
different multi-vector subspaces. In NVS tasks, where the input consists of raw images lacking
geometric structure, directly employing such equivariant models may not be straightforward. However, integrating GA into the GTA mechanism could potentially enhance the network’s expressivity,
warranting further investigation.
18

Published as a conference paper at ICLR 2024

B

A DDITIONAL EXPERIMENTAL RESULTS

B.1

T RAINING CURVES ON CLEVR-TR AND MSN-H ARD .

10 4
0.0 0.5 1.0 1.5 2.0

Steps

1e6

0.01

0.0

0.5

1.0

Steps

1.5

0.006

0.003

2.0
1e6

PSNR

10 3

40
38
36
34
32
30

MSE

PSNR

MSE

10 2

0

1

2

Steps

3

4

25.5
25.0
24.0
23.0
22.0
21.0

1e6

0

1

2

Steps

3

4

1e6

Fig. 10: Training and validation curves. Left: CLEVR-TR, Right: MSN-Hard.
B.2

R ESULTS WITH HIGHER RESOLUTION

Table 9 shows results on RealEstate10k with 384x384 resolutions (1.5 times larger height and widththan in Table 3). We see that GTA also improves over the baseline model at higher resolution.
Table 9: 384×384 resolution on RealEstate10K
PSNR↑ LPIPS ↓ SSIM↑
Du et al. (2023)
Du et al. (2023) + GTA (Ours)

B.3

21.77
22.77

0.316
0.290

0.848
0.864

R ESULTS WITH 3 CONTEXT VIEWS

We train models with 3-context views and show the results in Table 10. We see that GTA is also
better with context views more than 2.
Table 10: Results with different numbers of context views on RealEstate10K
2-view PSNR↑

3-view PSNR↑

21.65
22.85

21.88
23.22

Du et al. (2023)
Du et al. (2023) + GTA (Ours)

B.4

ROBUSTNESS TO CAMERA NOISE

Table 11 shows results on CLEVR-TR with a presence of camera noise. We train RePAST and GTA
with camera noise added to each camera extrinsic of the second view. We perturb camera extrinsics
by adding Gaussian noise to the coefficients of the SE(3)-Lie algebra basis. The mean and varianceof the noise is set to (m, σ) = (0, 0.1) during training. GTA shows better performance than RePAST
regardless of the noise level.
Table 11: Test PSNRs with camera noise on CLEVR-TR and MSN-Hard. σtest indicates thenoise strength at test time. † No noise injection during training.

σtest

CLEVR-TR
0.01
0.1

MSN-Hard
0.01
0.1

RePAST (Safin et al., 2023)
SRT+GTA (Ours)

35.26
36.66

22.76
24.06

19

35.14
36.57

22.60
24.16

Published as a conference paper at ICLR 2024

Table 12: Test metrics on CLEVR-TR.

APE
RPE
SRT
RePAST
GTA (Ours)

PSNR↑

LPIPS ↓

SSIM↑

33.66
36.06
33.51
37.27
39.63

0.161
0.159
0.158
0.119
0.108

0.960
0.971
0.960
0.977
0.984

Table 14: Computational time to perform one gradient step, encode a single scene, and rendera single entire image on MSN-Hard (top) and RealEstate10K (bottom). All time values are
expressed in milliseconds (ms). As for one gradient step time, we only measure time for forwardbackward props and weight updates and exclude data loading time. We measure each time on asingle A100 with bfloat16 precision for MSN-Hard and float32 precision for RealEstate10K.
MethodOne gradient step

Encoding

Rendering

SRT
RePAST
GTA

296
394
379

5.88
7.24
17.7

16.4
21.4
20.9

One gradient step

Encoding

Rendering

619
806

49.8
74.3

1.42×103
2.05×103

MethodDu et al. (2023)
GTA

B.5

P ERFORMANCE WITH DIFFERENT RANDOM SEEDS

We observe that the performance variance of different random weight initializations is quite small,as shown in Fig. 11, which displays the mean and standard deviation across 4 different seeds. We
see that the variance is relatively insignificant compared to the performance difference between thecompared methods. Consequently, the results reported above are statistically meaningful.
B.6

P ERFORMANCE DEPENDENCE ON THE REFERENCE COORDINATESTable 13 highlights the importance of coordinate-choice invariance. “SRT (global coord)” is trainedwith camera poses that have their origin set to always be in the center of all objects. This setting
enables the model to know how far the ray origin is from the center of the scene, therefore enabling
the model to easily find the position of the surface of objects that intersect with the ray. We see that
SRT’s performance heavily depends on the choice of reference coordinate system. Our model is,by construction, invariant to the choice of reference coordinate system of cameras and outperforms
even the privileged version of SRT.
Table 13: Test PSNRs in a setting where global coordinates are shared across scenes. All
numbers show test PSNRs and are produced with models trained for 1M iterations. Note that GTA
is invariant to the reference coordinates of the extrinsics, and the performance is not affected by the
choice of the reference coordinate system.
MethodCLEVR-TR

MSH-Hard

32.97
37.93
38.99

23.15
24.20
24.58

SRT
SRT (global coord)
GTA wo SO(3)20Published as a conference paper at ICLR 2024

CLEVR-TR

23.0

34

22.5

32

22.0

PSNR

PSNR

36

30
28
26
24

0.0

0.5

1.0

Steps

GTA
RePAST
1.5 2.0

MSN-Hard
GTA
RePAST

21.5
21.0
20.5
20.0

0.0 0.5 1.0 1.5 2.0

Steps

1e5

1e5

Fig. 11: Mean and standard deviation plots of validation PSNRs on CLEVR-TR and MSNHard. Due to the heavy computation requirements for training, we only trained models with 200,000
iterations and measured the validation PSNRs during the course of the training.

B.7

A NALYSIS OF ATTENTION PATTERNS

We conducted an analysis on the attention matrices of the encoders trained on MSN-Hard. We foundthat the GTA-based model tends to attend to features of different views more than RePAST, which
we show in Fig. 12. Furthermore, we see that GTA not only correctly attends to the respective
patches of different views, but also can attend to object level regions (Fig. 8 and 13). Surprisingly,
these attention patterns are seen at the very beginning of the course of the encoding process: thevisualized attention maps are obtained in the 2nd attention layer. To evaluate how well the attention
maps α weigh respective object features across views, we compute a retrieval-based metric withinstance segmentation masks of objects provided by MSN-Hard. Specifically, given a certain layer’s
attention maps α:
1. We randomly sample the i-th query patch token with 2D position p ∈ {1, ..., 16} × {1, ..., 16}.
∗∗
2. We compute the attention map ᾱi ∈ [0, 1]5 16 16 averaged over all heads.
3. We then identify which object belongs to that token’s position by looking at the corresponding
8 × 8 region of the instance masks. Note that multiple objects can belong to the region.
4. For each belonging object, we compute precision and recall values with 1[ᾱi > t] as prediction
and 0–1 masks of the corresponding object as ground truth on all context views, by changing thethreshold value t ∈ [0, 1].
5. In the final step, we calculate a weighted average of the precision and recall values for eachobject. To determine the weight of each object, we consider the number of pixels assigned tothat object’s mask within the 8x8 region. We then normalize these weights so that their sum
equals to be one.
We collect multiple precision and recall values by randomly sampling scenes and patch positions
2000 times and then compute the average of the collected precision-recall curves. In Fig. 14, weshow averaged precision-recall curves. Table 8 shows the area under the precision-recall curves(PR-AUCs) of each layer. We see that the GTA-based model learns well-aligned attention maps
with the ground truth object masks for every layer.
B.8

C OMPUTATIONAL TIME

We measure the time to perform one-step gradient descent, as well as encoding and decoding foreach method. Table 14 shows that the computational overhead added by the use of GTA is comparable to RePAST on MSN-Hard. In contrast to GTA and RePAST-based models which encode
positional information into every layer, SRT and Du et al. (2023) add positional embeddings only
to each encoder and decoder input. As a result, the computational time of SRT for one-step gradient21

Published as a conference paper at ICLR 2024

GTA

RePAST

Fig. 12: Visualization of view-to-view attention maps. The (i, j)-th element of each 5x5 matrix
represents the average of attention weights between all pairs of each query token of the i-th context view and each key token of the j-th context view. The (l, m)-th panel shows the weight of them-th head at the l-th layer. Yellow and dark purple cells indicate high and low attention weight,
respectively. A matrix with high diagonal values means that the corresponding attention head attends within each view while with high non-diagonal values means the corresponding attention headattends across views.

descent is around 1.3x faster than RePAST and GTA, and that of Du et al. (2023) is 1.3x faster thanGTA.

CE XPERIMENTAL SETTINGS

C.1

D ETAILS OF THE SYNTHETIC EXPERIMENTS IN S ECTION 3.1

We use 10, 000 training and test scenes. For the intrinsics, both the vertical and horizontal sensor
width are set to 1.0, and the focal length is set to 4.0, leading to an angle of view of 28°.
For optimization, we use AdamW (Loshchilov & Hutter, 2017) with weight decay 0.001. For eachPE method, we trained multiple models with different learning rates of {0.0001, 0.0002, 0.0005}
and found 0.0002 to work best for all models, and hence show results with this learning rate. We
use three attention layers for both the encoder and the decoder. The image feature dimension is
32 × 32 × 3. This feature is flattened and fed into a 2 layer-MLP to be transformed into the same
dimensions as the token dimension d. We also apply a 2 layer-MLP to the output of the decoder toobtain the 3, 072 dimensional predicted image feature. The token dimensions d are set to 512 for
APE and RPE. As we mention in the descriptions of the synthetic experiment, ρg is composed ofblock concatenation of 3 × 3 rotation matrices, and we set d to 510 for GTA, which is divisible by
3. Note that there is no difficulty with the case where d is not divisible by 3. In that case, we canapply ρg only to certain components of vectors whose dimensions are divisible by 3 and apply notransformation to the other dimensions. This corresponds to applying a trivial representation, i.e.,
the identity matrix, to the remaining vectors.
The RPE-based model we designed is a sensible model. For example, if bQ = bK and the set ofthree-dimensional vector blocks of bQ forms an orthonormal basis, then the inner product of thetransformed query and key bias vectors becomes the trace of the product of the rotation matrices:hρ(r)bQ , ρ(r0 )bK i = tr(RT R0 ). tr(AT B) is a natural inner product for matrices, by which we
can bias the attention weight based on the inner-product-based similarity of matrices. Hence, we
initialize each of the biases with vectorized identity matrices.
C.2

E XPERIMENTAL SETTINGS IN S ECTION 4

Table 15 shows dataset properties and hyperparameters that we use in our experiments. We train
with 4 RTX 2080 Ti GPUs on CLEVR-TR and with 4 Nvidia A100 GPUs on the other datasets.
22

Published as a conference paper at ICLR 2024

GTA

RePASTGTARePASTGTARePAST

Fig. 13: Additional attention map visualizations.
Layer= 1

0.2
0.0
0.0

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6
0.4
0.2

0.2

0.4

0.6

Recall

0.8

1.0

5

0.0
0.0

0.6
0.4
0.2

0.2

0.4

0.6

Recall

0.8

1.0

0.0
0.0

Precision

0.4

4Precision

Precision

0.6

3Precision

RePAST
GTA

0.8

Precision

1.0

2

0.6
0.4
0.2

0.2

0.4

0.6

Recall

0.8

1.0

0.0
0.0

0.6
0.4
0.2

0.2

0.4

0.6

Recall

0.8

1.0

0.0
0.0

0.2

0.4

0.6

RecallFig. 14: Precision-recall curves of the attention matrices of each encoder layer.

23

0.8

1.0

Published as a conference paper at ICLR 2024

Table 15: Dataset properties and architecture hyperparameters. # target pixels indicate how
many query pixels are sampled for each scene during training. ∗ We use 12 heads for the attention
layers in SRT and 8 heads in RePAST and GTA because 12 head models do not fit into our GPU
memory with those methods. † The decoder’s attention layers only have a single head. Also, thetoken dimensions in the decoder are set to 128 for query-key vectors and 256 for value vectors.
dataset

CLEVR-TR MSN-Hard RealEstate10k# Training scenes# Test scenes
Batch size
Training steps
Learning rate
# Context views# Target pixels# Self-attention layers in the encoder# Cross-attention layers in the decoder# Heads in attention layers
Token dimensions
MLP dimensions

20,0001,000
322,000,000
1e-4
2
512
5
2
6
384
768

1,000,000
10,00064
4,000,000
1e-4
5
2,048
5
2
12/8∗
768
1,536

ACID

66,837
10,974
7,192
1,910
48
48
300,000
200,000
5e-4
2
192
12
2
12†
768†
3,072

Table 16: Performance comparison between numbers reported in Safin et al. (2023) and our
reproduced numbers. Note that Safin et al. (2023) uses 4x larger batch size than available in our
experimental setting (4 A100s). The number of iterations for which we train each model is the sameas Safin et al. (2023).
PSNR↑ LPIPSVGG/Alex ↓ SSIM↑
SRT (Sajjadi et al., 2022b)
RePAST (Safin et al., 2023)

24.56
24.89

NA/0.223
NA/0.202

0.784
0.794

SRT
RePAST
SRT+GTA (Ours)

24.27
24.48
25.72

0.368/0.279
0.348/0.243
0.289/0.185

0.741
0.751
0.798

CLEVR-TR and MSN-Hard CLEVR-TR is synthesized by using Kubric (Greff et al., 2022).
The resolution of each image is 240 × 320. The camera poses of the dataset include translation,
azimuth, and elevation transformations. The camera does not always look at the center of the scene.
MSN-Hard is also a synthetically generated dataset.
Up to 32 objects sampled from
ShapeNet (Chang et al., 2015) appear in each scene. All 51K ShapeNet objects are used for thisdataset, and the training and test sets do not share the same objects with each other. MSN-Hard
includes instance masks for each object in a scene, which we use to compute the attention matrixalignment score described in Section 4 and Appendix B.7. The resolution of each image is 128×128.
We basically follow the same architecture and hyperparameters of the improved version of SRT
described in the appendix of Sajjadi et al. (2022a), except that we use AdamW (Loshchilov &Hutter, 2017) with the weight decay set to the default parameter and dropout with a ratio of 0.01 at
every attention output and hidden layers of feedforward MLPs.
Since there is no official code or released models available for SRT and RePAST, we train bothbaselines ourselves and obtain almost comparable but slightly worse results (Table 16). This isbecause we train the models with a smaller batch size and target ray samples than in the original
setting due to our limited computational resources (4 A100s). Note that our model, which is also
trained with a smaller batch size, still outperforms the original SRT and RePAST models’ scores.
RealEstate10k and ACID Both datasets are sampled from videos available on YouTube. At thetime we conducted our experiments, some of the scenes used in Du et al. (2023) were no longer
available on YouTube. We used scenes 66, 837 and 10, 974 training scenes and 7, 192 and 1, 910
test scenes for RealEstate10k and ACID, respectively. The resolution of each image in the original
24

Published as a conference paper at ICLR 2024

Table 17: Comparison between results reported in Du et al. (2023) (Top) and our reproduced
results (Bottom).
RealEstate10k
PSNR↑ LPIPS↓ SSIM↑

PSNR↑

ACID
LPIPS↓SSIM↑

Du et al. (2023)

21.38

0.262

0.839

23.63

0.364

0.781

Du et al. (2023)
Du et al. (2023) + GTA (Ours)

21.65
22.85

0.284
0.255

0.822
0.850

23.35
24.10

0.334
0.291

0.801
0.824

Input rays
Encoder

CNN

Transformer(self-attention)γ

Context images

OutputMLP

γTransformer(crossattention)

Decoder

Target ray

Fig. 15: Scene representation transformer (SRT) rendering process. The encoder E consistingof a stack of convolution layers followed by a transformer encoder translates context images into a
set representation S. The decoder D predicts an RGB pixel value given a target ray and S. In ourmodel, every attention layer in both the encoder and decoder is replaced with GTA. We also remove
the input and target ray embeddings from the input of the encoder and decoder, respectively. We
input a learned constant vector to the decoder instead of the target ray embeddings.
sequences is 360 × 640. For training, we apply downsampling followed by a random crop andrandom horizontal flipping to each image, and the resulting resolution is 256 × 256. For test time,
we apply downsampling followed by a center crop to each image. The resolution of each processed
image is also 256 × 256. We follow the same architecture and optimizer hyperparameters of Du
et al. (2023). Although the authors of Du et al. (2023) released the training code and their model on
RealEstate10k, we observed that the model produces worse results than those reported in their work.
The results were still subpar even when we trained models with their code. As a result, we decided
to train each model with more iterations (300K) compared to the 100K iterations mentioned in theirpaper and achieved comparable scores on both datasets. Consequently, we also trained GTA-basedmodels for 300K iterations as well.
C.2.1

S CENE REPRESENTATION TRANSFORMER (SRT)

Encoding views: Let us denote Ncontext -triplets of input view images and their associated camera
Ncontext
information by I := {(Ii , ci , Mi )}i=1, where Ncontext is the number of context views, Ii ∈RH×W ×3 is the i-th input RGB image, and ci ∈ R4×4 , Mi ∈ R3×3 are a camera extrinsic and acamera intrinsic matrix associated of the i-th view. The SRT encoder E encodes the context of views
into scene representation S and is composed of a CNN and a transformer Etransformer . First, a 6layer CNN ECNN is applied to a ray-concatenated image I 0 of each view to obtain (H/D)×(W/D)resolution features:0
Fi = ECNN (Ii0 ) ∈ R(H/D)×(W/D)×d , Iihw
= Iihw ⊕ γ(rihw )(18)

where d is the output channel size of the CNN, and D is the downsampling factor, which is set to
8. γ is a Fourier embedding function that transforms ray r = (o, d) ∈ R3 × S into a concatenationof the Fourier features with multiple frequencies. Each ray rihw is computed from given camera’s
extrinsic and intrinsic parameters (ci , Mi ). Here, “⊕” denotes vector concatenation.
25

Published as a conference paper at ICLR 2024

Next, a transformer-based encoder Etransformer processes the flattened CNN features of all views
together to output the scene representation:
Ncontext ∗ (H/D)∗ (W/D)
Ncontext ∗ (H/D)∗ (W/D)
S := {si }i=1
= Etransfomer {fi }i=1(19)where {fi } is the set of flattened CNN features.

Rendering a view: Given the scene representation S and a target ray r∗ , the decoder D outputs an
RGB pixel:âr∗ = D(γ(r∗ ), S) ∈ R3 .

(20)

where γ is the same function used in Eq. (18). The architecture of D comprises two stacks of across-attention block followed by a feedforward MLP. The cross-attention layers determine which
token in the set S to attend to, to render a pixel corresponding to the given target ray. The output of
the cross-attention layers is then processed by a 4-layer MLP, to get the final RGB prediction. The
number of hidden dimensions of this MLP is set to 1536.
Optimization: The encoder and the decoder are optimized by minimizing the mean squared error
between given target pixels ar and the predictions:X
L(E, D) =kar∗ − âr∗ k22 .
(21)
r∗

C.2.2

D ETAILS OF THE ARCHITECTURE AND LOSS OF D U ET AL . (2023)

Du et al. (2023) proposes an SRT-based transformer NVS model with a sophisticated architecture. The major differences between their model and SRT are that they use a dense vision transformer (Ranftl et al., 2021) for their encoder. They also use an epipolar-based sampling techniqueto select context view tokens, a process that helps render pixels efficiently in the decoding process.
We use the same optimization losses for training models based on this architecture as Du et al.
(2023). Specifically, we use the L1 loss between target and predicted pixels on RealEstate10k andACID. We also use the following combined loss after the 30K-th iterations on ACID.
L1 (P, P̂ ) + λLPIPS LLPIPS (P, P̂ ) + λdepth Ldepth (P, P̂ )(22)

where P, P 0 ∈ R32×32×3 are target and predicted patches. LLPIPS is the perceptual similarity metricproposed by Zhang et al. (2018). Ldepth is a regularization loss that promotes the smoothness of
estimated depths in the model. Please refer to Du et al. (2023) for more details. On RealEstate10k,
we found that using the combined loss above deteriorates reconstruction metrics. Therefore, we
train models on RealEsatate10k solely with the L1 loss for 300K iterations.
C.2.3

APE- AND RPE- BASED TRANSFORMERS ON CLEVR-TR

For the APE-based model, we replace the ray embeddings in SRT with a linear projection of the
combined 2D positional embedding and flattened SE(3) matrix. To build an RPE-based model,
we follow the same procedure as in Section 3.1 and apply the representations to the bias vectors
appended to the QKV vectors. Each bias dimension is set to 16 for the σcam and 16 for σh and σw .
The multiplicities and frequency parameters are determined as described in Section 3.2. {s, u, v}
is set to {4, 1, 1} and {f } is set to {1, ..., 1/23 } for both σh and σw . Table 12 shows an extended
version of Table 2, which includes LPIPS (Zhang et al., 2018) and SSIM performance.
C.2.4

I MPLEMENTATION OF OTHER PE METHODS

Frustum positional embeddings (Liu et al., 2022): Given an intrinsic K ∈ R3×3 , we transformthe 2D image position of each token by K −1 (x, y, 1)T . We follow Liu et al. (2022) and generate
points at multiple depths with the linear-increasing discretization (Reading et al., 2021), where each
depth value at index i = 1, ..., D is computed bydmin +

dmax − dmini(i + 1)
D(D + 1)26

(23)

Published as a conference paper at ICLR 2024

Fig. 16: Frustum points on CLEVR-TR. The black point indicates the origin (0, 0, 0). Each object
is sampled with its center in the range of ∈ [−4, 4] × [−4, 4] × {t/2} where t is the height of theobject.
where [dmin , dmax ] is the full depth range and D is the number of depth bins. Examples of thegenerated 3D points are visualized in Fig. 16. The concatenation of 3D points of multiple depth ateach pixel is further processed by a learned 1-layer MLP, and added to input.
Modulated layer normalization (Hong et al., 2023; Liu et al., 2023a): Modulated layer normalization (MLN) modulates and biases each token feature x by using vector features γ, β each ofwhich encodes geometric information. In Liu et al. (2023a), each token’s geometric information is a
triplet of a camera transformation, velocity, and time difference of consecutive frames. However, in
our NVS tasks, the last two information does not exist. Thus, the vectors simply encode the cameratransformation. Each γ, β is computed by: γ = ξγ (vec(E −1 )), β = ξβ (vec(E −1 )) where vec flattens the input matrix and ξγ,β are learned linear transformations. Each token x is transformed withγ and β as follows:
x0 = γ
where

LN (x) + β(24)denotes element-wise multiplication.

Geometry-biased transformers (GBT) (Venkat et al., 2023): GBT biases the attention matrixof each layer by using the ray distance. Specifically, suppose each token associates with a rayr = (o, d) ∈ R3 × S 2 . GBT first converts r into plücker coordinate r0 = (d, m) where m = o × d.
Then the ray distance between two rays r0q = (dq , mq ) and r0k = (dk , mk ) linked to each query
vector q and key vector k is computed by: q k k q |d ·m +d ·m |if dq × dk 6= 0
q ×dk ||2
dist(r0q , r0k ) = kdq||d
(25)
q
k
k
k2
 (m −mq /s)+d
if dq = sdk , s 6= 0.
2
kd k2

The GBT’s attention matrix is computed by:softmax(QK T − γ 2 D(Q, K)),
N ×N

0Qi

(26)

0Kj

where D(Q, K) ∈ R, Dij (Q, K) = dist(r , r ). γ ∈ R is a learned scaler parameterthat controls the magnitude of the distance bias. Following Venkat et al. (2023), in addition tothis bias term, we also add a Fourier positional embedding computed with the plücker coordinate
representation of the ray at each patch in the encoder and at each pixel in the decoder.
Element-wise multiplication: In this approach, for each token with a geometric attribute g, we
first concatenate the flattened SE(3) homogeneous matrix and flattened SO(2) image positional
representations with multiple frequencies. The number of frequencies is set to the same number
as in GTA on CLEVR-TR. The concatenated flattened matrices are then linearly transformed tothe same dimensional vectors as each Q, K, V . Then these vectors are element-wise multiplied toQ, K, V and the output of Attn in Eq. (6) in a similar way to GTA.
RoPE+FTL (Su et al., 2021; Worrall et al., 2017): RoPE (Su et al., 2021) is similar to GTA but
does not use the SE(3) part (extrinsic matrices) as well as transformations on value vectors. In thisapproach, we remove SE(3) component from the representations. Also, we remove the transformations on the value vectors from each attention layer. As an implementation of FTL (Worrall et al.,27

Published as a conference paper at ICLR 2024

2017), we apply SE(3) matrices to the encoder output to get transformed features to render novel
views with the decoder.
C.3

2D IMAGE GENERATION WITH D I T (P EEBLES & X IE , 2023)

RoPE (Su et al., 2021) is a method commonly used to encode positional information in transformer
models. GTA and RoPE are similar but differ in that, in GTA, group transformations are applied to
the value vectors in addition to the query and key vectors, leading to improvements in our NVS tasks
compared to models without this transformation. To further investigate the effectiveness of the valuetransformation, we conduct a 2D image generation experiment. We will describe the experimental
setting in the following. We also opensource the code for this experiments in the same repository as
our NVS experiments, and please refer to it for further details.
Following the experimental setup of DiT (Peebles & Xie, 2023), we use a transformer-based denoising network for image generation on ImageNet (Russakovsky et al., 2015). The image resolution
is set to 256x256, and we choose the DiT-B/2 model as our baseline. Since the original DiT model
does not adopt RoPE encoding, we trained models with both RoPE and GTA positional encodings.
We use the same representation matrix ρg for both RoPE and GTA, which is written as follows:ρg := σh (θh ) ⊕ σw (θw ).

(27)

Here, the notation of each symbol is the same as in the main section. The representation designof each σh and σw follows the original work of RoPE (Su et al., 2021). Training of each model isconducted for 2.5M iterations (approximately 500 epochs) with batch size of 256. We experimentwith mixed-precision training (BFloat16), but observed instability when using RoPE and GTA. To
address this, we adopt RMSNorm (Zhang & Sennrich, 2019) applied to each Q and K vector,
with which we find that no instability is made throughout the training. We report in Table 5 (Right)
inception scores and FIDs with classifier-free guidance and its scale set to 1.5. We show comparisonsof generated images in Section E.

28

Published as a conference paper at ICLR 2024

D

R ENDERED IMAGES

Context images

APE

RPE

SRT

RePAST

GTA

Ground truth

Context images

APE

RPE

SRT

RePAST

GTA

Ground truth

Fig. 17: Qualitative results on CLEVR-TR.
29

Published as a conference paper at ICLR 2024

Context images

SRT

RePAST

GTA

Ground truth

GTA

Ground truth

GTA

Ground truth

Context images

SRT

RePAST

Context images

SRT

RePAST

Fig. 18: Qualitative results on MSN-Hard.

30

Published as a conference paper at ICLR 2024

Context images

Du et al. (2023)

GTA

Fig. 19: Qualitative results on RealEstate10k.

31

Ground truth

Published as a conference paper at ICLR 2024

Context images

Du et al. (2023)

GTA

Fig. 20: Qualitative results on ACID.

32

Ground truth

Published as a conference paper at ICLR 2024E

G ENERATED IMAGES OF D I T S

DiT + GTA

DiT + RoPE

DiT (Peebles & Xie, 2023)

Fig. 21: Class-conditional generation on ImageNet. Labels and noises are randomly sampled.
33

Published as a conference paper at ICLR 2024

DiT + GTA

DiT + RoPE

DiT (Peebles & Xie, 2023)

Fig. 22: Generated images with class label ‘Goldfish’34

Published as a conference paper at ICLR 2024

DiT + GTA

DiT + RoPE

DiT (Peebles & Xie, 2023)

Fig. 23: Generated images with class label ‘Tree frog’35

Published as a conference paper at ICLR 2024

DiT + GTA

DiT + RoPE

DiT (Peebles & Xie, 2023)

Fig. 24: Generated images with label ‘Boston bull’36

Published as a conference paper at ICLR 2024

DiT + GTA

DiT + RoPE

DiT (Peebles & Xie, 2023)

Fig. 25: Generated images with label ‘Peacock’37