Where Am I and What Will I See: An Auto-Regressive Model for Spatial Localization and View Prediction
Authors: Junyi Chen
Di Huang
Weicai Ye
Wanli Ouyang
Tong He
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Models
Novel View Synthesis
Camera Pose Estimation
Primary area: generative models

Abstract: Spatial intelligence is the ability of a machine to perceive, reason, and act in three dimensions within space and time.
Recent advancements in large-scale auto-regressive models have demonstrated remarkable capabilities across various reasoning tasks. However, these models often struggle with fundamental aspects of spatial reasoning, particularly in answering questions like "Where am I?" and "What will I see?". While some attempts have been done, existing approaches typically treat them as separate tasks, failing to capture their interconnected nature. In this paper, we present **G**enerative **S**patial **T**ransformer (GST), a novel auto-regressive framework that jointly addresses spatial localization and view prediction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction. The proposed innovative camera tokenization method enables the model to learn the joint distribution of 2D projections and their corresponding spatial perspectives in an auto-regressive manner. This unified training paradigm demonstrates that joint optimization of pose estimation and novel view synthesis leads to improved performance in both tasks, for the first time, highlighting the inherent relationship between spatial awareness and visual prediction.

Published as a conference paper at ICLR 2025
Where Am I and What Will I See: AN AUTO-REGRESSIVE MODEL FOR SPATIAL LOCAL- IZATION AND VIEW PREDICTION
Junyi Chen Di Huang Weicai Ye?
Wanli Ouyang²
Tong He²
Shanghai Jiao Tong University Juny ichnees jtu. edu.co en
*Shanghai Al Lab
i.eiil...o com
WHERE
WHAT WILL SEE
12
Figure 1: We introduce Generative Spatial Transformer (GST), which has unified the image repre- sentation and camera pose representation within the realm of 3D vision, enabling the autoregressive generation of results in another modality given an observed image and specific modality.
ABSTRACT
Spatial intelligence is the ability of machine to perceive, reason, and act three dimensions within space and time. Recent advancements in large-scale auto- regressive models have demonstrated remarkable capabilities across various rea- soning tasks. However, these models often struggle with fundamental aspects of spatial reasoning, particularly in answering questions like "Where am I?" and "What will I see?". While some attempts have been done, existing approaches typically treat them as separate tasks, failing to capture their interconnected na- present Generative Spatial Transformer (GST), a novel
ture. In this paper, rege-regressive framework that jointly addresses spatial localization and view pre- diction. Our model simultaneously estimates the camera pose from a single image and predicts the view from a new camera pose, effectively bridging the gap be- tween spatial awareness and visual prediction. The proposed innovative camera tokenization method enables the model to learn the joint distribution of 2D pro- jections and their corresponding spatial perspectives in an regresseee manner. This unified training paradigm demonstrates that joint optimization of pose esti-
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
mation and novel view synthesis leads to improved performance in both tasks, for the first time, highlighting the inherent relationship between spatial awareness and visual prediction. Project page: https://sotamak/ Ir github.io/gst.
INTRODUCTION
Spatial Intelligence is critical piece of the Al puzzle, as it encompasses the ability to understand the spatial relationships between objects and scenes. "Where am I?" and "What will I see?" are two fun- damental questions used to test the spatial capability of intelligent agents. For humans, commencing from the observation of an image, they can infer spatial layouts and predict unseen aspects of the environment. This spatial ability enables humans to easily orient themselves in space and to envi- sion observations from different perspectives within a given space. Therefore, expanding existing general intelligence into the realm of 3D space, enabling it to effectively answer the aforementioned two foundational questions, is crucial step in the development of spatial intelligence.
Modern auto- uuto-reegessive models (Brown, 2020; Touvron et al., 2023) demonstrate exceptional intelli- gence due their advanced architecture, enabling effective long-range dependency modeling. This capacity empowers large language models (LLM) to exhibit outstanding intellectual performance across various domains (Kondratyuk et al., 2023; Team, 2024). To enable the model to answer "Where am I?" and "What will I see?" effectively, we endeavor to leverage the strong modeling capabilities of auto-regrest auto-regressive models to spatial intelligence. These two questions correspond to two classic tasks in the 3D domain: spatial localization and view prediction. Prior research has tradition- ally treated the tasks of generating novel views from a given location (Liu et al., 2023; Tewari et al., 2023; Chan al,, 2023) and estimating camera poses from varied perspectives (Wang et al., 2023; Zhang et al., 2024) as distinct tasks, typically employing separate models for each task. Neverthe- less, they are closely interconnected in human spatial cogmition, as individuals often subconsciously integrate them during spatial reasoning without subjectively distinguishing between them.
To bridge this gap, we propose Generative Spatial Transformer (GST), model inspired by the ob- servations of human spatial reasoning processes. Rays entering the eye form image signals; hence, 2D image the projection of 3D space from a given viewpoint position and direction. Building upon this notion, we introduce, for the first time, the concept of utilizing the camera as a new modality into the training an auto-regressive model. Specifically, we leverage Plücker coordinates to transform the camera into camera map akin to an image, and convert it into token sequence by applying a tokenization method similar to that used for image. To address the uncertainties associated with scene scale and unseen regions, we employ an auto-regressive approach to construct joint dis- tribution of novel views and camera locations given an initial observation. This joint distribution inherently encapsulates two posterior probability distributions, one for novel views and another for camera locations, and introduces two conditional probability distributions. This approach contrasts with directly modeling two completely different distributions using the same model.
Experiments demonstrate that modeling a single joint distribution leads to a more stable optimiza- tion process compared modeling two distinct distributions. This stability is achieved without compromising the final convergence accuracy, ultimately leading to better results in both novel view synthesis and relative camera pose estimation tasks through the introduction of redundant objectives. Furthermore, by integrating additional target distributions, our model can effectively complete novel tasks such as sampling valid camera poses from an observation, generating images under no camera conditions. This approach has been shown to significantly improve the model's understanding of the intricate nuances present within 3D spatial environments.
In summary, our contribution lies in introducing GST, the first model capable of concurrently per- forming both novel view synthesis and relative camera pose estimation within a unified framework. Drawing inspiration from human spatial reasoning, we design GST to model the joint distribution of images and camera poses, enabling it to effectively integrate the training objectives of both tasks. Extensive experiments demonstrate that GST achieves state-of-the th- performance in synthesizing a single novel view in a feed-forward manner while accurately estimating the relative camera pose between two frames, establishing new benchmark for spatial intelligence in vision-based systems.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 AUTO-REGRESSIVE MODELS
The core concept of auto-regressive models is establish the probability distribution of future se- quences given the current sequence. Building upon this principle, powerful large language models (Brown, 2020; Touvron al., 2023) have emerged, demonstrating their ability to tackle a variety of challenging language tasks. To extend these capability to multimodal tasks, researchers have explored tokenization methods (Esser al., 2021; Yu al., 2024; Zeghidour et al., 2021) that are specifically tailored to different data modalities. By tokenizing data from various modalities, re- searchers can construct diverse task sequences and leverage the next-token prediction mechanism of language models for training (Team, 2024; Kondratyuk et al., 2023). This approach has shown promising results in enhancing multimodal capabilities.
2.2 NOVEL VIEW SYNTHESIS
In recent years, significant strides (Rombach et al., 2022; Ramesh al., 2022; Saharia et al., 2022) have been made in the field of image generation. Numerous scholarly investigations posit that these modalities effectively encapsulate intricate 3D prior knowledge. One notable approach, 3ero-I-to-3 (Liu et al., 2023), introduces a methodology that begins with single image and incorporates the relative camera pose as a contextual factor define the conditional distribution of novel views, yield- ing promising results. However, generalizing this approach to real 3D environments remains area requiring further scholarly inquiry. Additionally, DFM (Tewari et al., 2023) conceptualizes the syn- thesis of new perspectives within the exorbitant computational demands inherent in the denoising process and the resource- shource-intensiee nature of volume rendering impede the seamless scalability of this framework. In contrast, CAT31) (Gao et al., 2024) enhances this process by concatenating the representation of each view's camera using Plücker ray notation with the image channels. It employs diffusion model to characterize the conditional distribution of multiple views based on specific camera configuration.
a solution to stochastic inverse problems. Nevertheless,
2.3 CAMERA POSE ESTIMATION
Estimating camera poses from sparse views poses a significant challenge, as sparse images often lack sufficient information to accurately determine the position and orientation of the camera. COLMAP (Schönberger & Frahm, 2016) ccomplishes camera pose estimation by detecting and matching fea- tures in images, estimating relative camera poses using robust algorithms, refining poses through bundle adjustment. This is a computationally intensive process and performs poorly with sparse viewpoints. RelPose (Zhang et al., 2022) addresses this issue by leveraging an energy-based model to amalgamate relative rotations into a set of camera poses. PoseDiffusion (Wang et al., 2023) uti- lizes diffusion models directly sample camera parameters. Building upon this foundation, Ray Diffusion (Zhang et al., 2024) represents a notable advancement. While it still employs diffusion model. this approach generates camera rays targets, demonstrating superior precision compared to directly predicting camera parameters.
3 METHOD
We consider the challenge of simultaneously sampling novel view images and their corresponding camera poses given single input image. Diverging from prior research, our focus lies in uncovering the inherent consistency between these two tasks rather than alternately training the two objectives during the training process. Our approach starts by tokenizing the image and camera spatial posi- tions, merging two codebooks to ensure the model treats both modalities equally (Sec 3.1.. We then proceed to train a generative network to model the joint distribution of these components (Sec 3.2).
3.1 TOKENIZATION
Image Tokenization. In approach, employ VQGAN (Esser et al., 2021) image tok- enizer, comprising the following components: an encoder that maps an image x e R # 23 to a feature map f R and a quantizer containing codebook of l d-dimensional vectors. The
3

---

Published as a conference paper at ICLR 2025
Observe
Camera Mup
Sample
Image Tokenizer
Camera Tokenizer
Image Tokenizer
GST
Novel View Synthesis
Imagee De DeTTokenizerr
Canara Adaa
Camera Pose Estimation
De- To-Toommmeerr
Camera
Figure 2: Illustration of the GST Upon providing an observed image, task category, and the target camera position the other view image, GST autonomously generates the desired outcome in an auto-regressive manner. The training process GST comprises two significant phases: (1) training of the image and camera tokenizer, (2) training of the auto-regressive model.
quantization process involves selecting the nearest vector from the codebook, based on Euclidean distance, for each d-dimensional vector at every position in f- The index the codebook serves as the image encoding, and the quantized feature map is then reprojected back to the image space through a decoder.
Concerning constraints the quality of image reconstruction, both Le reconstruction loss and LPIPS perceptual loss Lp are utilized, along with incorporating an adversarial loss La. Conse- quently, a discriminator alternateyy trained during this process, with Aa representing the weight of the adversarial loss:
C. - ()) Lx(, +dC((())
(1)
Due to the non-differentiabiliiy of quantization, the straight-through estimator is employed to prop- agate gradients from the decoder to the encoder:
z = sg|= f
where sg[.] stands for the stopgradient operator (Van Den Oord et al., 2017). The training loss of the codebook is defined the proximity of the embedding of the force codebook the features output by the encoder. The utilization of the stopgradient operator prevents the gradient from propagating back the encoder. To address this, a loss term is introduced enforce the feature vectors extracted from the encoder to approach those in the codebook, with the weighting adjusted by the parameter 3:
bug ||g|//| - +112+111 sg >g(=]]]
The final loss function comprises a combination of image reconstruction loss and codebook param- eter constraint loss.
Liu et al. (2023) directly utilizes the azimuth and elevation changes in camera poses as model inputs, yet such simplistic parameterization proves challenging extend to real-world scenarios. Following Zhang et al. (2024), we opt to utilize Plücker rays for the dense parameterization of the camera pose.
Camera Parameterization.
Specifically, for a camera extrinsic matrix, the spatial position and orientation of the camera can be derived. Each pixel the image plane corresponds to ray emanating from the camera origin,

---

Published as a conference paper at ICLR 2025
o)
p(i.
plc M)
Distribution
Condition
(a) Relative camera estimation
(b) Novel view Synthesis
(c) Ours
Figure 3: Comparison of target distributions for different tasks. Previous methods have tradi- tionally constructed unimodal target distributions for novel view synthesis and camera estimation tasks. However, GST has introduced joint distribution for both the image and the corresponding camera poses.
denoted as r (o, d). Plücker rays represent this ray 6-dimensional set of Plücker coordinates (ox d), encapsulating both the position and direction of the ray in space. Upon converting all rays emanating from the camera origin and intersecting with image pixels into Plücker coordinates, we can obtain tensor of the same size the image, referred to the camera map. The camera map can be utilized infer the camera matrix through a reverse derivation process (Zhang et al., 2024).
Camera Tokenization. enable auto-regressive model handle both camera and image modal- ities concurrently, have adopted consistent tokenization method for processing camera rays, akin to image tokenization. This approach involves leveraging a modified version of the VQVAE (Van Den Oord et al., 2017) with reduced convolutional network depths in both the encoder and decoder, following a training procedure analogous to that of the image tokenizer, albeit without the discriminator component. By adjusting the initial size of the camera map, derivation of camera tokens that align in size with the image tokens, facilitating seamless integration within the model architecture.
the eventual
3.2 JOINT DISTRIBUTION MODELING
Backbone. Previously, formulated the entire training approach as auto-regressive problem, which could be addressed using modern large language model (LLM) techniques. Here, we em- ployed the training paradigm of next-token prediction effectively integrate the capabilities of advanced large language models for future applications. We adopt Sun et al. (2024) codebase, and implemented techniques from current LLM, such as QK-Norm (Henry et al., 2020), RMSNorm (Zhang & Sennrich, 2019), and leveraged the 2D form of rotary positional embeddings (RoPE) (Su al., 2024) to operate on the image and camera map embeddings.
Training Target. In the pursuit of unifying two divergent training tasks p(ilc, o) and p(cli, o), a straightforward strategy entails the direct alternation between the respective training objectives throughout the training regimen. Nonetheless, the discemible dissimilarity between the disparate probability distributions can precipitate pronounced instability in training dynamics. redress this quandary, our focus has pivoted towards the joint distribution governing novel view images and camera poses, as illustrated in the figure 3. This distribution inherently encapsulates the training requirements for these dual tasks:
p(i,clo) p(ile, olpcco) p(cli, o)p(i
Upon completing training, during the inference stage, it proves adequate stochastically sample from the prescribed prior probabilities p(ioo) or p(clo) under conditional settings (cither through manual intervention automated model-driven processes) generate results corresponding the
5

---

Published as a conference paper at ICLR 2025
p(lc.
attmaaely jimnly
LARGE
p(c
Gral Nord
0.40
alternately
0.35
plic
0.10
aherutely
piistly
0.25
jointly
Triining
Triiiing
(a)
(b)
Figure 4: The impact of joint distribution training. (a) By transforming two distributions into joint distribution, the larget of the model can be unified. (b) (c) We trained using the same initial model parameters and hyperparameters, averaging the loss and gradient norms every 50 steps, and observed that joint distribution training yields more stable training trajectory compared to alterna- tively training two objectives.
alternate modality, thereby realizing the objective of unifying the dual tasks within a singular model. Crucially, the circumvention of the necessity to oscillate between the training of vastly differ- ent training objectives ensures more stable trajectory in training losses and gradient norms, as illustrated the figure
In our training stage, we tokenize the initial observed image along with the image i and camera corresponding to random sampled viewpoints, resulting three token sequences (L., 4i and 1,) of equal length. These sequences are concatenated to form fixed-length token sequence denoted as s. Further details will be discussed in the supplementary materials. Subsequently, a decoder-only transformer denoted G' and parameterized by is trained. The generation target is formulated
|s|
((0) =
Σ
logp(s)ls)...... (5-11);;)))))))))))
(5)
11/+1
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. We trained GST auto-regressive model and camera tokenizer on four datasets with multi- view images and camera pose annotations: Objaverse (Deitke et al., 2023), CO3D (Reizenstein et al., 2021), RealEstate 10k (Zhou et al., 2018), and MVImgNet (Yu et al., 2023). These datasets encom- pass 3D object models, real-world environments, and object-centric scenes. Objaverse exclusively offers object 3D models, for which we utilized the rendering outputs from Liu et al. (2023) and employed the filtered object ID list provided by Tang et al. (2024). We performed center cropping on all training images to ensure obtaining token sequences of the same length.
Baseline. We adopted the feed-forward method Zero-I-to-3 (Liu al., 2023), and its associated models: Zero-1-to-3 XL, trained on the larger-scale dataset Objaverse-XL (Deitke et al., 2024); Zero-1-to-3 NVS, fine-tuned by ZeroNVS (Sargent et al., 2023) real-image datasets. Our test set comprises two parts: A randomly selected non-overlapping subset from Objaverse, distinct from the training set; And a subset extracted from CO3D, following the settings of ZeroNVS. We se- lected various methods (Zhang et al., 2024; 2022; Lin et al., 2023; Wang et al., 2023) for evaluating the camera pose estimation performance of GST, following the experimental setup raydiffusion (Zhang et al., 2024), conducting quantitative experiments on the relative camera posc estimation of 2 images on CO3D (Reizenstein et al., 2021).
We utilized the image tokenizer from LlamaGen (Sun et al., 2024) along with its autorregressive model with 1.4 billion parameters to initialize our model weights, and made some slight modifications to the architecture like adding QK-Norm (Henry et al., 2020) to the at- tention operations. Throughout the training process, the weights of the image tokenizer were kept constant, while all parameters of the auto-regressive model were trained. We employed structure
Implementation details.

---

Published as a conference paper at ICLR 2025
similar the image tokenizer to construct camera tokenizer, albeit with fewer parameters. The parameter quantities for each model detailed in table la. The base learning rate for training the camera tokenizer is set at 10 with batch size of 128. The auto-regressive model also starts with a base learning rate of 10-1, which later decreases to 10-5 in the later stages of training. The hatch size for the auto-regressive model is 192, with gradient accumulation performed every steps. Both models utilize the AdamW optimizer with a gradient clipping threshold set at 1.0.
Table I: We present the selection of the model parameter size and the camera tokenizer codebook size that we used for our all experiments.
(a) Model Parameters.
(b) Camera Tokenizer Codcbook.
Models
Parameters
Size
Dim
usaget
Image tokenizer
77 M
1024
65.1 1%
Camera tokenizer
22 M
2048
34.6%
Auto-regressive model
1.4 B
2048
4
90.4%
2048
13.1%
4096
21.3%
4096
77.0%
4.2 NOVEL VIEW SYNTHESIS
We conducted comparison between GST and the comparative methods (Liu al., 2023; Sargent et al,, 2023) on Objaverse and CO3D. As shown in the figure 5 and table 2, even though GST was trained on subset of Objaverse, achieved superior results compared Zero-I-to-3, which was trained on a several orders of magnitude larger dataset than ours.
This advantage is also evident in the quantitative metrics displayed in the table
4.3 RELATIVE CAMERA POSE ESTIMATION
We tested GST on the CO3D (Reizenstein al., 2021) benchmark evaluate its performance in estimating relative camera poses. Here, we trained the model using a setting with only two frames, this setting is considered the most challenging yet fundamental for this task. As shown in the table 3, we achieved the best generalization under this setting (Unseen Categories). Given the joint training of GST across multiple datasets, a subsampled CO3D training was utilized to optimize training efficiency. This may explain the slightly lag behind in the accuracy of training categories compared to the approach presented in Zhang et al. (2024).
4.4 ABLATION STUDY
Camera Tokenizer Designs. The size of the camera tokenizer's codebook and the length of the quantized vectors are crucial factors that influence the performance subsequent models. We trained our camera tokenizer the dataset used for training the auto-regressive model, instead of randomly sampling camera positions in space. While the latter approach could better represent all camera positions space, it would reduce the usage of the camera tokenizer's codebook during the training of the auto-regressive model. This is because some randomly sampled, unconventional camera positions would consume a portion of the tokenizer's training resources, even though these positions would not appear during auto-regressive model training. Moreover, we aim for the model
Table 2: The quantitative results of novel view synthesis. GST outperforms Zero- 1-o-33, Zero- 1-
to-3 XL in terms of LPIPS and SSIM metrics Objaverse dataset. However, Zero-1-to-3 achieves higher PSNR scores compared to GST due to its training on the complete Objaverse dataset, which includes our test set. Furthermore, the GST significantly outperforms Zero- 1-to-3 NVS across all three quantitative metrics, attributable to our carefully crafted camera condition.
LPIPS
PSNR
SSIM t
LPIPS J
PSNR
SSIM
Objaverse Dataset
CO3D Dataset
Zero-I-to-3
0.135
14.77
Zero-1-to-3 NVS
0.515
13.4
0.407
1-to---to-3- XL
0.141
14.53
0.834
GST (ours)
0.419
15.6
0.456
GST (ours)
0.085
13,95
0.871

---

Published as a conference paper at ICLR 2025
Observation
Target GT
Zcro-I-to-3 (a) Objaverse
Zero-I-to-3 XL
GST (ours)
Observation
Target GT
Zero-I-to-3 NVS
GST (ours)
(b) CO3D
Visualization comparison of novel view synthesis. Among these methods, GST demon- strates the highest quality of generated results. Additionally, due to the design of the camera condi- tion, GST is capable of producing the most accurate images for the specific viewpoints.
Figure 5:
Table Camera Rotation Accuracy on CO3D (@ 15"). Here report the proportion of relative camera rotations that are within 15 degrees of the ground truth. Results for all comparative methods referenced from Ray Diffusion.
Methods
Seen Categories
Unseen Categories
RelPose
56.0
48.6
PoseDiffusion
75.7
63.2
RelPose++
81.8
69.8
Ray Diffusion
91.8
83.5
GST (Ours)
86.6
85.1
to learn a camera distribution that aligns with the dataset's distribution. This alignment is expected to help the model gain spatial understanding from observations, mentioned below.
In the table 1b, we present the usage of the codebook under different parameters. As the distribution of cameras can be better learned compared images, observed very low reconstruction losses for all parameters. Consequently, usage became a critical selection criterion. is notable that as the size and dimension of the codebook increase, we observe a trend of increasing usage followed by decrease, aligning with observations from previous work Sun et al. (2024). Ultimately, we selected parameters with size of 2048 and dimension of 4 for our final model.
Different Camera Parameterizations. In Section 3.1, we mentioned that we represent the camera using a method based on Plücker coordinates. Prior to this, we had explored another direct tokeniza- tion approach: representing the relative camera's 's rotation matrix as a three- dimeeesdiiniiinall vector in terms of Fuler angles, and adding an offset ensure all elements of this vector are positive. After quantization into positive integers, we can then use a codebook of size 360 to represent each di- mension. Ultimately, only 3 tokens are needed to represent this rotation angle. We only conducted this ablation experiment on a subset of the object-level dataset Objaverse, as we did not incorporate transformations of camera positions for the sake of simplification. As shown in the results depicted in the figure 6, it is evident that using only 3 tokens novel view synthesis makes challenging to generate ideal viewpoints. In other words, there is a need introduce more detailed conditions improve the quality of the generated outputs.
condition in the auto-regressive model for
8

---

wall, where the sampled camera locations Published as a conference paper at ICLR 2025
Camera map w/o tokenizer
Observation
Target GT
Euler Angle
Camera map
Figure 6: Through our experiments with different camera conditions, observed that employing token-wise conditioning and tokenizing the cameras alongside the images yielded the best genera- tion results for the auto-regressive model.
Cal
Looking
Looking
level
the
Figure 7: We present the results of automatically sampled camera distributions using GST, obtained from observing the same object placed in different locations and under various perspectives. The figure showcases three distinct scenarios: (a) top-down view of the object, where the sampled cameras are predominantly top-down; (b) frontal view of the object, where the sampled cameras are primarily horizontal; and (c) the object positioned effectively avoid the obstruction.
We also explored scenario where only the camera is used as a condition without requiring the model output the camera. Specifically, instead of training a camera tokenizer, the Plücker coordinates obtained from the camera are encoded using sine and cosine functions and fed into a small MIP (Mildenhall et al., 2020). This MLP, initially trained concurrently with the auto-regressive model, was subsequently frozen after predetermined number of training steps. As shown in the figure 6, we obtained visual results from this model, indicating that tokenization continues to exhibit the best performance in the generative process of the rugo-regressvve model.
Joint Distribution Training vs Alternately Training. As mentioned in the section 3.2, training joint distribution tends to have more stable training trajectory compared to alternating between training two separate targets. This stability is crucial in the training of large transformers, as instabil- ity in loss can lead loss explosion during auto-regressive training, aligning with experimental observations. Furthermore, we have quantitatively compared the two approaches in the table 4, re- vealing that training on the joint distribution outperforms alternately training for the selected two
Table 4: compared the performance of two training methods distinct tasks. Our findings indicate that training the joint distribution leads to superior results compared to alternating training on the distributions.
Pose Estimation
Visual Prediction
15a
@ 30 t
LPIPS
PSNR t
SSIM
GST (alternately)
84.6
92.8
0.554
10.97
0.328
GST (jointly)
85.7
93.5
0.500
11.90
0.358

---

Published as a conference paper at ICLR 2025
Observation
Target GT
Observation
Figure 8: Uncondition image generation. Training unconditional image generation each scenario can assist the model in construct- ing the distribution of images within that con- text.
Jointly
Alternately
Figure 9: By employing training the joint distribution, GST exhibits enhancements in im- age quality and spatial positional accuracy.
tasks. This observation underscores the notion that establishing a well-defined objective distribution can positively impact both the training process and the ultimate outcomes.
Introduction of Excess Distribution. As mentioned in Section 3.2, despite introducing an addi- tional distribution, our model not only avoids performance degradation but also achieves enhanced performance while ensuring training stability. straightforward interpretation is that our conditional prior distribution further assists the model in truly understanding the space.
As illustrated in the figure 7, we capture real-world object from various angles and positions, al- lowing GST to sample valid camera distributions from p(clo) for each scenario. For images captured from a top-down perspective (a), GST predominantly sampled cameras with a top-down viewpoint. Similarly, for objects viewed from frontal angle (b), GST preferentially sampled cameras with a frontal perspective. Notably, scenarios involving obstacles (c), GST effectively avoided these obstructions and sampled reasonable camera positions. These results, achieved without any manual intervention, further demonstrate GST's ability to accurately comprehend the spatial layout from observed images.
As depicted in the figure 8, we employ an alternative distribution for sampling p(i by GST, which compels the model learn the unconditional image distribution within the same scenario. This approach, as discussed in Ho & Salimans (2022), intertwines unconditional loss within the conditional generation training process strike balance between sample quality and diversity. This conclusion has also been validated in the figure 9.
CONCLUSION
In this paper, we propose the Generative Spatial Transformer (GST). To the best of our knowledge, this is the first work that connects novel view synthesis with camera pose estimation. Our method treats the camera as bridge between 2D projections and 3D space by introducing camera tokenizer and including the camera as a new modality in training the auto-regressive model. Furthermore, we propose joint distribution the training target, enabling diverse task completion and boosting the model's 's spatial understanding. These advancements not only broaden the model's 's capabilities but also set the stage for future strides in spatial intelligence.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was done during Junyi Chen's internship at Shanghai Artificial Intelligence Laboratory. This work is supported by the National Key R&D Program of China (2022DD016010)), and Shang- hai Artificial Intelligence Laboratory.
REFERENCES
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models, 2023. URL httpppp://aaxi.../ wwwpaawwwww...aaaaaa/o//aa//aaae
2304.02602.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEF/CVF conference computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405. 10314, 2024.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.04245, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL UtLptss:://aaxi
org/abs// 2aab//20772 2207.227722 12598.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, al. Videopoet: large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125. 2023.
Amy Lin, Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse- view observations, 2023. URL
hbtp 04926.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot one image 3d object. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9298-9309, 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, 2020. URL 08934..
https: //apsii///xvv..////////s org/
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Jeremy Reizenstein, Roman Shapovalov. Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10901-10911, 2021.
11

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-006929 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 33336479-36994 2022.
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310. 17994, 2023.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- from-motion motion revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0rXiv:2406.065555, 2024.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05004. 2024.
Chameleon Team. Chamelcon: Mixed-modal carly-fusion foundation models. arXiv preprint arXiv:2405.09878, 2024.
Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Josh Tenenbaum, Frédo Du- rand, Bill Freeman, and Vincent Sitzmann. Diffusion with forward models: Solving stochastic inverse problems without direct supervision. Advances in Neural Information Processing Systems, 36:12349-1233 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rX 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Jinnyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estimation via diffusion-aided bundle adjustment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9773- 177339973, 2023.
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation, 2024. URL ht : /tt///axvvv//////////////////aaa wwwivioorrraaae /2310 05737.
Xianggang Yu, Mutian Xu. Yidan Zhang. Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvingnet: A large-scale dataset of multi-view images. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 9150-9161, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec, 2021. URL /ttpttps:/aaxxi.ioo/o wwwpaawwwwww..////////o/////// 03312.
h
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose:
Predicting probabilistic relative rotation for single objects in the wild. In European Conference on Computer Vision, pp. 592-611. Springer, 2022.
Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tul- siani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.14811 2024.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images, arXiv preprint arXiv:/805.098177 2018.
13


Reviews:
[{"review_id": "x1DzmR0Mh0", "reviewer": "Reviewer_VC2z", "summary": "This paper aims to do view prediction and pose estimation with a single autoregressive model. The paper claims that \u0013humans do this task effortlessly, \u0013and that their own model does it effortlessly too. \u0013The method is inspired by LLMs, by which the authors mean: it is autoregressive. The paper claims to be the first to tokenize camera data. The model itself is enormous: >1.4B parameters. It achieves good results.", "questions": "Please interpret some of my stated \"weaknesses\" as questions.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "This paper's figures are very well designed. Good color coordination across the paper.", "weaknesses": "The paper says that, given an image of a scene, humans can \"effortlessly reconstruct the entire scene\". I don't think this is true. Can the authors give some support for this claim?\n\nThe paper says that it wants the model to \"effortlessly\" estimate camera poses and estimate novel views. What does this mean? How can we distinguish effortless vs effortful models?\n\nIn talking about pose estimation vs. view prediction, the paper says \"human cognition does not perceive these processes as isolated entities\". I think it's quite clear that this is false. If we did not perceive these as separate tasks, how could we even talk about them so distinctly?\n\nThe paper says that their approach is \"a model designed to align its understanding of 3D space with that of humans\". Given that I question the paper's description of human cognition, I also question whether it's a good idea to design a model based on this shaky foundation.\n\nThe paper makes a major claim about novelty by saying \"we introduce, for the first time, the concept of tokenizing the camera\". It is not the first time. The authors can read, for example, \"Input-level inductive biases for 3d reconstruction\" (CVPR 2022). There are probably dozens of papers that convert camera information into tokens; checking the papers that cite the input-level biases paper should reveal many references. The paper goes on to say, \"Specifically, we leverage Plucker coordinates to transform the camera into a camera map akin to an image\". This has also been done before: check \"Cameras as Rays: Pose Estimation via Ray Diffusion\" (ICLR 2024). I can see that this is in fact cited in a later part of the paper, so perhaps there was some mixup in the writing, and the author of this section is not aware of the related work and what the actual contributions are. Nonetheless, this is a serious issue. \n\nOverall, it is unfortunate for a work like this, which appears to be otherwise quite well polished, to make wild and unsupported statements about human cognition, and to claim novelty on techniques that have been published before (and even gained re-use across follow-up work). It is possible that this paper actually has novel parts and useful contributions, but the amount of false (or unsupported) and misleading material is overwhelming. \n\nthe Al puzzle -> the AI puzzle", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2ebMXU1sD", "reviewer": "Reviewer_AKff", "summary": "This paper introduces the Generative Spatial Transformer (GST), an innovative autoregressive framework capable of simultaneously handling spatial localization and view prediction tasks. By introducing a novel camera tokenization method, GST learns the joint distribution of 2D projections and spatial perspectives during training, thereby improving the performance of camera pose estimation and novel view synthesis. Experiments demonstrate that GST achieves state-of-the-art performance in these tasks, highlighting the intrinsic connection between spatial awareness and visual prediction.", "questions": "My main concerns are as follows:\nLine 268. I am curious about the effect of the concatenation order of the three tokens on the experimental results in this problem.\nLine 342, GST only uses part of the data set for training, but I want to know what the results will be if the same experimental settings are maintained as Zero-1-to-3.\nIn line 375, the author does not elaborate on the definition of Unseen Categories. Of course, I can understand that due to the experimental settings, there may be some disadvantages compared to these methods.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First, the paper's approach and the scientific questions it raises are novel and intriguing.\nSecond, the method proposed in the paper can simultaneously estimate camera pose from a single image and predict the view from a new camera pose, effectively bridging the gap between spatial awareness and visual prediction. Interesting validations are shown in Figure 7.\nFinally, the writing of the paper is clear and accessible, with excellent explanations from the motivation to the introduction of the method, and the provided conceptual diagrams are easy to understand.", "weaknesses": "I did not find any obvious shortcomings. I only have one suggestion: when citing the methods of other articles, it would be better for the author to briefly introduce them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9maw9V22n", "reviewer": "Reviewer_3EtM", "summary": "This paper proposes to jointly model the distribution of camera poses and novel views via a shared Generative Spatial Transformer (GST).", "questions": "1. How does the proposed method deal with scale differences of camera poses from different datasets? Is any normalization used for ground truth camera poses across datasets?\n2. Can authors provide more details about training resources? For example, how many GPUs are used, and how long has the model been trained?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of jointly modeling the distribution of camera poses and novel views via a Generative Spatial Transformer is cool and novel.\n2. The writing is clear, and the figures are nice.\n3. This paper presents comprehensive ablation studies to justify model designs.", "weaknesses": "1. Insufficient evaluation:\n    1. Novel view synthesis: The model has been trained on real-world images (CO3D, RealEstate10k, MVImgNet), but the evaluation is only on Objaverse (where the object shows simple and unrealistic textures). Can authors (quantitatively and qualitatively) compare to other baselines in terms of novel view synthesis on CO3D as well? One possible baseline is Zero123, fine-tuned on real-world images by ZeroNVS [1]. \n    2. Multi-image condition: Can the proposed approach synthesize novel views/estimate camera poses conditioned on multiple images? It'd be interesting to see its flexibility by showing these results (e.g., comparing with baselines, say Ray Diffusion with 3-8 images).\n    3. How does the proposed approach compare with the state-of-the-art method, DUSt3R [2]?\n2. Unfair comparison: In Tab.3, the proposed method achieves higher accuracy on unseen categories than the baseline. However, this could be explained by the more training datasets used by the proposed method, while the baselines, e.g., Ray Diffusion, are trained only on CO3D. Can the authors train the model on the same data used by Ray Diffusion and compare with it?\n3. Minor issue: I think the two arrows in Tab.4 are in the wrong direction (Rot@15, Rot@30). \n\n\n[1] Sargent et al. ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image, 2024. \n\n[2] Wang et al. Dust3r: Geometric 3d vision made easy, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gzoznm09p7", "reviewer": "Reviewer_xnP6", "summary": "The authors introduced a auto-regressive framework to simutaneously estimate camera pose of a given image, and synthesize novel views at new camera poses. The authors achieve this via modeling the joint distribution of camera frustums and their image projections in a unified training paradim.", "questions": "1. As the results are presented as training only on a subset of Objaverse, do the authors have any expectations on how the method will scale?\n2. For other camera rotation accuracies (5 degree or 10 degree), will GST's performance still be competitive to state of the art methods?\n3. What's the rough GPU memory and latency performance of the proposed pipeline relative to existing work (would this be a potential downsides of autoregressive modeling)? \n4. Would it be possible to leverage the model's understanding of the joint distribution of camera angles and views for 3D reconstruction/NVS from a set of potentially uncaliberated images?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. How the authors leverage VQVAE to tokenize camera views consistent with image tokenization is considered novel.\n2. The effective adoptation of language models for autoregressive image/camera pose predictictions is interesting.\n3. The choice to model the joint distribution of camera views and frustums within the same model seems clean and effective in combining the task of novel view synthesis and camera estimation.\n4. Descent visual illustrations and writing.", "weaknesses": "1. The NVS results seems to be mostly on par or slightly better than existing state of the art models only based on the training of \"a subset of objaverse,\" and I am curious of the particular choice of the data subset used in training. What's stopping the authors from training on the same dataset as Zero-1-to-3 to make a more fair comparison? \n2. The camera rotation accuracy metric of 15 degree is rather a relatively coarse metric, and the authors did not present the camera translation accuracies in any kind.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)