Scaling Sequence-to-Sequence Generative Neural Rendering
Authors: Shikun Liu
Kam Woh Ng
Wonbong Jang
Jiadong Guo
Junlin Han
Haozhe Liu
Yiannis Douratsos
Juan Camilo Perez
Zijian Zhou
Khanh Chi Phung
Tao Xiang
Juan-Manuel Perez-Rua
Venue: ICLR 2026 Poster (2026)
Keywords: 3D vision
Novel View Synthesis
Generative Neural Rendering
Primary area: generative models
TL;DR: We introduce Kaleido, a family of sequence-to-sequence rectified flow generative models designed for photorealistic, unified object- and scene-level neural rendering.

Abstract: We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido is driven by the principle of treating 3D as a specialised sub-domain of video, which we formulate purely as a sequence-to-sequence image synthesis task. Through a systemic study of scaling sequence-to-sequence generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views conditioned on any number of reference views via a masked autoregressive framework; and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large-scale video data for pre-training, which significantly improves spatial consistency and reduces reliance on scarce, camera-labelled 3D datasets --- all without any architectural modifications. Kaleido sets a new state-of-the-art on a range of view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the quality of per-scene optimisation methods in many-view settings. For supplementary materials, including Kaleido's generated renderings and videos, please refer to our website: https://shikun.io/projects/kaleido.

Published as a conference paper at ICLR 2026
SCALING SEQUENGE-TO-SEQURNCE GENERATIVE NEURAL RENDERING
Shikun Liu", Kam Woh Ng". Wonbong Jang, Jiadong Guo, Junlin Han, Haozhe Liu, Yiannis Douratsos, Juan C. Pérez, Zijian Zhou, Chi Phung, Tao Xiang & Juan--anuel Pérez-Rúa Meta AI "Equal Contribution {shikun, kamwohng @eeea. com
ABSTRACT
We present Kaleido, family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido is driven by the principle of treating 3D specialised sub-domain of video, which formulate purely a sequence-tor lsquence-to-sequenee tequuene image synthesis task. Through systemic study of scaling sequence-to-sequenee generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views condi- tioned on any number of reference views via masked autoregressive framework: and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large- scale video data for pre-training, which significantly improves spatial consistency and reduces reliance scarce, camera-labelled 3D datasets all without any architectural modifications. Kaleido sets new state-of-the-an on a range view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the qual- ity of per-scene optimisation methods in many-view settings. For supplementary materials, including Kaleido's generated renderings and videos, please refer to our eidd.
website: https: //s::///// //shi kun. o co/projectee
Figure 1: Kaleido is generative rendering engine that creates photorealistic views in any style from any number of reference images (white boxes) with full 6-DoF camera control.
INTRODUCTION
Rendering and view synthesis are foundational to 3D computer vision and graphics, driving applica- tions across virtual reality, cinematic effects, roboties, and autonomous driving. By allowing scene

---

Published as a conference paper at ICLR 2026
to be rendered from arbitrary viewpoints based on limited set of reference views, view synthesis mimics the adaptability of human vision - the ability to construct and reconstruct a coherent 3D understanding of our surroundings.
While deep learning, fuelled by massive datasets and scalable model designs, has achieved great success in language and 2D vision, its progress in 3D vision for general-purpon rendering has been comparatively slow. We argue this stems from two persistent and interconnected bottlenecks:
1. A Fragmented Landscape of 3D Representations. 3D vision lacks consensus the right 3D representation, with methods spanning explicit structures like voxels (Wu et al., 2015), point- based representations such point clouds (Qi al., 2017; Guo al., 2020) and 3D Gaussian Splatting (Kerbl et al., 2023), to implicit functions like neural fields (Mildenhall et al., 2020; Xie al., 2022). This fragmentation has prevented the focused, collective effort required scale a powerful architecture for any single representation, as development remains divided across incompatible data formats.
2. The High Cost of 3D Data. 3D datasets are scarce and difficult to obtain primarily because their creation is guided by the principle of strict 3D consistency. Achieving this level of precision requires either hand-crafting 3D synthetic meshes (Deitke et al., 2023b;a) or employing bun- dle adjustment and global alignments (Hartley & Zisserman, 2003) for slow multi-view camera labelling, making the data acquisition process slow, costly, and fundamentally difficult to scale.
As a direct consequence of these challenges, the 3D vision community has yet converge on a scalable paradigm for 3D modelling. The combination of fragmented research efforts and restric- tive data requirements has prevented the kind of focused, large-scale investment that enabled the dramatic architectural scaling and performance gains seen in language and 2D vision.
We believe these limitations, taken together, point to fundamental oversight:
3D perception is not a geometric problem, but a form of visual common sense.
The human ability to perceive 3D structure emerges from extensive observation of the world, from maintaining precise 3D model in the mind. For example, humans can interpret 3D geometry in optical illusions (e.g. in M.C. Escher's impossible structures and the Ponzo illusion), without hav- ing a physically accurate 3D representation or even a correct sense of depth. Accordingly, we argue that an ideal rendering system should not aim to explicitly model perfect geometric consistency, but instead to learn an implicit representation by capturing the statistical patterns of the extensive visual experience of the world.
Building on this insight, we introduce Kaleido, a scalable architecture for generative neural ren- dering. We design Kaleido as a type of spatial generative model that does not encode any explicit 3D structures. Instead, Kaleido inherits spatial perception and visual common sense directly from large-scale video data, purely in a data-driven way, similar how modern large language models acquire textual common sense from large-scale corpora without relying on explicit linguistic rules. This leads to our central hypothesis, inspired by the success of domain-specific models fine-tuned from pre-trained language models (e.g., coding models in Roziere et al. (2023); Anil et al. (2023); Chen al. (2021)), we believe that a powerful ceneral-purpose rendering model can be created by treating 3D as a specialised sub-domain of video. To put it simply.
We observed
large-scale corpus data structured code data general-purpose purpose coding model large-scale video data + structured 3D data a a general general-purpose purpose rendering model
We hypothesise
To realise this hypothesis, we reformulate rendering as a sequenice-to-sqqueec problem, specifi- cally as a pose- conditiodioon,d image-to-image synthesis task. We first establish unified, geometri- cally consistent representation of space and time as the core of our model design. This is achieved with a positional encoding design that extends the original Rotary Positional Encoding (RoPE) (Su et al., 2021) to parametrise all 2D, 3D, and temporal positions relatively, within the dot-product self-attention of transformer (Vaswani et al., 2017). This foundational design enables Kalcido to learn rich world representations from large-scale, unstructured video data and then perform efficient transfer learning with much smaller-scale, structured multi-view 3D data, all within the same model without any task-specific architectural changes.
Building this unified representation, Kaleido naturally benefits from scalable architectures and powerful generative techniques developed for language and vision. Specifically, Kaleido adopts a scalable transformer architecture inspired by Diffusion Transformer (DiT) (Peebles & Xie, 2023)
2

---

Published as a conference paper at ICLR 2026
The quick brown fox jumps
lazy
Space:
Time:
Sequence:
Rendering Model
Video Model
Language Model
dog
Figure 2: Rendering as Sequence-fo-Sequence Image Modelling, We propose that neural ren- dering can framed sequence-to-sequeee task, unifying design with language and video generation. In this formulation, a transformer (Vaswani et al., 2017) learns to generate image tokens conditioned on their spatial positions, similar to how language models condition on token positions in sequence, and video models condition temporal positions across frames.
and Llama-3 (Dubey et al., 2024), performing generative modelling via rectified flow objective (Liu et al., 2023c; Esser et al., 2024) within masked autoregressive framework (Li et al., 2024b; Fan et al., 2025; Liu al., 2025a).
Finally, we identify that rectified flow SNR samplers commonly used for text-to- -ittt---imaeevviee gener- ation are suboptimal for the precise pose conditioning required in rendering. We therefore introduce an improved, noise-biased sampling strategy and other key architectural adjustments to ensure stable and efficient scaling. Through extensive systemic studies, we validate these designs and highlight primary contributions:
We introduce the Kaleido family of Spatial Generative Models (SGMs), which can perform uni- fied object- and scene-level view synthesis from any number of reference views to any number of target views with full 6-DoF camera control. This is enabled by the following designs:
(a) A simple decoder-only rectified flow transformer that considers generative rendering sequence-ro-seqquencc task.
(b) A unified positional encoding design that seamlessly processes both 3D and video data within single, unchanged architecture.
(c) An effective scaling recipe for both model size and resolution, supported with tailored SNR sampler and solutions for training instability.
2. Kaleido generates high-resolution images (up to 1024px) across diverse aspect ratios, achieving state-of-the-art results on numerous view synthesis and 3D reconstruction benchmarks. Most notably, in many-view settings, Kaleido is the first zero-shot generative model to match the ren- dering quality per-scene optimisation methods such as Instant-NGP (Müller et al., 2022).
RELATED WORK
Generative 3D Modelling and View Synthesis Generative 3D modelling has rapidly evolved from synthesising isolated objects to composing entire, complex scenes. Pioneering text-to-31) works like Shap-E (Jun & Nichol, 2023) and Score Distillation Sampling (SDS) based methods like DreamFusion (Poole et al., 2022) laid the groundwork for single-object synthesis, inspiring wave of research focused on high-fidelity object generation (Liang et al., 2024; Tang et al., 2024; Wang et al., 2023b; Shi et al., 2024). More recently. the frontier has expanded to scene generation, with approaches ranging from procedural construction (Sun al.. 2023; Raistrick et al., 2023) di- rect compositional scene optimisation (Li et al., 2024a; Lu et al., 2024; Bai et al., 2023), A common thread in many of these works is the reliance on SDS to refine an explicit 3D representation.
Generative view synthesis models (Liu et al., 2023b;a;d; Shi et al., 2023) have emerged alongside this trend, but often face their own limitations. These methods typically struggle with multi-view consistency, are designed for fixed number of reference (often one) and target views, and frequently rely on the same complex, two-stage SDS pipelines to enforce geometric coherence. Conversely, Kaleido's 's sequence-to se-uueeeeee design naturally handles arbitrary number of both reference and target views, allowing it to generate spatially consistent views directly without requiring any post- processing optimisation stages like SDS.
3

---

Published as a conference paper at ICLR 2026
Sequence-to-SSqquene Generative View Synthesis Our work formulates generative view syn- thesis as a sequuence-t--eqqenne modelling problem, built upon a pure transformer architecture. A eritical challenge when applying transformers this domain is effectively encoding camera posi- tions. Recent advancements have introduced RoPE-style encodings (Su et al., 2021) to parameterise 6-DoF camera extrinsics, with notable examples including CaPE (Kong et al., 2024), GTA (Miyato et al., 2024), and also camera intrinsics in more recent work (Li et al., 2025). Kaleido builds directly on this direction, leveraging a GTA-based framework create a unified representation for both multi-view 3D poses and temporal video positions.
While other ceeenneeto--queenee methods like CAT3D (Gao al., 2025). EscherNet (Kong al., 2024) and SEVA (Zhou et al., 2025) have shown impressive results, their foundations lie in text- to-image latent diffusion models that use U-Net backbones. This reliance on a convolutional archi- tecture is known to scale less effectively than pure transformers. Furthermore, these models often require 3D-specific learnable components, such as Plücker ray encodings for camera poses and a separate vision encoder for reference views. In contrast, Kaleido adheres to pure transformer design from first principles, which results in a simpler, cleaner design that unifies 3D and video modelling, without any 3D-specific architectural modifications.
Camera-Conditions Video Generation Kaleido relies on video data for large-scale pre-training and is therefore conceptually related to recent line of work on conditioodd video gener- ation. However, the two fields address fundamentally different tasks. Camera-conditionel video generation is designed to tackle the conditional temporal prediction problem, typically contin- uous camera trajectories and often with strong constraints. For instance, MotionCtrl (Wang et al., 2024b) and CameraCtrl (He et al., 2025a) are text-to-video pipelines that do not use reference im- ages as NVS models do. CameraCtrl II (He et al., 2025b) can perform image-to-videss but it is constrained single reference image, which must also be the starting frame. ReCamMaster (Bai et al., 2025) requires the reference and target sequences to be of the same temporal length.
In contrast, Kaleido a generative neural rendering model that handles a 3D spatial problem. Its goal is to perform flexible "any-to-any" view synthesis: generating any number of target views from arbitrary 6-DoF poses, conditioned on any number of reference views. Kaleido's 's task therefore not directly comparable to these 4D (3D+time) temporal prediction models. We do believe, however, that a unified model which can flexibly control both space and time is an important and exciting direction for future work.
Additional related works are listed in Appendix A.
KALEIDO: THE DESIGN DETAILS
3.1
BACKGROUND AND NOTATIONS
Kaleido considers rendering and video generation within a unified requence-to-sequene framework, which is to estimate the conditional distribution of set of target views given set of reference views:
XT x(X |2 PR, pT)
(1)
Here, the conditioning set consists N reference views X = { } and their corresponding
positions pd = R 1- The target set consists of M target views X = {a]-1:>) } and their positions PT = {PT j=1.M }. The positions P are defined flexibly depending on their data modality. For 3D data, each P E SE(3) represents a 6-DoF camera pose. For video data, each P N represents a temporal position (i.e., frame index). For efficiency, within each iteration, we sample fixed total of V views, and choose N reference and M target views such that N + M V.
Kaleido is a latent rectified flow model (Rombach et al., 2022; Ma et al., 2024; Esser et al., 2024) that operates on spatially compressed image tokens. We first use a pre-trained VAE (Kingma & Welling, 2014) (with an 8 x compression rate and 16 latent channels) to encode all reference and target images into a latent space: {2a, _TT ((")).
Following the rectified flow formulation (Liu et al., 2023c; Lipman et al., 2023), we then construct linear interpolation path between each target latent =" z (from the data distribution Po) and standard normal noise latent ~ (0, T) (from the noise distribution p1):
z = (1 t)= + te, where E 10, 1] and V2T E TT.
(2)
4

---

Published as a conference paper at ICLR 2026
Objoverse (PSNR)
UCOSD (PSNR)
Throughput
Tar.
1Rof.
Ret. Tar.
(R batches sec)
Baseline
12:17
20.02
16.77
(1 Architacture Design care (SSUL cdd
13.02
21.25
14.63
17.27
160
(iO Spatial Positional Encoding tuie incare ons po sunt de curq
11.93
22.03
13:65
17.54
(i0 View Sampling Strategy
15.13
21.11
15:16
17:85
148
Temporal Attention Dwep diletiin perry > teep Mik diiw je--i
15.55
18.45
142
(o) Audiaay Features jom
15.96
21.90
15.01
10.01
158
Timestep Condition Design AddM
15.86
15.81
(vi) Attention Registers Regitturs Hop 1012) degiire (Tip fAct 2907
15.93
22.12
15.77
19.08
150
(vill Timestep Training Sampling Odl Timestop Inforence Sampling
23.75
16:03
19.01
18.09
23.05
17:03
19.79
138
with Video Pre-training
17:18
Figure 3: Kaleido Design Ablations. We report PSNR and training throughput and evaluate perfor- mance in two settings: 5 to I and 5 to reference target views. We broadly split our designs into four categories: the Kaleido architecture design spaces (i-v); scaling stability techniques to handle large activations (vi-vii); training and inference timestep sampling strategies (viii-ix); and the role of video pre-training (x). The arrow (-+ indicates the progression from our initial baseline design to our final, optimised design choice.
vision transformer (Dosovitskiy et al., 2020) then processes the combined sequence of clean refer- ence latents 2 R and noised target latents 277 We tokenise the latents using a patch size of 2 x 2 (for a combined spatial compression of 16 x 16), which we found provides an optimal trade-off between generation quality and inference speed. Kaleido is trained with a standard flow matching objective (Lipman et al., 2023), applied only to the target latents Z to estimate a velocity field between Po and Pl. conditioned as defined in Eq. as a form of masked autoregression (Li et al., 2024b). To analyse the scalability, we present three model variants: Kaleido-Small, Kaleido-Medium, and Kaleido-Large Large (Kaleido), with detailed architectures and training strategies introduced next.
3.2 KALEIDO ARCHITECTURE DETAILS AND TRAINING STRATEGIES
In this section, we present a comprehensive ablation study to identify the key architectural and train- ing strategies for scaling sequence- sequence-to-sqquene to-sequence generative neural rendering. To facilitate this explo- ration efficiently, we used our Kaleido-Small model for all experiments, training each configuration for 100K steps on a mixture of the synthetic Objaverse (Deitke et al., 2023b) and the in-the-win wild uCO3D (Liu et al., 2025b) datasets. Our main findings from this greedy search are summarised in Fig. 3, with the final architecture shown in Fig. 4. Full quantitative results and explorations of alternative designs are detailed in Appendix D, with additional design details in Appendix B.
3.2. DESIGNING KALEDIO'S DESIGN SPACES
We begin by exploring Kaleido's 's architectural design spaces and training strategies. Our Kaleido- Small's s starting point is a vanilla DiT-L/SiT-L architecture (Peebles & Xie, 2023; Ma et al., 2024) within rectified flow framework, whose scaling properties have been well established in image and video generation (Esser et al., 2024; Polyak et al., 2024; Chen et al., 2025).
(i) Improved Architecture Design with Llama We first incorporate recent architectural ad- vances from state-of-ther the-trt art sequence-to to-eueeen---seeeeeeeeee language models like Llama-3 (Dubey et al., 2024). Specifically, we replace the standard GLU activations in our transformer's feed-forward layers with SwiGLU (Shazeer, 2020) and swap multi-head attention (MHA) for the more efficient grouped-query attention (GQA) (Ainslie et al., 2023). These simple modifications yield consistent performance gains across our experiments without increasing computational overhead.
(ii) Unified Positional Encoding for Space and Time One of the critical design decisions in Kaleido is unified positional encoding that seamlessly represents 2D, 3D, and temporal positions within a single, consistent design. Specifically, we introduce parameter- free encoding scheme that extends the principles of RoPE-style relative encodings (Su et al., 2021) and Geometric Transfor-
5

---

Published as a conference paper at ICLR 2026
Reshape & Linear
Scale
MotMul
SwiGLU
Geometric Transform
Scale, Shift
Attention
LayerNorm
Linear
Linear
Linear
MatMul
Scale
Temporra Windoww Seif-Attention
Reshape
1ogg
Iv. (HK), (wi), C3 (0/W)/ K*2V,
pt
p-]
Scale, Shift
LayerNorm
Query
Key
Value
Query
Key
Value
SD Data: 20 5D Camera Positions
Scale
S0(2)
SO(2)
SE(3)
Spatial Self-Attertion
Image Widthh
Image Height
SD Camera Position
Scale, Shift
LayerNorm
Inear
2D RoPE
3D CaPE
Diffusion Timestep t0,0,t,
Latent Tokeniser
Temporal
SO(2)
SD(2)
S0(2)
Image
Image Height
Temporal
Reference Latents
Target Latents
RoPE
RoPE
DIIOOw2 Festures
Figure 4: Kaleido Architecture Design Details. Kaleido is simple, scalable decoder-only trans- former that processes clean reference and noised target latents for eqquence-to-sqquene generative neural rendering. During training, single timestep is sampled per network via AdaIN layers. Within each attention block, we apply a unified positional encoding, which consistently represents all 2D, 3D, and temporal positions. This enables single architecture to seamlessly learn from both video and multi-view 3D data without modification.
and integrated into the
mation Attention (GTA) (Miyato et al., 2024), which we adapt and generalise to create a unified representation for space and time, with all positional transformations constructed within bounded range. This design allows Kaleido to process both multi-view 3D and video data without any archi- tectural modifications. Our ablations confirm that this unified design outperforms (more significantly in multi-view settings) both simpler baseline (2D RoPE 3D CaPE without value-transformation and the Plücker embeddings used other leading sequence-to-see tot-tto--eenceee rendering models (Gao et al., 2025; Zhou et al., 2025; Jin et al., 2024). Detailed designs are illustrated in Appendix B.1.
(iii) Principled View Sampling Improves Generalisation To improve generalisation, duce a new view sampling strategy, an oflen-overluoked design aspect in prior models that typically use fixed-view training (Jin et al., 2024; Kong et al., 2024; Gao et al., 2025). Our key insight that the rendering task is more constrained and easier with more reference views; therefore, train- ing should emphasise the more challenging few-view scenarios. To achieve this, we propose an exponential sampling distribution, (n)) Xe-An where X = In(,), where its probability density halves as the number of reference views n increases. By combining this distribution with random attention masking, our model is exposed all possible combinations of (n, m) reference and target view pairs such that n + m E [2, VJ. Our experiments confirm that this approach significantly out- performs both fixed-view and uniform sampling, providing superior trade-off between single- and multi-view conditioning.
intro-
(iv) Expanded Perception Field with Window Attention Our baseline model processes token sequence of shape V x H x W using a factorised attention mechanism: Spatial Attention followed by Temporal Attention, which efficiently processes token sequences but limits cross-view interactions. To improve this, we introduce windowed cross-view attention, where each query token attends to a local K K window in all other frames, significantly enhancing cross-view view feature exchange. This design maintains scalability, with complexity of O(V²K*) that is far more efficient than full attention (/222W?) W?). Our experiments show consistent performance gains with larger window sizes, leading us K 4 for our Small/Medium models and K 8 for our Large model.

---

Published as a conference paper at ICLR 2026
(v) Integration of Auxiliary Visual Features We study the integration of auxiliary visual fea- tures from pre-trained networks to enhance 3D perception. Our findings show that features from DINOv2 (Oquab et al., 2024) further improve Kalcido's depth estimation in-the-wild images, leading to more accurate renderings. These pre-trained semantic features performed similarly to, and sometimes slightly better than, pre-computed depth or surface normals, which encode explicit scene geometry built on top of the same DINOv2 model. We also observed that larger DINOv2 models provide additional, albeit marginal, performance gains. Based on this trade-off, we pair the feature extractor with our model size: we use DINOv2 with ViT-B backbone for Kaleido-Smin -aaal and Medium, and DINOv2 with ViT-L backbone for our Large model.
3.2.2 MASSIVE ACTIVATIONS IN RECTIFIED FLOW TRANSFORMERS
During our initial scaling experiments, we traced severe training instability on high-resolution im- ages to the emergence of massive activations within the transformer layers. While massive activa- tions have been studied extensively in language models (Sun al., 2024) (often called "attention sinks" (Xiao et al., 2024: Gu et al., 2025)) and in visual representation learning (Darcet et al., 2024), their behaviour within diffusion or rectified flow models remains largely unexplored. In this section, we provide the first empirical findings in this context, illustrated in Fig. 5.
Magaimade
Magnituee
Megnitude
10
Top 0.001%
Top 1
15192
512 who Ray.
24353
10
who Beg.
10a
0.01%
10
256 who Reg.
Reg.
10F
Top 0.19
219.75
174.13
10º
256 I Rey
835
w/ 32 Reg.
397.75
19.75
363,5
10a
238.75
12
18
24
12 Layer Index
Layer Index
Training Steps
(a) Massive activations are very few in numbers.
(b) The activations grow faster and larger with more tokens.
(c) Learnable registers effectively reduce massive activations.
Figure 5: Visual Analysis of Massive Activations in a Rectified Flow Transformer. We analyse the massive activation phenomenon in our model. (a) We show that massive activations are sparse. (b) They grow aggressively with model depth and training time. (c) Appending learnable register tokens effectively suppresses these activations to stable, low magnitude.
We also found that these massive activations emerge most prominently when training mixture of synthetic and real-world data, where they act as global information aggregators as observed in VITs (Darcet al., 2024) reconcile the different rendering logic required for each domain. To resolve the training instability, we adopt the solution from Sun et al. (2024) and append a single, learnable "register" token the keys and values in each attention layer, as confirmed in ablation (vii). This simple modification proved highly effective, consistently reducing activation magnitudes stable level of ~300. While we explored alternative solutions in ablation (vi), such as modifying the timestep conditioning (Bozic et al., 2021; Sun et al., 2025; Tang et al., 2025), they were far less effective and resulted in lower overall performance.
3.2.3 TAILORED RECTIFIED FLOW SNR SAMPLERS FOR GENERATIVE RENDERING
Prior rectified flow models for text-to-image/videe generation (Esser al., 2024; Polyak et al., 2024) typically use logit-normal sampling (Atchison & Shen, 1980) to focus training on intermedi- ate timesteps & E [0, 1]. However, we found this approach to be suboptimal for neural rendering. We hypothesise this is because rendering is highly constrained generative problem, where the output must be consistent with reference images, unlike the much less constrained nature of text-to- image/video synthesis. This insight suggests that rendering models should focus more heavily on the early, high-noise timesteps where the coarse scene structure is established.
Our ablations validate this hypothesis. In ablation (viil), we found that sampling distributions shifted towards the noise end consistently outperform the standard baselines by a large margin. While a heavily shifted uniform distribution performed well, we choose shifted Mode sampling as our final training sampler design because it provides a superior balance between the critical early timesteps and the intermediate steps. To complement this, in ablation (ix), we align our inference process by adopting a linear-quadratic schedule, which concentrates more denoising steps in the initial, high-noise part of the trajectory. We found that this design combination, using a biised
7

---

Published as a conference paper at ICLR 2026
sampler for both training and inference, delivered the single most significant performance improve- ment across all of our Kaleido design ablations. Detailed designs are illustrated Appendix B.2.
3.2.4 VIDEO PRE-TRAINING IMPROVES 3D EFFICIENCY
Finally, ablation (x), we validate our core hypothesis of treating 3D as a sub-domain of video. The results confirm that pre-training on large-scale video data significantly boosts the efficiency of 3D fine-tuning, with 100K and 200K pre-training steps yielding 1.3x and 2.0x improvements, respectively. This demonstrates that stronger video foundation model directly translates to faster convergence on view synthesis tasks.
3.3 FRAME INTERPOLATION AS ZERO-SHOT SPATIAL UPSAMPLER
Since view 3D data lacks the temporal consistency required for typical video VAEs, Kaleido uses an image-bases VAE, which makes generating dense video sequences computationally expen- sive. To overcome this, we employ an efficient two-stage pipeline: Kaleido first renders sparse set of keyframes along a camera path, and then separate, lightweight interpolation model based on FiLM (Reda et al., 2022) predicts the intermediate frames. This hybrid approach mitigates high memory costs and effectively emulates temporal video decoder, allowing for the efficient produc- tion of smooth, high-frame-ran video sequences.
EXPERIMENTS
We designed three variations of our model: Small, Medium, and Large, with increasing parameter counts to demonstrate the scalability of our architecture. The design choices for each model are summarised in Table 1, Hereafter, we refer our largest model simply as Kaleido and the entire collection as the Kaleido family.
Table 1: Kaleido Family Architecture Details. We detail the key hyper-parameters the number of layers, hidden embedding size, number of query and key/value heads, the window size used in temporal attention, the choice of auxiliary DINOv2 encoder, and the total parameter count.
Layers
Hidden
Query
Window
Aux. Encoder
Params.
Kaleido-Small Kaleido-Medium
24
1024
16
DINOv2 B (86M)
653M
1280
DINOv2-B
1.2B
Kaleido
40
1792
28
7
8
DINO (30IM)
3.1B
The Kaleido family is trained on a diverse mixture of object-level and scene-level datasets, featuring synthetic 3D object mesh renderings and real-world indoor and outdoor scenes. We outline our detailed training configurations and evaluation strategies in Appendix E.
4.1 RESULTS ON NOVEL VIEW SYNTHESIS
Compared to Generative NVS Methods We first evaluate Kaleido's zero-shot performance on standard novel view synthesis (NVS) benchmarks. For object-level synthesis, compare against leading methods such as SV3D (Voleti et al., 2024) and EscherNet (Kong et al., 2024) on datasets including 003D (Wu al., 2023), GSO-30 (Downs et al., 2022), and RTMV (Tremblay et al., 2022). For scene-level synthesis, we compare against the state-of-the-art rendering model SEVA (Zhou al., 2025) on LLFF (Mildenhall et al., 2019), Mip-NeRF 360 (Barron al., 2022), and Tanks and Temples (Knapitsch et al., 2017). To ensure fair comparisons, we match evaluation resolutions and follow SEVA's strategy for handling scale ambiguity in single-view renderings.
In Table 2, our results demonstrate that Kaleido is both highly efficient and scalable. Even our smallest model, Kaleido-Sman (0.6B), matches or surpasses larger baselines like SEVA (1.5B) and SV3D (2.3B). As we scale up. performance consistently improves, with our largest model decisively outperforming all competitors, achieving gains as high as +7.8 dB PSNR on GSO-30 (10 views) and +1.3 dB PSNR on Mip-NeRF 360 (6 views). Full SSIM/LPIPS metrics are provided in the Appendix F, and with additional high-resolution generations are shown in Fig. 6.
Compared to Per-Scene Optimisation Methods Next, we evaluate the upper bound of Kaleido's rendering precision when provided with many reference views. For this analysis, we compare against two state-of-the-e of-the-art scene-specific optimisation methods: Instant-NGP (Müller et al., 2022) and 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023). As these methods are optimised per-scene, they represent strong performance ceiling. We also include generative NVS baselines that can
8

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot PSNR Performance with Generative Methods. Kaleido achieves state-of- art NVS performance across all object- and scene-level benchmarks, with particularly dominant results in many-view settings. Notably, our Kaleido-Smin model consistently matches or outper- forms all baselines despite having significantly fewer model parameters.
RTMV
LLFF
Mip-NeRF
Tanks Temples
e Ket. Views
2
10
3
10
3
3
1
6
Data Type
Object
Object
Scene
256
Views
SoTA Model
SV3D
ExcherNet
ExcherNee
SEVA
SEVA
SEVA
Resils (PSNR1)
19.28
20.24
22.91
24.09
25.09
25.00
10.56
12,66
13.59
14:52 15.55
12,93
15.78
16.70
11.28
12.65
13,80
14.72
Kaleido Small
18:58
23.73
26.20
29.11
51.66
13.57
17.18
1841
19.57
21.35
15.81
13.13
14.13
15.20
Mediid--Meelam
20.78
20.32
25.78
28.01
30.94
32.94
13.78
19.41
21.09
22.53
14.86
20.40
16447
17.50
11:36
13.04
14.43
15:47
Kalcido
20.1
14.61
Reference Image
Kaleido Generations
Figure 6: In-the-Wild Single-Vies View Rendering with Kaleido. We showcase Kaleido's zero-shot generative capabilities challenging in-the-wild images. From single input view (first column), Kaleido generates sequence of photorealistic novel views along a circular, object-centric camera trajectory. The examples feature complex scenes with diverse objects and structures, demonstrating Kaleido's 's remarkable generalisation and high-fidelity rendering quality.
accept flexible number of reference views: EscherNet, evaluated on the NeRF-Synthetic dataset (Mildenhall et al., 2020) with 256px resolution; and SEVA, evaluated on the LLFF and Mip-NeRF 360 datasets (Mildenhall al., 2019; Barron et al., 2022) with 512px resolution.
In Fig. 7, we can observe that generative baselines such as EscherNet and SEVA initially outperform per-scene methods few-view settings, their performance quickly plateaus. In contrast, Kaleido's performance continues to scale with more views, mirroring the trend of scene-specific methods. This superior zero generalisation creates a widening performance gap over other generative models.
When all available reference views are used, Kaleido's performance the NeRF-Synthetic dataset nearly on par with Instant-NGP. On the more challenging LLFF and Mip-NeRF 360 datasets, Kaleido surpasses Instant-NGP, marking the first time zero-shot generative model has matched the quality of a sfate-of-the-arr, per-scene optimised method. Given that per-scene methods are often sensitive to camera coordinate systems, Kaleido's robust, data-driven performance highlights the immense potential zero-shot solutions for general general-purpose purpose rendering. A detailed analysis of the computational trade-offs between generative and scene-specific methods is provided in Appendix C.

---

Published as a conference paper at ICLR 2026
PSNR
PSNR
PSNR
36
32
24
28
20
31.72
21-
20
18.15
18
16
19.79
10
3DXIS
Eacher Net
TEXGS
SEVA
12
3DGS
SEVA
Irstant SGF Kalesco
Instan-NGP NGP Kaleido
Itstun-NGP
Kaleido
Reference
(a) NeRF Synthetic [256 Res.]
(b) LLFF [512 Res.]
(c) Mip-NeRF 360 [512 Res.
Figure 7: PSNR Performance with Per-Scene Optimisation Methods. Kaleido's performance scales consistently with more reference views. It significantly outperforms other generative NVS baselines, with the performance gap widening as more views are provided. Notably, when given all available reference views, Kaleido surpasses Instant-NGP on both scene-level datasets.
4.2 RESULTS ON 3D RECONSTRUCTION
We evaluate Kaleido's 3D reconstruction capabilities on the GSO-30 dataset by applying the NeuS2 framework (Wang et al., 2023a) to Kaleido's generated views. We compare against a diverse set of image-to-3D baselines, including direct generation methods like Point-E (Nichol al., 2022) and Shape-E (Jun & Nichol, 2023), optimisation- based methods like DreamGaussian (Tang et al., 2024), and other view-synthesis-baeed approaches like One-2-3-45 (Liu al., 2023a) and Syne- Dreamer (Liu et al., 2023d). Following the protocol of EscherNet, we generate fixed set of 36 object-centric views to serve as imput for the reconstruction. For fair comparison, all methods evaluated at 256px resolution, and we additionally report Kaleido's performance at 1024px to showcase its high-resolution capabilities.
Table 3: 3D Reconstruction Performance GSO-30. We measure reconstruction quality using Chamfer Distance (CD, lower is better) and Volumetric IoU (VIoU, higher is better). Kaleido clearly surpasses EscherNet by large margin, demonstrating 5x greater view efficiency. The quality is further improved when using higher-resolution renderings from Kaleido.
View
2 Views
Views
Views
10 Views
CD L
VloU
VIoUt
CD 4
VioU
VloU
CD
VioU f
Point E
0.0447
0,2503
Shope-E
0.0448
0,3762
One-2.3.45
0.0667
DreanGaussan
0.0459
0.4531
SyneDreamer
0.0400
0,5220
NeuS
0.0366
0.5352
0.7264
EscherNet
0.0314
0.5974
0.0215
0.6868
0.0190
0.7189
0.0175
0.7423
0.0167
0.7478
Kaleido
0.0214
0.7785
0.0113
0.7960
0.0100
0.8118
Kaleido 11024 Res.)
0.0183
0.7006
0.0118
0.7851
0.0104
0.8053
0.0091
0.8290
0.0086
0.8418
In Table 3, Kaleido again achieves the best performance in 3D reconstruction, outperforming direct image-to-3D models, our NeuS baseline, and EscherNet. The results highlight Kaleido's remarkable rendering efficiency and precision. With just 2 views, our model has surpassed the reconstruction quality that EscherNet achieves with 10 views. This superiority is more evident qualitatively in Fig. 10. Given the same 256px resolution, Kaleido's generated meshes clearly better. At 1024px resolution, the reconstructed textures are incredibly detailed and sharp. appearing close to the ground truth and suggesting exciting new applications for high-fidelity, few-shot 3D reconstruction.
Finally, as Kaleido relies on a video data prior, it conceptually related to camera-conditioned video generation models. We discuss these relationships and provide comparative results in Appendix H.
CONCLUSION
In this paper, we introduced Kaleido, family of generative models that reframes neural rendering as a sequence-to-sequeee unified representation of space and time, Kaleido exhibits strong scaling and achieves state-of-the- art performance on view synthesis and 3D reconstruction benchmarks. Most notably, Kaleido the first zero-shot generative model match the quality of per-scene optimisation methods, marking significant step towards a universal rendering engine. A discussion of the model's limitations and future work is included in Appendix I.
problem. By gradually modernising the transformer architecture with a
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This research was conducted with a commitment to responsible Al practices. All data used train our models was sourced from either publicly available academic datasets was legally licensed from commercial providers, ensuring compliance with data usage rights. Our research was con- ducted with careful consideration for ethical principles, and the best of our knowledge, it does not present issues related to privacy, security, legal compliance, or research integrity.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have provided comprehensive details of our methodol- ogy. This includes detailed descriptions of our training data, evaluation strategies, and hyperparam- eters in Appendix E. complete breakdown of our architectural design is provided Section 3.2 and Appendix B.
REFERENCES
Joshua Ainslie, James Lee- Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Jhon Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 1980.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hisn Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. Vd3d: Taming large video diffusion transformers for 3d camera control. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Haotian Bai, Yuanhuiyi Lyu, Lutao Jiang, Sijia Li, Haonan Lu, Xiaodong Lin, and Lin Wang. Com- ponerf: Text Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv preprint arXiv:2303.1 13843, 2023.
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang. Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai. Pengfei Wan, et al. Recammaster: Camera-comtrolled generative rendering from a single video. In Proceedings of the International Conference Computer Vision (ICCV), 2025.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.15127, 2023.
Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Pro- cessing Systems (NeurIPS), 2021.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative in- teractive environments. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
11

---

Published as a conference paper at ICLR 2026
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang. Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, et al. Goku: Flow based video generative foundation models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE Transactions on Puttern Analysis and Machine Intelligence (PAMI), 2007.
Deepmind. Genie3.
models/, 2025.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances Neural Information Processing Systems (NeurIPS), 2023a.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of an- notated 3d objects. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations (ICIR), 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In Proceedings of the IEFE International Conference Robotics and Automation (ICRA), 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning (ICMI.), 2024.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. Proceedings of the International Conference on Learning Represen- fations (ICLR), 2025.
12

---

Published as a conference paper at ICLR 2026
Sara Fridovich-Keil Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Yasutaka Furukawa, Carlos Hernández, et al. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision, 2015.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin Brualla, Pratul Srinivasan, Jonathan Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Nearal Information Processing Systems (NeurIPS), 2025.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2020.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge University Press, 2003.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for video diffusion models. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2025a.
Hao He, Ceyuan Yang, Shanchuan Lin, Yinghao Xu, Meng Wei, Liangke Gui, Qi Zhao, Gordon Wetzstein, Lu Jiang, and Hongsheng Li. Cameractrl ii: Dynamic scene exploration via camera- controlled video diffusion models. In Proceedings of the International Conference on Computer Vision (ICCV), 2025b.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposhum on User interface software and technology, 2011.
Wonbong Jang and Lourdes Agapito. Nvist: In the wild new view synthesis from a single image with transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. In Pro- ceedings of the International Conference on Learning Representations (ICLR), 2024.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the International Conference Learning Representations (ICLR), 2014.
Arno Knapitsch, Jaesik Park, Qian-Y Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (TOG), 2017.
Xin Kong, Shikun Liu, Xiaoyang Lyu. Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Escher- net: A generative model for scalable view synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
13

---

Published as a conference paper at ICLR 2026
Haoran Li, Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang. Lik-hang Lee, and Peng Yuan Zhou. Dreamscene: 3d gaussian- based text-to-3d scene generation via formation pat- tern sampling. In Proceedings of the European Conference on Computer Vision (ECCV), 2024a.
Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, and Angjoo Kanazawa. Cameras as relative positional encoding. arXiv preprint arXiv:2507.1044 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Kon- stantinos N Plataniotis, Sergey Tulyakov, and Jian Ren. Wonderland: Navigating 3d scenes from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2025.
Yixun Liang. Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning- based 3d vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2023.
Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan Camilo Perez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked auto-regressive diffusion for video generation at scale. Transactions on Machine Laerning Research (TMLR),
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In Proceedings of the International Conference on Computer Vision (ICCV), 2023b.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In Proceedings of the International Conference on Learning Representations (ICLR), 2023c.
Xingchen Liu, Piyush Tayal, Jianyuan Wang, Jesus Zarzar, Tom Monnier, Konstantinos Tertikas, Jiali Duan, Antoine Toisoul, Jason Y Zhang, Natalia Neverova, et al. Uncommon objects in 3d. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Syncdreamer: Generating multiview-consister images from a single-view image. arXiv preprint arXiv:2309.03455 2023d.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference Learning Representations (ICIR), 2019.
Fan Lu, Kwan- Yee Lin, Yan Xu, Hongsheng Li, Guang Chen, and Changjun Jiang. Urban architect: Steerable 3d urban scene generation with layout prior. arXiv preprint 2rXiv:2404.06788. 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion-batt based generative models with scalable interpolant transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
14

---

Published as a conference paper at ICLR 2026
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions Graphics (TOG), 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. Gta: A geometry-aware atten- tion mechanism for multi-view transformers. In International Conference Learning Represen- tations (ICLR), 2024.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Transactions on Graphics (TOG), 2022.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: A versatile and accurate monocular slam system. IEEE Transactions on Robotics and Automation, 2015.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.0875// 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Laerning Research (TMLR), 2024.
William Peebles and Saining Xie. Scalable diffussin models with transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Ma, Ching- Yao Chuang, et al. Movie gen: A cast media founda- tion models. arXiv preprint arXiv:2410.13720. 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv: /4988, 2022.
Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, et al. Infinite photorealistic worlds using pro- cedural generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code laama: Open foundation models for code. arXiv preprint 2r88v: 2023.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from from-motion revisited. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan- Man-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
15

---

Published as a conference paper at ICLR 2026
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXlv:2002.05202, 2020.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15110 2023.
Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi- view diffusion for 3d generation. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding. Neurocomputing. 2021.
Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 3d-gpt: Procedural 3d modeling with large language models. In Proceedings of the International Confer- ence on 3D Vision (3DV), 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In Conference on Language Modeling (CoLM), 2024.
Qiao Sun, Zhicheng Jiang, Hanhong Zhao, and Kaiming He. Is noise conditioning necessary for de- noising generative models? In Proceedings of the International Conference Machine Learning (ICML), 2025.
Bingda Tang, Boyang Zheng, Sayak Paul, and Saining Xie. Exploring the deep fusion of large language models and diffusion transformers for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In Proceedings ofe International Conference on Learning Representations (ICLR), 2024.
Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Thomas Müller, Charles Loop, Nathan Morrical, Koki Nagano, al. Runv: ray-traced multi- view synthetic dataset for novel view synthesis. arXiv preprint arXiv:2205.07055. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in Neural Informa- lion Processing Systems (NeurIPS), 2017.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2024a.
Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the International Conference Computer Vision (ICCV), 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In Proceedings of SIGGRAPH, 2024b.
16

---

Published as a conference paper at ICLR 2026
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary 3d object dataset for realistic percep- tion, reconstruction and generation. In Proceedings of the IEEE Conference Computer Viston and Puttern Recognition (CVPR), 2023.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, 2022.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng. Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2025.
Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025a.
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos. In Proceedings of the International Conference on Computer Vision (ICCV), 2025b.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang. Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video diffusion models for high- fidelity novel view synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2025c.
Jensen Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusson models. Proceedings of the International Conference on Computer Vision (ICCV), 2025.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics (TOG), 2018
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
From 2D to 3D and Camera Parameters Reconstructing 3D geometry and camera parameters from 2D images is foundational problem in computer vision. Classical approaches like Structure from Motion (SfM) (Hartley & Zisserman, 2003; Schönberger et al., 2016) and Simultaneous Local- isation and Mapping (SLAM) (Davison al., 2007; Mur-Artal et al., 2015; Izadi et al., 201 I) have been highly successful, but they are limited by their need to optimise each scene from scratch and their struggles with non-overlapping views. More recently, learnin-bbased methods have emerged to address these limitations. Models like DUS13R (Wang et al., 2024a) and VGGT (Wang et al., 2025) introduce feed-forward pointmap regression, enabling end-to-end 3D reconstruction that gen- eralises across scenes, While these methods represent a significant step forward, their reliance on direct geometric regression means they cannot effectively infer content in occluded regions. No- tably, Kaleido's s fully generative design allows it to predict plausible, spatially consistent content for occluded regions, key advantage over both classical and modern regression- based techniques.
Multi- View Stereo, Neural Rendering, and Novel View Synthesis Traditional Multi-view Stereo (MVS) (Furukawa et al., 2015; Schönberger et al., 2016) reconstructs 3D surfaces by tri- angulating features across multiple viewpoints. This principle was revolutionised by Neural Radi- ance Fields (NeRF) (Mildenhall et al., 2020), which uses volume rendering and coordinate MLPs to achieve photorealistic novel view synthesis. A plethora of follow-up works have focused on improv- ing the speed and quality of this per-scene optimisation paradigm (Müller et al., 2022; Fridovich- Keil al., 2022; Chen et al., 2022; Kerbl al., 2023). More recently, feed-forward transformer-basee models have emerged (Hong et al., 2023; Jang & Agapito, 2024; Jin et al., 2024), which can directly predict 3D primitives or render novel views from limited inputs. However, as deterministie models, they fundamentally struggle with the inherently probabilistic nature of inferring occluded regions.
Generative Video and World Models Kaleido's 's methodology deeply connected to recent ad- vancements in generative video and the emerging paradigm of world models. This progress is largely driven by a dominant technical stack combining diffusion or rectified flow models with transformer architectures, a foundation shared by many recent generative video model designs (Chen et al., 2025; Blattmann et al., 2023; Chen et al., 2024b;a; Yang et al., 2025). The increasing capabilities of video generation have also positioned it stepping stone towards building world models systems that learn an internal model of the world to simulate physical interactions and predict future states. This trajectory is evident in models that focus on controllability and interactivity. For instance, the Navigation World Model (Bar et al., 2025) predicts future observations to facilitate planning, while frameworks like WonderWorld (Yu et al., 2025a) and GameFactory (Yu al., 2025b) generate ex- plorable 3D environments. Most notably, the Genie series (Bruce et al., 2024; Deepmind, 2025) creates interactive environments with persistent spatial memory and real-time promptable world events, marking significant advance toward truly immersive and dynamic virtual worlds.
Kaleido contributes to this broader pursuit of world modelling from a different perspective. Instead of focusing on temporal dynamics agent-based interactivity, Kaleido approaches world modelling through the lens neural rendering, prioritising spatial consistency and generation flexibility. flexibbiil This unique approach allows it to operate across a spectrum of realities: with many reference views, it produces a grounded reality through faithful reconstruction; while with few views, it creates a gen- erated reality with plausible unseen details. This unique capability to seamlessly transition between reconstruction and creative generation marks a distinct and intriguing path toward creating truly versatile and navigable virtual worlds.
18

---

Published as a conference paper at ICLR 2026
B ADDITIONAL DETAILS ON KALEIDO DESIGNS
B.I UNIFIED POSITIONAL ENCODING DESIGN DETAILS
Our unified positional encoding represents different positions as follows: 2D image positions, pixel coordinates are mapped a pair of angles (0,, ), representing element in SO(2) SO))), where 0,. E [0, 2x) distributed uniformly from the top-left to the bottom-right patches; In temporal positions, frame indices
single angle 0, SO(2), with values interpolated linearly from the start to the end of a clip. In 3D camera poses, 6-DoF camera extrinsics c = [ i (with rotation R and translation t) are represented an element in SE(3), following the design in CaPE (Kong et al., 2024).
similarly mapped
We then define unified geometric attribute g for each token, depending on its data modality:
For 3D data: For video data: g == (0p, Ow 02) E S0(2) x SO(2) x SO(2).
g := (0h, ow c) E SO(2) SO(2) SE(3)
(3)
(4)
Within the GTA framework, these components are used to construct a block-diagonal transformation matrix Pg that is applied to each token's feature vector e E Rd. The construction of Pg varies for different attention blocks, allocating the feature dimension d as follows: In Spatial Attention, we apply only the 2D position embeddings (0n, Ow.). which are expanded into d/4 4 distinct frequency bands, with dimensions allocated to image height and width components based a 1:1 ratio; In Temporal/3D Attention, the 2D embeddings (0n, Ow.) are expanded into d/8 frequency bands. For video data, the temporal embedding 0, is expanded into d/4 frequency bands. For 3D data, the pose embedding is repeated to fill the remaining dimensions. The total dimensions are allocated to image height, width, and temporal/3D components based 1:1:2 ratio.
Finally, we normalise the camera translation element such that its maximum norm across all views in given scene is 1. This ensures all positional transformations remain within bounded range, which we found is crucial for stable training to handle different scene scales.
B.2 RECTIFIED FLOW SNR SAMPLER DESIGN DETAILS
Timestep Sampler for Training We evaluate three timestep sampling distributions, which were previously explored in SD3 (Esser al., 2024), to assess their effect learning efficiency and generalisation for neural rendering:
Uniform: A standard uniform distribution 1),
This distribution passes a value u ~ M(m,s) s) through sigmoid function. The mean m and standard deviation s control the sampling bias. A positive mean (m 0) focuses training on high-noise timesteps (closer to t = 1), while negative mean (m < 0) focuses on low-noise timesteps (closer = 0).
Logit-Normal:
Mode: This distribution warps a uniform sample u ~ U((0 1) with the mapping - u - 5 (cos"(Ew) - 1 + u). The parameter s controls the sampling focus: > 0 favors the midpoint (t = 0.5), while s < 0 favors the endpoints (t = 0 and = 1).
We apply a modulation function, m(t, a - + (a 1) t), to skew each base distribution. A shifting factor pushes the probability mass towards the noise end of the trajectory 1). The resulting probability densities are visualised in Fig. 8.
In ablation (viii), our results showed that all distributions shifted towards the noise end signifi- cantly outperformed the unshifted baselines. While heavily shifted uniform distribution performed marginally best, we chose shifted Mode sampling as our final design because it provides superior balance between the critical early timesteps and the intermediate steps.
Timestep Sampler for Inference In ablation (ix), we introduce a linear-quadratic sampling schedule for inference to complement our training sampler. This schedule works as follows: for a total of S denoising steps, the first 5/2 timesteps are sampled linearly, while the remaining S/2 steps are sampled quadratically. This non-uniform schedule effectively concentrates more discrete denoising steps in the high-noise region of the trajectory, aligning the inference process with our training strategy and leading to substantial performance gains.
19

---

Published as a conference paper at ICLR 2026
Deasity
Deasity
Density
1.5
2.5
0.6
0.6
0.3
0.5
0.2
0.8
0.2
Sampled e
Sampled
Sampled r
(a) Logit- Logit-Normal (Bascline)
(b) Mode [Scalc=0.8, 8,, Shift=1]
(c) Mode [Scale=0.8, Shift-3]
Deasity
Deasity
Deasity
2.4
3.2
2.4
0.4
0.2
0.6
0.2
Sampled
Sampled
Sampled
(d) Uniform [Shift=1]
(c) Uniform [Shift=3]
(f) Uniform [Shift=5]
Figure 8: Probability Densities of Different Timestep Sampler. We visualise the PDFs for the samplers evaluated in our ablation study, where 1 the noise end and
the data end. (a) Logit-Normal: The standard baseline, which concentrates probability mass the middle of the timestep range and has diminished density near the endpoints; (b) Mode: Similar to Logit-Norma but maintains a positive density the endpoints. (d) Uniform: A standard uniform distribution, which samples all timesteps with equal probability: (c), (e), and (f) show the Mode and Uniform distributions shifted towards the high-noise end study the effect of noise-biased sampling.
C MEMORY CONSUMPTION AND RUNNING LATENCIES
In this section, we compare the memory consumption and inference latency of Kaleido against the generative and deterministic baselines evaluated in Section 4.1. In Table 4, the per-scene optimisa- tion methods (Instant-NGP and 3DGS) operate in distinct efficiency tier. They run significantly faster than generative approaches, they do not require iterative denoising steps; 3DGS, in partic- ular, achieves the fastest rendering speeds by not relying on the neural parameterisation entirely.
However, within the class of generative models, Kaleido demonstrates superior scaling efficiency. Notably, Kaleido-Medium is slightly larger than EscherNet in parameter count but scales much bet- ter with resolution (due to the efficient design of factorised spatial and temporal window attention), requiring only 10% of the rendering time on 1024 resolution. While our current design priori- tises rendering quality and model scalability, narrowing the inference speed gap with deterministic, scene-specific methods remains an important direction for future work.
Table 4: Memory Consumption and Inference Latency Comparisons. We report the averaged per-frame rendering time for a sequence of 25 frames (1 reference 24 target views) across both generative and deterministic methods at various resolutions. For generative models, the reported times reflect the full inference process based on the required denoising steps specified the original papers. To evaluate memory efficiency, we additionally report the maximum number of frames for all generative methods that can generated concurrently on a single GPU. benchmarks were conducted on single NVIDIA A100.
Generative (soc. per frames max frames)
Deterministic (sec. per frame)
EscherNet [0.9R] (50 Steps)
SEVA [1.3]] (50 Steps)
Kaleido Small
Kaleido-Medium Medium
Kaleido
[0.6B] (24 Stops)
[1.2B] (24 Stops)
[3.1B]
InstanINGP
3DGS
(24 Steps)
1256 x 256
0.58s/ 1601
0.42s/ / 1473
0.55s 231
1.1s 201
2,7s 161
0.35g
0.0025s
512 512
6.1
1.5s 369
1.6%/ /
3.1s/ 103
4.6x
0.35s
0.00255
1024 1024]
92s 101
8.66 93
5.48 55
10.0s
17.6s/35
0.67%
0.00344s
20

---

Published as a conference paper at ICLR 2026
D KALEIDO DESIGN ABLATIVE QUANTITATIVE RESULTS
Table presents the full quantitative results supporting the ablation study in Fig. including all
alternative design decisions we explored. To provide a comprehensive comparison, we report per- formance across three settings: one-to-five, five-to-one, and five-to-five seference-to-targee views.
Objaverse
BCO3D
Training Throughput
-
Objaverse Single
21.23
uCO3D Single Baseline
14.89
19.19
16.97
Objaverse uCO3D - Joint Baseline
12.17
19.10
20.02
14,66
18.71
16.77
160
(i) Architecture Design Vanilla DiT)
DTT I.lama3 (SwiGLU GQA)
13.02
20.18
21.23
14.63
19.31
17.27
160
(ii) Spatial Positional Encoding (2D RoPE 3D CaPE]
RaPE.
17.61
GTA [2D RoPE 3D CaPE)
11.93
21.05
22.03
13.65
20. 04.84
17.54
148
(iil) View Sampling Strategies [Fixed 6-+6]
Uniform Sampling w/o Masking
12.18
22.22
21.65
14.22
21.36
18.34
150
Unilfomm Sampling Masking
Expomential Sampling Masking
15.13
21.41
21.11
15.16
20.25
17.83
148
(vv) Temporal Attention Design (Temmporl Attention (K=1))
Full Attention
14.54
22.62
22.40
15.00
20.75
18.28
5%
Temporal Window Attention (K
15.45
21 60
21.04
15.40
20.43
18.15
146
Temporal Window Attention (K
15.67
22.25
21.79
15.55
20.81
18:45
142
Temporal Window Attention (K 8)
15.73
22.60
22.54
15.85
21.39
19.15
103
(V) Auxillary Features [None]
DINOv2 [DiT-B]
15.86
22.28
21.90
15.81
21.09
18.81
138
DiNOv2 [DiT.I.]
22.65
MetaDepth [DiT-L]
13.82
22.28
22, 22
15.76
21.26
18.65
135
MetaNormals [DIT-L]
15.77
22.32
22.
22
15.59
21.20
18.85
135
(vi) Timestep Conditioning Design [AdaL.N-Zero, Top Act.: 15192]
Shift Only [Top Act.; 6820]
16.07
21.56
21:51
15.97
20.56
18.28
144
No Timestep [Top 1 Act.: 4312.1
16.26
20.85
21.00
16.15
20.38
18.38
148
(vil) Attention Registers (No Registers, Top 1 Act.: 15192)
1 Register [Top Act.: 397.75)
15.93
22.27
22. 22.12
15.77
21.02
19.03
138
8 Registers [Top Act.: 279.75]
15.26
22.04
21.94
15.72
20.75
18.55
138
Registers [Top Act:; 238.75]
15.07
21.88
21.54
15.66
20.59
18.27
138
(viti) Timestep Sampling Training Strategy [LogitNorm [0,1]]
Uniform [Shift= 1|
17.58
22.01
21.96
15.90
20.49
18.57
138
Uniform [Shift = 3]
18.27
23.64
23.
16.39
21.74
19.21
138
18.94
Made [Scale 0.8)
17.39
22.06
22.08
15.99
20.75
18.68
138
Mode (Scale 0.8, Shift
18.19
24.06
23. 23.75
16.03
21.76
19.11
138
(ix) Timestep Sampling Inference Sampling [Linspace [1, 999]] Trailing 980]
17.95
23.66
23.51
16.70
21.83
19.43
138
LinearQuadratic [1. 9991
18.09
23.87
23.95
17.03
22.15
19.79
138
(x) with Video Pre-training [No video Pre-training) Video Pre tr-irrining 100K Steps (1.3x BAL)
18.
24.22
24.30
17.11
22.23
20. 10
138
Video Pre-training 200K Steps (2x Eff.)
18.28
24.55
24.60
17.18
22.43
20.15
138
Table 5: Quantitative Results for Kaleido Design Ablations. We report the complete quantitative results (PSNR, higher is better) corresponding to the ablation study in Fig. 3. Performance is eval- uated in one-to-five, five-to-one, and five-to-five reference-to-targe view settings. Our final design choice for each component is marked in red.
21

---

Published as a conference paper at ICLR 2026
E ADDITIONAL DETAILS OF KALEIDO TRAINING ANSEGURAATION AND EVALUATION STRATEGIES
Training Datasets Kaleido is trained on a diverse mixture of object-level and scene-level datasets. For object-level data, we use ShutterStock 3D, our licensed collection of synthetic 3D meshes, which we render with object-centric camera poses under varied lighting conditions; and uCO3D (Liu et al., 2025b), which includes real- world objects with estimated poses. For scene-level data, we combine several datasets: RealEstate (Zhou et al., 2018), which features indoor room scenes; DL3DV (Ling et al., 2024), which features both indoor and outdoor scenes; and a filtered subset of ShutterStock Video. This licensed video subset is curated to include only static scenes, and then labelled with a pose estimator VGGT (Wang et al., 2025).
In summary, our 3D fine-tuning dataset consists of approximately 1.5M object sequences and 2M scene sequences. For the initial video pre-training stage, we leverage the full, unfiltered Shutterstock Video dataset, comprising 34M video clips.
Training Strategy Our training process follows two-stage, progressive-resolution curriculum. First, we pre-train Kaleido exclusively on video data at a fixed 256px resolution. We then fine- tune combined multi-view 3D datasets, progressively increasing the resolution from 256px to 512px, and finally to 1024px. In our 1024px fine-tuning, we introduce aulli-aspeet--atio training (including 1:1, 4:5, 5:4, 16:9, and 9:16) to enable flexible resolution generation.
All Kaleido model variants are trained using the same datasets, with the AdamW optimiser (Loshchilov & Hutter, 2019) and weight decay of 0.01. In each training iteration, we randomly sample a total of 12 frames per sequence. These frames are then partitioned into reference and target views according the view sampling strategy in Sec. 3.2.1.
The learning rate is chosen based on the training stage. For the initial video pre-training and the first stage of 3D fine-tuning (both at 256px resolution), apply learning rate of 10-4. For the subsequent high-resolution 3D fine-tuning stages (512px and 1024px resolution), we decrease the learning rate 10-5 To train our larger models at large resolutions, we incorporate FSDP sharding and activation checkpointing.
We use fp16 6 mixeedpreccssion in all training stages, as we find it crucial for stable convergence; while bf16 6 consistently leads to unstable training. Our largest Kalcido model is trained for two weeks on 512 NVIDIA H100 GPUs Additional srper-parameters are listed in Table 6.
Stage (Vidco data) [256 x 256]
Stage 2 (3D data)
Stage 3 (3D data) [512 512]
Stage 4 (3D data) [1021 mixed AR]
[256 x 256]
Batch Size
# Steps
Batch Size # Steps
Batch Size
# Steps
Batch Size # Steps
Kaleido-Small
1024
700K
1024
300K
256
100K
256
100K
Kaleido-Medium
1024
700K
1024
300K
256
100K
256
100K
Kaleido
2048
700K
2048
500K
256
100K
256
100K
Table 6: Kaleido Training Pipeline. Kaleido's 's training follows a multi-stage curriculum. The model is first pre-trained large-scale video dataset and is then fine-tuned combined multi- view 3D datasets, with the image resolution progressively increased from 256px up to 1024px. In the final stage, we sample images with mixed aspect ratios enable flexible resolution generation. Larger batch sizes are used for our largest Kaleido model to validate scaling laws.
Evaluation Strategy We evaluate Kaleido on both view synthesis (Section 4.1) and 3D recon- struction benchmarks (Section 4.2). All evaluation datasets were held out and not used during model training- For all experiments, we report the zero-shot performance of Kaleido without any per-dataset fine-tuning. Unless otherwise specified, all generations use classifier-free guidance scale of 1.5. To ensure a fair comparison with prior work, the frame interpolation model is not used in these evaluations. In NVS benchmarking, we match our model's resolution to the baselines, us- ing our 256px checkpoint against EscherNet (evaluated at 256px) and our 512px checkpoint against SV3D and SEVA (both evaluated at 576px). For single-view evaluations address scale ambiguity by sweeping camera translations along each and all axes (from 0.1 to 2.0) and reporting the best result, following the protocol of SEVA.
scene-level datasets,
22

---

Published as a conference paper at ICLR 2026
F ADDITIONAL RESULTS FOR NOVEL VIEW SYNTHESIS
We provide additional quantitative metrics for our few-view NVS benchmarks shown in in Table and additional quantitative examples for our many-view NVS comparisons shown in Fig. 9,
RIMV
Mip-NeRR
Temples
Ref. Views
2
10
6
Eval. Duta Type
Object
Object
Multi Object 256
Scene
Seene
Scone 512
Eval. Resoluton
$12
255
512
512
Eval. Tar. Views
20
10 EischerNctt
27
35
SoLA Model
SV311
EscherNet
SEVA
SEVA
SEEV
Results (I
0.158
0.095
0.064
0.052 0.043 0.036 0410 0.301 0.258 0.222 0.185 0.389
0.181 0.573
0.364
0.319 0.571
0.463 0.587
0.328
Kaleido-Small
0.144
0.125
D.04
0.029
0.019
0.332
0.204
0.166
0.130
0.323
0.528
0.318
0.385
0.325
Kaleido-Medium
0.289
0.363
Results (SSIMT)
0,80
0,884
0.988
00918
0.927
0.355
0.518
0.585
0.611
0,633
0.657
0,384
0.002
0,282
0.377
0.395
0,342
0.385
0.427 0,452
Kalrido-Smalll
0.873
0.867
0.938
0.954
0.584
0.670
0.703
0.341
0.221
0.113
0.362
0.313
0403
0.444
Kaltido-Medium
0.530
0.885
0.33
0.948
0.963
0.75
0.59]
0.697
0.231
0.778
0.827
0.359
0.645
0.271
0.347
0.400
0.35]
0.359
0.419
0.459
Kaleido
0.531
0.895
0.988
0.954
0,966 0.988 0.610
0.704
0.738
0.781
0.836
0.375
0.639
0.248
0.361
0.433
0,333
0.368
0.429 0.479
Table Zero-shot SSIM/LPIPS Performance with Generative Methods. Kaleido achieves state- of-the-art performance across all object- and scene-level benchmarks, with SSIM and LPIPS metrics consistent with the superior PSNR performance reported Table
Iestant-NGD [S Views]
3DGS Views]
EscherNet [5 Views]
Kaleido (5 Views]
Ground Truth
DGs [10 Views]
3DGS 110 Views!
SEVA 110 Views]
Kaleido [10 Views]
Ground Ground-Truth
Figure 9: Qualitative Comparison on NeRF- (256px, top) and LLFF (512px, bottom). With more reference views, Kaleido demonstrates superior rendering precision compared other generative baselines, with more accurate texture details and pose alignment. Furthermore, it avoids the representation-basse artefacts sometimes present in the per-scene optimisation methods, high- lighting the robustness of its learned, data-driven prior.
23

---

Published as a conference paper at ICLR 2026
G ADDITIONAL VISUALISATIONS ON 3D RECONSTRUCTIONS
We include additional visualisation of our reconstructed object meshes in Fig. 10. The results show that Kaleido's s generated views lead to significantly higher-quality meshes. Notably, at 1024px reso- lution, Kaleido's 's reconstructions is nearly close to the ground-truth, capturing fine geometric details and producing sharp, realistic textures.
Referece Images
EscherNei
Kaleido
Kaleido [1024 Res.]
Ground Truth
12
11
10
2
3
9
4
4
6
5
7
6
5
Figure 10: Visualisation of 3D Reconstructions with 3 Reference Views. Kaleido's precise ren- derings enable high-fidelity 3D mesh reconstruction using NeuS2. When leveraging 1024px render- ings, the resulting meshes exhibit incredibly detailed textures, accurately capturing fine features like the numbers on the clock and the intricate patterns on the backpack.
24

---

Published as a conference paper at ICLR 2026
H ADDITIONAL RESULTS FOR CAMERA-CONDITIONE VIDEO GENERATION
Kaleido leverages large-scale video data improve novel view synthesis, a strategy that is concep- tually related to recent camera-conditioned video generation models. While Kaleido is designed for distinct problem of "any-to-any" spatial view synthesis, there an overlap in capabilities when the task is restricted to specific setting: generating video sequence along continuous target tra- jectory from a single reference view. this section, we compare Kaleido against video generation models within this shared domain.
Baselines and Datasets In Table 8, we compare Kaleido with a range of state-of-the-art camera- conditioned video models: Wonderland (Liang al., 2025), ViewCrafter (Yu et al., 2025c), VD3D Bahmani et al. (2025), and MotionCtrl (Wang et al., 2024b). We evaluate on two chal- lenging in-the-will datasets: DL3DV-140 (Ling al., 2024) and Tanks and Temples (Knapitsch et al., 2017), strictly following the evaluation protocol described in Wonderland:
D133V-140: We randomly select 300 video clips from the test set. For each video, the first frame serves as the reference image, and the subsequent n camera poses are used as conditions. set n = 48 to align with the Wonderland's setting.
Tanks and Temples: We randomly sample 100 video clips from all 14 scenes using the same strategy.
Since both datasets lack dense pose annotations for every frame, we use COLMAP (Schönberger & Frahm, 2016; Schönberger et al., 2016) to generate ground-truth poses for the source videos.
Evaluation Metrics
We evaluate performance using two criteria, following Wonderland's ' setting:
We measure Rotation error (Rerr) and Translation error (Terr). To compute this, we estimate the camera poses of the generated videos using COLMAP, align the coordinate system relative the first frame, and normalise the scale for comparison against the input condition.
Caidan-Guidance Precision:
Visual Similarity: We assess image quality using PSNR, SSIM, and LPIPS between the gen- erated frames and the ground-truth views. For reliable comparison, we evaluate these metrics over the first 14 frames. This is due the generated content naturally diverges from the ground truth as the scene progresses, rendering pixel-wise metrics less indicative generation quality for longer sequences.
Results As shown in the Table 8, Kaleido clearly outperforms all camera camera-conditionn conditioned video mod- els. The improvement particularly significant in camera precision. Despite the baseline models being specialised for video generation along continuous trajectories, Kaleido has been shown to ad- here to the target camera path with much higher accuracy. This highlights Kalcido's 's generality and superior rendering precision,
when applied to tasks outside its primary design scope.
Table 8: Comparison with Condtr--Cndd Video Models. We evaluate camera precision (Revr. Ter.) and visual similarity (LPIPS, PSNR, SSIM) against state-of-the-art video generation baselines. Kaleido significantly outperforms all video models in camera precision, indicating supe- rior geometric consistency, while maintaining visual quality on par with the state-of- state-of-the-art This demonstrates Kaleido's robust generalisation, even compared models specifically designed for continuous video trajectory generation.
Dataset
Metrics
Method
Beark
Terrl
LPIPSI
PSNRT
SSIMT
DL3DV-140
MotionCtrl
0.467
1.114
0.309
14.35
0.385
VD3D
0.094
0.237
0.259
16.28
0.487
ViewCrafter
0.092
0.243
0.237
17.10
0.519
Wonderland
0.061
0.130
0,218
17.56
0.543
Kaleido
0.011
0.026
0.232
18.09
0.544
Tanks and Temple
MotionCtrl
0.834
1.501
0,312
14.58
0.386
VD3D
0.117
0.292
0,284
15.35
0.467
ViewCrafter
0.125
0.306
0.245
16.20
0.506
Wonderland
0.094
0.172
0.221
16.87
0.529
Kaleido
0.016
0.086
0.197
17.97
0.528
25

---

Published as a conference paper at ICLR 2026
I LIMITATIONS AND FUTURE WORK
Despite its strong performance, Kaleido has several limitations that open exciting avenues for future research:
Fixed Camera Intrinsics Kaleido currently does not model camera intrinsics, which prevents from generating effects like dolly-zooms, capability present in models like SEVA (Zhou et al., 2025). Future work could explore incorporating intrinsic parameterisation, potentially through an- other form of RoPE-based positional encoding designs (Li et al., 2025), allow for more flexible camera control.
Degraded Generations with Large Viewpoint Changes While Kaleido maintains excellent spa- tial consistency. its generated views can sometimes lack semantic plausibility when the viewpoint change extreme. This suggests that while video pre-training builds a strong geometric foundation, it may not provide the diverse semantic knowledge required for high-fidelity single-image realism. Integrating priors from large-scale -ittto-imaae/vviee models could be a promising direction to address this limitation.
Towards Faster Rendering Kaleido's generation time scales with the number of input views, and it is far from real-time. To fully bridge the gap with efficient scene-specific methods like 3D Gaussian Splatting, future work will focus on improving inference speed through techniques like step distillation or architectural optimisations.
Towards 4D Generation Our unified positional encoding for space and time provides a natu- ral foundation for true 4D generation. A promising future direction to extend Kaleido to pre- cisely control scenes across both space and time, enabling generative modelling of dynamic, four- dimensional worlds.
26


Reviews:
[{"review_id": "05xxcZktqm", "reviewer": "Reviewer_F7xZ", "summary": "Method Kaleido solved the object- and scene- level synthesis through the sequence-to-sequence generation technology. And they mainly borrowed the video generation model to achieve this and assisted some structural designs like activation function to improve the performance.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The results show that this method have advantage in both few- and many- view settings, and they can match the performance of classical 3D representations under the many-view setting.\n\n- This paper explored many detailed model designs including the positional encoding and architecture designs through ablation studies.", "weaknesses": "- Adopting the video generation to achieve novel view synthesis is a common solution, and many methods still adopt the video pretrained model to improve the performance. So the contribution on the video pretrained model is unconvincing.\n\n- Once this model unified the 3D and video, so how about the performance on the 4D scene.\n\n- The demo results still have obvious inconsistency in the generated novel view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRWhcDywet", "reviewer": "Reviewer_Q5DW", "summary": "This paper presents a sequence-to-sequence method to achieve multi-view image synthesis, and they construct the pipeline from the positional encoding to the video pretraining. The final results show that this model has advantage in both single or multiple view input.", "questions": "As mentioned in the Strengths and Weaknesses, I am inclined to give a borderline-accept score; however, I would be happy to raise it if the authors address these main concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n\n- Proposed the unified positional encoding to seamlessly represent multi-view and temporal positions.\n\n- The presentation structure is reasonable and most design modules have corresponding ablation studies.\n\n- The results with single or multiple view inputs show the advantage in both 3D and novel view synthesis.", "weaknesses": "Weakness:\n\n- Missing the discussion on the running time and memory consumption (especially with the classical reconstruction representations like 3DGS). This kind of generation methods has the advantage on the generation of the nun-seen part, but it still needs huge memory consumption and long running time.\n\n- From the proposed demos, the generated results still have obvious ghosting and artifacts (e.g., the generation attempts at the same location is inconsistent), this is still the inherent disadvantage of this kind of generation methods.\n\n- How about the results when the input views and the target views have large difference in perspective? Maybe this challenging situation can further prove the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4DyWaem0d", "reviewer": "Reviewer_xBaB", "summary": "This paper presented a method to generate novel view images from input views using the sequence-to-sequence generative model. They conducted numerous validate experiments to design the final model, such as the positional encodding, model architecture, and video pretraining, which can achieve the certain consistent view synthesis from both sparse and many view conditions.", "questions": "Missing the information of the memory and runtime comparisons with both previous generative methods and reconstruction methods like 3DGS, and how many input views can be processed at most and the corresponding runtime time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The model design of this paper is motivated by many ablation studies. This paper designed the model mainly from five aspects, including the positional encodding, activation function and so on, and most designs have the corresponding ablation studies.\n2. This paper conducted both 3D reconstruction and novel view synthesize comparisons with existing methods and show its advantage.\n3. The results with many-view setting show that it can achieve more consistent results compared with existing methods.", "weaknesses": "1. Adopting the generative model to synthesis the novel view images is a common solution and is just one posible solution to achieve the final reconstruction of 3D object or scene. It has clear advantage in challenging situation like generating the non-seen views, but on the contraty, it has clear shortcoming in the consistency rendering results and efficiency.\n2. To my knowledge, many previous methods have treated the multi-view synthesis as the video generation, and they can also generative any number of target views conditioned by any number of input views. And the declared novelty in this field is not clear to me. \n3. In the many-view setting, seems like luck the experiments with long sequence where the inputs not just have numerous views but have long trajectory like SEVA.\n4. Most of the technologies used may not seem novel, but the key is to choose the right combination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHCnxFgnx9", "reviewer": "Reviewer_7YYk", "summary": "This paper proposes Kaleido, a diffusion transformer for neural rendering. The model is jointly trained on 2D video and 3D data to take image(s) and output images from novel viewpoints. The paper proposes a new positional encoding to facilitate joint training on 2D video and 3D data.", "questions": "I am not very convinced by the paper. It makes a big deal out of jointly training on 2D and 3D data, but it is not clear what the advantage or novelty of this work is. There are many camera-controlled video diffusion models that can just be used for joint fine-tuning. The architecture of Kaleido looks very similar to that of a regular diffusion transformer.\n\nI would like authors to address the following questions:\n\n- How does this compare to camera-controlled video diffusion models qualitatively and quantitatively? And why would someone not just jointly fine-tune a video model on 2D video data and multi-view data?\n- Why are the viewpoint changes limited so much?\n- Why is the geometry often flat?\n\nI am not very optimistic that I can be convinced to accept this paper, since it is missing critical comparisons and the results are not very convincing. Moreover, the storyline is not convincing to motivate why we need this from-scratch-trained model. But still happy to see a rebuttal.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- New positional encoding layer: This work combines RoPE for 2D video and 3D to jointly train on both data modalities.", "weaknesses": "- Limited viewpoint changes: The synthesized videos don’t show very large viewpoint change or show any synthesized “new” content not available in the input. It seems like the method requires many images to increase the viewpoint change. Though, methods like CAT3D can synthesize large viewpoint changes from a single image. Moreover, recent camera-controlled video models can synthesize very long trajectories.\n- Flat geometry: Most generated trajectories have very flat geometry without good depth.\n- Missing comparisons with camera-controlled video diffusion: This work looks conceptually similar to camera-controlled video diffusion models but comparisons with those kind of works are missing and the key differences in the approach are not clear. Moreover, in that case it would be possible to just use MegaSaM [1] or ViPE [2] to annotate all 2D videos with camera poses and directly train on large-scale video data for novel view synthesis.\n- Simple datasets: The model is trained with limited scale 3D datasets. Is it a current issue that the model is not cross-generalizing from the synthetic 3D datasets to the video dataset distribution?\n\n[1] Li et al., MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos, CVPR 2025 \\\n[2] Huang et al., ViPE: Video Pose Engine for 3D Geometric Perception, arXiv 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)