4K4DGen: Panoramic 4D Generation at 4K Resolution
Authors: Renjie Li
Panwang Pan
Bangbang Yang
Dejia Xu
Shijie Zhou
Xuanyang Zhang
Zeming Li
Achuta Kadambi
Zhangyang Wang
Zhengzhong Tu
Zhiwen Fan
Venue: ICLR 2025 Spotlight (2025)
Keywords: 4D Generation
Panoramic Video
Panoramic Gaussian Splatting
Primary area: generative models

Abstract: The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the requirements of VR/AR applications that need free-viewpoint, 360$^{\circ}$ virtual views where users can move in all directions. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360$^{\circ}$ views at 4K (4096 $\times$ 2048) resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of 3D Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360$^{\circ}$ images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we propose Dynamic Panoramic Lifting to elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of 4K for the first time. Project page: https://4k4dgen.github.io/.

Published as a conference paper at ICLR 2025
4K4DGEN:
PANORAMIC 4D GENERATION AT 4K RESOLUTION
Renjie Li Panwang Pan Bangbang Yang" Dejia Xu* Shijie Zhou", Xuanyang Zhang', Zeming Li², Achuta Kadambi Zhangyang Wang", Zhengzhong Tu?, Zhiwen Fan² Byttedance, University of Texas Austin, University of California, Los Angeles,
Texas A&M University
https: / /444dgen. github io/
paulpanwang@gmai.. com
Novel View
Novel View
Novel View #3
Novel View
Panoramic 4D Generation at 4K Resolution
Figure 1: 4K4DGen takes static panoramic image with resolution of 4096x2044 and allows animation through user interaction or an input mask, transforming the static panorama into dynamic Gaussian Splatting. 4K4DGen supports the rendering of novel views at various timestamps, enriching immersive virtual exploration.
ABSTRACT
The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from single perspective image, failing meet the requirements of VR/AR applications that need free- viewpoie 360 virtual views where users can move in all directions. In this work, we tackle the challenging task of elevating single panorama to immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360 views at 4K (4096 2048) resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set 3D Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scale annotated 4D data and models, especially in panoramic formats, we propose novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360 images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, elevate the panoramic video into 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, generation at a resolution of 4K for the first time.
propose Dynamic Panoramic Lifting
achieve high-quality Panorama-to-4D
Equal contribution t: Project lead; Corresponding author.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
With the increasing growth of generative techniques (Rombach et al., 2022; Blattmann et al., 2023a), the capability to create high-quality assets has the potential to revolutionize content creation across VR/AR and other spatial computing platforms. Unlike 2D displays such as smartphones or tablets, ideal VR/AR content must deliver an immersive and seamless experience, enabling 6-DoF virtual tours and supporting nigh-resolution 4D environments with omnidirectional viewing capabilitiess. De- spite significant advancements in the generation of images, videos, and 3D models, the development of panoramic 4D content has lagged, primarily due the scarcity of well-annotated, high-quality 4D training data. Even in the most relevant field of 4D generation, existing works mainly focus on generating or compositing object-level contents (Bahmani et al., 2024; Lin et al., 2024), which are often in low-resolution (e.g., below 1080p) and cannot fulfill the demand of qualified immersive experiences. Based on these observations, we propose that an ideal generative tool for creating immersive environments should possess the following properties: (i) the generated content should exhibit high perceptual quality, reaching high-resolution (4K) output with dynamic elements (4D); (ii) the 4D representation must be capable of rendering coherent, continuous, and seamless 360 panoramic views in real time, supporting efficient 6-DoF virtual tours. However, creating diverse, high-quality 4D panoramic assets presents two significant challenges: (i) the scarcity of large-scale, annotated 4D data, particularly in panoramic formats, limits the training of specialized models. (ii) achieving both fine-grained local details and global coherence in 4D and 4K panoramic views is difficult for existing 2D diffusion models. These models, typically trained on perspective images with narrow fields of view (FoV), cannot be easily adapted to the expansive scopes of large panoramic images (see Exp. 4.3). On another front, video diffusion models (An et al., 2023) trained with web-scale multi-modal data have demonstrated versatile utility as region-based dynamic priors, and Gaussian Splatting (Kerbl et al., 2023) has shown efficient capabilities in modeling 4D environment. Thus, we address the large-scale, omnidirectional dynamic scene generation (4D panoramic gener- ation) problem by utilizing the generative power of diffusion models to animate static panoramic images, transforming them into realistic, dynamic scenes that can support immersive, 360 viewing experiences. To achieve this, we propose to elevate the dynamic panoramic video to 4D environment assets using a set of dynamic Gaussians, which can be seamlessly integrated into VR/AR platforms for real-time rendering and interaction.
In this paper, we introduce 4K4DGen, a novel framework designed to enable the creation of panoramic 4D environments at resolutions up to 4K. 4K4DGen addresses the key challenges of maintaining consistent object dynamics across the entire 360 field-of-view (FoV) in panoramic videos, while preserving both spatial and temporal coherence the video transitions into fully interactive 4D en- vironment. Specifically, we propose the Panoramic Denoiser, which animates 360 FoV panoramic images by denoising spherical latent codes corresponding to user-interacted regions. The Panoramic Denoiser leverages a well-trained diffusion model originally designed for narrow-FoV perspective images, enabling the generation of 360 dynamic panoramas while ensuring global coherence and continuity throughout the entire panorama. To transform the omnidirectional panoramic video into 4D environment, we introduce Dynamic Panoramic Lifting, which corrects scale discrepancies using a depth estimator enriched with perspective prior knowledge to generate panoramic depth maps. Additionally, employs dimeddepeneent 3D Gaussians optimized with spatial-temporal geometry aligmment to ensure cross-frame consistency in dynamic scene representation and rendering. By adapting generie 2D statistical patterns from the perspective domain to the panoramic format and effectively regularizing Gaussian optimization with geometric principles, we achieve high-quality 4K panorama-tu -o-aaam--tt-DDD-- content generation with photorealistic novel-view synthesis capabilities. Our contributions can be summarized as follows.
We introduce 4K4DGen. the first framework capable of generating high-resolution (up to 4096x2048) 4D omnidirectional assets without the need for annotated 4D data.
We propose the Panoramic Denoiser, which transfers generative priors from pre-trained 2D perspective diffusion models to the panoramic space, enabling consistent animation of panoramas with dynamic scene elements.
We introduce Dynamic Panoramic Lifting, method that transforms dynamic panoramic videos into dynamic Gaussians, incorporating spatial-temporal regularization to ensure cross cross-frame consistency and coherence.
2

---

Published as a conference paper at ICLR 2025
Per Bota Kiewwe Rownbdeg
(4i Passarsic Dencier Roeds
360' Panomma
Giterative
Framme A
Figure 2: Panoramic Denoiser adapts diffusion priors from the perspective domain to the panoramic domain by simultancously denoising perspective views and integrating them into spherical latents at each denoising step. This approach ensures consistent animation across multiple views.
RELATED WORK
Diffusion-bassee Image and Video Generation. Recent advancements have significantly expanded the capabilittiss of generating 2D images using diffusion models, as evidenced in several studies (Dhari- wal & Nichol, 2021; Nichol et al., 2021; Podell al., 2024; Ramesh et al., 2022; Saharia al., 2022). Notably, Stable Diffusion (Rombach et al., 2022) optimizes diffusion models (DMs) within the latent spaces of autoencoders, striking an effective balance between computational efficiency and high image quality. Beyond text conditioning, there is increasing emphasis on integrating additional control signals for more precise image generation (Mou al., 2024; Zhang et al., 2023). For example, ControlNet (Zhang et al., 2023) enhances the Stable Diffusion encoder to seamlessly incorporate these signals. Furthermore, the generation of images with consistent perspective views gaining attention, such the training-based techniques like (Tang et al., 2023; Höllein et al., 2024), or the sampling-basee techniques like (Song et al., 2023; Bar- Tal et al., 2023; Lee et al., 2023; Quattrini et al., 2025). Diffusion models are also extensively applied in video generation, as demonstrated by various recent works (Ge et al., 2023; Ho et al., 2022; Wang et al., 2023a; Wu et al., 2023b; 2024b; Zhou al., 2022). For instance, Imagen Video (Ho al., 2022) utilizes series of video diffusion models to generate videos from textual descriptions. Similarly, Make-A- LateeAAVicee -ideo (Singer et al., 2023)
advances diffusion-based text-to-image model to create videos without requiring paired text-video data. Magic Video (Zhou et al., 2022) employs frame-wise adaptors and a causal temporal attention module for text-to-video synthesis. Video Latent Diffusion Model (VLDM) (Blattmann et al., 2023b) incorporates temporal layers into 2D diffusion model to generate temporally coherent videos.
3D/4D Large-scale Generation. In recent 3D computer vision, a large-scale scene is usually represented as implicit or explicit fields for its appearance (Mildenhall et al., 2020; Kerbl et al., 2023), geometry (Peng et al., 2020; Wang et al., 2023b; Huang et al., 2023), and semantics (Kerr et al., 2023; Zhou et al., 2024a; Qin et al., 2024). We mainly discuss the 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) based generation here. Several works including DreamGaussian (Tang et al., 2024), Dreamer (Yi et al., 2024), GSGEN (Chen et al., 2023), CG3D (Vilesov et al., 2023), and DiffSplat (Lin al., 2025) employ 3DGS to generate diverse 3D objects and lay the foundations for compositionality, while LucidDreamer (Chung et al., 2023), Text2lmmersion (Ouyang et al., 2023), GALA3D (Zhou al., 2024c). RealmDreamer (Shriram al., 2024), and DreamScene360 (Zhou et al., 2024b) aim to generate static large-scale 3D scenes from text. Considering the current advancements in 3D generation, investigations into 4D generation using 3DGS representation have also been conducted. DreamGaussian4D (Ren et al., 2024) accomplishes 4D generation based on a reference image. AYG (Ling et al., 2023) equips 3DGS with dynamic capabilities through deformation network for text-to-4D generation. Besides, Efficient4D (Pan et al., 2024) and 4DGen (Yin et al., 2023) explore video-to-4D generation, and utilize SyncDreamer (Liu et al., 2023) to produce multi-view images from input frames pseudo ground truth for training dynamic 3DGS. 4K4D (Xu et al., 2024) is righrresolution reconstruction technique that extends 3DGS to model complex human motion with detailed backgrounds while achieving real-time rendering speed.
3

---

Published as a conference paper at ICLR 2025
Panoramic Representation. A panorama is image that captures wide, unbroken view of an area, typically encompassing a field of vision much wider than what a standard photo would cover, providing more immersive representation of the subject. Recently, novel view synthesis using panoramic representation has been widely explored. For instance, PERF (Wang et al., 2024a) trains panoramic neural radiance field from single panorama to synthesize 360 novel views. 360Roam (Huang et al., 2022) proposed learning an omnidirectional neural radiance field and progres- sively estimating 3D probabilistic occupancy map to speed up volume rendering. OmniNeRF (Gu et al., 2022) introduced an end-to-end framework for training NeRF using only 360 RGB images and their approximate poses. PanoHDR-NeHE (Gera et al., 2022) learns the full HDR radiance field from a low dynamic range (LDR) omnidirectional video by freely moving standard camera around. In the realm of 3DGS, 360-GS (Bai et al., 2024) takes panorama images and 2D room layouts as scene priors to reconstruct the panoramic Gaussian radiance field. DreamScene360 (Zhou et al., 2024b) achieves text-to-3D Panoramic Gaussian Splatting by utilizing monocular depth priors to regularize the Gaussian optimization.
METHODOLOGY
Taking a single panoramic image as input, the goal of 4K4DGen is to generate a panoramic 4D environment capable of rendering novel views from arbitrary angles and at various timestamps. Our approach initially constructs panoramic video and then elevates it into series of 3D Gaussians, enabling efficient splatting for flexible rendering. Naïvely animating projected perspective images, however, often results in unnatural motion and inconsistent animations. To overcome this, our method propose the denoising of projected spherical latents, ensuring consistent animation of the panoramic video from the original image, detailed in Sec. 3.3.
Moreover, directly converting multiple perspective images from different timestamps into 4D fre- quently leads degraded geometry and visible artifacts (see Sec. 4.3). We address this by applying spatial-temporal geometry fusion to lift the panoramic video, as described in Sec. 3.4. The complete pipeline of 4K4DGen is illustrated in Fig. 3.
Animating
lepet 4k 361 Pasmara
" D'mppatii (fragee Latenti Pairs
Propective Images o
Bensie:
Despctivv Latees
Fanoramic 40 Gaamone
Readeal video el sand viows
Spatal Tempon
Alignrazt
Lifting
Figure 3: Overall Pipeline. Beginning with a static panorama as input, the Animating Phase generates panoramic video by first mapping the panorama into spherical latent space, followed by denoising within the perspective space, fusing back to the spherical latent space at each step, and finally transforming it into the panoramic space. In the 4D Lifting Phase, series of dynamic Gaussians is employed to lift the panoramic video into a 4D representation, ensuring both spatial and temporal consistency.
4

---

Published as a conference paper at ICLR 2025
3.1 PRELIMINARIES
Latent Diffusion Models (LDMs). LDMs (Rombach et al., 2022) consist of a forward procedure q and backward procedure p. The forward procedure gradually introduces noise into the initial latent code E pxxuxx, where 2o = E() is the latent code of image I within the latent space of a VAE, denoted by E. Given the latent code at step - 1, the g procedure described as q (ft-1) = (22; v - 1, Be I). Conversely, the backward procedure p. aimed at progressively removing noise, defined as po ( (2+) = N (1o(( t), Eo (2+, t)). In practical applications, images are generated under the condition y. by progressively sampling from JT down to zo. Recently, image-to- video (12V) generation has been realized (Guo et al., 2024; Dai et al., 2023) by extending the latent code with an additional frame dimension and performing decoding at each frame. The denoising procedure is succinctly represented as zt-1 = (z I). where 24, It-1 E R (xxxx wxxc represent the sampled latent codes and / the conditioning image. Recently, image-to-viee video (I2V) generation has been achieved (Guo et al., 2024; Dai et al., 2023) by extending the latent code with an additional frame dimension and performing decoding each frame. The denoising procedure is succinetly expressed as Tt-1 = ( 1). where Ft. 2t-1 E Rxxhxxxx represent the sampled latent codes, and I represents the conditioning image.
Omnidirectional Panoramic Representation. Panoramic images or videos, denoted I, are typically represented using equirectangular projections, forming an H x W x C' matrix, where H and W denote the image resolution and C' represents the number of channels. While this format preserves the matrix structure, making it consistent with planar images captured by conventional cameras, introduces distortions, especially noticeable near the polar regions of the projection. To mitigate these distortions, we adopt a spherical representation for panoramas, where pixel values are defined on a sphere S {d (I,,), y, =), y, = R [d] 1}. For more precise definition of the projection, we represent matrix-like like images using a mapping E₁ [-1, + RCC which normalizes the image coordinates into the range [0, 1]. Thus, for any given pixel (z, y) E [-1, 1J², the corresponding pixel value is determined by & (ar, y). We define the spherical representation of panoramas using the field Sy 2 -> TRC,, where Sy (dl gives the pixel value given direction (x. y, z). The relationship between the spherical and equirectangular representations is established through the following projection formula:
Si (I, (x, E, arccos
y VI- -
aresin 2
(I)
For perspective images, we define a virtual camera centered at the origin. The rays for each pixel determined through ray casting, as described in (Mildenhall al., 2020), where each ray d is represented by r(z, y, f, u. 8, R). This representation takes into account the focal length f. the z-axis direction u, the image plane size and the camera's 's rotation along the z-axis R. Consequently, for a given panorama I, the perspective image can be projected using these camera parameters (f, u, s, R) as:
Ep (z S₁ o (x,y, s, R).
In this paper, we fix the focal length f. the image plane size and the rotation R. We denote the process of projecting the panorama into perspective image i, based on the camera's z-axis direction u, as i (1)
3.2 INCONSISTENT PERSPECTIVE ANIMATION
Large-scale pre-trained 2D models have shown remarkable generative capabilities in creating images and videos, benefiting from vast multi-modal training data gathered from the Internet. However, acquiring high-quality 4D training data is considerably more challenging, and no current 4D dataset reaches the scale of those available for images and videos. Therefore, our approach aims to utilize the capabilities of video generative models to produce consistent panoramic 360 videos, which are then elevated to 4D. Nonetheless, the availability of panoramic videos is significantly more limited compared to planar perspective videos. Consequently, mainstream image-to-video (I2V) animation techniques may not perform optimally for panoramic formats, and the resolution of the videos remains constrained, as illustrated in Fig. 5 (b) and Tab. 2. Alternatively, the animator can be applied to perspective images. but this introduces inconsistencies across different projected views, as depicted in Fig. (c)
5

---

Published as a conference paper at ICLR 2025
3.3 CONSISTENT PANORAMIC ANIMATION
Limited by the scarcity of 4D training data in panoramic format, and given that large diffusion models are primarily trained on planar perspective videos, directly applying 2D perspective denoisers presents challenges in generating seamless panoramic videos with proper equirectangular projection, due to inconsistent motion across different views and the domain gap between spherical and perspective spaces. This constraint has driven to develop panoramic video generator in spherical space that leverages priors from general image-to-video (I2V) animation techniques, as shown in Fig. 2. Consequently, starting from static input panorama, we animate it into panoramic video, as demonstrated in the "Animating Phase" section of Fig. 3.
Spherical Latent Space. To generate panoramic video from a static panorama, we build up the poroois-ii-latnt--spee schema (An al., 2023; Blattmann al., 2023a; Dai et al., 2023) in a spherical context. For general video generation, noisy latent sample is progressively denoised using DDPM (Ho et al., 2020), conditioned on a static input image, and subsequently decoded into a video sequence by a pre-trained VAF decoder. However, in 4K4DGen, unlike the method for generating perspective planar videos, both the latent code and the static panorama input are represented on spheres. We start with the initial panoramic latent code S S - RLxe, where L denotes the number of video frames and c the channels per frame, A novel Panoramic Denoiser is then applied to generate the clean panoramic latent code 5 conditioned on the static input panorama I RR xW Subsequently, the equirectangular projection, as introduced in Sec. 3.1, projects the clean panoramic latent code into the matrix-like latent code zd Bxx with h and w representing the resolution of the latent code. Each hthh video frame 1 in pixel space is decoded by the pre-trained VAE decoder as 1 D (2 :))
Build the Panoramic Denoiser. We leverage a pre-trained perspective video generative model (Dai et al., 2023) to build our Panoramic Denoiser. This video generator takes perspective image i E 1RPxXPM xc and an imitial latent code 2" E IR Xp x(xxe) as inputs, progressively denoising the latent code 2T' clean state 2 through a denoising function z²-1 ( i). Here, Ps and pw represents the resolution of the latent code, PH and pw the resolution of the conditioning image, e the number of channels, and L the video length. Our goal to transform the initial noisy panoramic latent code 5 into the clean state 5 ensuring that each perspective view is appropriately animated while maintaining global consistency. The underlying intuition is that each perspective view undergoes its respective denoising process, the perspective video will feature meaningful animation. Moreover, two perspective views overlap, they will align with each other (Jiménez, 2023; Bar- Bar-Tal& et al., 2023; Lugmayr et al., 2022) produce seamless global animation.
Given a static input panorama I and an initial spherical latent code 5 52 + Lxe we progressively remove noise employing project-and-fie and-fuee procedure each denoising step. Specifically, the spherical latent code at the fth denoising step, St S + Lxx is projected into multiple perspective latent codes Z = {zj. .i...., ......... J. where each o = (S* dz) E RPs xp= x (Ixxc) represents the 4th perspective latent code projected in the equirectangular format detailed in Sec. 3.1. Each perspective latent code is then denoised by one step using a pre-trained perspective denoiser, denoted as =f- (< ix), where ia = (I, d&) RPa xe is the perspective conditioning image projected from the panorama I. Subsequently, we optimize the spherical latent code 5t g + RIxe step t 1 by fusing all the denoised perspective latent codes z
z Formally, the denoising procedure at step I, denoted as St = VSS" I). encompasses the following operations:
V (8' argmin Ede (s, d) & ()(St,)), (L, II.
(3)
3.4 DYNAMIC PANORAMIC LIFTING
We define the panoramic video as = 12., 1_),
consisting of L frames. The video is divided into overlapping perspective videos {vo, Vi,..., Un ). each captured from specific camera directions (di,... d J. collectively encompassing the entire span of the panoramic video V. Subsequently, we estimate the geometry of the 4D scene by fusing the depth maps through apatial-temporal geometry alignment. Following this, we describe our methodology for 4D representation and the subsequent rendering procedure.

---

Published as a conference paper at ICLR 2025
Supervision from Spatial-Tempore Geometry Alignment. To transition from 2D video to 3D space, we utilize a monocular depth estimator (Ranftl et al., 2021), inspired by advancements in (Zhou et al., 2024b), to estimate the scene's geometric structure. Nonetheless, depth maps generated for each frame and perspective might lack spatial and temporal consistency. To address this, we implement Opatial-Tempont Geometry Alignment using pre-trained depth estimator 1opxxwx3_42
θ j + IR hx applied to perspective images. Our objective is to amalgamate n perspective depth maps D. ((1(1,, dd)) into cohesive panoramic depth map D for each frame 1&, ensuring spatial and temporal continuity. We express these depth maps as spherical representation sp.....Sh For enhanced optimization, we assign n scale factors o E R and shifting parameters of E RRx each perspective depth map. The comprehensive depth map D is then optimized jointly with these parameters and 3. The formal objective is structured as follows:
s = argmin s
E :E(I.....)
Adepth + facale + AchifffShifft
(4)
where Ldepth = softplus(a ( d4) - (S) + 8/ is the depth supervision term, Lecale +|+||
I|
ll - I1+ || softplus(a (a+) -1ll - the regularize term for and Lehin Lr LTv(()) 118 3,
the regularize term for 3 where LTv is the TV regularization.
4D Representation and Rendering. We represent and render the dynamic scene using T sets of 3D Gaussians. Each set, corresponding to a specific timestamp t, is denoted as G, = This definition aligns with the methods described in (Bah-
{ (P'' qi si ci, of) l 1, , n.. mani et al., 2024), which also provides a fast rasterizer for rendering images based on these Gaussian sets and given camera parameters. Consistent with Sec. 3.1, while the camera intrinsics remain fixed, we parameterize the camera extrinsics through a position E IR3 and an orientation d E S?. The training process is structured in two stages: initially, we directly supervise the 3D Gaussians using the panoramic videos. Let camera extrinsics p- and camera direction d. Let L, denote the bth frame of the panoramic video. We optimize the h Gaussian set G2 using the following objective:
p, d) represent the rasterized image from Gaussian set G. utilizing
[ Crgh + Atemporal Cremporal Asem Csem + AgeoCges
(5)
where the RGB supervision term Ligb = AL1 + (1 ASSIM is the same as 3D-GS (Kerbl et al., 2023), and the temporal regularize term Lremporal written as:
Cremporal IR(G, R(G)-1,0,d:)]
(6)
Then, we adopt the distillation loss and geometric regularization used in (Zhou al., 2024b), the distillation loss is defined as follows: Leem = 1 cos (CLS(R(G,, 0, d.)), CLS(R(G, 8,. d))).. where 5, E [-a,af a is the disturbing vector, CLS(.) the feature extractor such as DINO (Oquab et al., 2023), and cos(;, -> the value of two vectors. The geometric regularization is defined as Cov(R (...........r((ddd..)). (.)(((1d))) where RD is the rendered depth, Cov(..)) (G. )) Var(O()(F.d.)))" the covariance, and Var(-) the variance.
follows: Egeo =
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Implementation Details. For perspective images, we uniformly select 20 directions u on the sphere S² as the z-axis of 20 cameras. In each experiment, the image plane size is set at 0.6 x 0.6, with focal length f = 0.6 and a resolution of 512 x 512. Rotation along the z-axis is kept at zero for all cameras, ensuring that the up-axis for the ith camera aligns with the (O, u, z) plane. During the animating phase, we utilize the perspective denoiser O instantiated as the Animate-anything model (Dai et al., 2023), which fine-tunes the SVD model (Blattmann al., 2023a). In the Spatial-Temporee Geometric Alignment stage of the lifting phase, the depth estimator θ is implemented using MiDaS (Ranftl et al., 2021; Birkl et al., 2023). All experiments are executed on single NVIDIA 100 GPU with 80 GB RAM.
7

---

Published as a conference paper at ICLR 2025
Oupp
Rard
does 5U% sarl dana nads wasmpa alical wdads
the mile, cartle teddiope, paoomi new cler sce wat pultic talad
axit neteckrone, diaid art refectioss morrhh leniscape famay, naur nury forst tock and white ruriiie
BOC DE
cleas. the Hi penc inc., aasa. wibrar colrr summer thamaaio tangd, panorneic pereiactive dear weatee
Figure 4: Comparison between 4K4DGen and AD-Cinemagraphy. We present the input static panorama (Pano RGB), the corresponding text prompts, and the rendered results from different views and at various timestamps. 4K4DGen (Ours) effectively generates 4D scenes that are both spatially and temporally consistent, while AD-Cinemagraphy (3D-Cin.) suffers from ghosting artifacts the middle frames.
Evaluation. As there is no ground truth 4D scene data available, we render videos at specific test camera poses from the synthesized 4D representation and employ non-reference video/image quality assessment methods for quantitative evaluation of our approach. For the test views, we select random cameras with 0 as part of our testing camera set. We then introduce disturbances as described in Sec. 3.4, applying disturbance factor of = 0.05 at these selected views. Datasets, The task of generating 4D panoramas from static panoramas is new, and thus, no pre-existing datasets are available. line with previous large-scale scene generation works (Zhou et al., 2024b; Yu et al., 2024), evaluate methodology using dataset of 16 panoramas generated by text-to-panorant diffusion models (Yang et al., 2024). Baselines. Current SDS-based methods (Wu et al., 2024a; Zhao et al., 2023) are limited generating object-centered assets and do not support outward-facing scene generation. We compare our method with the optical-flowe JD-Cinemagraphy (3D-Cin.) (Li et al., 2023b) (both the "circle" and "zoom-in" mode), by inputting the static panorama and projecting the output onto perspective images. Metrics. It is challenging to evaluate the visual quality without a ground-trute reference. We evaluate the rendered perspective videos regarding both the frame and video visual quality. For frame quality, We employ the LLM- based visual scorer Q-Align (Wu et al., 2023a) (IQ Scorer and IA Scorer) evaluate the quality of individual frames. For video quality, we use the Q-Align video model (VQ) as the quality scorer. Additionally, we conduct user studies to further evaluate the results. In this paper, there are two types of user studies: (I) User Choice (UC), where participants are asked to compare and select the best video from candidates generated by different methods, and (2) User Agreement (UA), where participants assess whether specific properties present in the videos generated by particular approach.
3D dynamic image technique,
4.2 RESULTS
Quantitative Results. We show the quantitative comparison between 4K4DGen and 3D- Cinemagraphy (Li et al., 2023a) in Tab. 1. 4K4DGen consistently achieves better performance in the M.---based Q-Align metric regarding the image quality (IQ), image aesthetic (IA), and the
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with 3D-Cinemagraphhy We compare our method with 3D-Cinemagraphy using rendered images from 4D representations. The IQ, IA, and VQ models represent the image quality scorer, image aesthetic scorer, and video quality scorer, respectively, within the Q-Align assessment framework. Our method, 4K4DGen, consistently achieves superior performance in both image and video quality across these metrics. Furthermore, 4K4DGen performs better our user studies in terms of visual quality (Quality), motion amplitude (Amplitude), and the motion naturalness (Naturalness). Please refer to D.2 for further details.
Method
Q-Align (Q)) 1 Q-Aliga dAJ
Q-Aliga (VQ)
Quality (UC)
Amplitud: (UC)
Naturalness (UC)
Cinemagraphy (zoom-in)
3I>CCinemagrphy (circle)
0:48
0.40
0.58
12%
32.0%%
21. 1%
Ours (holistic pipeline)
0.66
0.44
0.62
81%
38.6%
59.2%
video quality (VQ). Besides, 4K4DGen is preferred by the users considering the video quality, motion amplitude, and motion naturalness.
Qualitative Results. We present qualitative comparison between 4K4DGen and 3D- Cinemagraphy (3D-Cin.) on the rendered images from 4D representations. Since the performance of 3D-Cin. similar under the "circle" and "zoom-in" settings Tab. I, we use the "circle" setting to represent 3D-Cin. in Fig. 4. As shown in the figure, 4K4DGen produces high-quality perspec- tive videos that maintain consistency across both time and views, whereas 3D-Cin. struggles with generating ghosting artifacts in the middle frames.
4.3 ABLATION STUDIES
We conduct ablation studies for both the animating and lifting phases of our methodology. In the animating phase, we perform evaluation on 2D animated videos with different strategies, and highlight the importance of our spherical denoise strategy by replacing it with two basic animation techniques. In the lifting phase, we analyze the impact of excluding the Spatial-Temport Temporal Geometry Alignment process and the temporal loss during the optimization 4D representations.
Animating Phase. For analyzing the strategies in the animating phase, shown in Tab. Q-Align (visual quality scorer), view-consistency (user agreement), motion amplitude (user choice), and motion naturalness (user choice) to evaluate the 2D animated videos. For the details of the user studies, please refer to the Appendix D.2. To animate the panorama into a panoramic video, straightforward approach is to apply animators directly to the entire panorama. However, we observed that this strategy often results in minor motion, as shown in Fig. 5 (b) and Tab. 2 (Animate Pano. with small motion amplitude and less naturalness). This issue arises due two main reasons: (1) animators are typically trained on perspective images with a narrow field of view (FoV), whereas panoramas have a 360 FoV with specific distortions under the equirectangular projection; (2) our panorama is high-resolution (4K), which exceeds the training distribution of most 2D animators and can easily cause out-of of--ooo-ommoy issues, even with an 80GB VRAM graphics card. Thus the
View1
View2
View1
View2
View1
View2
Spherical Denoiser
(b) Animate Panorama
(c) Animate Pers. Image (inconestien Morion)
(Consstent Sufficient Modio))
(Small Merion)
Figure 5: Comparison to Different Animators: Animators trained primarily on perspective images tend to produce limited motion when applied to panoramas, and the resolution may be limited. On the other hand, animating perspective images individually can lead to inconsistencies between overlapping views.

---

Published as a conference paper at ICLR 2025
t=0
0.5
1=0.5
Mella
(4) mylt
HallM
Figure 6: Ablating Lifting Phase: (Left) The w/o Temmp variant (column d) produces renderings with flashing stripes. Zoomed-in details of the flashing stripe region are highlighted in (e). (Right) Without spatial-temporal geometry alignment, the geometry in the smoke area of the volcano for w/o STA variant (column g) appears less consistent compared to the full model (column f).
panoramas have to be down-sampled to a lower resolution (2K), causing a loss of details. To this end, seek animate on perspective views. Applying the animator perspective views offers benefits such as reduced distortion and inputs that suit the domain of the animator, allowing for smooth animation of high-resolution panoramas. However, animating perspective images separately can introduce inconsistencies hetween overlapping perspective views, as illustrated in Fig. 5 (c) and Tab. (Animate Pers.). To resolve this challenge, we propose simultaneously denoising all perspective views and fusing them at each denoising step. in the spherical latent space, which capitalizes on the benefits of animating perspective views while ensuring cross-viis riess-view consistency. The results are displayed in Fig. 5 (a) and Tab. 2.
Lifting Phase. We conduct ablation studies on the Temporall Geometry Alignment (STA) module and the temporal loss during the lifting phase, as shown Fig. 6.
Table 2: Different Animation Strategies the Animating Phase. We analyze the efficacy of animation strategies by evaluating the animated 2D videos in different ways. Animating the entire panorama results in worse motion and reduced resolution (first row), as indicated by the Amplitude and Naturalness metrie. Conversely, animating from perspective views leads to inconsistencies across different views (second row), as supported by the Q-Align metric and the "View-consistency (UA)" study. 4K4DGen capitalizes the generative ability from perspective animating priors while enabling cross-view consistent motion between different perspectives, which achieves the best motion naturalness and amplitudes among all the settings (third row).
Animator
Max Pano. Res.
Align (VQ)
View consistency (UA)f Amplitude (UC)
Naturalness (UC)
Animate Pano.
20-88 1021
0.82
26.8%
17.8%
Animate Pers.
4096 2018
0.64
33%
32.4%
39.3%
(Animating Phase)
0.85
40.8%
42.9%
5 CONCLUSION
Conclusion. We have proposed a novel framework 4K4DGen, allowing users to create high-quality 4K panoramic 4D content using text prompts, which delivers immersive virtual touring experiences. To achieve panorama-to-4/] even without high-quality 4D training data, integrate generic 2D prior models into the panoramic domain. Our approach involves a two-stage pipeline: initially generating panoramic videos using a Panoramic Denoiser, followed by 4D elevation through a Ppatial-Temport Geometry Alignment mechanism to ensure spatial coherence and temporal continuity.
Limitation. First, the quality of temporal animation the generated 4D environment mainly relies on the ability of the pre-trained 12V model. Future improvements could include the integration of more advanced 2D animator. Second, since our method ensures spatial and temporal continuity during the 4D elevation phase, it is currently unable to synthesize significant changes in the environment, such as the appearance of glowing fireflies or changing weather conditions. Third, the high-resolution and dependent representation of the generated 4D environment necessitates substantial storage capacity, which could be optimized in future work using techniques such as model distillation and pruning.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jie An, Songyang Zhang. Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.084 08477, 2023.
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy; Text-to-4d generation using hybrid score distillation sampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
Jiayang Bai, Letian Huang, Jie Guo, Wen Gong, Yuanqi Li, and Yanwen Guo. 360-gs: Layout-guided panoramic gaussian splatting for indoor roaming. arXiv preprint arXiv:2402.00763 2024.
Omer Bar- Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusiin: Fusing diffusion paths for controlled image generation. International conference on machine learning, 2023.
Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3.1 model 700 for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 14460, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint wrXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Jacyoung Chung, Suyoung Lee, Hyeongjin Nam, Jacrin Lee, and Kyoung Mu Lee. Luciddreamer: Domain- generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine-grained open domain image animation with motion guidance, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
Mengyang Feng, Jinlin Liu, Miaomiao Cui, and Xuansong Xie. Diffusion360: Seamless 360 degree panoramic image generation based on diffusion models. arXiv preprint arXiv:2311.13144 2023.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang. Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 12230-222942 2023.
Pulkit Gera, Mohammad Reza Karimi Dastjerdi, Charles Renaud, PJ Narayanan, and Jean-François Lalonde. Casual indoor hdr radiance capture from omnidirectional images. British Machine Vision Conference, 2022.
Kai Gu, Thomas Maugey, Sebastian Knorr, and Christine Guillemot. Omni- neural radiance field from 360 image captures. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEFE, 2022.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adyances in neural information processing systems, 33:6840-6851, 2020.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXtv:2210.023333, 2022.
Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yi Yu Tseng, Christian Richardt, Michael Zollhöfer, and Matthias Nießner. Viewdiff: 3d-consistent image generation with text- to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Huajian Huang, Yingshu Chen, Tianjian Zhang, and Sai-Kit Yeung. 360roam: Real-time indoor roaming using geometry-awee aware 360 radiance fields. SIGGRAPH Asia Conference Proceedings, 2022.
Jiahui Huang, Zan Gojeie, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4369-4364 4379, 2023.
Álvaro Barbero Jiménez. Mixture of diffusers for scene composition and high resolution image generation. arXiv preprint arXiv:2300. 024112 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühllrr and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. Advances in Neural Information Processing Systems, 36:50648- 50660, 2023.
Xingyi Li, Zhiguo Cao, Huiqiang Sun, Jianming Zhang, Ke Xian, and Guosheng Lin. 3d cinemagra- phy from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4595 4605, 2023a.
Xingyi Li, Zhiguo Cao, Huiqiang Sun, Jianming Zhang, Ke Xian, and Guosheng Lin. 3d cinemagra- phy from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4595-4605, June 2023b.
Chenguo Lin, Yuchen Lin, Panwang Pan, Xuanyang Zhang, and Yadong Mu. Instructlay- out: Instruction-driven 2d and 3d layout synthesis with semantic graph prior, 2024. URL https: //arxii//rxiv.../// org/ abs (ooo/2407007880
Chenguo Lin, Panwang Pan, Bangbang Yang, Zeming Li, and Yadong Mu. Diffsplat: Repurposing image diffusion models for scalable gaussian splat generation, 2025. URL aittps://arxit
org/abs/ /2501.2 16764.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis, Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. International Conference on Learning Representations, 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 11461-11471, 2022.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020.
12

---

Published as a conference paper at ICLR 2025
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for texa-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4296-4304, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. International conference on machine learning, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023.
Hao Ouyang, Kathryn Heal, Stephen Lombardi, and Tiancheng Sun. Text2immersion: Generative immersive scene with 3d gaussians. arXiv preprint arXiv:23/2.09222, 2023.
Zijie Pan, Zeyu Yang, Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single-view video. arXiv preprint (tt::2001.084423 2024.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part III 16, pp. 523-540. Springer, 2020.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. International Conference on Learning Representations, 2024.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, and Rita Cucchiara. Merging and splitting diffusion paths for semantically coherent panoramas. In European Conference on Computer Vision, pp. 231-251. Springer, 2025.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEF/CVF international conference computer vision, pp. 12179-12188, 2021.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaus- sian4d: Generative 4d gaussian splatting. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 1..36479-36999 2022.
Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with inpainting and depth difffusion arXiv preprint arXiv:2404.07199, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. International Conference on Learning Representattons, 2023.
Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang, Feng Tang, Junsong Yuan, and Zhao Yang. Roomdreamer: Text-driven 3d indoor scene synthesis with coherent geometry and texture. In Proceedings of the 3 lst ACM International Conference on Multimedia, pp. 6898-6906, 2023.
13

---

Published as a conference paper at ICLR 2025
Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion. arXiv preprint arXhv:2411.04926, 2024.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Genera- tive gaussian splatting for efficient 3d content creation. International Conference on Learning Representations, 2024.
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with cortespondence-wware diffusion. Advances in Neural Information Processing Systems, 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:/8/2.01177, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. International Conference on Learning Representatioos((CLRR) 2019.
Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. Cg3d: Compositional generation for text-to-3d via gaussian splatting. arXiv preprint arXiv:2311.17907 2023.
Guangcong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang, Chen Change Loy, and Ziwei Liu. Perf: Panoramic neural radiance field from single panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint 0:X1v:2308.06571, 2023a.
Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, and Jian Zhang. 360dvd: Controllable panorama video generation with 360-degree video diffusion model. In Proceedings of the IÊEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6913-6922 2024b.
Zhen Wang, Shijie Zhou, Jeong Joon Park, Despoina Paschalidou, Suya You, Gordon Wetzstein, Leonidas Guibas, and Achuta Kadambi. Alto: Alternting latent topologies for implicit 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 259-270, 2023b.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024a.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Chunyi Li, Liang Liao, Annan Wang, Erli Zhang, Wenxiu Sun, Qiong Yan, Xiongkuo Min, Guangtai Zhai, and Weisi Lin. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint a2 17090, 2023a.
Jay Zhangjie WU, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633, 2023b.
Ruiqi Wu, Liangyu Chen, Tong Yang. Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn motion pattern for few-shot-based video generation. Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, 2024b.
Zhen Xu, Sida Peng. Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
14

---

Published as a conference paper at ICLR 2025
Bangbang Yang. Wenqi Dong, Lin Ma, Wenbo Hu, Xiao Liu, Zhaopeng Cui, and Yuewen Ma. Dreamspace: Dreaming your room space with text-driven panoramic texture propagation. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp. 660-660. IEEE, 2024.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.1722: 2023.
Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William Freeman, and Jiajun Wu, Wonderworld: Interactive 3d scene generation from single image. arXiv preprint arXiv:2406.09399, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong. Zhenguo Li, and Gim Hee Lee. Animate 124: Animating image 4d dynamic scene. arXiv preprint arXiv:2311.14600 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11008 2022.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang. and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024a.
Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj. Suya You, Zhangyang Wang, and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting. European Conference on Computer Vision, 2024b.
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, and Ming-HHuar Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting, International conference on machine learning, 2024c.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
Due to space constraints in the main draft, include supplementary details and experimental results the appendix. Specifically, in Sec. B, we provide details about the acquisition process for the static panoramas used in experiments. In Sec. C, we offer further explanation of the implementation for both the animation and lifting phases. Finally, in Sec. D, we describe the experimental setup and present additional results.
B ACQUISITION OF PANORAMAS
The static panoramas used in the dataset of the main draft are generated by a text-to-panorama diffusion model, fine-tuned from stable diffusion (Rombach et al., 2022) on SUN360. Similar to (Yang et al., 2024; Feng et al., 2023), this model follows three steps: circular blending, super- resolution, and refinement. The panoramas are initially at a resolution of 6144 x 3072 and then down-sampled 4096 2048 using the bi-linear interpolation.
C IMPLEMENTATION DETAILS
In this section, we introduce the implementation details of the panoramic animator and the 4D lifting procedure.
Implementation of Spherical Representing For the spherical representation, the continuous spherical mapping S₁ S2 RC is instantiate as discrete point P = {pi), which is uniformly sampled from the sphere S,- We first initialize a icosahedron with 20 triangle faces {feli 1, 203 approximate real sphere S2, Then uniformly sample point set P. on each face fi and union all the point sets together as P = 1 We then obtain the discrete point set P by projecting onto the sphere $2 by P = (pt/ Ps P).
Panoramic Animation Phase For the Panoramic Animator, we set the video length L = 14. the channel number e 9, the latent code size (h, w) 1 (H, W), the perspective image size PH = Pw AW. The sphere is uniformly divided into 20 perspective views, each with 80 FOV. For the denoiser, the max denoising step is 25. For the continuous optimization in Eq. 3, calculate each latent vector at each point on the sphere by taking the weighted average on the latent vectors of knn points that projected from the corresponding pixel the perspective views, the weights are the inversed distances between the sphere point and the projected points. We conduct the spherical denoising for the first 10 denoising steps and then the spherical latent is projected to the cquirectangular form and denoised using sliding window, to avoid noises introduced by interpolation. Aettmate-Ayythnng (Dai al., 2023). The masks required by
The perspective denoiser is initiated the denoiser are given by bounding boxes defined by user clicks.
Dynamic Panoramic Lifting Phase In the lifting phase, similar to the animation phase, we choose the perspective view number n = 20, each with 80 FOV. Each perspective view has a square shape, Ри Pw W. where W is the width of the original static panorama. In the Spatial-Temponee Geometric Aligmment stage, the depth estimator θ implemented using MiDaS (Ranftl et al., 2021; Birkl et al., 2023). The depth map from the perspective image scaled according the projection of the unit-length ray direction onto the camera orientation d. Formally, if the pre-scaled depth at point E P. introduced above, the scaled depth should be d//lp]| Additionally, for scenes without distinct boundaries, such as the sky, depth values for distant elements assigned a finite value to support optimization.
Optimization Details The hyper-parameter for optimization are set follows: Adepth
1, Ascale = 0.1, Ashift = 0.01. We conduct Spatial-Tempont Geometry Aligmment optimization over 3000 iterations, with Ascale and Aehint to zero during the first 1500 iterations. For the 4D represen- tation training stage, Gaussian parameters are optimized over 10000 iterations for each time stamp
The pyrameparr for this stage defined \rgb 1, Atemporal A Agoo 0.05,
16

---

Published as a conference paper at ICLR 2025
and the disturbance vector range is varied at 0.05, 0.1, and 0.2 during the 5400, 6600, and 9000 iterations, respectively.
D EXPERIMENTAL DETAILS
D.1 THE PROCEEDING TIME OF FER-GENERATION
Animating Phase. We provide the time and GPU cost to animate a single video at different resolutions in the Table
Table 3: The Proceeding Time of Animating Phase.
Resolution
GPU Usage (GB)
Time Cost (Minutes/frame))
1024 512 (1K) 2048 x 1024 (2K)
9.38
0.89
29.3
2.99
4096 x 2048 (4K)
73.41
22.31
Lifting Phase. We provide the time and GPU cost lift single frame 4K, 2K, and IK resolution. The results are shown the Table 4.
Table 4:
The Proceeding Time of Lifting Phase.
Resolution
GPU Usage (GB)
Time Cost (Minutes/frame))
1024 x 512 (1K) 2048 x 1024 (2K)
7.59
19
12.07
22
4096 x 2048 (4K)
31.27
33
The computational cost increases rapidly with the resolution, making 4K generation highly chal- lenging. We will put more engineering effort to accelerate the generation pipeline in the future.
D.2
USER STUDY DETAILS
D.2.1 USER STUDY FOR VIDEO QUALITY
We conducted two user studies, gathering a total 84 questionnaires from 42 users. For the "Quality (UC)"column in Tab. 1, we collected 42 questionnaires, each containing eight questions. Each question asked users to choose the bests video in term of visual quality from the perspective videos provided by different models. The choice (UC) score of method is the percentage of times the method's video was selected as the best one, out of a total of 336 questions. Thus, the UC scores for (UA)" column in Tab. 2, we collected another
all methods sum 100%. For the "View-Consistens 42 questionnaires, with each questionnaire containing eight questions. Each question presented two videos from different views, both generated by the same method, and users were asked determine whether the two videos were niew-consistent. The user agreement (UA) score is the percentage of video pairs marked view-consistent out of all the video pairs generated by the method. The UA scores do not necessarily sum to 100%. In the Quality column of Tab. 1. among the 336 questions, users selected 4K4DGen 272 times, 3D-Cin. (circle) 40 times, and 3D-Cin. (zoom-in) 24 times. In the View-consistency column of Tab. 2, 118 out of 168 video pairs generated by "Our" were marked as consistent, while 56 out of 168 pairs from "Animate Pers" were considered consistent.
D.2.2 USER STUDY FOR VIDEO MOTION
Since the quantitative evaluation of motion quality remains an open problem in our tasks, we hereby conducted supplemented user studies for the "4D generation task" and "Animating Phase", which considers the motion's 's naturalness and amplitude.
Motion's 's naturalness: the motion of the generated view should natural to human's under- standing, avoiding abrupt pixel changes across frames.
17

---

Published as a conference paper at ICLR 2025
Motion's 's amplitude: the motion trajectory of the scene's subjects should have adequate and realistic magnitude.
The user studies of Amplitude column and the Naturalness column are all conducted in the "user choice" (UC) way. For each Amplitude and Naturalness column in Tab. and Tab. 2, we collect 320 questions from 20 participants. Each questions contains three videos from the different three methods, users are asked to select the best one or more videos from the provided set that exhibit noticeably greater amplitude (for the Amplitude columns) or superior naturalness (for the Naturalness columns). Since users could select more than one video per question, the UC metric was normalized based on the total number of selections. For example, if method A, B, C are selected Ra. Riy and ne times, respectively, the UC metric for them should be
and
Th +-6 +ng
ny Pa tmm Re Po+t
D.3 MORE QUANTITATIVE RESULTS
We present quantitative results additional 32 scenes randomly sampled from WEB360 dataset (Wang et al., 2024b). As shown the Table 5, 4K4DGen consistently outperforms the baseline methods across the quantitative metrics.
Table 5: Comparison with 3D-Cinemagraphy in WEB360 Dataset. We adopt the FVD (Unterthiner et al., 2019) and KVD (Unterthiner et al., 2018) to evaluate the generated panoramic video, which is the intermediate result from the animating phase. The IQ, IA, and VQ models represent the image quality scorer, image aesthetic scorer, and video quality scorer, respectively, within the Q-Align assessment framework.
Method
FVD 4
KVD
Q-Align (IQ) t Q-Align (IA)
Q-Align (VQ) f
AD-Cinemagraphy (zoom-in)
307
5,86
0.65
0.57
0.70
3D-Cinemagraphy (circle)
309
5.72
0.65
0.57
0.70
4K4DGen
218
1.76
0.73
0.64
0.77
D.4 COMPARISONS WITH MORE BASELINE METHIOSS
To further address your primary concern, we adopt two types of baseline methods: 4D Object Generation and 4D Scene Generation, to compare with the proposed method. Note that the panoramic 4D generation is still underexplored due to the scarcity of annotated data and the lack of well-trained prior models tailored for panorama format. As result, we find that existing methods cannot achieve similar quality as 4K4DGen in this task.
4D Object Generation methods. Following your suggestion, we have devoted substantial engi- neering efforts to adapt the 4D object generation framework 4DGen (Yin et al., 2023) to our camera settings, shown in the qualitative results in Figure of the revised paper. It demonstrates that recent 4D object generation methods struggle to generate scene-level content due to inherent domain gaps between objects and scenes. Our method overwhelmingly outperforms 4DGen in terms of quantitative evaluations (c.g., Image Quality, Image Aesthetics, and Video Quality) and qualitative evaluations.
4D Generation Methods for Scene. We also construct two 4D scene generation baselines: (1) We equip LucidDreamer (Liu et al., 2023) with our animator. We follow the authors' setting, using ZoeDepth and its inpainting model (Stable Diffusion) to expand invisible views for each timestamp, and then we use our backbone animator to animate and optimize the Gaussians. (2) We employed a very recent 4D scene generation technique DimensionX (Sun et al., 2024). We use the default configuration, employing its lora ('orbit left") model to generate novel views and 3D structures. Since the DimensionX's 's T-Director is currently unavailable, we leveraged the same backbone animator from approach to provide temporal guidance for its 4D representation in the 4D generation stage. Compared with existing methods, including 4DGen, LucidDreamer, and DimensionX, our method consistently achieves higher quantitative results, demonstrating the efficacy of the proposed method.
18

---

Published as a conference paper at ICLR 2025
Table Comparison with 4D Generation Methods.
Method
Q-Align (IQ)
Q-Align (IA) f
Q-Align (VQ)
4DGen (object)
0.19
0.20
0.29
Co--enmamarrphy (zoom-in)
0.47
0.38
0.57
3D- DD-Cimmmagrpphy (circle)
0.48
0.40
0.58
LucidDreamer + Our Animator
0.44
0.41
0.58
DimensionX Our Animator
0.55
0.42
0.60
4K4DGen
0.66
0.44
0.62
D.5 MORE QUALITATIVE RESULTS
We provide additional qualitative results in Figure 7. Furthermore, we highly recommend viewing the video renderings of 4K4DGen and comparisons baseline models in the supplementary static HTML page for more comprehensive and immersive experience.
We adapt the 4D object generation framework 4DGen (Yin et al., 2023) to our specific settings and present the qualitative results in Figure 8, which indicate that the generated object varies significantly form from 4K4DGen's scene outputs. We also compare OmniNeRF (Gu al., 2022)'s 's optimized geometry with 4K4DGen. The corresponding depth maps are shown in Figure 9. It can be evidently demonstrated that 4K4DGen attains sharper geometric results. We provide the renderings of a lifted 3D scene where user walked along street in Figure 10, Notice the roof highlighted by green bounding boxes in (a) and (b). When the user walks nearer and more area the roof observed, implies the necessity of the lifted 3D structure.
E ETHICS AND REPRODUCIBILITY STATEMENT
Ethics Statement. Our research enables the generation of 4D digital scenes from single panoramic image, which advantageous for various applications such as AR/VR, movie production, and video games. This technology distinctly excels in creating high-resolution 4D scenes up 4K, significantly enhancing user experiences. However, there is potential for misuse in the creation deceptive content or privacy violations, which contradicts our ethical intentions. These risks can be mitigated through a combination of regulatory and technical strategies, such as watermarking.
Reproducibility. We provide sufficient implementation details to reproduce our methodology in Sec. C, including the details of spherical denoiser, panoramic animator, dynamic panoramic lifting, etc. We provide 16 Sec. 4's panoramas and Sec. D.4's 32 panoramas in the revised supplementary material. Furthermore, we will make our panorama datasets and related code publicly available in the future.
19

---

Published as a conference paper at ICLR 2025
==
View
sailing ship, monochrome, digital art, swirls, surreal, sea, waves, reflections, mountain landscape,
Prompts
View
fantasy, stylized water, starry sky, circular frame, high contrast, black and white, maritime
t=0.5
View
fireworks, cityscape, sunset, panorama, digital art, vibrant colors, futuristic, urban, celebration, skyscrapers, streets, glowing lights, dusk, orange sky, purple hues, artistic, wide angle...
View
t=0.5
View
Pana RGB
beach, seascape, waves, sandy shore, sallboat, horizon, clouds, blue sky, panoramic view, tranquil scene, digital painting, soft focus, pastel colors, wide-angle, nature, outdoor, no people, calm sea, coastline, vacation mood
Prompte
View 2
1
View
Pana RGB
desert, sunset, sand dunes, yucca plants, mountain range, clouds, sky, warm color palette, panoramic view, digital art, serene, tranquil, nature, vast landscape, purple sky, orange hues, twilight, no people, scenic beauty, subtle shadows
Prompts
View 2.
Figure 7: Visualizations: We provide more visual results. For each shown case we provide the input panorama, corresponding text prompt, and the rendering from two perspective views.
20

---

Published as a conference paper at ICLR 2025
Ours
4DGen
Ours
4DGen
Input
Frame
Prame
Frame 10
Frame
Figure 8: Results of 4DGen: 4DGen (Yin al., 2023) focuses on object generation and struggles to generate scenes.
4K4DGen Depth
OmniNeRF Depth
RGB Reference
Figure 9; Results of OmniNeRF: The optimized geometry of OmniNeRF (Gu et al., 2022) is not as sharp as 4K4DGen.
21

---

Published as a conference paper at ICLR 2025
(a) Stand far
(b) Stand near
Figure 10: Occlusion in True 3D Structure: When the user walks nearer, the more area of the roof (highlighted by the green box) will be observed. It is hard to implement such effect without the lifted 3D structure.
22


Reviews:
[{"review_id": "vA42NJmNC2", "reviewer": "Reviewer_pfwC", "summary": "This paper presents a method by which a static pano image is animated to produce a video pano, then subsequently lifted to 3D using 3DGS.  The authors claim the following contributions:\n* 4K4DGen, a method for lifting a static pano to dynamic 3DGS\n* A panoramic denoiser which generates consistent animated perspective views \n* A dynamic panoramic lifting which transforms the video pano to dynamic 3DGS", "questions": "* Have the authors considered using a more recent depth model than MiDaS?  It's pretty old at this point.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* compelling use case\n* qualitative results look good\n* quantitative metrics show improvements over baselines\n* well written", "weaknesses": "* The animations each have small spatial extent.  I didn't see an example that illustrated that you could have many of these overlapping perspective animations and for a large scale animation to emerge.  e.g., It would be much more impressive if the authors could show the car in the desert where the car drives around the center of project.\n\n* Some of the generated videos have significant artifacts.  For example, the fireworks in the third row of the supplemental has fireworks that don't move and speckled artifacts that are static.\n\n* The motion isn't always globally consistent.  For example, I'm not sure the clouds in row 7 of the supplemental make sense.\n\n* I think overall, the idea of applying this existing model to perspective patches is neat, but I'd be concerned this work would be quickly superseded by one where the diffusion model was fine-tuned on 360 videos.\n\n* The lifting phase doesn't seem sufficiently evaluated.  It would have been nice to have something to inspect in the supplemental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VP2PVxc6N8", "reviewer": "Reviewer_iVVu", "summary": "This paper proposes a novel framework to generate 4K4G panoramic video rendered from 3D Gaussian representation. The key technical insights is to use pre-trained 2D perspective camera to denoise the panorama by first rendering it into perspective image patch and then fusing it back into the sphere space. Once a panoramic video is created, a lifting process built on single image depth prediction method is to turn each frame into 3D Gaussian representation so that we can navigate inside the panorama. Both quantitative and qualitative experiments verified the effectiveness of each technical components and overall the framework achieves impressive video generation results.", "questions": "My major question is on the necessities of the dynamic panoramic lifting and whether that actually improves the quality when we move the camera and look around. In the submitted video, we can see a little bit camera movements but I wonder if that can be mimiced but simply textured the panorama to a large enough sphere. Also when handling the dynamic scene, why not consider all the dynamic Gaussian papers and instead use static Gaussian with a temporal constraints? If we use dynamic Gaussian representation, will that cause any differences?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. As far as I know, this is the first paper to generate high-resolution 4D panorama videos, which may have many applications in VR. \n\n2. The proposed method is technically solid and easy to follow. The consistent panoramic animation part is similar to prior works that use 2D diffusion model to generate panorama but the paper replaces 2D diffusion models with 2D video diffusion model. The dynamic panoramic lifting part is also reasonable. \n\n3. While evaluating generative models is always a difficult problem and there is no prior works solving this problem, this paper still manages to build a set of benchmarks and a reasonable baseline to show the effectiveness of the proposed framework, which should be appreciated. \n\n5. The paper is well-written and easy to follow.", "weaknesses": "I am not an expert in using diffusion models for video generation and I haven't followed recent progress in that direction. The consistent panoramic animation part looks solid and reasonable to me. I only have some minor questions that are not necessarily the weakness of the paper. \n\n1. Eq. (3): While this equation is similar to how we generated static panorama using diffusion models for 2D perspective image, I wonder whether that will cause different Gaussian noise distribution at the overlap regions. Since we are essentially average the Gaussian noise from two denoised latent code, will that cause the variance of the noise at the overlap region to be smaller compared to non-overlapping area? Will that cause artifacts in image generation?\n\n2. My second question is how significant the dynamic panoramic lifting can help us render 3D video. Given that all the examples shown in the paper have very large-scale scene and the perturbation $\\alpha$ is very small in training, I would guess whether we lift the panorama into 3D or not may not make a big difference and approximating the whole 3D structure with a sphere may give us similar results. I assume the comparison is shown in Figure 6(c) but it is very difficult for me to see the difference. \n\n3. Also I wonder why don't we choose recent dynamic Gaussian works to model the video. I assume that is a more systematic way to handle this problem and may give us more temporal consistent view synthesis results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J6wfj6E7by", "reviewer": "Reviewer_2Ek6", "summary": "The paper proposes an end-to-end solution for generating realistic 4D panoramic videos from static panoramas at 4K resolution for the first time. The paper uses pretrained image to video models to convert the static 2D panoramas to videos and then dynamically (with spatio-temporal alignment) lift them to 3D space resulting in the 4D content. \nThe paper demonstrates the appropriateness of using a spherical latent space for the task and how the panoramic denoiser utilizes that. The lifting uses a geometry alignment model (pre-trained depth estimator) and optimize DreamScene360's objective with an additional temporal loss. \n\nThe framework achieves high fidelity generations, both, visually and quantitatively. The user-study also aligns well with the results.", "questions": "My main concern is the lack of baselines in the paper but the problem statement, conceptualization and results are very convincing hence the accept rating.\n\nQuestion(s):\nQ1. What is the processing times per generation on the A100 used? I would also be interested in knowing how memory intensive it is.\nQ2. If more baselines are added (see weaknesses), I would be willing to upgrade my recommendation score to 8.\n\nSuggestion(s):\n* L295: Fix typo: biuld -> build\n* The space is low but can the authors somehow manage to split the Evaluation paragraph starting on L407? (split at the each underlined term)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novelty: This is the first paper that allows seamless (spatio-temporally aligned) 4K resolution generation of 4D content.\n2. The spherical latent space operation alleviates a lot of visual artifacts and this paper is evidence that it is suitable for Panoramic generation tasks.\n3. The model components, intuition and math is well grounded.\n4. The fidelity of results is promising. Quantitative metrics suggest the same. (The resulting panorama videos provided on the supplementary page are extremely visually pleasing!)", "weaknesses": "1. Why are Efficient4D and 4DGen not used as baselines that potentially fail on Panoramas as there is no spatial-temporal alignment mechanism there? It would make the paper much stronger. \n2. It would have been nice to see comparisons for the 4D lifting phase to see how the proposed is better than existing methods like: OmniNeRF. This way the speed of the proposed method can also be highlighted as a strength of the paper.\n3. More visual results in the main text would be nice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrWPZiuYX6", "reviewer": "Reviewer_jzFB", "summary": "The paper first proposes a method, 4D4KGen, which can generate immersive 360-degree, 4K panoramic 4D videos from a single 4K static panoramic image. The method first uses the pre-trained perspective video generative model AnimateAnything to create a panoramic video through a project-and-fuse scheme with a spherical latent space. Next, the pretrained monocular depth estimator MiDaS is used to create a panoramic depth map with a spherical representation. Finally, the 4D animated panoramic videos are rendered using reconstructed 3DGS. Qualitative results show that the proposed method can effectively generate immersive 360-degree 4D panoramic videos.", "questions": "(1) How much time and GPU memory are required to generate a single 4K video?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The paper first proposes animatable 360-degree 4K panoramic images. \n\n(2) To create 360-degree panorama with fixed-size pretrained video generation and depth estimation models, the authors carefully design the projection function. \n\n(3) The qualtative results shows that great spatial and temporal consistency.", "weaknesses": "(1) I understand that there is no metric to evaluate 360-degree panoramic images, but there is insufficient basis for measuring FID and KID between the generated frames and the corresponding perspective images projected from the static panoramas. In particular, these metrics may be meaningless when the reference set is small, and in this paper, only 16 panoramic videos were generated. Proposing and measuring such less meaningful metrics, especially when this paper aims to serve as a baseline for future 4D 360-degree panorama video generation, does not seem beneficial for subsequent research. Additionally, if we want to assess the fidelity of the generated video, we could measure FVD [1] and KVD [1]. Therefore, I recommend removing FID and KID.\n\n(2) I don't know how long the proposed method takes, but generating only 16 videos seems too few. To demonstrate that the proposed method works generally on many 4K 360-degree panoramic images, it would be better to generate more videos (at least 30 or more) to measure quantitative results.\n\n(3) In Sec. 4.3 Ablation Studies, a straightforward approach would be to apply animators directly to the entire panorama as the direct baseline, measured by the Q-Align metrics (IQ, IA, and VQ) and also through a user study. If 2D animators cause out-of-memory issues, we can simply upsample each frame of the videos using a super-resolution (upsampling) model such as StableSR [2].\n\n(4) In Fig. 6, Ablation of the Lifting Phase, I find it difficult to visually discern the difference between the proposed temporal regularization and Spatial-Temporal Geometry Alignment (STA). The authors should have included a video of this in the supplementary materials or shown a more definitive difference in the figure.\n\n(5) Since the paper's topic is closely related to panoramic image generation, the following papers [3, 4, 5] should be mentioned in the related work section.\n\n(6) Please double-check the citations (bibtex) to confirm which papers were accepted by specific conferences. Too many papers published in conferences are listed as arXiv.\n\n\nThe proposed method demonstrates good quality in the new research field of 4D panoramic video generation, but it lacks convincing evidence for the choice of pipeline components and is missing many essential elements that a paper should fundamentally include. I believe it lacks the basic qualities necessary for publication in a top-tier ML conference ICLR.\n\n\n[1] Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric & Challenges, arXiv preprint arXiv:1812.01717, 2018.\n\n[2] Wang et al., Exploiting Diffusion Prior for Real-World Image Super-Resolution, IJCV2024\n\n[3] Bar-Tal et al., MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, ICML 2023\n\n[4] Lee et al., SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions, NeurIPS 2023\n\n[5] Quattrini et al., Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)