4K4DGen: Panoramic 4D Generation at 4K Resolution

## Abstract

Abstract The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the requirements of VR/AR applications that need free-viewpoint, 360 ∘ virtual views where users can move in all directions. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360 ∘ views at 4K (4096 × \times × 2048) resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of dynamic Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360 ∘ images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we propose Dynamic Panoramic Lifting to elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of 4K for the first time.

## 1 Introduction

With the increasing growth of generative techniques (Rombach et al., ; Blattmann et al., ), the capability to create high-quality assets has the potential to revolutionize content creation across VR/AR and other spatial computing platforms. Unlike 2D displays such as smartphones or tablets,
ideal VR/AR content must deliver an immersive and seamless experience, enabling 6-DoF virtual tours and supporting high-resolution 4D environments with omnidirectional viewing capabilities.
Despite significant advancements in the generation of images, videos, and 3D models, the development of panoramic 4D content has lagged, primarily due to the scarcity of well-annotated, high-quality 4D training data.
Even in the most relevant field of 4D generation, existing works mainly focus on generating or compositing object-level contents (Bahmani et al., ; Lin et al., ), which are often in low-resolution (e.g., below 1080p) and cannot fulfill the demand of qualified immersive experiences.
Based on these observations, we propose that an ideal generative tool for creating immersive environments should possess the following properties:
(i) the generated content should exhibit high perceptual quality, reaching high-resolution (4K) output with dynamic elements (4D);
(ii) the 4D representation must be capable of rendering coherent, continuous, and seamless 360^∘ panoramic views in real time, supporting efficient 6-DoF virtual tours.
However, creating diverse, high-quality 4D panoramic assets presents two significant challenges:
(i) the scarcity of large-scale, annotated 4D data, particularly in panoramic formats, limits the training of specialized models.
(ii) achieving both fine-grained local details and global coherence in 4D and 4K panoramic views is difficult for existing 2D diffusion models. These models, typically trained on perspective images with narrow fields of view (FoV), cannot be easily adapted to the expansive scopes of large panoramic images (see Exp. 4.3).
On another front, video diffusion models (An et al., ) trained with web-scale multi-modal data have demonstrated versatile utility as region-based dynamic priors, and Gaussian Splatting (Kerbl et al., ) has shown efficient capabilities in modeling 4D environment.
Thus, we address the large-scale, omnidirectional dynamic scene generation (4D panoramic generation) problem by utilizing the generative power of diffusion models to animate static panoramic images, transforming them into realistic, dynamic scenes that can support immersive, 360^∘ viewing experiences. To achieve this, we propose to elevate the dynamic panoramic video to 4D environment assets using a set of dynamic Gaussians, which can be seamlessly integrated into VR/AR platforms for real-time rendering and interaction.

In this paper, we introduce 4K4DGen, a novel framework designed to enable the creation of panoramic 4D environments at resolutions up to 4K. 4K4DGen addresses the key challenges of maintaining consistent object dynamics across the entire 360^∘ field-of-view (FoV) in panoramic videos, while preserving both spatial and temporal coherence as the video transitions into a fully interactive 4D environment.
Specifically, we propose the Panoramic Denoiser, which animates 360^∘ FoV panoramic images by denoising spherical latent codes corresponding to user-interacted regions. The Panoramic Denoiser leverages a well-trained diffusion model originally designed for narrow-FoV perspective images, enabling the generation of 360^∘ dynamic panoramas while ensuring global coherence and continuity throughout the entire panorama.
To transform the omnidirectional panoramic video into a 4D environment, we introduce Dynamic Panoramic Lifting, which corrects scale discrepancies using a depth estimator enriched with perspective prior knowledge to generate panoramic depth maps. Additionally, it employs time-dependent 3D Gaussians optimized with spatial-temporal geometry alignment to ensure cross-frame consistency in dynamic scene representation and rendering.
By adapting generic 2D statistical patterns from the perspective domain to the panoramic format and effectively regularizing Gaussian optimization with geometric principles, we achieve high-quality 4K panorama-to-4D content generation with photorealistic novel-view synthesis capabilities.
Our contributions can be summarized as follows.

- •
We introduce 4K4DGen, the first framework capable of generating high-resolution (up to 4096$\times$2048) 4D omnidirectional assets without the need for annotated 4D data.
- •
We propose the Panoramic Denoiser, which transfers generative priors from pre-trained 2D perspective diffusion models to the panoramic space, enabling consistent animation of panoramas with dynamic scene elements.
- •
We introduce Dynamic Panoramic Lifting, a method that transforms dynamic panoramic videos into dynamic Gaussians, incorporating spatial-temporal regularization to ensure cross-frame consistency and coherence.

Figure: Figure 2: Panoramic Denoiser adapts diffusion priors from the perspective domain to the panoramic domain by simultaneously denoising perspective views and integrating them into spherical latents at each denoising step. This approach ensures consistent animation across multiple views.
Refer to caption: x2.png

## 2 Related Work

#### Diffusion-based Image and Video Generation.

Recent advancements have significantly expanded the capabilities of generating 2D images using diffusion models, as evidenced in several studies (Dhariwal & Nichol, ; Nichol et al., ; Podell et al., ; Ramesh et al., ; Saharia et al., ). Notably, Stable Diffusion (Rombach et al., ) optimizes diffusion models (DMs) within the latent spaces of autoencoders, striking an effective balance between computational efficiency and high image quality. Beyond text conditioning, there is increasing emphasis on integrating additional control signals for more precise image generation (Mou et al., ; Zhang et al., ). For example, ControlNet (Zhang et al., ) enhances the Stable Diffusion encoder to seamlessly incorporate these signals. Furthermore, the generation of multi-view images is gaining attention, with techniques like MVDiffusion (Tang et al., ) or Geometry Guided Diffusion (Song et al., ) processing perspective images with a pre-trained diffusion model. Diffusion models are also extensively applied in video generation, as demonstrated by various recent works (Ge et al., ; Ho et al., ; Wang et al., ; Wu et al., ; ; Zhou et al., ).
For instance, Imagen Video (Ho et al., ) utilizes a series of video diffusion models to generate videos from textual descriptions. Similarly, Make-A-Video (Singer et al., ) advances a diffusion-based text-to-image model to create videos without requiring paired text-video data.
MagicVideo (Zhou et al., ) employs frame-wise adaptors and a causal temporal attention module for text-to-video synthesis. Video Latent Diffusion Model (VLDM) (Blattmann et al., ) incorporates temporal layers into a 2D diffusion model to generate temporally coherent videos.

#### 3D/4D Large-scale Generation.

In recent 3D computer vision, a large-scale scene is usually represented as implicit or explicit fields for its appearance (Mildenhall et al., ; Kerbl et al., ), geometry (Peng et al., ; Wang et al., ; Huang et al., ), and semantics (Kerr et al., ; Zhou et al., ; Qin et al., ). We mainly discuss the 3D Gaussian Splatting (3DGS) (Kerbl et al., ) based generation here. Several works including DreamGaussian (Tang et al., ), GaussianDreamer (Yi et al., ), GSGEN (Chen et al., ), and CG3D (Vilesov et al., ) employ 3DGS to generate diverse 3D objects and lay the foundations for compositionality, while LucidDreamer (Chung et al., ), Text2Immersion (Ouyang et al., ), GALA3D (Zhou et al., ), RealmDreamer (Shriram et al., ), and DreamScene360 (Zhou et al., ) aim to generate static large-scale 3D scenes from text. Considering the current advancements in 3D generation, investigations into 4D generation using 3DGS representation have also been conducted. DreamGaussian4D (Ren et al., ) accomplishes 4D generation based on a reference image. AYG (Ling et al., ) equips 3DGS with dynamic capabilities through a deformation network for text-to-4D generation. Besides, Efficient4D (Pan et al., ) and 4DGen (Yin et al., ) explore video-to-4D generation, and utilize SyncDreamer (Liu et al., ) to produce multi-view images from input frames as pseudo ground truth for training a dynamic 3DGS.
4K4D (Xu et al., ) is a high-resolution reconstruction technique that extends 3DGS to model complex human motion with detailed backgrounds while achieving real-time rendering speed.

#### Panoramic Representation.

A panorama is an image that captures a wide, unbroken view of an area, typically encompassing a field of vision much wider than what a standard photo would cover, providing a more immersive representation of the subject. Recently, novel view synthesis using panoramic representation has been widely explored. For instance, PERF (Wang et al., ) trains a panoramic neural radiance field from a single panorama to synthesize 360^∘ novel views. 360Roam (Huang et al., ) proposed learning an omnidirectional neural radiance field and progressively estimating a 3D probabilistic occupancy map to speed up volume rendering. OmniNeRF (Gu et al., ) introduced an end-to-end framework for training NeRF using only 360^∘ RGB images and their approximate poses. PanoHDR-NeRF (Gera et al., ) learns the full HDR radiance field from a low dynamic range (LDR) omnidirectional video by freely moving a standard camera around. In the realm of 3DGS, 360-GS (Bai et al., ) takes 4 panorama images and 2D room layouts as scene priors to reconstruct the panoramic Gaussian radiance field. DreamScene360 (Zhou et al., ) achieves text-to-3D Panoramic Gaussian Splatting by utilizing monocular depth priors to regularize the Gaussian optimization.

## 3 Methodology

Taking a single panoramic image as input, the goal of 4K4DGen is to generate a panoramic 4D environment capable of rendering novel views from arbitrary angles and at various timestamps. Our approach initially constructs a panoramic video and then elevates it into a series of 3D Gaussians, enabling efficient splatting for flexible rendering. Naïvely animating projected perspective images, however, often results in unnatural motion and inconsistent animations. To overcome this, our method propose the denoising of projected spherical latents, ensuring consistent animation of the panoramic video from the original image, as detailed in Sec. 3.3.

Moreover, directly converting multiple perspective images from different timestamps into 4D frequently leads to degraded geometry and visible artifacts (see Sec. 4.3). We address this by applying spatial-temporal geometry fusion to lift the panoramic video, as described in Sec. 3.4. The complete pipeline of 4K4DGen is illustrated in Fig. 3.

Figure: Figure 3: Overall Pipeline. Beginning with a static panorama as input, the Animating Phase generates a panoramic video by first mapping the panorama into a spherical latent space, followed by denoising within the perspective space, fusing back to the spherical latent space at each step, and finally transforming it into the panoramic space. In the 4D Lifting Phase, a series of dynamic Gaussians is employed to lift the panoramic video into a 4D representation, ensuring both spatial and temporal consistency.
Refer to caption: x3.png

### 3.1 Preliminaries

#### Latent Diffusion Models (LDMs).

LDMs  (Rombach et al., ) consist of a forward procedure $q$ and a backward procedure $p$. The forward procedure gradually introduces noise into the initial latent code $x_{0}\in{}^{h\times w\times c}$, where $x_{0}=\mathcal{E}(I)$ is the latent code of image $I$ within the latent space of a VAE, denoted by $\mathcal{E}$. Given the latent code at step $t-1$, the $q$ procedure is described as $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\bm{I})$. Conversely, the backward procedure $p$, aimed at progressively removing noise, is defined as $p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(\mu_{\theta}(x_{t},t),\Sigma_{\theta}(x_
{t},t))$. In practical applications, images are generated under the condition $y$, by progressively sampling from $x_{T}$ down to $x_{0}$. Recently, image-to-video (I2V) generation has been realized  (Guo et al., ; Dai et al., ) by extending the latent code with an additional frame dimension and performing decoding at each frame. The denoising procedure is succinctly represented as $x_{t-1}=\Phi(x_{t},I)$, where $x_{t},x_{t-1}\in{}^{l\times h\times w\times c}$ represent the sampled latent codes and $I$ the conditioning image.
Recently, image-to-video (I2V) generation has been achieved (Guo et al., ; Dai et al., ) by extending the latent code with an additional frame dimension and performing decoding at each frame. The denoising procedure is succinctly expressed as $x_{t-1}=\Phi(x_{t},I)$, where $x_{t},x_{t-1}\in{}^{l\times h\times w\times c}$ represent the sampled latent codes, and $I$ represents the conditioning image.

#### Omnidirectional Panoramic Representation.

Panoramic images or videos, denoted as $I$, are typically represented using equirectangular projections, forming an $H\times W\times C$ matrix, where $H$ and $W$ denote the image resolution and $C$ represents the number of channels. While this format preserves the matrix structure, making it consistent with planar images captured by conventional cameras, it introduces distortions, especially noticeable near the polar regions of the projection. To mitigate these distortions, we adopt a spherical representation for panoramas, where pixel values are defined on a sphere
$\mathbb{S}^{2}=\{\bm{d}=(x,y,z)|x,y,z\in\real\wedge|\bm{d}|=1\}$.
For a more precise definition of the projection, we represent matrix-like images using a mapping $\mathcal{E}_{I}:[-1,1]^{2}\rightarrow{}^{C}$, which normalizes the image coordinates into the range $[0,1]$. Thus, for any given pixel $\left(x,y\right)\in[-1,1]^{2}$, the corresponding pixel value is determined by $\mathcal{E}_{I}\left(x,y\right)$.
We define the spherical representation of panoramas using the field $\mathcal{S}_{I}:\mathbb{S}^{2}\rightarrow{}^{C}$, where $\mathcal{S}_{I}\left(\bm{d}\right)$ gives the pixel value at a given direction $\bm{d}=(x,y,z)$. The relationship between the spherical and equirectangular representations is established through the following projection formula:

$$ $\mathcal{S}_{I}\left(x,y,z\right)=\mathcal{E}_{I}\left(\frac{1}{\pi}\arccos \frac{y}{\sqrt{1-z^{2}}},\frac{2}{\pi}\arcsin z\right).$ (1) $$

For perspective images, we define a virtual camera centered at the origin. The rays for each pixel are determined through ray casting, as described in  (Mildenhall et al., ), where each ray $\bm{d}$ is represented by $\bm{r}(x,y,f,\bm{u},\bm{s},R)$. This representation takes into account the focal length $f$, the z-axis direction $\bm{u}$, the image plane size $\bm{s}$, and the camera’s rotation along the z-axis $R$. Consequently, for a given panorama $I$, the perspective image $P$ can be projected using these camera parameters ($f,\bm{u},\bm{s},R$) as:

$$ $\mathcal{E}_{P}\left(x,y\right)=\mathcal{S}_{I}\circ\bm{r}\left(x,y,f,\bm{u}, \bm{s},R\right).$ (2) $$

In this paper, we fix the focal length $f$, the image plane size $\bm{s}$, and the rotation $R$. We denote the process of projecting the panorama $I$ into a perspective image $i$, based on the camera’s $z$-axis direction $\bm{u}$, as $i=\gamma(I,\bm{u})$.

### 3.2 Inconsistent Perspective Animation

Large-scale pre-trained 2D models have shown remarkable generative capabilities in creating images and videos, benefiting from vast multi-modal training data gathered from the Internet. However, acquiring high-quality 4D training data is considerably more challenging, and no current 4D dataset reaches the scale of those available for images and videos. Therefore, our approach aims to utilize the capabilities of video generative models to produce consistent panoramic 360^∘ videos, which are then elevated to 4D. Nonetheless, the availability of panoramic videos is significantly more limited compared to planar perspective videos. Consequently, mainstream image-to-video (I2V) animation techniques may not perform optimally for panoramic formats, and the resolution of the videos remains constrained, as illustrated in Fig. 5 (b) and Tab. 2.
Alternatively, the animator can be applied to perspective images. but this introduces inconsistencies across different projected views, as depicted in Fig. 5 (c)

### 3.3 Consistent Panoramic Animation

Limited by the scarcity of 4D training data in panoramic format, and given that large diffusion models are primarily trained on planar perspective videos, directly applying 2D perspective denoisers presents challenges in generating seamless panoramic videos with proper equirectangular projection, due to inconsistent motion across different views and the domain gap between spherical and perspective spaces.
This constraint has driven us to develop a panoramic video generator in spherical space that leverages priors from general image-to-video (I2V) animation techniques, as shown in Fig. 2.
Consequently, starting from a static input panorama, we animate it into a panoramic video, as demonstrated in the "Animating Phase" section of Fig. 3.

#### Spherical Latent Space.

To generate panoramic video from a static panorama, we build up the denoise-in-latent-space schema (An et al., ; Blattmann et al., ; Dai et al., ) in a spherical context. For general video generation, a noisy latent sample is progressively denoised using DDPM  (Ho et al., ), conditioned on a static input image, and subsequently decoded into a video sequence by a pre-trained VAE decoder.
However, in 4K4DGen, unlike the method for generating perspective planar videos, both the latent code and the static panorama input are represented on spheres. We start with the initial panoramic latent code $S^{T}:\mathbb{S}^{2}\rightarrow{}^{L\times c}$, where $L$ denotes the number of video frames and $c$ the channels per frame. A novel Panoramic Denoiser is then applied to generate the clean panoramic latent code $S^{0}$, conditioned on the static input panorama $I\in{}^{H\times W}$. Subsequently, the equirectangular projection, as introduced in Sec. 3.1, projects the clean panoramic latent code into the matrix-like latent code $Z^{0}\in{}^{h\times w\times L\times c}$, with $h$ and $w$ representing the resolution of the latent code. Each $k^{\rm{th}}$ video frame $I^{k}$ in pixel space is decoded by the pre-trained VAE decoder as $I^{k}=\mathcal{D}(Z^{0}[:,:,k,:])$.

#### Build the Panoramic Denoiser.

We leverage a pre-trained perspective video generative model (Dai et al., ) to biuld our Panoramic Denoiser.
This video generator takes a perspective image $i\in{}^{p_{H}\times p_{W}\times c}$ and an initial latent code $z^{T}\in{}^{p_{h}\times p_{w}\times(L\times c)}$ as inputs,
progressively denoising the latent code $z^{T}$ to a clean state $z^{0}$ through a denoising function $z^{t-1}=\Phi(z^{t},i)$.
Here, $p_{h}$ and $p_{w}$ represents the resolution of the latent code, $p_{H}$ and $p_{W}$ the resolution of the conditioning image, $c$ the number of channels, and $L$ the video length.
Our goal is to transform the initial noisy panoramic latent code $S^{T}$ into the clean state $S^{0}$, ensuring that each perspective view is appropriately animated while maintaining global consistency. The underlying intuition is that if each perspective view undergoes its respective denoising process, the perspective video will feature meaningful animation. Moreover, if two perspective views overlap, they will align with each other  (Jiménez, ; Bar-Tal et al., ; Lugmayr et al., ) to produce a seamless global animation.

Given a static input panorama $I$ and an initial spherical latent code $S^{0}:\mathbb{S}^{2}\rightarrow{}^{L\times c}$, we progressively remove noise employing a project-and-fuse procedure at each denoising step. Specifically, the spherical latent code at the $t^{\rm{th}}$ denoising step, $S^{t}:\mathbb{S}^{2}\rightarrow{}^{L\times c}$, is projected into multiple perspective latent codes $\mathcal{Z}^{t}=\{z^{t}_{1},z^{t}_{2},\ldots,z^{t}_{n}\}$, where each $z^{t}_{k}=\gamma(S^{t},\bm{d}_{k})\in{}^{p_{h}\times p_{w}\times(L\times c)}$ represents the $k^{\rm{th}}$ perspective latent code projected in the equirectangular format detailed in Sec. 3.1. Each perspective latent code is then denoised by one step using a pre-trained perspective denoiser, denoted as $z_{k}^{t-1}=\Phi(z^{t}_{k},i_{k})$, where $i_{k}=\gamma(I,\bm{d}_{k})\in{}^{p_{H}\times p_{W}\times c}$ is the perspective conditioning image projected from the panorama $I$. Subsequently, we optimize the spherical latent code $S^{t-1}:\mathbb{S}^{2}\rightarrow{}^{L\times c}$ at step $t-1$ by fusing all the denoised perspective latent codes ${z_{k}^{t-1}}$. Formally, the denoising procedure at step $t$, denoted as $S^{t-1}=\Psi(S^{t},I)$, encompasses the following operations:

$$ $\Psi\left(\mathcal{S}^{t},I\right)=\operatornamewithlimits{argmin}_{\mathcal{S }}\mathbb{E}_{\bm{d}\in\mathbb{S}^{2}}\|\gamma(\mathcal{S},\bm{d})-\Phi\left( \gamma(\mathcal{S}^{t},\bm{d}),\gamma(I,\bm{d})\right)\|.$ (3) $$

### 3.4 Dynamic Panoramic Lifting

We define the panoramic video as $V=\{I^{1},I^{2},\ldots,I^{L}\}$, consisting of $L$ frames. The video is divided into overlapping perspective videos $\{v_{0},v_{1},\ldots,v_{n}\}$, each captured from specific camera directions $\{\bm{d}_{1},\ldots,\bm{d}_{n}\}$, collectively encompassing the entire span of the panoramic video $V$. Subsequently, we estimate the geometry of the 4D scene by fusing the depth maps through spatial-temporal geometry alignment. Following this, we describe our methodology for 4D representation and the subsequent rendering procedure.

#### Supervision from Spatial-Temporal Geometry Alignment.

To transition from 2D video to 3D space, we utilize a monocular depth estimator  (Ranftl et al., ), inspired by advancements in  (Zhou et al., ), to estimate the scene’s geometric structure. Nonetheless, depth maps generated for each frame and perspective might lack spatial and temporal consistency. To address this, we implement Spatial-Temporal Geometry Alignment using a pre-trained depth estimator $\Theta:{}^{h\times w\times 3}\rightarrow{}^{h\times w}$, applied to perspective images. Our objective is to amalgamate $n$ perspective depth maps $D^{K}_{i}=\Theta(\gamma(I^{k},\bm{d}_{i}))$ into a cohesive panoramic depth map $D^{k}$ for each frame $I^{k}$, ensuring spatial and temporal continuity. We express these depth maps as a spherical representation ${\mathcal{S}_{D}^{1},\ldots,\mathcal{S}_{D}^{L}}$. For enhanced optimization, we assign $n$ scale factors $\alpha_{i}^{k}\in\real$ and shifting parameters $\beta_{i}^{k}\in{}^{h\times w}$ to each perspective depth map. The comprehensive depth map $D^{k}$ is then optimized jointly with these parameters $\alpha$ and $\beta$. The formal objective is structured as follows:

$$ $\centering\mathcal{S}_{D}^{k}=\operatornamewithlimits{argmin}_{\mathcal{S}} \operatornamewithlimits{\mathbb{E}}_{i\in\left\{1,...n\right\}}\lambda_{\rm{ depth}}\mathcal{L}_{\rm{depth}}+\lambda_{\rm{scale}}\mathcal{L}_{\rm{scale}}+ \lambda_{\rm{shift}}\mathcal{L}_{\rm{shift}}.\@add@centering$ (4) $$

where $\mathcal{L}_{\rm{depth}}=\|\operatorname{softplus}(\alpha_{i}^{k})\Theta(
\gamma(I^{k},d_{i}))-\gamma(\mathcal{S})+\beta_{i}^{k}\|$ is the depth supervision term, $\mathcal{L}_{\rm{scale}}=\|\alpha_{i}^{k}-\alpha_{i}^{k-1}\|+\|\operatorname{
softplus}(\alpha_{i}^{k})-1\|$ the regularize term for $\alpha$, and $\mathcal{L}_{\rm{shift}}=\mathcal{L}_{\rm{TV}}(\beta_{i}^{k})+\|\beta_{i}^{k}-
\beta_{i}^{K-1}\|$ the regularize term for $\beta$ where $\mathcal{L}_{\rm{TV}}$ is the TV regularization.

#### 4D Representation and Rendering.

We represent and render the dynamic scene using $T$ sets of 3D Gaussians. Each set, corresponding to a specific timestamp $t$, is denoted as $G_{t}=\{\left(\bm{p}_{t}^{i},\bm{q}_{t}^{i},\bm{s}_{t}^{i},\bm{c}_{t}^{i},o_{t
}^{i}\right)|i=1,\ldots,n\}$. This definition aligns with the methods described in  (Bahmani et al., ), which also provides a fast rasterizer for rendering images based on these Gaussian sets and given camera parameters. Consistent with Sec. 3.1, while the camera intrinsics remain fixed, we parameterize the camera extrinsics through a position $\bm{p}\in{}^{3}$ and an orientation $\bm{d}\in\mathbb{S}^{2}$. The training process is structured in two stages: initially, we directly supervise the 3D Gaussians using the panoramic videos. Let $\mathcal{R}(G,\bm{p},\bm{d})$ represent the rasterized image from Gaussian set $G$, utilizing camera extrinsics $\bm{p}=0$ and camera direction $\bm{d}$. Let $I_{t}$ denote the $t^{\rm{th}}$ frame of the panoramic video. We optimize the $t^{\rm{th}}$ Gaussian set $G_{t}$ using the following objective:

$$ $\mathcal{L}=\lambda_{\rm{rgb}}\mathcal{L}_{\rm{rgb}}+\lambda_{\rm{temporal}} \mathcal{L}_{\rm{temporal}}+\lambda_{\rm{sem}}\mathcal{L}_{\rm{sem}}+\lambda_{ \rm{geo}}\mathcal{L}_{\rm{geo}}$ (5) $$

where the RGB supervision term $\mathcal{L}_{\rm{rgb}}=\lambda\mathcal{L}_{1}+(1-\lambda)\mathcal{L}_{\rm{SSIM}}$ is the same as 3D-GS  (Kerbl et al., ), and the temporal regularize term $\mathcal{L}_{\rm{temporal}}$ written as:

$$ $\mathcal{L}_{\rm{temporal}}=\sum\limits_{i=1}^{n}\|\mathcal{R}(G_{t},\bm{0}, \bm{d}_{i})-\mathcal{R}(G_{t-1},\bm{0},\bm{d}_{i}))\|$ (6) $$

Then, we adopt the distillation loss and geometric regularization used in  (Zhou et al., ), the distillation loss is defined as follows: $\mathcal{L}_{sem}=1-\operatorname{cos}\left\langle\operatorname{CLS}(\mathcal{
R}(G_{t},\bm{0},\bm{d}_{i})),\operatorname{CLS}(\mathcal{R}(G_{t},\bm{\delta}_
{p},\bm{d}_{i}))\right\rangle$,
where $\bm{\delta}_{p}\in[-\alpha,\alpha]^{3}$ is the disturbing vector, $\operatorname{CLS}(\cdot)$ the feature extractor such as DINO (Oquab et al., ), and $\operatorname{cos}\langle\cdot,\cdot\rangle$ the $\operatorname{cos}$ value of two vectors. The geometric regularization is defined as follows: $\mathcal{L}_{geo}=1-\frac{\operatorname{Cov}(\mathcal{R}_{D}(G_{t},\bm{0},\bm{
d}_{i}),\Theta(\gamma(I,\bm{d}_{i})))}{\sqrt{\operatorname{Var}(\mathcal{R}_{D
}(G_{t},\bm{0},\bm{d}_{i}))\operatorname{Var}(\Theta(\gamma(I,\bm{d}_{i})))}}$, where $\mathcal{R}_{D}$ is the rendered depth, $\operatorname{Cov}(\cdot,\cdot)$ the covariance, and $\operatorname{Var}(\cdot)$ the variance.

## 4 Experiments

Figure: Figure 4: Comparison between 4K4DGen and 3D-Cinemagraphy. We present the input static panorama (Pano RGB), the corresponding text prompts, and the rendered results from different views and at various timestamps. 4K4DGen (Ours) effectively generates 4D scenes that are both spatially and temporally consistent, while 3D-Cinemagraphy (3D-Cin.) suffers from ghosting artifacts in the middle frames.
Refer to caption: x4.png

### 4.1 Experimental Settings

#### Implementation Details.

For perspective images, we uniformly select 20 directions $\bm{u}$ on the sphere $\mathbb{S}^{2}$ as the z-axis of 20 cameras. In each experiment, the image plane size $\bm{s}$ is set at $0.6\times 0.6$, with a focal length $f=0.6$ and a resolution of $512\times 512$. Rotation along the z-axis is kept at zero for all cameras, ensuring that the up-axis for the $i^{\rm{th}}$ camera aligns with the $(O,\bm{u}_{i},\bm{z})$ plane.
During the animating phase, we utilize the perspective denoiser $\Phi$, instantiated as the Animate-anything model  (Dai et al., ), which fine-tunes the SVD model  (Blattmann et al., ). In the Spatial-Temporal Geometric Alignment stage of the lifting phase, the depth estimator $\Theta$ is implemented using MiDaS  (Ranftl et al., ; Birkl et al., ).
All experiments are executed on a single NVIDIA A100 GPU with 80 GB RAM.

#### Evaluation.

As there is no ground truth 4D scene data available, we render videos at specific test camera poses from the synthesized 4D representation and employ non-reference video/image quality assessment methods for quantitative evaluation of our approach. For the test views, we select random cameras with $\bm{p}=0$ as part of our testing camera set. We then introduce disturbances as described in Sec. 3.4, applying a disturbance factor of $\alpha=0.05$ at these selected views.
Datasets. The task of generating 4D panoramas from static panoramas is new, and thus, no pre-existing datasets are available. In line with previous large-scale scene generation works (Zhou et al., ; Yu et al., ), we evaluate our methodology using a dataset of 16 panoramas generated by text-to-panorama diffusion models.
Baselines. Current SDS-based methods  (Wu et al., ; Zhao et al., ) are limited to generating object-centered assets and do not support outward-facing scene generation. We compare our method with the optical-flow-based 3D dynamic image technique, 3D-Cinemagraphy (3D-Cin.) (Li et al., ) (both the “circle” and “zoom-in” mode), by inputting the static panorama and projecting the output onto perspective images.
Metrics. It is challenging to evaluate the visual quality without a ground-truth reference.
We assess the rendered perspective videos based on both frame and video visual quality.
For frame quality, we use distribution-based metrics such as FID (Heusel et al., ) and KID (Bińkowski et al., ), which calculate the distance between generated frames and the corresponding perspective images projected from the static panoramas.
We also employ the LLM-based visual scorer Q-Align (Wu et al., ) to evaluate the quality of individual frames.
For video quality, we use the Q-Align video model as the quality scorer. Additionally, we conduct user studies to further evaluate the results.
In this paper, there are two types of user studies: (1) User Choice (UC), where participants are asked to compare and select the best video from candidates generated by different methods, and (2) User Agreement (UA), where participants assess whether specific properties are present in the videos generated by a particular approach.

**Table 1: Comparison with 3D-Cinemagraphy. The IQ, IA, and VQ models represent the image quality scorer, image aesthetic scorer, and video quality scorer, respectively, within the Q-Align assessment framework. Our method, 4K4DGen, consistently achieves superior performance in both image and video quality across these metrics. Furthermore, the majority of participants in our user study rated 4K4DGen as the best in terms of visual quality.**
| Method | FID $\downarrow$ | KID ($\times 10^{-2}$) $\downarrow$ | Q-Align (IQ) $\uparrow$ | Q-Align (IA) $\uparrow$ | Q-Align (VQ) $\uparrow$ | Quality (UC) $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| 3D-Cinemagraphy (zoom-in) | 57.05 | 1.4 | 0.47 | 0.38 | 0.57 | 7% |
| 3D-Cinemagraphy (circle) | 57.78 | 1.3 | 0.48 | 0.40 | 0.58 | 12% |
| 4K4DGen | 16.59 | 0.2 | 0.66 | 0.44 | 0.62 | 81% |

### 4.2 Results

#### Quantitative Results.

We also show the qualitative comparison between 4K4DGen and 3D-Cinemagraphy (Li et al., ) in Tab. 1, considering both frame and video quality. In terms of frame quality, 4K4DGen achieves significantly better performance on both distribution-based metrics and LLM-based metrics. In terms of video quality, 4K4DGen achieves better Q-Align score, and is selected as the method with best visual quality by 81% of users in our study.

#### Qualitative Results.

We present a qualitative comparison between 4K4DGen and 3D-Cinemagraphy (3D-Cin.). Since the performance of 3D-Cin. is similar under the "circle" and "zoomin" settings in Tab. 1, we use the "circle" setting to represent 3D-Cin. in Fig. 4. As shown in the figure, 4K4DGen produces high-quality perspective videos that maintain consistency across both time and views, whereas 3D-Cin. struggles with generating ghosting artifacts in the middle frames.

Figure: Figure 5: Comparison to Different Animators: Animators trained primarily on perspective images tend to produce limited motion when applied to panoramas, and the resolution may be limited. On the other hand, animating perspective images individually can lead to inconsistencies between overlapping views.
Refer to caption: x5.png

### 4.3 Ablation Studies

We conduct ablation studies for both the animating and lifting phases of our methodology. In the animating phase, we highlight the importance of our spherical denoise strategy by replacing it with two basic animation techniques. In the lifting phase, we analyze the impact of excluding the Spatial-Temporal Geometry Alignment process and the temporal loss during the optimization of 4D representations.

#### Animating Phase.

To animate the panorama into a panoramic video, a straightforward approach is to apply animators directly to the entire panorama. However, we observed that this strategy often results in minor motion, as shown in Fig. 5 (b) and Tab. 2 (Animate Pano.). This issue arises due to two main reasons: (1) animators are typically trained on perspective images with a narrow field of view (FoV), whereas panoramas have a 360^∘ FoV with specific distortions under the equirectangular projection; (2) our panorama is high-resolution (4K), which exceeds the training distribution of most 2D animators and can easily cause out-of-memory issues, even with an 80GB VRAM graphics card. Thus the panoramas have to be down-sampled to a lower resolution (2K), causing a loss of details.
Thus, we seek to animate on perspective views. Applying the animator to perspective views offers benefits such as reduced distortion and domain-appropriate input for the animator, allowing for smooth animation of high-resolution panoramas. However, animating perspective images separately can introduce inconsistencies between overlapping perspective views, as illustrated in Fig. 5 (c) and Tab. 2 (Animate Pers.). To resolve this challenge, we propose simultaneously denoising all perspective views and fusing them at each denoising step, in the spherical latent spaace, which capitalizes on the benefits of animating perspective views while ensuring cross-view consistency. The results are displayed in Fig. 5 (a) and Tab. 2 (Ours).

#### Lifting Phase.

We conduct ablation studies on the Spatial-Temporal Geometry Alignment (STA) module and the temporal loss during the lifting phase. Our findings indicate that removing the STA module leads to a degradation in geometric quality, as shown in Fig. 6 (c). Additionally, omitting the temporal loss introduces artifacts in certain frames, potentially resulting in flickering, as demonstrated in Fig. 6 (b).

Figure: Figure 6: Ablation of the Lifting Phase. Omitting temporal regularization during the optimization of 3D Gaussians results in the appearance of artifacts. The absence of Spatial-Temporal Geometry Alignment causes the degradation of geometric structures.
Refer to caption: x6.png

**Table 2: Different Animation Strategies in the Animating Phase. While the three strategies achieves similar visual quality (Q-Align), animating the entire panorama results in minor motion and reduced resolution (first row). Conversely, animating from perspective views leads to inconsistencies across different views (second row). This is supported by the “user agreement (UA)” study, where 70% of participants identified our method as view-consistent, in contrast to only 33% who consider animation from perspective views to be view-consistent.**
| Animater | Max Pano. Res. | Q-Aline(VQ) $\uparrow$ | Motion$\uparrow$ | View-consistency (UA)$\uparrow$ |  |
| --- | --- | --- | --- | --- | --- |
| Animate Pano. | $2048\times 1024$ | 0.82 | 0.30 | - |  |
| Animate Pers. | $4096\times 2048$ | 0.64 | 0.91 | 33% |  |
| Ours | $4096\times 2048$ | 0.85 | 1.27 | 70% |  |

## 5 Conclusion

#### Conclusion.

We have proposed a novel framework 4K4DGen, allowing users to create high-quality 4K panoramic 4D content using text prompts, which delivers immersive virtual touring experiences.
To achieve panorama-to-4D even without high-quality 4D training data, we integrate generic 2D prior models into the panoramic domain. Our approach involves a two-stage pipeline: initially generating panoramic videos using a Panoramic Denoiser, followed by 4D elevation through a Spatial-Temporal Geometry Alignment mechanism to ensure spatial coherence and temporal continuity.

#### Limitation.

First, the quality of temporal animation in the generated 4D environment mainly relies on the ability of the pre-trained I2V model. Future improvements could include the integration of a more advanced 2D animator.
Second, since our method ensures spatial and temporal continuity during the 4D elevation phase, it is currently unable to synthesize significant changes in the environment, such as the appearance of glowing fireflies or changing weather conditions.
Third, the high-resolution and time-dependent representation of the generated 4D environment necessitates substantial storage capacity, which could be optimized in future work using techniques such as model distillation and pruning.

## Appendix A Appendix

Due to space constraints in the main draft, we include supplementary details and experimental results in the appendix. Specifically, in Sec. B
, we provide details about the acquisition process for the static panoramas used in our experiments. In Sec. C, we offer further explanation of the implementation for both the animation and lifting phases. Finally, in Sec. D, we describe the experimental setup and present additional results.

## Appendix B Acquisition of Panoramas

The static panoramas used in the dataset of the main draft are generated by a text-to-panorama diffusion model, fine-tuned from stable diffusion  (Rombach et al., ) on SUN360. Similar to  (Feng et al., ), this model follows three steps: circular blending, super-resolution, and refinement. The panoramas are initially at a resolution of $6144\times 3072$ and then down-sampled to $4096\times 2048$ using the bi-linear interpolation.

## Appendix C Implementation Details

In this section, we introduce the implementation details of the panoramic animator and the 4D lifting procedure.

#### Implementation of Spherical Representing

For the spherical representation, the continuous spherical mapping $\mathcal{S}_{I}:\mathbb{S}^{2}\rightarrow{}^{C}$ is instantiate as discrete point set $\mathcal{P}=\{p_{i}\}$, which is uniformly sampled from the sphere $\mathcal{S}_{I}$.
We first initialize a icosahedron with 20 triangle faces $\{f_{i}|i=1,\cdots,20\}$ to approximate a real sphere $\mathbb{S}^{2}$. Then we uniformly sample a point set $P_{i}$ on each face $f_{i}$ and union all the point sets together as $\hat{\mathcal{P}}=\cup_{i=1}^{20}P_{i}$. We then obtain the discrete point set $\mathcal{P}$ by projecting $\hat{\mathcal{P}}$ onto the sphere $\mathbb{S}^{2}$ by $\mathcal{P}=\{{p_{i}}/{\|p_{i}\|}~{}|~{}p_{i}\in\hat{\mathcal{P}}\}$.

Figure: Figure 7: Visualizations: We provide more visual results. For each shown case we provide the input panorama, corresponding text prompt, and the rendering from two perspective views.
Refer to caption: x7.png

#### Panoramic Animation Phase

For the Panoramic Animator, we set the video length $L=14$, the channel number $c=9$, the latent code size $(h,w)=\frac{1}{8}(H,W)$, the perspective image size $p_{H}=p_{W}=\frac{1}{4}W$. The sphere is uniformly divided into 20 perspective views, each with 80^∘ FOV. For the denoiser, the max denoising step is $25$. For the continuous optimization in Eq. 3, we use a close form, where each latent vector at each point on the sphere is the average of the latent vectors of the corresponding pixel on the perspective views that overlap it. The perspective denoiser is initiated as Animate-Anything Dai et al. (). The masks required by the denoiser are given by bounding boxes defined by user clicks.

#### Dynamic Panoramic Lifting Phase

In the lifting phase, similar to the animation phase, we choose the perspective view number $n=20$, each with 80^∘ FOV. Each perspective view has a square shape, $P_{H}=P_{W}=\frac{1}{4}W$, where $W$ is the width of the original static panorama. In the Spatial-Temporal Geometric Alignment stage, the depth estimator $\Theta$ is implemented using MiDaS  (Ranftl et al., ; Birkl et al., ). The depth map from the perspective image is scaled according to the projection of the unit-length ray direction onto the camera orientation $\bm{d}$. Formally, if the pre-scaled depth is $d$ at point $p\in\hat{\mathcal{P}}$ introduced above, the scaled depth should be $d/\|p\|$.
Additionally, for scenes without distinct boundaries, such as the sky, depth values for distant elements are assigned a finite value to support optimization.

#### Optimization Details

The hyper-parameters for optimization are set as follows: $\lambda_{\rm{depth}}=1,\lambda_{\rm{scale}}=0.1,\lambda_{\rm{shift}}=0.01$. We conduct Spatial-Temporal Geometry Alignment optimization over 3000 iterations, with $\lambda_{\rm{scale}}$ and $\lambda_{\rm{shift}}$ set to zero during the first 1500 iterations.
For the 4D representation training stage, Gaussian parameters are optimized over 10000 iterations for each time stamp $t$. The hyper-parameters for this stage are defined as $\lambda_{\rm{rgb}}=1,\lambda_{\rm{temporal}}=\lambda_{\rm{sem}}=\lambda_{\rm{
geo}}=0.05$, and the disturbance vector range $\alpha$ is varied at $0.05$, $0.1$, and $0.2$ during the $5400$, $6600$, and $9000$ iterations, respectively.

## Appendix D Experimental Details

### D.1 User Study Details

We conducted two user studies, gathering a total of 84 questionnaires from 42 users. For the “Quality (UC)”column in Tab. 1, we collected 42 questionnaires, each containing eight questions.
Each question asked users to choose the bests video in term of visual quality from the perspective videos provided by different models. The user choice (UC) score of a method is the percentage of times the method’s video was selected as the best one, out of a total of 336 questions. Thus, the UC scores for all methods sum to 100%.
For the “View-Consistency (UA)” column in Tab. 2, we collected another 42 questionnaires, with each questionnaire containing eight questions.
Each question presented two videos from different views, both generated by the same method, and users were asked to determine whether the two videos were view-consistent. The user agreement (UA) score is the percentage of video pairs marked as view-consistent out of all the video pairs generated by the method. The UA scores do not necessarily sum to 100%.
In the UC column of Tab. 1, among the 336 questions, users selected 4K4DGen 272 times, 3D-Cin. (circle) 40 times, and 3D-Cin. (zoomin) 24 times. In the UA column of Tab. 2, 118 out of 168 video pairs generated by “Our” were marked as consistent, while 56 out of 168 pairs from “Animate Pers” were considered consistent.

### D.2 More Results

We provide additional qualitative results in Fig. 7. Furthermore, we highly recommend viewing the video renderings of 4K4DGen and comparisons to baseline models in the supplementary static HTML page for a more comprehensive and immersive experience.

## Appendix E Ethics and Reproducibility Statement

#### Ethics Statement.

Our research enables the generation of 4D digital scenes from a single panoramic image, which is advantageous for various applications such as AR/VR, movie production, and video games. This technology distinctly excels in creating high-resolution 4D scenes up to 4K, significantly enhancing user experiences. However, there is potential for misuse in the creation of deceptive content or privacy violations, which contradicts our ethical intentions. These risks can be mitigated through a combination of regulatory and technical strategies, such as watermarking.

#### Reproducibility.

We provide sufficient implementation details to reproduce our methodology in Sec. C, including the details of spherical denoiser, panoramic animator, dynamic panoramic lifting, etc. Furthermore, we will release the code in the future.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * Have the authors considered using a more recent depth model than MiDaS?  It's pretty old at this point.
soundness: 3
strengths: * compelling use case
* qualitative results look good
* quantitative metrics show improvements over baselines
* well written
summary: This paper presents a method by which a static pano image is animated to produce a video pano, then subsequently lifted to 3D using 3DGS.  The authors claim the following contributions:
* 4K4DGen, a method for lifting a static pano to dynamic 3DGS
* A panoramic denoiser which generates consistent animated perspective views 
* A dynamic panoramic lifting which transforms the video pano to dynamic 3DGS
weaknesses: * The animations each have small spatial extent.  I didn't see an example that illustrated that you could have many of these overlapping perspective animations and for a large scale animation to emerge.  e.g., It would be much more impressive if the authors could show the car in the desert where the car drives around the center of project.

* Some of the generated videos have significant artifacts.  For example, the fireworks in the third row of the supplemental has fireworks that don't move and speckled artifacts that are static.

* The motion isn't always globally consistent.  For example, I'm not sure the clouds in row 7 of the supplemental make sense.

* I think overall, the idea of applying this existing model to perspective patches is neat, but I'd be concerned this work would be quickly superseded by one where the diffusion model was fine-tuned on 360 videos.

* The lifting phase doesn't seem sufficiently evaluated.  It would have been nice to have something to inspect in the supplemental.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: My major question is on the necessities of the dynamic panoramic lifting and whether that actually improves the quality when we move the camera and look around. In the submitted video, we can see a little bit camera movements but I wonder if that can be mimiced but simply textured the panorama to a large enough sphere. Also when handling the dynamic scene, why not consider all the dynamic Gaussian papers and instead use static Gaussian with a temporal constraints? If we use dynamic Gaussian representation, will that cause any differences?
soundness: 3
strengths: 1. As far as I know, this is the first paper to generate high-resolution 4D panorama videos, which may have many applications in VR. 

2. The proposed method is technically solid and easy to follow. The consistent panoramic animation part is similar to prior works that use 2D diffusion model to generate panorama but the paper replaces 2D diffusion models with 2D video diffusion model. The dynamic panoramic lifting part is also reasonable. 

3. While evaluating generative models is always a difficult problem and there is no prior works solving this problem, this paper still manages to build a set of benchmarks and a reasonable baseline to show the effectiveness of the proposed framework, which should be appreciated. 

5. The paper is well-written and easy to follow.
summary: This paper proposes a novel framework to generate 4K4G panoramic video rendered from 3D Gaussian representation. The key technical insights is to use pre-trained 2D perspective camera to denoise the panorama by first rendering it into perspective image patch and then fusing it back into the sphere space. Once a panoramic video is created, a lifting process built on single image depth prediction method is to turn each frame into 3D Gaussian representation so that we can navigate inside the panorama. Both quantitative and qualitative experiments verified the effectiveness of each technical components and overall the framework achieves impressive video generation results.
weaknesses: I am not an expert in using diffusion models for video generation and I haven't followed recent progress in that direction. The consistent panoramic animation part looks solid and reasonable to me. I only have some minor questions that are not necessarily the weakness of the paper. 

1. Eq. (3): While this equation is similar to how we generated static panorama using diffusion models for 2D perspective image, I wonder whether that will cause different Gaussian noise distribution at the overlap regions. Since we are essentially average the Gaussian noise from two denoised latent code, will that cause the variance of the noise at the overlap region to be smaller compared to non-overlapping area? Will that cause artifacts in image generation?

2. My second question is how significant the dynamic panoramic lifting can help us render 3D video. Given that all the examples shown in the paper have very large-scale scene and the perturbation $\alpha$ is very small in training, I would guess whether we lift the panorama into 3D or not may not make a big difference and approximating the whole 3D structure with a sphere may give us similar results. I assume the comparison is shown in Figure 6(c) but it is very difficult for me to see the difference. 

3. Also I wonder why don't we choose recent dynamic Gaussian works to model the video. I assume that is a more systematic way to handle this problem and may give us more temporal consistent view synthesis results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: My main concern is the lack of baselines in the paper but the problem statement, conceptualization and results are very convincing hence the accept rating.

Question(s):
Q1. What is the processing times per generation on the A100 used? I would also be interested in knowing how memory intensive it is.
Q2. If more baselines are added (see weaknesses), I would be willing to upgrade my recommendation score to 8.

Suggestion(s):
* L295: Fix typo: biuld -> build
* The space is low but can the authors somehow manage to split the Evaluation paragraph starting on L407? (split at the each underlined term)
soundness: 3
strengths: 1. Novelty: This is the first paper that allows seamless (spatio-temporally aligned) 4K resolution generation of 4D content.
2. The spherical latent space operation alleviates a lot of visual artifacts and this paper is evidence that it is suitable for Panoramic generation tasks.
3. The model components, intuition and math is well grounded.
4. The fidelity of results is promising. Quantitative metrics suggest the same. (The resulting panorama videos provided on the supplementary page are extremely visually pleasing!)
summary: The paper proposes an end-to-end solution for generating realistic 4D panoramic videos from static panoramas at 4K resolution for the first time. The paper uses pretrained image to video models to convert the static 2D panoramas to videos and then dynamically (with spatio-temporal alignment) lift them to 3D space resulting in the 4D content. 
The paper demonstrates the appropriateness of using a spherical latent space for the task and how the panoramic denoiser utilizes that. The lifting uses a geometry alignment model (pre-trained depth estimator) and optimize DreamScene360's objective with an additional temporal loss. 

The framework achieves high fidelity generations, both, visually and quantitatively. The user-study also aligns well with the results.
weaknesses: 1. Why are Efficient4D and 4DGen not used as baselines that potentially fail on Panoramas as there is no spatial-temporal alignment mechanism there? It would make the paper much stronger. 
2. It would have been nice to see comparisons for the 4D lifting phase to see how the proposed is better than existing methods like: OmniNeRF. This way the speed of the proposed method can also be highlighted as a strength of the paper.
3. More visual results in the main text would be nice.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: (1) How much time and GPU memory are required to generate a single 4K video?
soundness: 3
strengths: (1) The paper first proposes animatable 360-degree 4K panoramic images. 

(2) To create 360-degree panorama with fixed-size pretrained video generation and depth estimation models, the authors carefully design the projection function. 

(3) The qualtative results shows that great spatial and temporal consistency.
summary: The paper first proposes a method, 4D4KGen, which can generate immersive 360-degree, 4K panoramic 4D videos from a single 4K static panoramic image. The method first uses the pre-trained perspective video generative model AnimateAnything to create a panoramic video through a project-and-fuse scheme with a spherical latent space. Next, the pretrained monocular depth estimator MiDaS is used to create a panoramic depth map with a spherical representation. Finally, the 4D animated panoramic videos are rendered using reconstructed 3DGS. Qualitative results show that the proposed method can effectively generate immersive 360-degree 4D panoramic videos.
weaknesses: (1) I understand that there is no metric to evaluate 360-degree panoramic images, but there is insufficient basis for measuring FID and KID between the generated frames and the corresponding perspective images projected from the static panoramas. In particular, these metrics may be meaningless when the reference set is small, and in this paper, only 16 panoramic videos were generated. Proposing and measuring such less meaningful metrics, especially when this paper aims to serve as a baseline for future 4D 360-degree panorama video generation, does not seem beneficial for subsequent research. Additionally, if we want to assess the fidelity of the generated video, we could measure FVD [1] and KVD [1]. Therefore, I recommend removing FID and KID.

(2) I don't know how long the proposed method takes, but generating only 16 videos seems too few. To demonstrate that the proposed method works generally on many 4K 360-degree panoramic images, it would be better to generate more videos (at least 30 or more) to measure quantitative results.

(3) In Sec. 4.3 Ablation Studies, a straightforward approach would be to apply animators directly to the entire panorama as the direct baseline, measured by the Q-Align metrics (IQ, IA, and VQ) and also through a user study. If 2D animators cause out-of-memory issues, we can simply upsample each frame of the videos using a super-resolution (upsampling) model such as StableSR [2].

(4) In Fig. 6, Ablation of the Lifting Phase, I find it difficult to visually discern the difference between the proposed temporal regularization and Spatial-Temporal Geometry Alignment (STA). The authors should have included a video of this in the supplementary materials or shown a more definitive difference in the figure.

(5) Since the paper's topic is closely related to panoramic image generation, the following papers [3, 4, 5] should be mentioned in the related work section.

(6) Please double-check the citations (bibtex) to confirm which papers were accepted by specific conferences. Too many papers published in conferences are listed as arXiv.


The proposed method demonstrates good quality in the new research field of 4D panoramic video generation, but it lacks convincing evidence for the choice of pipeline components and is missing many essential elements that a paper should fundamentally include. I believe it lacks the basic qualities necessary for publication in a top-tier ML conference ICLR.


[1] Unterthiner et al., Towards Accurate Generative Models of Video: A New Metric & Challenges, arXiv preprint arXiv:1812.01717, 2018.

[2] Wang et al., Exploiting Diffusion Prior for Real-World Image Super-Resolution, IJCV2024

[3] Bar-Tal et al., MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation, ICML 2023

[4] Lee et al., SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions, NeurIPS 2023

[5] Quattrini et al., Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas, ECCV 2024

Meta-review: The paper presents an approach to lift 2D panorama images to 3D and animate them.
It builds on existing work for animation and depth estimation, and proposes an adoption to panoramic data.
To handle scarcity of training data, the paper leverages 2D diffusion models.

The reviewers appreciated the novelty of the proposed work and the qualitative results. 
The paper was considered well written. The method is quantitatively evaluated against reasonable baselines.

The main concerns were raised by jzFB who remarked a limited set of experimental results and those being an insufficient basis for measuring FID and KID. Further concerns regarded ablation experiments and comparisons to baselines. All of these concerns were addressed during the discussion phase with the reviewers finally recommending to unanimously accept the paper.

Decision: Accept (Spotlight)