Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models

## Abstract

Abstract 3D meshes are widely used in computer vision and graphics because of their efficiency in animation and minimal memory footprint.
They are extensively employed in movies, games, AR, and VR, leading to the creation of a vast number of mesh sequences.
However, creating temporally consistent and realistic textures for these mesh sequences remains labor-intensive for professional artists.
On the other hand, video diffusion models have demonstrated remarkable capabilities in text-driven video generation, enabling users to create countless video clips based solely on their imagination.
Despite their strengths, these models often lack 3D geometry awareness and struggle with achieving multi-view consistent texturing for 3D mesh sequences.
In this work, we present Tex4D , a zero-shot approach that integrates inherent 3D geometry knowledge from mesh sequences with the expressiveness of video diffusion models to produce multi-view and temporally consistent 4D textures.
Given an untextured mesh sequence and a text prompt as inputs, our method enhances multi-view consistency by synchronizing the diffusion process across different views through latent aggregation in the UV space.
To ensure temporal consistency, we leverage prior knowledge from a conditional video generation model for texture synthesis.
However, straightforwardly combining the video diffusion model and the UV texture aggregation leads to blurry results. We analyze the underlying causes and propose a simple yet effective modification to the DDIM sampling process to address this issue.
Additionally, we introduce a reference latent texture to strengthen the correlation between frames during the denoising process.
To the best of our knowledge, Tex4D is the first method specifically designed for 4D scene texturing.
Extensive experiments demonstrate its superiority in producing multi-view and multi-frame consistent videos based on untextured mesh sequences.
Project page: https://tex4d.github.io .

## 1 Introduction

3D meshes are widely used in 3D modeling, computer-aided design (CAD), animation, and computer graphics due to their low memory footprint and efficiency in animation.
Visual artists, game designers, and movie creators frequently create numerous animated 3D mesh sequences for various applications.
However, creating temporally consistent and realistic textures for these mesh sequences remains labor-intensive and requires significant professional expertise.

On the other hand, recent advancements in generative models have democratized content creation and demonstrated impressive performance in image and video synthesis.
For instance, video generation models (Ho et al., ; Esser et al., ; Li et al., ; He et al., ; Yu et al., ; Zhou et al., ; Hong et al., ; Yang et al., ; Zhang et al., ; Xing et al., ; Chen et al., ; ) trained on large-scale video datasets (Bain et al., ; Schuhmann et al., ) allow users to create realistic video clips from various inputs such as text prompts, images, or geometric conditions.
However, these text-to-video generation models, which are trained solely on 2D data, often struggle with spatial consistency when applied to multi-view image generation (Tang et al., ; Shi et al., ; Liu et al., ; Weng et al., ; Long et al., ; Shi et al., ; Kwak et al., ; Tang et al., ; Voleti et al., ) or 3D object texturing (Cao et al., ; Liu et al., ; Richardson et al., ; Chen et al., ; Huo et al., ).

To address these limitations, two main approaches have been developed.
One approach (Richardson et al., ; Chen et al., ; Cao et al., ) focuses on resolving multi-view inconsistency in static 3D object texturing by synchronizing multi-view image diffusion processes and enforcing UV space consistency.
While these methods produce multi-view consistent textures for static 3D objects, they do not address the challenge of generating temporally consistent textures for mesh sequences.
Another approach (Guo et al., ; Lin et al., ; Peng et al., ) aims to generate temporally consistent video clips based on the rendering (e.g., depth, normal or UV maps) of an untextured mesh sequence. To encourage temporal consistency, these methods modify the attention mechanism in 2D diffusion models and utilize inherent correspondences in a mesh sequence to facilitate feature synchronization between frames. Although these techniques can be adapted for multi-view image generation by treating camera pose movement as temporal motion, they usually produce inconsistent 3D texturing due to insufficient exploitation of 3D geometry priors.

In this paper, we introduce a novel task: 4D scene texturing. Given an animated untextured 3D mesh sequence and a text prompt, our goal is to generate textures that are both temporally and multi-view consistent.
Different from existing works, we fully leverage 3D geometry knowledge from the mesh sequence to enforce multi-view consistency.
Specifically, we develop a method that synchronizes the diffusion process from different views through latent aggregation in the UV space.
To ensure temporal consistency, we employ prior knowledge from a conditional video generation model for texture synthesis and introduce a reference latent texture to enhance frame-to-frame correlations during the denoising process.
However, naively integrating the UV texture aggregation into the video diffusion process derails the denoising process, leading to blurry results.
To resolve this issue, we analyze the underlying causes and propose a simple yet effective modification to the DDIM (Song et al., ) sampling process.
Additionally, we propose to synthesize a dynamic background along with the textures of the given mesh sequences, which not only creates a complete 4D scene but also fully exploits the prior knowledge embedded in the video diffusion model. Our method is computationally efficient thanks to its zero-shot nature. The textured mesh sequence can be rendered from any camera view, thus supporting a wide range of applications in content creation.

We evaluate our method on various animated mesh sequences and show its superior performance compared to state-of-the-art baseline methods.
The key contributions of our work are as follows:

- •
We present Tex4D, a novel zero-shot pipeline for generating high-fidelity textures that are temporally and multi-view consistent, utilizing pre-trained text-to-video diffusion models conditioned on untextured low-fidelity mesh sequences.
- •
We develop a simple and effective modification to the DDIM sampling process to address the variance shift issue caused by multi-view texture aggregation.
- •
We introduce a reference UV blending mechanism to establish correlations during the denoising steps, addressing self-occlusions, and synchronizing the diffusion process in invisible regions.
- •
Our method is not only computationally efficient, but also demonstrates comparable if not superior performance to various state-of-the-art baselines.

## 2 Related Work

#### Video Stylization and Editing

Text-to-video diffusion models have shown remarkable performance in the field of video generation. These models learn motions and dynamics from large-scale video datasets using 3D-UNet to create high-quality, realistic, and temporally coherent videos.
Although these approaches show compelling results, the generated videos lack fine-grained control, inhibiting their application in stylization and editing. To solve this issue, inspired by ControlNet (Zhang et al., ), SparseCtrl (Guo et al., ) trains a sparse encoder from scratch using frame masks and sparse conditioning images as input to guide a pre-trained video diffusion model. CTRL-Adapter (Lin et al., ) proposes a trainable intermediate adapter to efficiently connect the features between ControlNet and video diffusion models.

Meanwhile, Tumanyan et al. () observed that the spatial features of T2I models play an influential role in determining the structure and appearance, Text2Video-Zero (Khachatryan et al., ) uses a frame-warping method to animate the foreground object by T2I models and Wu et al. (); Ceylan et al. (); Qi et al. () propose utilizing self-attention injection and cross-frame attention to generate stylized and temporally consistent video using DDIM inversion (Song et al., ).
Subsequently, numerous works (Zhang et al., ; Cai et al., ; Yang et al., ; Geyer et al., ; Eldesokey & Wonka, ) generate temporally consistent videos utilizing T2I diffusion models by spatial latent alignment without training. However, the synthesized videos usually show flickerings due to the empirical correspondences, such as feature embedding distances and UV maps, which are insufficient to express the continuous content in the latent space. Another line of work (Singer et al., ; Bar-Tal et al., ; Blattmann et al., ; Xu et al., ; Guo et al., ) is to train additional modules on large-scale video datasets to construct feature mappings, for example, Text2LIVE (Bar-Tal et al., ) applies test-time training with the CLIP loss, and MagicAnimate (Xu et al., ) introduced an appearance encoder to retain intricate clothes details.

#### Texture Synthesis

With the rapid development of foundation models, researchers have focused on applying their generation capability and adaptability to simplify the process of designing textures and reduce the expertise required.
To incorporate the result 3D content with prior knowledge, earlier works (Khalid et al., ; Michel et al., ; Chen et al., ) jointly optimize the meshes and textures from scratch with the simple semantic loss from the pre-trained CLIP (Radford et al., ) to encourage the 3D alignment between the generated results and the semantic priors. However, the results show apparent artifacts and distortion because the semantic feature cannot provide fine-grained supervision during the generation of 3D content.

DreamFusion (Poole et al., ) and similar models (Lin et al., ; Wang et al., ; Po & Wetzstein, ; Metzer et al., ; Chen et al., ) distill the learned 2D diffusion priors from the pre-trained diffusion models (Rombach et al., ) to synthesize the 3D content by Score Distillation Sampling (SDS).
These methods render 2D projections of the 3D asset parameters and compare them against reference images, iteratively refining the 3D asset parameters to minimize the discrepancy of the target distribution of 3D shapes learned by the diffusion model.
Although these approaches enable people without expertise to generate detailed 3D content by textual prompt, their results are typically over-saturated and over-smoothed, hindering their application in actual cases.
Another line of optimization-based methods (Yu et al., ; Zeng et al., ; Bensadoun et al., ) turned to fuse 3D shape information, such as vertex positions, depth maps and normal maps, with the pre-trained diffusion model by training separate modules on 3D datasets. Still, they require a specific UV layout process to achieve plausible results.

Recently, TexFusion (Cao et al., ) and numerous zero-shot methods (Liu et al., ; Richardson et al., ; Chen et al., ; Huo et al., ) have shown significant success in generating globally consistent textures without additional 3D datasets. Based on depth-aware diffusion models, they sequentially inpaint the latents in the UV domain to ensure the spatial consistency of latents observed across different views. Then, they decode the latents from multiple views and finally synthesize the RGB texture through backprojection.

However, these methods primarily focus on generating static 3D assets and do not account for temporal changes in the final visual presentation, such as in videos. Our work introduces a training-free framework that enables multi-view consistent video generation based on animated meshes. To the best of our knowledge, this is the first approach to synthesize multi-view and multi-frame consistent textures for mesh sequences.

## 3 Preliminaries

#### Video Diffusion Prior.

In this paper, we adopt CTRL-Adapter (Lin et al., ) as our prior model to provide dynamic information.
CTRL-Adapter aims to adapt a pre-trained text-to-video diffusion model to condition various types of images such as depth or normal map sequences.
The key idea behind CTRL-Adapter is to leverage a pre-trained ControlNet (Zhang et al., ) and to align its latents with those of the video diffusion model through a learnable mapping module.
Intuitively, the video diffusion model generates temporally consistent video frames that capture dynamic elements like character motions and environmental lighting, while the ControlNet further enhances this capability by allowing the model to condition on geometric information, such as depth and normal map sequences.
This makes CTRL-Adapter particularly effective in providing a temporally consistent texture prior for our 4D scene texturing task.
Specifically, we leverage the depth-conditioned CTRL-Adapter model. Given a sequence of depth images denoted as $\{D_{1},...,D_{K}\}$ and a text prompt $\bm{\mathcal{P}}$, CTRL-Adapter (denoted as $\bm{\mathcal{C}}$) synthesizes a frame sequence $F$ by $F=\bm{\mathcal{C}}(\{D_{1},...,D_{K}\},\bm{\mathcal{P}})$.

#### DDIM Sampling.

DDIM (Song et al., ) is a widely used sampling method in diffusion models due to its superior efficiency and deterministic nature compared to DDPM (Ho et al., ).
To enhance numerical stability and prevent temporal color shifts in Video Diffusion Models (VDMs), numerous models (Zhang et al., ; Ho et al., ) employ a learning-based sampling technique known as v-prediction (Salimans & Ho, ).
At each denoising step, the DDIM sampling process for the latents (denoted as $\bm{z}_{t}$) can be described as follows:

$$ $\bm{z}_{t-1}=\sqrt{\alpha_{t-1}}\cdot\hat{\bm{z}}_{0}(\bm{z}_{t})+\sqrt{1- \alpha_{t-1}}\cdot{\bm{\epsilon}}_{\theta}(\bm{z}_{t}),$ (1) $$

$$ $\quad\hat{\bm{z}}_{0}(\bm{z}_{t})=\frac{\bm{z}_{t}-\sqrt{1-\alpha_{t}}\cdot{ \bm{\epsilon}}_{\theta}}{\sqrt{\alpha_{t}}},\quad{\bm{\epsilon}}_{\theta}(\bm{ z}_{t})={\bm{\epsilon}}_{\theta},$ (2) $$

where $\alpha_{t}$ represents the noise variance at time step $t$, ${\bm{\epsilon}}_{\theta}$ is the estimated noise from the U-Net denoising module, which is expected to follow $\mathcal{N}(0,\mathcal{I})$, and $\hat{\bm{z}}_{0}(\bm{z}_{t})$ denotes the predicted original sample (i.e., the latents at timestep 0).
After the v-parameterization, the predicted original sample $\hat{\bm{z}}_{0}(\bm{z}_{t})$ and the predicted epsilon ${\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ are computed as follows:

$$ $\hat{\bm{z}}_{0}(\bm{z}_{t})=\sqrt{\alpha_{t}}\cdot\bm{z}_{t}-\sqrt{1-\alpha_{ t}}\cdot{\bm{\epsilon}}_{\theta},\quad{\bm{\epsilon}}_{\theta}(\bm{z}_{t})= \sqrt{\alpha_{t}}\cdot{\bm{\epsilon}}_{\theta}+\sqrt{1-\alpha_{t}}\cdot\bm{z}_ {t}.$ (3) $$

In this paper, we leverage an enhanced DDIM sampling process in video diffusion models, along with a multi-view consistent texture aggregation mechanism to synthesize 4D textures.

## 4 Method

Given an animated, untextured mesh sequence and a text prompt, our goal is to generate multi-view and multi-frame consistent texture for each mesh that aligns with both the text description and motion cues.
To optimize computational efficiency, instead of processing all video frames, we uniformly sample $K$ key frames from the video and synthesize textures specifically for these key frames. The textures for the remaining frames are then generated by interpolating the key frame textures.
Formally, given $K$ animated meshes at the key frames ($\{M_{1},...,M_{K}\})$, along with a text description $\bm{\mathcal{P}}$, our method produces a sequence of temporally and spatially consistent UV maps denoted as $\{UV_{1},...,UV_{K}\}$, in a zero-shot manner.

Figure: Figure 2: Overview of our pipeline. Given a mesh sequence and a text prompt as inputs, Tex4D generates a UV-parameterized texture sequence that is both globally and temporally consistent, aligning with the prompt and the mesh sequence. We sample multi-view video sequences using a depth-aware video diffusion model. At each diffusion step, latent views are aggregated into UV space, followed by multi-view latent texture diffusion to ensure global consistency. To maintain temporal coherence and address self-occlusions, a Reference UV Blending module is applied at the end of each step. Finally, the latent textures are back-projected and decoded to produce RGB textures for each frame.
Refer to caption: x2.png

Previous texture generation methods  (Richardson et al., ; Chen et al., ; Cao et al., ) typically inpaint and update textures sequentially using pre-defined camera views in an incremental manner.
However, these approaches rely on view-dependent depth conditions and lack global spatial consistency, often resulting in visible discontinuities in the assembled texture map. This issue arises from error accumulation during the autoregressive view update process, as noted by Bensadoun et al. ().
To resolve these issues, rather than processing each view independently, recent methods (Liu et al., ; Huo et al., ; Zhang et al., ) propose to generate multi-view textures simultaneously through diffusion, and then aggregate them in the UV space at each diffusion step.
In this work, we similarly leverage the UV space as an intermediate representation to ensure multi-view consistency during texture generation.

### 4.1 Overview

As shown in Fig. 2, given a sequence of $K$ meshes, we start by rendering the mesh at $V$ predefined, uniformly sampled camera poses to obtain multi-view depth images (denoted as $\{D_{1,1},...,D_{1,K},D_{2,1}...,D_{V,K}\}$), which serve as the geometric conditions.
To generate textures for each mesh, we initialize $V\times K$ noise images sampled from a Normal distribution (denoted as $\{\bm{z}^{1,1},...,\bm{z}^{1,K},\bm{z}^{2,1},...,\bm{z}^{V,K}\}$).
Additionally, we initialize an extra noise map sequence $\{\bm{z}_{b}^{1},...,\bm{z}_{b}^{K}\}$ for the backgrounds learning. This noise map corresponds to the texture of a plane mesh that is composited with the foreground object at each diffusion step (See Sec. 4.3).
Next, for each view $v\in\{1,...,V\}$, we apply the video diffusion model (Lin et al., ) discussed in Sec. 3 to simultaneously denoise all latents and obtain multi-frame consistent images as $\{I^{1,v},...,I^{K,v}\}=\bm{\mathcal{C}}(\{D_{1,v},...,D_{K,v}\},\bm{\mathcal{
P}})$, where $\bm{\mathcal{P}}$ is the provided text prompt.
Finally, we un-project and aggregate all denoised multi-view images for each mesh to formulate temporally consistent UV textures.

However, applying the video diffusion model independently to each camera view often results in multi-view inconsistencies. Inspired by (Liu et al., ; Huo et al., ; Zhang et al., ), we aggregate the multi-view latents of each mesh in the UV space at every diffusion step. We then render multi-view consistent latents for the subsequent denoising steps. To simultaneously generate a dynamic background and fully exploit prior in the video diffusion model, we composite the rendered foreground latents with the background latents at each diffusion step. This aggregation process is discussed in detail in Sec. 4.2.
Nonetheless, such a simple aggregation method introduces blurriness in the final results. In Sec. 4.3, we analyze the underlying causes and propose a simple yet effective method to enhance the denoising process.
Additionally, we create and leverage a reference UV to handle self-occlusions and further improve temporal consistency in Sec. 4.4.

### 4.2 Multi-view Latents Aggregation in the UV Space

We describe how to aggregate multi-view latent maps in the UV space.
Taking frame $k\in\{1,...,K\}$ as an example, we aggregate the multi-view latents $\{\bm{z}^{1,k},\dots,\bm{z}^{V,k}\}$ in the UV space by:

$$ $\mathcal{T}^{k}\left(\bm{z}^{k}\right)=\frac{\sum_{v=1}^{V}\mathcal{R}^{-1}({ \bm{z}}^{v,k},c_{v})\odot\text{cos}\left(\theta^{v}\right)^{\alpha}}{\sum_{v=1 }^{V}\text{cos}\left(\theta^{v}\right)^{\alpha}},$ (4) $$

where $\mathcal{R}^{-1}$ represents the inverse rendering operator that un-projects the latents to the UV space, thus $\mathcal{R}^{-1}({\bm{z}}^{v,k},{\bm{c_{v}}})$ produces a partial latent UV texture from view $v$, $\text{cos}(\theta^{v})$ is the cosine map buffered by the geometry shader, recording the cosine value between the view direction and the surface normal for each pixel, $\alpha$ is a scaling factor, and ${\bm{c_{v}}}$ denotes one of the predefined cameras.
After multi-view latents aggregation, we obtain multi-view consistent latents by rendering the aggregated UV latent map using $\tilde{\bm{z}}^{v,k}=\mathcal{R}\left(\mathcal{T}^{k};{\bm{c}}_{v}\right)$, where $\mathcal{R}$ is the rendering operation.

Figure: Figure 3: Ablation studies on the multi-view denoising algorithm and backgrounds. (a) Aggregating $\hat{\bm{z}}_{0}(\bm{z}_{t})$, ${\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ in Eq. 1 into UV space. (b) Aggregating $\bm{z}_{t-1}$ in Eq. 1 into UV space. (c) Replacing learnable background with white background. (d) Our results. See Sec. 5.3 for details.
Refer to caption: x3.png

### 4.3 Multi-frame Consistent Texture Generation

The aggregation process discussed above yields multi-view consistent latents $\{\tilde{\bm{z}}^{v,k}\}$ for the subsequent denoising steps.
However, this simple aggregation and projection strategy leads to a blurry appearance as shown in Fig. 3(b).
This issue arises primarily because the aggregation process depicted in Eq. 4 derails the DDIM denoise process.
Specifically, the estimated noise ${\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ for each step in Eq. 1 is expected to follow $\mathcal{N}(0,\mathcal{I})$, but Eq. 4 indicates that after aggregating multi-view latents, the expected norm of variance of the noise distribution would be less than $\mathcal{I}$. We denote this as the “variance shift” issue caused by the texture aggregation.

To resolve this issue, we rewrite the estimated noise $\epsilon_{\theta}$ for a latent as the combination of the t-step latent $\bm{z}_{t}$ and the estimated latent $\hat{\bm{z}}_{0}(\bm{z}_{t})$ at step 0, thus the v-paramaterized predicted epsilon ${\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ in Eq. 3 can be equally expressed as follow:

$$ $\displaystyle{\bm{\epsilon_{\theta}}}$ $\displaystyle=\left(\sqrt{\alpha_{t}}\cdot\bm{z}_{t}-\hat{\bm{z}}_{0}(\bm{z}_{ t})\right)/\sqrt{1-\alpha_{t}}$ (5) $\displaystyle{\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ $\displaystyle=\sqrt{\alpha_{t}}\cdot{\bm{\epsilon_{\theta}}}+\sqrt{1-\alpha_{t }}\cdot\bm{z}_{t}$ $\displaystyle=\sqrt{\frac{\alpha_{t}}{1-\alpha_{t}}}\cdot(\sqrt{\alpha_{t}}\bm {z}_{t}-\hat{\bm{z}}_{0}(\bm{z}_{t}))+\sqrt{1-\alpha_{t}}\cdot\bm{z}_{t}.$ $$

In practice, we carry out this denoising technique in the UV space. Specifically, we first compute the original texture map (i.e., texture map at step 0, denoted as $\hat{\mathcal{T}}_{0}$) by aggregating the predicted original multi-view image latents through Eq. 4. The noisy latent texture map at time step $t$ (denoted as $\mathcal{T}_{t}$) can be similarly computed. We then run one desnoising step by:

$$ $\mathcal{T}_{t-1}=\sqrt{\alpha_{t-1}}\cdot\hat{\mathcal{T}}_{0}+\sqrt{1-\alpha _{t-1}}\left(\sqrt{\frac{\alpha_{t}}{1-\alpha_{t}}}\cdot(\sqrt{\alpha_{t}} \mathcal{T}_{t}-\hat{\mathcal{T}}_{0})+\sqrt{1-\alpha_{t}}\cdot\mathcal{T}_{t} \right).$ (6) $$

Through experimentation, we observe that background optimization plays a crucial role in fully exploiting the prior within the video diffusion model.
As shown in Fig. 3(c), using a simple white background leads to blurry results.
This may be attributed to a mismatch between the white-background images and the training dataset, which likely contains fewer such examples, affecting the denoising process.
To resolve this issue, we compute the final latents as the combination of the foreground latent $\tilde{\bm{z}}_{t-1}$ projected from the aggregated UV latents and the residual background latent $\bm{z}_{b,t-1}$ denoised by diffusion models.
Specifically, we composite the estimated latents in the $t-1$ step as follows:

$$ $\bm{z}_{t-1}=\tilde{\bm{z}}_{t-1}\odot\bm{\mathcal{M}}_{\text{fg}}+\bm{z}_{b,t -1}\odot\left(1-\bm{\mathcal{M}}_{\text{fg}}\right),\quad\tilde{\bm{z}}_{t-1}, \bm{\mathcal{M}}_{\text{fg}}=\mathcal{R}\left(\mathcal{T}_{t-1};{\bm{c}}_{v} \right),$ (7) $$

where $\bm{\mathcal{M}}_{\text{fg}}$ represents the foreground mask of the mesh.

To summarize, our diffusion process starts with $K\times(V+1)$ randomly initialized noise maps sampled (i.e., $\{\bm{z}_{T}^{1,k},\dots,\bm{z}_{T}^{V,k}\}$, for foreground, $\{\bm{z}_{b}^{1},...,\bm{z}_{b}^{K}\}$ for background) and denoise them into images simultaneously.
At each denoising step $t$ with the key frame $k$, we derive the estimated noises $\{{\bm{\epsilon}}_{t-1}^{1,k},\dots,{\bm{\epsilon}}_{t-1}^{V,k}\}$ using the video diffusion model
and calculate
the estimated original latent $\{\hat{\bm{z}}_{0}^{1,k},\dots,\hat{\bm{z}}_{0}^{V,k}\}$ by Eq. 2.
Then, we use Eq. 4 to aggregate the latents onto UV space.
Next, we utilize Eq. 6 to take the diffusion step in the UV space, and render the synchronized latents $\{\tilde{\bm{z}}_{t-1}^{1,k},\dots,\tilde{\bm{z}}_{t-1}^{V,k}\}$ from latent UVs $\{\mathcal{T}_{t-1}^{1},\dots,\mathcal{T}_{t-1}^{K}\}$ to ensure multi-view consistency.
Finally, we composite the denoised latent with the latents at step $t-1$ according to foreground masks by Eq. 7.

### 4.4 Reference UV Blending

While the video diffusion model ensures temporal consistency for latents from each view, consistency can sometimes diminish after aggregation in the texture domain.
This issue primarily stems from the view-dependent nature of the depth conditions and the limited resolution of latents, which can lead to distortions when features from different camera angles are combined onto the UV texture.
Additionally, self-occlusion during mesh animation often results in a loss of information in invisible regions.

To address these challenges, we introduce a reference UV map to provide additional correlations between latent textures across frames.
Specifically, the reference UV map is constructed by sequentially combining latent textures over time, with each new texture filling only the empty texels of the reference UV map.
Then, each texture is blended using the reference UV $\mathcal{T}_{\mathcal{UV}}$ with a mask $\bm{\mathcal{M}}_{\mathcal{UV}}$ that labels the visible region:

$$ $\mathcal{T}_{t}^{k}=\left(\left(1-\lambda\right)\cdot\mathcal{T}_{t}^{k}+ \lambda\cdot\mathcal{T}_{\mathcal{UV}}\right)\odot\bm{\mathcal{M}}^{k}_{ \mathcal{UV}}+\mathcal{T}_{\mathcal{UV}}\odot\left(1-\bm{\mathcal{M}}^{k}_{ \mathcal{UV}}\right),$ (8) $$

where $\lambda$ is the blending weight for the reference UV in the visible region, while the invisible region is simply replaced with the reference texture. We empirically set the blending weight to $0.2$ during our experiments.

Figure: Figure 4: Qualitative comparisons of multi-view video generation. We compare our method against Text2Video-Zero (Khachatryan et al., ), PnP-diffusion (Tumanyan et al., ), TokenFlow (Geyer et al., ), Gen-1 (Esser et al., ), LatentMan (Eldesokey & Wonka, ), and Generative Rendering (Cai et al., ). We generate videos in the front view and the side view (gray box) on SMPL model data (left column) and Mixamo dataset (right column). Our method manages to generate vivid videos that align with the textual prompts while preserving spatial and temporal consistency.
Refer to caption: x4.png

## 5 Experiments

#### Datasets.

We sourced our datasets from two primary repositories: human motion diffusion outputs and the Mixamo(^1^11https://www.mixamo.com/) and Sketchfab(^2^22https://sketchfab.com/) websites. We employed the text-to-motion diffusion model (HDM) (Tevet et al., ) to compare our approach with LatentMan (Eldesokey & Wonka, ), as LatentMan requires the SMPL model (Loper et al., ) to get corresponding features. For comparison with Generative Rendering (Cai et al., ), we obtained animated characters from the Mixamo platform and rendered them with different motions. Specifically, we first used Blender Community () to extract meshes, joints, skinning weights, and animation data from the FBX files. Then, we applied linear blend skinning to animate the meshes. For meshes without UV maps, we utilized XATLAS to parameterize the mesh and unwrap the UVs.

#### Baselines.

To the best of our knowledge, no existing studies directly address the task of multi-view consistent video generation guided by untextured mesh sequences, as our method does. Consequently, we adapted six recent methods and rendered the input (untextured mesh renders and depth maps) according to their configurations to serve as baselines. These baselines include DDIM inversion-based video stylization methods and video generation methods with different control mechanisms,
including PnP-Diffusion (Tumanyan et al., ), Text2Video-Zero (Khachatryan et al., ), TokenFlow (Geyer et al., ), Generative Rendering (Cai et al., ), LatentMan (Eldesokey & Wonka, ), and Gen-1 (Esser et al., ).
PnP-Diffusion is an image style transfer method that is conditioned on the attention feature of the input image by DDIM inversion.
We extended the method to stylize videos on a frame-by-frame basis for comparison, aligning with previous work (Geyer et al., ).
Built upon cross-frame attention, Text2Video-Zero guides the video by warping latents to implicitly enhance video dynamics, and we utilized their official extension, which supports depth control.
TokenFlow, Generative Rendering, and LatentMan study frame relations in latent space and establish feature correspondences through nearest neighbor matching, UV maps, and DensePose features, respectively.
Gen-1 is a video-to-video model that learns the structure of input videos and transforms the input content (untextured mesh renders) into stylized outputs.
Given the availability of the source code for Generative Rendering, we utilize the experimental results presented in their video demos for qualitative comparison.

#### Evaluation Metric.

Quantitatively assessing multi-view consistency and temporal coherence is still an unresolved problem. We perform a user study to assess overall performance, including the appearance, temporal coherence and spatial consistency, and the fidelity to prompt based on human preference. In addition, we compute the multi-view coherence via Fréchet Video Distance (FVD) (Unterthiner et al., ), a video-level metric that assesses temporal coherence, as utilized in previous approaches (Li et al., ; Xie et al., ).

**Table 1: Quantitative evaluation. We present FVD values and a comparison highlighting the percentage of user preference for our approach against other methods. Our method shows the best spatio-temporal consistency as measured by the FVD metric (Unterthiner et al., ). Users consistently favored Tex4D over all baselines.**
| Method | FVD ($\downarrow$) | Appearance Quality | Spatio-temporal Consistency | Consistency with Prompt |
| --- | --- | --- | --- | --- |
| Text2Video-Zero | 3078.94 | 89.33% | 91.78% | 91.55% |
| PnP-Diffusion | 1390.04 | 86.42% | 87.18% | 89.74% |
| TokenFlow | 1330.43 | 92.31% | 86.84% | 93.42% |
| Gen-1 | 3114.26 | 70.27% | 75.00% | 77.78% |
| LatentMan | 2811.23 | 86.57% | 86.57% | 81.82% |
| Ours | 1303.14 | - | - | - |

### 5.1 Qualitative Results

We present qualitative evaluation in Fig. 4.
LatentMan (Eldesokey & Wonka, ), Generative Rendering (Cai et al., ), TokenFlow (Geyer et al., ), and Text2Video-Zero (Khachatryan et al., ), which are based on T2I diffusion models with cross-frame attention mechanisms, exhibit significant flickering compared to other methods. This issue arises in part from the empirical and implicit correspondence mapping used to encourage the interframe latent consistency, and the correspondences in the latent space may not exactly match the RGB space. In contrast, our approach interpolates the frames between key frame textures in RGB space, effectively eliminating artifacts caused by latent manipulation.
PnP-Diffusion (Tumanyan et al., ), which edits frames independently, generates detailed and sophisticated appearances but suffers from poor spatio-temporal consistency due to the loss of temporal correlations in the latent space.
While Gen-1 (Esser et al., ) (fifth row) produces high-quality videos, it exhibits a jitter effect on the foreground and lacks spatio-temporal consistency.

Furthermore, we present additional multi-view results showcasing a variety of styles and prompts in Fig. 5.
Our denoising algorithm, driven by video diffusion models, effectively captures temporal variations over time. For instance, as highlighted in the green boxes in Fig. 5, our method accurately represents cloth wrinkles (Row 1) and changes in lighting (Rows 2 and 3).

### 5.2 Quatitative Evaluation

To quantitatively assess the effectiveness of our proposed method, we follow prior research (Eldesokey & Wonka, ; Geyer et al., ; Esser et al., ) and conduct a comprehensive A/B user study. Our study involved 67 participants who provided a total of 1104 valid responses based on six different scenes drawn from six previous works, with each scene producing videos from two different views. During each evaluation, participants were presented with rendered meshes and depth conditions viewed from two angles, serving as motion references. They were shown a pair of videos: one generated by our approach and the other from a baseline method. Participants were asked to select the method that exhibited superior performance in three criteria: 1) appearance quality, 2) spatial and temporal consistency, and 3) fidelity to the prompts. Table 1 summarizes the preference percentage of our method over the baseline methods.
Our method significantly surpasses state-of-the-art methods by a large margin.
In addition, our method achieves lower FVD that demonstrates better multi-view consistency in generated video clips.

Figure: Figure 6: Ablation study on the reference UV blending module. Without this module, the generated textures may lose consistency over time as highlighted in the red boxes.
Refer to caption: x6.png

### 5.3 Ablation Study

Ablation for texture aggregation. In Fig. 3 (a) and (b), we present two alternative texture aggregation methods. In Fig. 3 (a), we un-project $\hat{\bm{z}}_{0}(\bm{z}_{t})$ and ${\bm{\epsilon}}_{\theta}(\bm{z}_{t})$ into UV space for aggregation. In Fig. 3 (b), we map $\bm{z}_{t-1}$ to the UV space. Both these two approaches show inferior results compared to our method, which verifies the effectiveness of the proposed texture aggregation algorithm.

Ablation for UV blending module. In Sec. 4.4, we propose a reference UV blending schema to resolve the temporal inconsistency caused by latent aggregation.
To validate the effectiveness of this mechanism (See Sec. 4.4), we conduct an ablation study by disabling the reference UV blending module (setting $\lambda$ to 0).
As shown in Fig. 6, without the UV blending module, our method generates textures with noticeable distortions on the Joker’s face over time.

Ablation for background priors. Sec. 4.3 discusses the importance of including a plausible background and proposes to learn a dynamic background through diffusion.
To verify the effectiveness of this design, we replace the learnable background latents with an all-white background while keeping all other parts unchanged.
Fig. 3 (c) illustrates that this ablation experiment produces significantly blurrier textures compared to our full method, highlighting the importance of background learning.

## 6 Conclusions

In this paper, we present a zero-shot approach that generates multi-view, multi-frame consistent textures for untextured, animated mesh sequences based on a text prompt.
By incorporating texture aggregation in the UV space within the diffusion process of a conditional video diffusion model, we ensure both temporal and spatial coherence in the generated textures.
To address the variance shift introduced by texture aggregation, we propose a simple yet effective modification to the DDIM sampling algorithm.
Additionally, we enhance temporal consistency by introducing a reference UV map and develop a dynamic background learning framework to produce fully textured 4D scenes.
Extensive experiments show that our method can synthesize realistic and consistent 4D textures, demonstrating its superiority against state-of-the-art baselines.

## Appendix A Appendix

### A.1 Implementation Details

We utilize the CTRL-Adapter (Lin et al., ), trained on the video diffusion model I2VGen-XL (Zhang et al., ), as the backbone for generation, with the denoising steps set to $T=50$. Initially, we center the untextured mesh sequence and pre-define six different viewpoints around the Y-axis in the XZ-plane, uniformly sampled in spherical coordinates, along with an additional top view with an elevation angle of zero and an azimuth angle of $30^{\circ}$.
For latent initialization, we first sample Gaussian noise on the latent textures and then render 2D latent samples for each view to improve the coherence of the generated outputs. During denoising, we upscale the latent resolution to $96\times 96$ to reduce aliasing. We empirically set the blending coefficient to $0.2$. It takes approximately 30 minutes to generate a video with 24 key frames taken on a RTX A6000 Ada GPU. We decode the denoised latents in key frames to RGB images, and then un-project and aggregate these images to transform the latent UV maps to RGB textures as previous works (Liu et al., ; Cao et al., ; Huo et al., ). Finally, we interpolate the textures of the key frames at intervals of 3 to synthesize the final video clips.

### A.2 Denoising Algorithm of Our Method

We present the complete workflow of our method in Algorithm 1.
The reference UV map $\mathcal{T}_{\mathcal{UV}}$ is constructed by progressively combining latent textures over time, with each new texture filling only the unoccupied texels in the reference UV map.

Figure: Algorithm 1 Tex4D

### A.3 More Qualitative results

In Fig. 7, we present additional characters generated by Tex4D, showcasing the method’s effectiveness and its ability to generalize across a diverse array of styles and prompts.
We also evaluate Tex4D on non-human character animations in Fig. 8, demonstrating its robust generalization capabilities across various types of mesh sequences.
Additionally, we provide a supplementary video that includes baseline comparisons and multi-view results for all examples presented in the main paper.
For more examples, please kindly refer to our project page.

Figure: Figure 7: More qualitative results.
Refer to caption: x7.png

Figure: Figure 8: More qualitative results on non-human character animations.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Although the experiments provide some evidence of the proposed method’s effectiveness, several concerns remain:

1. Could the authors provide additional qualitative and quantitative comparisons in the ablation study? With only one demonstration, it is difficult to convincingly assess the effectiveness of the proposed method.

2. The authors suggest that video diffusion models struggle with multi-view consistent texturing for 3D mesh sequences due to a lack of 3D geometry awareness. However, the approach already uses a depth-aware video diffusion model, which inherently includes some geometric awareness. Why does this straightforward combination not achieve the desired consistency? Does this imply that depth-aware video diffusion models alone cannot guarantee multi-view consistency even with depth information? If so, could the authors provide performance metrics or visual comparisons showing results when using only the depth-conditioned video diffusion model as a prior? Additionally, for a single viewpoint, does the video diffusion model produce temporally consistent results? If not, visual examples would help clarify.

3. Since a mesh input is available, a straightforward approach could be to texture the mesh on the first frame using methods like Text2Tex or SceneTex, then animate the textured mesh. This method might improve efficiency and naturally maintain multi-view consistency across frames. How does this alternative approach compare in terms of both methodology and performance? An in-depth discussion of these differences would be beneficial.

4. The authors mention that for each predefined viewpoint, a sequence of K rendered meshes is used as input and individually textured by the depth-guided diffusion model. Could the authors clarify the motivation behind this setup? Since the videos are generated separately for each view, multi-view inconsistencies are expected. Why introduce this setup if it inherently leads to consistency issues at the start?

5. While using UV textures for each mesh can enhance multi-view consistency, this approach seems more like an averaging of multiple viewpoints to produce a smoother result. Can the authors elaborate on how this averaging mechanism ensures true multi-view consistency?

6. Given that the current method requires rendering V views for each mesh in the sequence, which may be computationally intensive, could the authors discuss the efficiency of the method? Details on the time required to process a sample would help assess the method's practicality.

7. It would be beneficial to include video visualizations or comparative examples to further illustrate the method's performance and effectiveness.
soundness: 3
strengths: See below.
summary: This paper introduces a novel framework for generating textures for mesh sequences. The authors utilize a depth-conditioned video diffusion model to ensure temporal consistency in videos generated from rendered mesh sequences for each predefined viewpoint. To achieve multi-view consistency, they adopt a UV space texture aggregation strategy. Additionally, they propose a modified sampling approach to address the issue of blurriness in the generated textures.
weaknesses: See below.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the paper, the authors mentioned that the mesh texture could be significantly influenced by the background, providing an example with a white background. I’m curious how the generated texture might look if a non-white background was used, especially one that contrasts strongly with the foreground object. How would such a background affect the consistency and quality of the generated texture?
soundness: 2
strengths: 1. The authors assert that this is the first method developed specifically for 4D scene texturing.
2. The authors introduce a multi-frame consistent texture generation technique, demonstrating improved consistency in results compared to baseline methods.
3. The paper is fluent and well-written, contributing to its readability and overall clarity.
summary: This paper proposed a 4D scene texturing approach by video diffusion models, a zero-shot pipeline for generating temporally and multi-view consistent textures. In order to aggregate multiview latents in UV space, they discovered the issue of "variance shift" caused by their aggregation, and proposed to modify DDIM sampling process to address the issue. By UV blending during denoising steps, the issue of self-occlusion is addressed and synchronized in invisible regions.
weaknesses: 1. The generated textures do not blend seamlessly with the background, creating a disjointed appearance that resembles separate foreground and background elements stitched together.
2. Despite claims of multi-view consistency, flickering effects are observed across different views, indicating instability in rendering.
3. Some of the compared methods, such as TokenFlow and Text2Video-Zero, do not utilize mesh or depth inputs, making direct comparisons less equitable.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Overall, the experimental results are quite satisfactory; however, there is a lack of explanation regarding the advantages of the pipeline compared to other pipelines using 2D poses and textured meshes.

Animation can drive the mesh. Are the position and rotation of the mesh manually controlled, such as in the second example on the first page?

How is the animated mesh obtained? We observe some temporal changes in Figure 5. Is this one of the contributions of your paper? How do you distinguish between temporal changes and temporal inconsistencies, as there are some temporal inconsistencies in your results?

Would using textured meshes yield better outcomes? What are the advantages of generating videos with untextured meshes compared to textured meshes?

What is the difference between using a 2D pose and an animated mesh?
soundness: 2
strengths: The paper is the first work to perform video generation based on animated mesh sequences, while its UV mapping strategy ensures multi-view consistency. The experimental results show significant advantages compared to some existing works.
summary: This paper introduces 4D scene texturing to generate textures that are consistent both temporally and across multiple views for animated mesh sequences. Tex4D, uses 3D geometry to synchronize diffusion processes, incorporates video generation model insights for temporal consistency, and modifies the DDIM sampling process to improve texture clarity.
weaknesses: Under the current pipeline, this work has yielded highly effective results. However, the importance of this pipeline should be further clarified, such as by comparing it with pipelines based on 2D poses or textured meshes. The paper should include more comprehensive comparisons to highlight the contribution of the pipeline. For example, is it a reasonable pipeline to first generate textured meshes and then use animated meshes for video generation?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Q: "While these methods produce multi-view consistent textures for static 3D objects, they do not address the challenge of generating temporally consistent textures for mesh sequences." I would appreciate further clarification on the motivation behind "generating temporally consistent textures" for mesh sequences. Could you provide examples where dynamic 4D texturing is essential and cannot be achieved through traditional methods?

Q: How does the model ensure robustness when dealing with varying UV mapping results?
- How sensitive is the method to different UV unwrapping techniques?
- Did the authors experiment with different UV mapping strategies, and if so, what were the results?
- Are there any limitations or best practices for UV mapping when using this method?
soundness: 3
strengths: # Strengths

- The paper is well-written, making it easy to understand and follow.
- The related works are sufficiently covered.
- Several experiments are conducted for demonstrating its effectiveness.
summary: # Summary

This paper focuses on creating temporally consistent and realistic textures for mesh sequences. The input is an untextured mesh sequence and a text prompt. To achieve this, a method named Tex4D is proposed, which is a zero-shot approach that integrates geometry information from mesh sequences with video diffusion models, specifically the depth-conditioned CTRL-Adapter (Lin et al., 2024), to produce multi-view and temporally consistent 4D textures. The model synchronizes the diffusion process across different views through latent aggregation in the UV space. Additionally, a reference latent texture is introduced to strengthen the correlation between frames during the denoising process.
weaknesses: Regarding the contribution and motivation, I still wonder about the specific meaning of "4D texturing" for the object and how this differs from a 3D model that is first textured and then animated using skinning techniques. Even for dynamic textures, one could also generate the dynamic texture for a 3D model and then animate the character through skinning. This approach seems useful **if** the mesh is also significantly dynamic, such as with topology changes. Further, when it comes to the video background, I noticed in the supplementary video that there are some dynamic effects from the proposed method, but they are not that significant. Could one first perform 3D texturing and then render it with a generative background video?

I think the following suggestions could be helpful in justifying the significance of the setting in this paper.
- Provide specific examples or use cases where 4D texturing offers advantages over traditional 3D texturing and animation.
- Clarify how the proposed method handles dynamic meshes with topology changes, if applicable.
- Compare the method with a pipeline of 3D texturing followed by rendering with a generative background video, highlighting any benefits of the presented "integrated" approach.

I also have concerns about the novelty of the paper. The entire pipeline can be seen as a depth-conditioned CTRL-Adapter for mesh sequence texturing with UV space aggregation, which feels like a straightforward composition of existing models. I would prefer to see a simple yet effective method of tackling a critical problem. However, as I am still uncertain about the meaning/significance of "4D texture," this makes me somewhat skeptical about the proposed pipeline.

I think the authors could provide some arguments for the novelty of the proposed method:
- Highlight the key technical innovations in the pipeline beyond the composition of existing models.
- Explain how the introduced method addresses specific challenges in 4D texturing that are not easily solved by existing methods.
- Provide a clearer definition and motivation for "4D texturing" to help readers understand its significance in the context of their work (similar to the previous questions).

I understand the difficulty in evaluating the results, but it would be helpful and necessary to conduct an evaluation of Appearance Quality, Spatio-temporal Consistency, and Consistency with Prompt via a user study - The quantitative evaluation is insufficient.

The following action can be taken:
- Conduct a user study evaluating the specific aspects, e.g., Appearance Quality, Spatio-temporal Consistency, and Consistency with Prompt, and compare the proposed method with previous models.

Limitations and Future Works should be included. 
- For instance, the authors may discuss the current limitations of their approach, such as some failure cases or more specifically, the types of meshes or textures it struggles with, etc.
- Potential future improvements or extensions to the method.
- Broader implications or applications of this work in related fields.


# Minor Comments

- "To resolve this issue, we analyze the underlying causes and propose a simple yet effective modification to the DDIM (Song et al., 2020) sampling process." In the introduction section, it would be beneficial to briefly explain how you achieved this.

Meta-review: This paper introduces a method for generating temporal and multi-view consistent textures for a mesh sequence in a training-free manner using a pretrained depth-conditioned video diffusion model. The proposed method builds upon previous 3D texture generation methods such as SyncMVD and TexGen, but adapts these ideas for 4D texture generation. The authors propose generating the foreground and background separately and then aggregating them. They also suggest a method to improve the denoising process and reduce blurry results. The experimental results include comparisons with previous methods, demonstrating a lower FVD score of the proposed method.

All reviewers gave negative feedback and were not convinced about the need for a 4D generation method, as it is also possible to generate 3D textures and animate the object. While the authors emphasized in the rebuttal that 4D generation can incorporate lighting changes, wrinkles, and dynamic effects, the additional results were not convincing to all the reviewers and the AC. Furthermore, although the authors claim that applying these changes in the graphics pipeline requires significant labor, if users can animate a mesh, changing lighting and adding dynamic effects would not be much more burdensome.

Regarding the technical contributions, from the AC's perspective, the proposed method appears to be a straightforward extension of the 3D texture generation approach to 4D using a video generative model, with some engineering techniques such as foreground/background separation. However, it does not seem to provide a sufficient technical contribution to meet the ICLR standards.

Decision: Reject