Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping
Authors: Zijie Pan
Jiachen Lu
Xiatian Zhu
Li Zhang
Venue: ICLR 2024 poster (2024)
Keywords: 3D generation
diffusion model
Primary area: generative models

Abstract: High-resolution 3D object generation remains a challenging task primarily due to the limited availability of comprehensive annotated training data. Recent advancements have aimed to overcome this constraint by harnessing image generative models, pretrained on extensive curated web datasets, using knowledge transfer techniques like Score Distillation Sampling (SDS).
Efficiently addressing the requirements of high-resolution rendering often necessitates the adoption of latent representation-based models, such as the Latent Diffusion Model (LDM). In this framework, a significant challenge arises: 
To compute gradients for individual image pixels, it is necessary to backpropagate gradients from the designated latent space through the frozen components of the image model, such as the VAE encoder used within LDM. However, this gradient propagation pathway has never been optimized, remaining uncontrolled during training.
We find that the unregulated gradients adversely affect the 3D model's capacity in acquiring texture-related information from the image generative model, 
leading to poor quality appearance synthesis.
To address this overarching challenge, we propose an innovative operation termed Pixel-wise Gradient Clipping (PGC) designed for seamless integration into existing 3D generative models, thereby enhancing their synthesis quality. Specifically, 
we control the magnitude of stochastic gradients by clipping the pixel-wise gradients efficiently,
while preserving crucial texture-related gradient directions.
Despite this simplicity and minimal extra cost, extensive experiments demonstrate the efficacy of our PGC
in enhancing the performance of existing 3D generative models
for high-resolution object rendering.

Published as a conference paper at ICLR 2024
ENHANCING HIGH-RESOLUTION 3D GENERATION THROUGH PIXEL-WISE GRADIENT CLIPPING
Zijie Pan', Jiachen Lu', Xiatian Zhu?, Li Zhang
School of Data Science. Fudan University University of Surrey
https: Fudan- zvg. g thub. 1o//o//////// PGPGC-33/
Fantasia3D (Chen ct.l. 2023b)
Ours
Figure 1: Blender rendering for textured meshes. Top: Fantasia3D (Chen et al., 2023b). Bottom: Ours. For each mesh in the top, we can find corresponding one in the bottom whose texture is generated conditioned on the same prompt. Our method generates more detailed and realistic texture and exhibits better consistency with input prompts.
ABSTRACT
High- High-resolution 3D object generation remains challenging task primarily due the limited availability of comprehensive annotated training data. Recent advance- ments have aimed overcome this constraint by harnessing image generative models, pretrained on extensive curated web datasets, using knowledge transfer techniques like Score Distillation Sampling (SDS). Efficiently addressing the re- quirements of high-resolution rendering often necessitates the adoption of latent representation-baat models, such the Latent Diffusion Model (LDM). In this framework, significant challenge arises: To compute gradients for individual im- age pixels, it is necessary to backpropagate gradients from the designated latent space through the frozen components of the image model, such as the VAE en- coder used within LDM. However, this gradient propagation pathway has never been optimized, remaining uncontrolled during training. We find that the unreg- ulated gradients adversely affect the 3D model's capacity in acquiring texture- related information from the image generative model, leading to poor quality ap- pearance synthesis. To address this overarching challenge, we propose an innova-
"Lii Zhang (lizhangfd@ fudan. edidan.edu.en) is the corresponding author.

---

Published as a conference paper at ICLR 2024
tive operation termed Pixel-wise Gradient Clipping (PGC) designed for seamless integration into existing 3D generative models, thereby enhancing their synthesis quality. Specifically, control the magnitude of stochastic gradients by clip- ping the pixel-wise gradients efficiently, while preserving crucial texture-related gradient directions. Despite this simplicity and minimal extra cost, extensive ex- periments demonstrate the efficacy of our PGC in enhancing the performance of existing 3D generative models for high-resolution object rendering.
INTRODUCTION
Motivated by the success of 2D image generation (Ho et al., 2020; Rombach et al., 2022), sub- stantial advancements have occurred in conditioned 3D) generation. One notable example involves using a pre-trained text-conditioned diffusion model (Saharia al., 2022), employing a knowledge distillation method named "score distillation sampling" (SDS), to train 3D models. The goal is to align the sampling procedure used for generating rendered images from the Neural Radiance Field (NeRF) (Mildenhall et al., 2021) with the denoising process applied to 2D image generation from textual prompts.
However, surpassing the generation of low-resolution images (e.g., 64x64 pixels) presents greater challenges, demanding more computational resources and attention to fine-grained details. To ad- dress these challenges, the utilization of latent generative models, such as the Latent Diffusion Model (LDM) (Rombach et al., 2022), becomes necessary as exemplified in (Lin et al., 2022; Chen et al., 2023b; Tsalicoglou et al., 2023; Wang et al., 2023; Zhu & Zhuang, 2023; Hertz et al., 2023).
Gradient propagation in these methods comprises two phases. In the initial phase, gradients prop- agate from the latent variable to the rendered image through a pre-trained and frozen model (e.g., Variational Autoencoder (VAE) or LDM). In the subsequent phase, gradients flow from the image to the parameters of the 3D model, where gradient regulation techniques, such activation func- tions and 1.2 normalization, are applied to ensure smoother gradient descent. Notably, prior research has overlooked the importance of gradient manipulation in the first phase, which is fundamentally pivotal in preserving texture-rich information in 3D generation.
We contend that neglecting pixel-wise gradient regulation in the first phase can pose issues for 3D model training and ultimate performance since pixel-wise gradients convey crucial information about texture, particularly for the inherently unstable VAE with the latest SDXL (Podell et al., 2023), used for image generation at the resolution of 1024 x 1024 pixels, as illustrated in the second column of Figure 2. The pronounced presence of unexpected noise pixel-wise gradients obscures the regular pixel-wise gradient, leading to blurred regular gradient. Consequently, this blurring effect causes the generated 3D model to lose intricate texture details or, in severe cases of SDXL (Podell et al., 2023), the entire texture altogether.
Motivated by these observations, in this study, we introduce a straightforward yet effective variant of gradient clipping, referred to as Pixel-wise Gradient Clipping (PGC). This technique is specifi- cally tailored for existing 3D generative models. Concretely, PGC truncates unexpected pixel-wise gradients against predefined thresholds along the pixel vector's direction for each individual pixel. Theoretical analysis demonstrates that when the clipping threshold is set around the bounded vari- ance of the pixel-wise gradient, the norm of the truncated gradient is bounded by the expectation of the 2D pixel residual. This preservation of the norm helps maintain the hue of the 2D image texture and enhances the overall fidelity of the texture. Importantly, PGC seamlessly integrates with existing SDS loss functions and LDM- based 3D generation frameworks. This integration results in significant enhancement in texture quality, especially when leveraging advanced image generative models like SDXL. (Podell et al., 2023).
(i) We identify a critical and generic issue in optimizing high- resolution 3D models, namely, the unregulated pixel-wise gradients of the latent variable against the rendered image. (ii) To address this issue, we introduce an efficient and effective approach called Pixel-wise Gradient Clipping (PGC). This technique adapts traditional gradient clipping to regu- late pixel-wise gradient magnitudes while preserving essential texture information. (iii) Extensive experiments demonstrate that PGC can serve generic integrative plug-in, consistently benefit- ing existing SDS and LDM-based 3D generative models, leading to significant improvements in high-resolution 3D texture synthesis.
Our contributions are as follows:
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
2D diffusion models Image diffusion models have made sigmificant advancements (Ho et al,, 2020; Balaji et al., 2022; Saharia et al., 2022; Rombach et al., 2022: Podell et al., 2023). Rom- bach et al. (2022) introduced Latent Diffusion Models (LDMs) within Stable Diffusion, using latent space for high-resolution image generation. Podell al. (2023) extended this concept in Stable Diffusion XL (SDXL) to even higher resolutions (1024 x 1024) with larger latent spaces, VAE, and U-net. Zhang & Agrawala (2023) enhances these models' capabilities by enabling the generation of controllable images conditioned on various input types. Notably, recent developments in 3D-awar 2D diffusion models, including 1e (Liu et al., 2023a). MVDream (Shi et al., 2023) and Sync- Dreamer (Liu et al., 2023b), have emerged. These models, also falling under the category of LDMs, can be employed to generate 3D shapes and textures by leveraging the SDS loss (Poole et al., 2023) or even reconstruction techniques (Mildenhall et al., 2021; Wang et al., 2021).
3D shape and texture generation using 2D diffusion The recent method TEXTure (Yu et al,, 2023) and Text2Tex (Chen et al., 2023a) can apply textures to 3D meshes using pre-trained text- to-image diffusion models, but they do not improve the mesh's shape. For the text-to-3D task, DreamFusion (Poole et al., 2023) introduced the SDS loss for generating 3D objects with 2D dif- fusion models. Magic3D (Lin et al., 2022) extended this approach by adding a mesh optimization stage based on Stable Diffusion within the SDS loss. Subsequent works have focused on aspects like speed (Metzer et al., 2022), 3D consistency (Seo et al., 2023; Shi et al., 2023), material proper- ties (Chen al., 2023b), editing capabilities (Li et al., 2023), generation quality (Tsalicoglou al., 2023; Huang et al., 2023b; Wu et al., 2023), SDS modifications (Wang et al., 2023; Zhu & Zhuang, 2023), and avatar generation (Cao et al., 2023; Huang et al., 2023a; Liao et al., 2023; Kolotouros al, 2023). All of these works employ an SDS-like loss with Stable Diffusion. In the image-to-3D context, various approaches have been explored, including those using Stable Diffusion (Melas- Kyriazi et al., 2023; Tang et al., 2023), entirely new model training (Liu et al., 2023a), and combi- nations of these techniques (Qian et al., 2023). Regardless of the specific approach chosen, they all rely on LDM-based SDS loss to generate 3D representations.
Gradient clippigg/orrmaliinng techniques Gradient eirpiing and normalization techniques have proven valuable in the training of neural networks (Mikolov, 2012; Brock al., 2021). Theoretical studies (Zhang et al., 2019; 2020; Koloskova et al., 2023) have extensively analyzed these methods. In contrast to previous parameter-wiss strategies, our focus lies on the gradients of model-rendered image. Furthermore, we introduce specifically crafted pixel-wise operations within the framework of SDS-based 3D generation. While a recent investigation by Hong et al. (2023) delves into gradient issues in 3D generation, it overlooks the impact of VAE in LDMs. In summary, we address gradient- related challenges contemporary LDMs and crucially propose a pipeline-agnosti method for enhancing 3D generation.
3 BACKGROUND
3.1 SCORE DISTILLATION SAMPLING (SDS)
The concept of SDS, first introduced by DreamFusion (Poole al., 2023), has transformed text-to- 3D generation by obviating the requirement for text-3D pairs. SDS comprises two core elements: a 3D model and pre-trained 2D text-to-image diffusion model. The 3D model leverages a differen- tiable function
g(0) to render images, with 0 representing the 3D volume.
DreamFusion leverages SDS to synchronize 3D rendering with 2D conditioned generation, as man- ifested in the gradient calculation:
VoLsDs(o.g(0) Er,c (((() (co (c 8) -)) Ox
00
In DreamFusion, the 2D diffusion model operates at a resolution of 64 x 64 pixels. To enhance quality, Magic3D (Lin et al., 2022) incorporates a 2D Latent Diffusion Models (LDM) (Rombach et al., 2022). This integration effectively boosts the resolution to 512 x 512 pixels, leading to an improved level of detail in the generated content.
3

---

Published as a conference paper at ICLR 2024
It's important to highlight that the introduction of LDM has subtle impact the SDS gradient.
This adjustment entails the incorporation of the gradient from the newly introduced VAE encoder, thereby contributing to overall improvement in texture quality:
VoLiDM-SDS(O.9(0) Re, wm(() (ea(ZE:))))) -c) O= dr
Or 00
As illustrated in the leftmost columns of Figure 2, results achieved in the latent space exhibit superior quality, highlighting potential issue with 02/Or that may impede optimization. However, due to the limited resolution the latent space, 3D results remain unsatisfactory. Consequently, it is imperative explore solutions address this challenge.
3.2 PARAMETER-WE WISE NORMALIZED GRADIENT DESCENT AND GRADIENT CLIPPING
To prevent gradient explosion during training, two common strategies are typically used: Normal- ized Gradient Descent (NGD) and Gradient Clipping (GC) (Mikolov, 2012). These approaches both employ a threshold value denoted as 0 to a stochastic gradient, but they vary their imple- mentation. For a stochastic gradient g: 0. /0,, where 0, represents parameter, parameter-wiss NGD can be expressed as
cg
0.+1 0, - normalizzo))
where normalize(g) == | +
(3)
where Tm 0 denotes the learning rate. In summary, when the gradient norm l|9: exceeds the threshold c, NGD constrains it to around c. However, when ||g:| | is below c, NGD retains a fraction of it. A limitation of NGD becomes evident when the gradient approaches the threshold c.
Gradient clipping comes in two primary variants: clipping-by-van and clipping-by-norm.
Gradient clipping-by-value involves truncating the components of the gradient vector g: if they surpass a predefined threshold. However, this method has a drawback, as it modifies the vector gradient's direction. This alteration in direction can influence the convergence behavior the opti- mization algorithm, potentially resulting in slower or less stable training.
Gradient clipping-by-nowe is performed in the following stochastic gradient descent iteration:
0.11 = 0: - clllppggg
where clip(g) == min (Ilgl), c)
min (Ilgl), e) u,
[gl|
where > 0 denotes the learning rate, ||و|| represents the norm of the gradient vector and stands for the unit vector. By applying this operation, we ensure that the magnitude of the gradient remains below the defined threshold c. Notably, it also preserves the gradient's s direction, offering solution the issues associated with the alternative method of clipping-by- byypine
4 METHOD
To mitigate the uncontrolled term 0z/Or, there two viable approaches. First, during the opti- mization of the Variational Autoencoder (VAE), the term 0z/Or can be regulated. Alternatively, control over the term 0z/0z can be exercised during the Score Distillation Sampling (SDS) proce- dure. These strategies provide practical solutions to tame the erratic gradient, enhancing the stability and controllability of model training.
4.1 VAE OPTIMIZATION REGULATION
Managing gradient control in VAE optimization can be difficult, particularly when it's impractical to retrain both the VAE and its linked 2D diffusion model. In such cases, an alternative approach, inspired by Latent-NeRF (Metzer et al., 2022), to train a linear layer that maps RGB pixels to latent variables. We assume a linear relationship between RGB pixels and latent variables, which allows for explicit gradient control. This control is achieved by applying L2-norm constraints to the projection matrix's norm during the training process.
4

---

Published as a conference paper at ICLR 2024
(A) Stable Diffusion (Rombach al., 2022) guidance
(8)
2D
Gradient
(b)
(c)
3D
(B) SIXL (Podell et al., 2023) as guidance
(A)
2D
Gradient
(b)
(0) 3D
(iii) Linear
(i) Latent
(ii) VAE
(iv) PNGD (v) PGC-V (vi) PGC-N
Approx
Figure 2: Visualization of 2D/3D results and typical gradients guided by different LDMs. (A) Stable Diffusion 2. 2-ase (Rombach et al., 2022) as guidance.
(B) SDXL. (Podell et al., 2023) as guidance. The text prompt is wooden car. For each case, we visualize (a) directly optimizing 2D image using SDS loss, alongside (b) the corresponding gradients; (c) optimizing texture field (Chen et al., 2023b) based on fixed mesh of car. We compare six gradient propagation methods: (i) Backpropagation of latent gradients, (ii) VAE gradients, (iii) linear approximated VAE
gradients, (iv) normalized VAE gradients, (v) our proposed PGC VAE gradients by value and (vi) by norm. O highlights gradient noise.
To elaborate, when dealing with RGB pixel vector e R and latent variable vector y RR we establish the relationship as follows:
Ax + b.
(5)
where A E R4x3 and b E IR serve as analogs to the VAE parameters.
For evaluation, we use ridge regression methods with the COCO dataset (Lin et al., 2014) to de- termine the optimal configuration. For optimizing SDS, we approximate the term 0z/Oz using the transposed línear matrix A This matrix is regulated through ridge regression, enabling controlled gradient behavior. Nevertheless, as illustrated in the appendix, this attempt approximate the VAE with linear projection falls short. This linear approximation cannot adequately capture fine texture details, thus compromising the preservation of crucial texture-related gradients.
4.2 SCORE DISTILLATION SAMPLING PROCESS REGULATION
As previously discussed in Section 3.2, parameter-wise gradient regularization techniques are com- monly employed in neural network training. Additionally, observe that regulating gradients the pixel level plays a crucial role in managing the overall gradient during the SDS process.
Traditionally, the training objective for 3D models is defined by 2D pixel residual, given by:
(an(0,x)
E[[l= =12)
5

---

Published as a conference paper at ICLR 2024
where z denotes the rendered image, while ž corresponds to the ground truth image. The objective to minimize the disparity between the rendered image and the ground truth image. Consequently, the update rule for the 3D model can be expressed as follows:
@LaD 0, @CaD dr
dx Of - 22/BB (xr 2)
O++1 0, a0,
Or 00,
a0,
For SDS on the latent variable, the gradient update for the 3D model can be simplified as:
Oe dæ
0c+1 = = - wott (Eo(Z): -
Ox de
Ox 00
= 0 E/e,
E, [ara - x1-11
(8)
In this context, we employ the expectation of pixel residuals, denoted as E, [xy 11. as a substi- tute for wtt (eo(zri y, e) az
When we compare the equation and equation 8, we observe that the difference between and & remains strictly constrained within the interval of [-1, 1] due to RGB restrictions. This constraint plays crucial role in stabilizing the training process for the 3D model. However, in the case of SDS with stochastic elements, the expectation of the 2D pixel residual, denoted as E, [xa - xT-1), is im- plicitly represented through the stochastic gradient (t) (col (24 y. t) - c) äz without such inherent regulation. Therefore, introduce two novel techniques: Pixel-wise Normalized Gradient Descent (PNGD) and Gradient Clipping (PGC).
4.3 PIXEL-WISE NORMALIZED GRADIENT DESCENT (PNGD)
PNGD incorporates normalized gradient to regulate the change in variable 2; It-1, as defined:
E, [x+ 24- E, [x, xt-il
PNGD effectively mitigates this issue by scaling down E, [x, xz-1] when the gradient is excep- tionally large, while preserving when it's sufficiently small.
However, PNGD's primary limitation becomes most evident when the gradient closely approaches the threshold c, especially in scenarios where texture-related information is concentrated near this threshold. In such cases, the gradient norm is significantly suppressed, approaching the threshold c, potentially resulting in the loss of crucial texture-related details.
4.4 PIXEL-WISE GRADIENT CLIPPING (PGC)
To overcome PNGD's limitation, we introduce clipped pixel-wise gradients to restrict the diver- gence between and It-1. According to Section 3.2, the Pixel-wise Gradient Clipping (PGC) method offers two variants: Pixel-wise Gradient hiiping--b--allle (PGC-V) and Pixel-wise Gradi- ent Elipping-by-not (PGC-N).
PGC-V involves directly capping the value E[[; - xt-i] when exceeds the threshold c. How- ever, this adjustment affects the direction of the pixel-wise gradient, leading to change in the correct 2D pixel residual direction. Consequently, this alteration can have a detrimental impact on the learning of real-world textures, illustrated in the fifth column of Figure 2.
PGC-N can be derived from equation and is expressed as follows:
Tr 21 1]
min [ [xr 21-1l c)
min ( [z+ I x1-1) (l.))l,c)
(10)
E [1 xt-1l
where u4 stands for the unit vector.
PGC offers an advantage in managing the "zero measure" of the set created by noisy pixel-wise gradients. It achieves this by filtering noisy gradients with negligible information while retaining those containing valuable texture information. illustrate the effectiveness of PGC, we establish the following assumption.

---

Published as a conference paper at ICLR 2024
Imput
Fantasia3D
+PGC
+SDXL
Ours
"a dragon holding a sword"
"a werewolf archer"
"a wooden car"
Figure 3: Comparison with baselines. With the meshes fixed, we compare methods: Fanta- sia3D (Chen et al., 2023b), Fantasia3D+SDDL (Podell et al., 2023), Fantasia3D+PGG and Fanta- sia3D+SDXL (Ours).
4.4.1 NOISE ASSUMPTION
There is a concern about whether gradient clipping could lead to excessive texture detail loss, as observed cases like linear approximation and PNGD, As shown in Figure 2's second column, have noticed that noisy out-of-boundary pixel-wise gradients points. In mathematical terms, we can assume that the gradient within the boundary almost everywhere, with the region of being out-of-houndary having zero measure. This corresponds to the uniform boundness assumption discussed in Kim et al. (2022), which asserts that the stochastic noise in the norm of the 2D pixel residual 2; _T-1 is uniformly bounded by for all time steps t: Pr |||,, x I aj 1. Furthermore, the bounded variance can be derived as E [[|x, xx-1ll] <
mainly limited to isolated
Now, applying Jensen Inequality to equation 10 by the convexity of L2 norm, have:
min [] (za - 1 |l, <) < min (E [[|a - xxl c) S min (a,c)
(11)
constrain the clipped gradient norm to remain roughly within the range of the 2D pixel residual norm, represented by a. This approach essential as it enables the preservation of pixel-wise gradient information without ex- cessive truncation. This preservation effectively retains texture detail while ensuring noise remains within acceptable limits.
Hence, by choosing a suitable threshold, denoted as σ,
4.5 CONTROLLABLE LATENT GRADIENTS
Improper gradients in the latent space result in failure scenarios. This can manifest as notice- able misalignment between the visualized gradients and object outlines the rendered images, causing texture mismatch with the mesh. To mitigate this issue, propose incorporating shape information into U-nets. Leveraging the provided mesh we apply a depth and/or normal control- net (Zhang& & Agrawala, 2023), substantially enhancing the overall success rate.
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
For all the experiments, we adopt the uniform setting without any hyperparameter tuning. Specif- ically, optimize the texture and/or signed distance function (SDF) fields Chen et al.

---

Published as a conference paper at ICLR 2024
Fantasia3D
+PGC
+SDXL
Ours w/o nmm
Ours
"A panda is dressed in armor holding a spear one hand and shield in the other."
"a castle on a car"
"an angry cat"
Figure 4: Comparison of using normal-SDS jointly with RGB-SDS. We compare 5 methods: Fantasia3D (Chen et al., 2023b), Fantasia&D+SDXL (Podell et al., 2023), Fantasia3++PGC, Fanta- sia3D+SDXL+PGC (Ours) and Fantasia3D=SDXL=PGGG w/o normal-SDS (Ours w/o nrm).
(2023b) for 1200 iterations two A6000 GPUs with batch size 4 by using Adam optimizer without weight decay. The learning rates are set to constant x 10-3 for texture field and x 10-5 for SDF field. For the sampling.
set the initial mesh normalized in [-0.8, 0.8)s. focal range [0.7, 1.35). radius range [2.0, 2.5), elevation range [-100, 45%] and azimuth angle range 10 360% In SDS, we CFG 100, t U (0.02, 0.5) and w(t) = of In PGC, we use PGC-N for PGC default and set the threshold 0.1. The clipping threshold is studied in Section A.3.
5.2 PGC ON MESH OPTIMIZATION
As Stable Diffusion (Rombach et al., 2022) and Stable Diffusion XL (SDXL) (Podell et al., 2023) demonstrate notable capabilities in handling high-resolution images, primary focus lies in eval- uating PGC's performance within the context of mesh optimization, with a specific emphasis on texture details. To conduct comprehensive comparisons, employ numerous mesh-prompt pairs to optimize both texture fields and/or SDF fields. Our experimental framework establishes the Fan- tasia3D's appearance stage, utilizing Stable Diffusion-L15 with depth-controlnet for albedo render- ing, as our bascline reference. Subsequently, we conduct series of methods, including Fanta- sia3D+PGC, Fantasia3D+SDXX (replace Stable Diffusion), and FantasiaDD-SDXL-PPC..
In the first setting where the meshes remain unchanged, the outcomes of these comparisons are pre- sented in Figure 3. noteworthy that PGC consistently enhances texture details when contrasted with the baseline. Notably, the direct replacement of Stable Diffusion with SDXL results in a con- sistent failure; however, the integration of PGC effectively activates SDXL's capabilities, yielding textured mesh of exceptional quality.
In the second setting, we allow for alterations in mesh shape through the incorporation of normal- SDS loss which replaces RGB image with normal image as the input of diffusion model, albeit at the expense of doubling the computation time. The results of these experiments are presented in Figure 4. Similar to the first setting, observe consistent enhancement in texture quality by using PGC. Furthermore, in terms of shape details, the utilization of normal-SD loss yields significantly more intricate facial features in animals. Interestingly, find that even if the change of shape is
8

---

Published as a conference paper at ICLR 2024
Baseline
+PGC
Stable-DreamHine (Tang, 2023)
PantasiaD--Geomee (Chen et al., 2023b)
Reference
Baseline
+PGC
Reference
Baseline
+PGC
Zero123 (Liu et al., 2023a)
Figure 5: PGC benefit various pipelines, including
(Tang, 2023). Fanta-
sia3D (Chen et al., 2023b) geometry stage and Zero123 (Liu et al., 2023a).
not particularly significant, minor perturbation of the input points coordinates of texture fields can enhance the robustness of optimization, resulting more realistic texture.
5.3 PGC BENEFITS VARIOUS PIPELINES
(Tang, 2023) with Stable Diffusion 2. base, Fantasia3D (Chen et al., 2023b) geometry stage with Stable Diffusion 2. base and Zero123-SDS (Liu et al., 2023a). These three pipelines cover a wide range of SDS applica- tions including both text-to-3d and image-to-3d tasks. As depicted in Figure 5, within the Stable- DreamFusion pipeline, PGC demonstrates notable improvements in generation details and success rates. In the case of Fantasia3D, PGC serves to stabilize the optimization process and mitigate the occurrence of small mesh fragments. Conversely, in the Zerol23 pipeline, the impact PGC on texture enhancement remains modest, primarily due the lower resolution constraint 256. How- ever, it is reasonable to anticipate that PGC may exhibit more pronounced effectiveness scenarios involving larger multi-view diffusion models, should such models become available in the future.
We also test PGC in various pipelines using LDM:
5.4
USER STUDY
We also conducted user study to evaluate our methods quantitatively. We put 12 textured meshes generated by 4 methods described in Section 5.2 on website so that users are able to conveniently rotate and scale 3D models for observation online and finally pick the preferred one. Among 15 feedback with 180 picks, ours received 84.44% preference while Fantasia3D w/ and w/o PGC only received 10.56% and 5% preference, respectively. Since Fantasia3D w/o PGC does generate any meaningful texture, no one picks this method. The results show that our proposed PGC greatly improves generation quality. More results can be found in supplementary materials.
CONCLUSION
In our research, we have identified critical and widespread problem when optimizing high- resolution 3D models: the uncontrolled behavior of pixel-wise gradients during the backpropagation of the VAE encoder's gradient. To tackle this issue, we propose an efficient and effective solution called Pixel-wise Gradient Clipping (PGC). This technique builds upon traditional gradient clipping but tailors it to regulate the magnitudes of pixel-wise gradients while preserving crucial texture in- formation. Theoretical analysis confirms that the implementation of PGC effectively bounds the norm pixel-wise gradients to the expectation of the 2D pixel residual. Our extensive experiments further validate the versatility of PGC general plug-in, consistently delivering benefits to isting SDS and LDM- 3D generative models. These improvements translate into significant enhancements in the realm of high-resolution 3D texture synthesis.

---

Published as a conference paper at ICLR 2024
Acknowledgments This work supported in part by STI2030-Major Projects (Grant No. 202 2021ZD0200244) National Natural Science Foundation of China (Grant No. 62106050 and 62376060), Natural Science Foundation of Shanghai (Grant No. 22ZR USyd-Fudan BISA Flagship Research Program and and Lingang Laboratory (Grant No. LG--QS-20220--07)
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint, 2022.
Andy Brock, Soham De, Samuel Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In ICML, 2021. 3
Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- K Wong. Dreamavatar: Text-and- shape guided 3d human avatar generation via diffusion models. arXiv preprint, 2023.
Hsin-Ying Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint, 2023a. 3
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. ICCV, 2023b. 1, 3, 8,
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint, 2023. 2
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 2, 3
Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of diffusion for robust text-to-3d generation. arXiv preprint, 2023. 3
Shuo Huang. Zongxin Yang, Liangting Li, Yi Yang, and Jia Jia. Avatarfusion: Zero-shot generation of clothing-decouplee 3d avatars using 2d diffusion. arXiv preprint, 2023a.
Yukum Huang, Jianan Wang, Yukai Shi, Xianbiao Qi, Zheng-Jun Zha, and Lei Zhang. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint, 2023b. 3
Taehun Kim, Kunhee Kim, Joonyeong Lee, Dongmin Cha, Jiho Lee, and Daijin Kim. Revisiting image pyramid structure for high resolution salient object detection. In ACCV, 2022.
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In ICMI. 2023. 3
Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Fieraru, and Cristian Sminchisescu. Dreamhuman: Animatable 3d avatars from text. arXiv preprint, 2023.
Yuhan Li, Yishun Dou, Yue Shi, Yu Lei, Xuanhong Chen, Yi Zhang, Peng Zhou, and Bingbing Ni.
Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint, 2023.
Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, and Michael J Black. Tada! text to animatable digital avatars. arXiv preprint, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint, 2022. 2, 3
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitiick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ruoshi I. Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmmkov,, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image 3d object. In ICCV, 2023a. 3,9
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Learning to generate multiview-cconiistern images from a single-view image. arXiv preprint, 2023b.
10

---

Published as a conference paper at ICLR 2024
Luke Melas-Kyy Kyriazi,,, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 { {'dee] reconstruction of any object from a single image. arXiv preprint, 2023. 3
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint, 2022. 3,4
Tomáš Mikolov. Statistical language models based on neural networks. PhD thesis, Bro University of Technology, 2012. 3, 4
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 2021. 2.3
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint, 2023. 2, 3,
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin Ying Lec, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high- quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint, 2023. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 5,
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 2,
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jachoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint, 2023. 3
Yichun Shi, Peng Wang. Jianglong Ye, Mai Long. Kejic Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint, 2023. 3
Jiaxiang Tang.
Text-to-3d with stable-diffusion, 2023. https://github.
com/ wtthawwwe//tabbl-erreaaatte
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from single image with diffusion prior. arXiv preprint, 2023.
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, and Federico Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint, 2023. 2,
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, 2021.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: fideliiiy and diverse text-to-3d generation with variational score distillation. arXiv preprint, 2023. 2,3
Jinbo Wu, Xiaobo Gao, Xing Liu, Zhengyang Shen, Chen Zhao, Haocheng Feng, Jingtuo Liu, and Errui Ding. Hd-fusion: Detailed text-to-3d 3d generation leveraging multiple noise estimation. arXiv preprint, 2023. 3
Xin Yu, Peng Dai, Wenbo Li, Lan Ma. Zhengzhe Liu, and Xiaojuan Qi. Texture generation 3d meshes with point-uv diffusion, arXiv preprint, 2023. 3
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. In NewrIPS, 2020.

---

Published as a conference paper at ICLR 2024
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In ICLR, 2019.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprini. 2023. 3,7
Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint, 2023. 2,
A APPENDIX
A.1 POTENTIAL RISKS AND SOCIAL IMPACTS
Every method that learns from data carries the risk of introducing biases. Our 3D generation model based on the text-to-inee totaaeo-mmae models that are pre-trained on the image and text data from the Internet. Work that bases itself our method should carefully consider the consequences of any potential underlying biases.
A.2 LINEAR APPROXIMATION FOR VAE
We provide linear approximation between image pixel æ E Rs and latent pixel z E R4: x Aoz bo and z = A æ b1, where
- 0.5537
1.8844
2.1757
1.65900
33.9900
1.7472
1.6805
0.3810
Ao
bu
0.6894
3.2756
-3.4658
0.3939
224990
1.3309
-0.1115
0.7896
T0.1956
-0.0910
0.0462
-0.15217
T0.5573
A1
0.2125
-0.0206
0.0401
-0.1215
b
0.5105
00.2208
0.0047
-0.0028
-0.1083
0..4635
Figure visualizes samples of fitted image-latent pair.
GT
fitted
(b)
Figure 6: Samples of linear approximation. (a) The results of VAE decoder approximation. (b) The results of VAE encoder approximation. For each case, top is the ground- -truth and bottom is the fitted result.
A.3 ABLATION ON CLIPPING THRESHOLD
Figure shows the texture quality is robust to different clipping thresholds.
12

---

Published as a conference paper at ICLR 2024
0.01
0.05
0.1
0.5
1.0
Figure 7: Ablation on clipping threshold. The thresholds are chosen from {0.01, 0.05, 0.1, 0.5, 1.0).
13


Reviews:
[{"review_id": "BjLPVClC0b", "reviewer": "Reviewer_EEyr", "summary": "In the field of high-resolution 3D object generation, the limited availability of training data necessitates the widespread use of knowledge transfer techniques. However, in such methods (SDS), it is challenging to control gradients at the pixel level when applying pretrained models. To address this issue, this paper proposes a method called Pixel-wise Gradient Clipping (PGC) to effectively control gradients and incorporate them into existing 3D generation models. The proposed PGC offers a straightforward approach that can greatly aid in rendering high-quality, high-resolution objects. This paper proposes a reasonable approach within the context of previous research efforts in the actively studied field of 3D object generation to achieve high-quality results in high resolution. It introduces not only regularization for Score Distillation sampling but also Pixel-wise normalized gradient descent (PNGD) to preserve the details of textures. These techniques aim to generate high-resolution outputs with excellent quality.", "questions": "1. I am curious if there were any side effects that arose from applying the proposed method to 3D object generation.\n2. I am curious if there are any experimental results for more unusual exceptional samples that were tested for comparison evaluation.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a clear and intuitive explanation of the proposed PGC and PNGD methods, showcasing high-quality results that align with these techniques. Additionally, considering the inherent ambiguity in defining metrics in the field, the paper presents indirect yet convincing numerical values through user studies. Furthermore, qualitative results are presented through experiments on mesh optimization, demonstrating significant performance improvements.", "weaknesses": "I think it would be better if there is an in-depth analysis regarding the gradient control aspect of the proposed method. It would be beneficial to have more thorough consideration and analysis on how the gradient is influenced by the proposed method compared to the vanilla model. Presenting a more comprehensive investigation and analysis in this regard would enhance the paper. It is disappointing that the stability of the gradient through PGC and PNGD is only demonstrated through the results without further analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WG7WbG05ko", "reviewer": "Reviewer_ezZ1", "summary": "This paper aims to address gradient-related issues in typical Latent Diffusion Models used for 3D generation. This paper finds that the unregulated gradients through image models are harmful for the 3D models to capture correct and fine textures. To solve this problem, this paper proposes Pixel-wise Gradient Clipping (PGC) that clips the pixel-wise gradients by norm. Experiments show improved performance over baselines such as Fantasia3D.", "questions": "1. Since this paper mostly focuses on addressing the gradient issues, I would suggest adding a paragraph in Sec. 2 to review general gradient-related techniques and their connections&differences with the proposed method. Example related papers include:\n[a] Zhang et al. Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity. ICLR 2020.\n[b] Zhang et al. Improved analysis of clipping algorithms for non-convex optimization. NeurIPS 2020.\n[c] Brock et al. High-Performance Large-Scale Image Recognition Without Normalization. ICML 2021. \n[d] Koloskova et al. Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees. ICML 2023.\n\n2. Minor suggestions: \n(1) Currently the texts that refer to Fig. 2 are scattered all around. It may improve the clarity a lot by adding a few sentences in the caption to explain different columns in Fig. 2.\n(2) Typo: last paragraph before Sec. 4.3: \"xt-1\" --> \"x_{t-1}\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow and understand. For example, section 3 provides enough background on score distillation sampling and gradient clipping to motivate the observations of optimization issue in existing models and the proposals on pixel-wise gradient clipping.\n\n2. This paper targets at an important goal: improving gradient flows that 3D models obtain from 2D diffusion models. This is critical because many existing text/image-to-3D models rely on pretrained 2D diffusion models to guide the optimization of a 3D model. An effective tool can potentially benefit lots of related work.\n\n3. The proposed PGC seem to be effective over a few baselines such as Fantasia3D+SDXL.", "weaknesses": "1. Insufficient ablation study\nThe paper reviews parameter-wise normalized Gradient Descent (NGD) and Gradient Clipping (GC) in Sec. 3.2 & 3.3 and proposes pixel-wise NGD and GC in Sec. 4.3 & 4.4. Pixel gradients are computed from parameter gradients, and bounded parameter gradients may result in bounded pixel gradients too -- an alternative of the proposed Sec. 4.3 & 4.4 to limit pixel gradient magnitudes. Therefore, parameter-wise NGD and GC would have served as good baselines for ablation study. However, the paper only showed results of pixel-wise NGD and GC but not results of parameter-wise NGD and GC or other simple ways to bound pixel gradients. \n\n2. Insufficient experiments to show that the proposed method \"benefit existing SDS and LDM-based 3D generative models\"\nThe paper claims \"PGC consistently benefit existing SDS and LDM-based 3D generative models\", but experiments only compared with two baselines: Stable-DreamFusion and (SDXL variant of) Fantasia3D, which are not published state-of-the-arts and not sufficient to support the claim. For example, ProlificDreamer [Wang et al. 2023] and improved Fantasia3D (https://github.com/Gorilla-Lab-SCUT/Fantasia3D \"Q7\") can produce more realistic texture results than the compared baselines. Will the proposed PGC still be effective on top these two methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dd1E9CJ5F8", "reviewer": "Reviewer_L63f", "summary": "The paper identifies a critical and generic issue in optimizing high-resolution 3D models by exploiting latent diffusion models. To address this problem, the paper analyzes the gradient propagation process and proposes a simple and effective gradient clipping technique. The paper demonstrates the proposed techniques can be used as a generic plug-in method to improve a body of works based on LDM.", "questions": "(1) In Section 4.2, it says \"we observe that the difference between x and \\hat{x} remains strictly constrained within the interval of (0, 1) due to RGB restrictions\". I think the range of the difference should be in [-1,1]. Did I miss something?\n\n(2) Can the author provide some insights on why \"this constraint plays a crucial role in stabilizing the training process for the 3D model\"?\n\n(3) Another way is to clip the gradient of the entire term, i.e. (eps_phi - eps) * dz/dx, rather than the dz/dx. Can the author comment on this alternative method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper identifies an important problem in LDM based 3D AIGC method, and proposes a simple and effective technique to solve this problem. I like the general idea of this paper. I think it can benefit a large amount of future work in this field.", "weaknesses": "(1) The most important Figure in the paper, i.e. Figure 2, is unclear to me.\n* Figure 2 is first referenced in Section 3.1. For me, it's very hard to understand the meaning and experiment setup of this Figure. What's the meaning of the three rows respectively? What's the task of Figure 2, is the mesh fixed while only optimizing the texture? Many important details are missed in my eyes; it makes me feel hard to follow the paper.\n\n(2) Some other figures and descriptions are unclear to me. More necessary details should be provided.\n* In Figure 6, what is the meaning of the 6 images in the right column? Is the gradient?\n* Can the author provide more illustrations to support Section 4.5?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)