Hash3D: Training-free Acceleration for 3D Generation

## Abstract

Abstract The evolution of 3D generative modeling has been notably propelled by the adoption of 2D diffusion models. Despite this progress,
the cumbersome optimization process per se
presents a critical hurdle to efficiency.
In this paper,
we introduce Hash3D,
a universal acceleration for 3D generation without model training.
Central to Hash3D is the insight that feature-map redundancy is prevalent in images rendered from camera positions and diffusion time-steps in close proximity. By effectively hashing and reusing these feature maps across neighboring timesteps and camera angles, Hash3D substantially prevents redundant calculations, thus accelerating the diffusion model’s inference in 3D generation tasks. We achieve this through an adaptive grid-based hashing.
Surprisingly, this feature-sharing mechanism
not only speed up the generation
but also enhances the smoothness and view consistency of the synthesized 3D objects. Our experiments covering 5 text-to-3D and 3 image-to-3D models, demonstrate Hash3D’s versatility to speed up optimization, enhancing efficiency by 1.3 ∼ 4 × 1.3\sim 4\times 1.3 ∼ 4 × . Additionally, Hash3D’s integration with 3D Gaussian splatting largely speeds up 3D model creation, reducing text-to-3D processing to about 10 minutes and image-to-3D conversion to roughly 30 seconds. The project page is at https://adamdad.github.io/hash3D/ .

###### Keywords:

## 1 Introduction

In the evolving landscape of 3D generative modeling, the integration of 2D diffusion models [, ] has led to notable advancements. These methods leverage off-the-the-shelf image diffusion models to distill 3D models by predicting 2D score functions at different views, known as score distillation sampling (SDS).

While this approach has opened up new avenues for creating detailed 3D assets, it also brings forth significant challenges, particularly in terms of efficiency. Particularly, SDS requires sampling thousands of score predictions at different camera poses and denoising timesteps from the diffusion model, causing a extensively long optimization, even for hours to create one object []. These prolonged duration create a significant obstacle to apply them in practical application products, calling for new solutions to improve its efficiency.

To mitigate this bottleneck, current efforts concentrate on three strategies. The first strategy trains an inference-only models [, , , , ] to bypass the lengthy optimization process. While effective, this method requires extensive training time and substantial computational resources. The second approach [, , ] seeks to reduce optimization times through enhanced 3D parameterization techniques. However, this strategy necessitates a unique design for each specific representation, presenting its own set of challenges. The third approach attempts to directly generate sparse views to model 3D objects, assuming near-perfect view consistency in generation [, ] which, in practice, is often not achievable.

Returning to the core issue within SDS, a considerable portion of computational effort is consumed in repeated sampling of the 2D image score function []. Motivated by methods that accelerate 2D diffusion sampling [, , ], we posed the question: *Is it possible to reduce the number of inference steps of the diffusion model for 3D generation?*

In pursuit of this, our exploration revealed a crucial observation: denoising outputs and feature maps from near camera positions and timesteps are remarkably similar. This discovery directly informs our solution, Hash3D, designed to reduce the computation by leveraging this redundancy.

At its core, Hash3D implements a space-time trade-off through a grid-based hash table. This table stores intermediate features from the diffusion model. Whenever a new sampled view is close to one it has already worked on, Hash3D efficiently retrieves the relevant features from the hash table. By reusing these features to calculate the current view’s score function, it avoids redoing calculations that have already been done. Additionally, we have developed a method to dynamically choose the grid size for each view, enhancing the system’s adaptability. As such, Hash3D not only conserves computational resources, but does so without any model training or complex modifications, making it simple to implement and efficient to apply.

Beyond just being efficient, Hash3D helps produce 3D objects with improved multi-view consistency. Traditional diffusion-based methods often result in 3D objects with disjointed appearances when viewed from various angles []. In contrast, Hash3D connects independently sampled views by sharing features within each grid, leading to smoother, more consistent 3D models.

Another key advantage of Hash3D is on its versatility. It integrates seamlessly into a diverse array of diffusion-based 3D generative workflows. Our experiments, covering 5 text-to-3D and 3 image-to-3D models, demonstrate Hash3D’s versatility to speed up optimization, enhancing efficiency by $1.3\sim 4\times$, without compromising on performance. Specifically, the integration of Hash3D with 3D Gaussian Splatting [] brings a significant leap forward, cutting down the time for text-to-3D to about 10 minutes and image-to-3D to roughly 30 seconds.

The contribution of this paper can be summarized into

- •
We introduce the Hash3D, a versatile, plug-and-play and training-free acceleration method for diffusion-based text-to-3D and image-to-3D models.
- •
The paper emphasizes the redundancy in diffusion models when processing nearby views and timesteps. This finding motivates the development of Hash3D, aiming to boost efficiency without compromising quality.
- •
Hash3D employs an adaptive grid-based hashing to efficiently retrieve features, significantly reducing the computations across view and time.
- •
Our extensive testing across a range of models demonstrates that Hash3D not only speeds up the generative process by $1.3\sim 4\times$, but also results in a slight improvement in performance.

## 2 Related Work

3D Generation Model.
The development of 3D generative models has become a focal point in the computer vision. Typically, these models are trained to produce the parameters that define 3D representations. This approach has been successfully applied across several larger-scale models using extensive and diverse datasets for generating voxel representation [], point cloud [, ], implicit function [], triplane [, ]. Despite these advances, scalability continues to be a formidable challenge, primarily due to data volume and computational resource constraints.
A promising solution to this issue lies in leveraging 2D generative models to enhance and optimize 3D representations. Recently, diffusion-based models, particularly those involving score distillation into 3D representations [], represent significant progress. However, these methods are often constrained by lengthy optimization processes.

Efficient Diffusion Model. Diffusion models, known for their iterative denoising process for image generation, are pivotal yet time-intensive. There has been a substantial body of work aimed at accelerating these models. This acceleration can be approached from two angles: firstly, by reducing the sampling steps through advanced sampling mechanisms [, , , ] or timestep distillation [, ], which decreases the number of required sampling steps. The second approach focuses on minimizing the computational demands of each model inference. This can be achieved by developing smaller diffusion models [, , ] or reusing features from adjacent steps [, ], thereby enhancing efficiency without compromising effectiveness. However, the application of these techniques to 3D generative tasks remains largely unexplored.

Hashing Techniques. Hashing, pivotal in computational and storage efficiency, involves converting variable-sized inputs into fixed-size hash code via *hash functions*. These code index a *hash table*, enabling fast and consistent data access. Widely used in file systems, hashing has proven effective in a variety of applications, like 3D representation [, , , ], neural network compression [, ], using hashing as a components in deep network [] and neural network-based hash function development [, , , ]. Our study explores the application of hashing to retrieve features from 3D generation. By adopting this technique, we aim to reduce computational overhead for repeated diffusion sampling and speed up the creation of realistic 3D objects.

## 3 Preliminary

In this section, we provide the necessary notations, as well as the background on optimization-based 3D generation, focusing on diffusion models and Score Distillation Sampling (SDS) [].

### 3.1 Diffusion Models

Diffusion models, a class of generative models, reverse a process of adding noise by constructing a series of latent variables. Starting with a dataset $\mathbf{x}_{0}$ drawn from a distribution $q(\mathbf{x}_{0})$, the models progressively introduce Gaussian noise over $T$ steps. Each step, defined as $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t
}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$, is controlled by $\beta_{1:T}$, values ranging from 0 to 1. The inherently Gaussian nature of this noise enables direct sampling from $q(\mathbf{x}_{t})$ using the formula $\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}
\bm{\epsilon},\quad\text{where}\quad\epsilon\sim\mathcal{N}(0,\mathbf{I})$ with $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.

The reverse process is formulated as a variational Markov chain, parameterized by a time-conditioned denoising neural network $\bm{\epsilon}(\mathbf{x}_{t},t,y)$, with $y$ being the conditional input for generation, such as text for text-to-image model [] or camera pose for novel view synthesis [].
The training of the denoiser aims to minimize a re-weighted evidence lower bound (ELBO), aligning with the noise:

$$ $\displaystyle\mathcal{L}_{\text{DDPM}}=\mathbb{E}_{t,\mathbf{x}_{0},\bm{ \epsilon}}\left[||\bm{\epsilon}-\bm{\epsilon}(\mathbf{x}_{t},t,y)||_{2}^{2}\right]$ (1) $$

Here, $\bm{\epsilon}(\mathbf{x}_{t},t,y)$ approximates the score function $\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t}|\mathbf{x}_{0})$. Data generation is achieved by denoising from noise, often enhanced using classifier-free guidance with scale parameter $\omega$: $\bm{\hat{}}{\epsilon}(\mathbf{x}_{t},t,y)=(1+\omega)\bm{\epsilon}(\mathbf{x}_{
t},t,y)-\omega\bm{\epsilon}(\mathbf{x}_{t},t,\emptyset)$.

Extracting Feature from Diffusion Model. A diffusion denoiser $\bm{\epsilon}$ is typically parameterized with a U-Net []. It uses $l$ down-sampling layers $\{D_{i}\}_{i=1}^{l}$ and up-sampling layers $\{U_{i}\}_{i=1}^{l}$, coupled with skip connections that link features from $D_{i}$ to $U_{i}$. This module effectively merges high-level features from $U_{i+1}$ with low-level features from $D_{i}$, as expressed by the equation:

$$ $\mathbf{v}_{i+1}^{(U)}=\text{concat}(D_{i}(\mathbf{v}_{i-1}^{(D)}),U_{i+1}( \mathbf{v}_{i}^{(U)}))$ (2) $$

In this context, $\mathbf{v}_{i}^{(U)}$ and $\mathbf{v}_{i+1}^{(D)}$ represent the up-sampled and down-sampled features after the $i$-th layer, respectively.

### 3.2 Score Distillation Sampling (SDS)

The Score Distillation Sampling (SDS) [] represents an optimization-based 3D generation method. This method focuses on optimizing the 3D representation, denoted as $\Theta$, using a pre-trained 2D diffusion models with its noise prediction network, denoted as $\bm{\epsilon}_{\text{pretrain}}(x_{t},t,y)$.

Given a camera pose $\bm{c}=(\theta,\phi,\rho)\in\mathbb{R}^{3}$ defined by elevation $\phi$, azimuth $\theta$ and camera distances $\rho$, and the its corresponding prompt $y^{c}$, a differentiable rendering function $g(\cdot;\Theta)$, SDS aims to refine the parameter $\Theta$, such that each rendered image $\bm{x}_{0}=g(\bm{c};\theta)$ is perceived as realistic by $\bm{\epsilon}_{\text{pretrain}}$. The optimization objective is formulated as follows:

$$ $\min_{\Theta}\mathcal{L}_{\text{SDS}}=\mathbb{E}_{t,\bm{c}}\left[\frac{\sigma_ {t}}{\alpha_{t}}\omega(t)\text{KL}\left(q^{\Theta}(\bm{x}_{t}|y_{c},t)\,\|\,p( \bm{x}_{t}|y_{c};t)\right)\right]$ (3) $$

By excluding the Jacobian term of the U-Net, the gradient of the optimization problem can be effectively approximated:

$$ $\nabla_{\Theta}\mathcal{L}_{\text{SDS}}\approx\mathbb{E}_{t,\bm{c},\bm{ \epsilon}}\left[\omega(t)(\bm{\epsilon}_{\text{pretrain}}(\bm{x}_{t},t,y^{c})- \bm{\epsilon})\frac{\partial\bm{x}}{\partial\Theta}\right]$ (4) $$

To optimize Eq. 4, we randomly sample different time-step $t$, camera $\bm{c}$, and random noise $\bm{\epsilon}$, and
compute gradient of the 3D representation, and update $\theta$ accordingly. This approach ensures that the rendered image from 3D object aligns with the distribution learned by the diffusion model.

Efficiency Problem. The main challenge lies in the need for thousands to tens of thousands of iterations to optimize Eq 4, each requiring a separate diffusion model inference. This process is time-consuming due to the model’s complexity. We make it faster by using a hash function to reuse features from similar inputs, cutting down on the number of calculations needed.

## 4 Hash3D

This section introduces Hash3D, a plug-and-play booster for Score Distillation Sampling (SDS) to improve its efficiency. We start by analyzing the redundancy presented in the diffusion model across different timestep and camera poses. Based on the finding, we present our strategy that employ a grid-based hashing to reuse feature across different sampling iterations.

### 4.1 Probing the Redundancy in SDS

Figure: Figure 2: Feature similarity extracted from different camera poses.
Refer to caption: x2.png

Typically, SDS randomly samples camera poses and timesteps to ensure that the rendered views align with diffusion model’s prediction. A critical observation here is that deep feature extraction at proximate $\bm{c}$ and $t$ often reveals a high degree of similarity. This similarity underpins our method, suggesting that reusing features from nearby points does not significantly impact model’s prediction.

Measuring the Similarity. Intuitively, images captured with close up camera and times results in similar visual information. We hypothesize that features produced by diffusion model would exhibit a similar pattern. In terms of the *temporal similarity*, previous studies[, ] have noted that features extracted from adjacent timesteps from diffusion models show a high level of similarity.

To test the hypothesis about the *spatial similarity*, we conducted a preliminary study using the diffusion model to generate novel views of the same object from different camera positions. In practice, we use Zero-123 [] to generate image from different cameras poses conditioned on single image input. For each specific camera angle and timestep, we extracted features $\mathbf{v}_{l-1}^{(U)}$ as the input of the last up-sampling layer at each timestep. By adjusting elevation angles ($\phi$) and azimuth angles ($\theta$), we were able to measure the cosine similarity of these features between different views, averaging the results across all timesteps.

Figure: Figure 3: By interpolating latent between generated views, we enable the synthesis of novel views with no computations.
Refer to caption: x3.png

The findings, presented in Figure 2, reveal a large similarity score in features from views within a $[-10^{\circ},10^{\circ}]$ range, with the value higher than 0.8. This phenomenon was not unique to Zero-123; we observed similar patterns in text-to-image diffusion models like Stable Diffusion []. These findings underscore the redundancy in predicted outputs within the SDS process.

Synthesising Novel View for Free. Exploiting redundancy, we conducted an initial experiment to create new views by simply reusing and interpolating scores from precomputed nearby cameras. We started by generating 2 images using Zero-123 at angles $(\theta,\phi)=(10^{\circ}\pm\delta,90^{\circ})$ and saved all denoising predictions from each timestep. Our goal was to average all pairs of predictions to synthesize a 3-$nd$ view at $(10^{\circ},90^{\circ})$ for free. We experimented with varying $\delta\in\{1^{\circ},5^{\circ},10^{\circ},20^{\circ}\}$, and compared them with the full denoising predictions.

Figure 3 demonstrates that for angles ($\delta$) up to $5^{\circ}$, novel views closely match fully generated ones, proving effective for closely positioned cameras. Yet, interpolations between cameras at wider angles yield blurrier images. Additionally, optimal window sizes vary by object; for example, a $\delta=5^{\circ}$ suits the ghost but not the capybara, indicating that best window size is sample-specific.

Based on these insights, we presents a novel approach: instead of computing the noise prediction for every new camera pose and timestep, we create a memory system to store previously computed features. As such, we can retrieve and reuse these pre-computed features whenever needed. Ideally, this approach could reduces redundant calculations and speeds up the optimization process.

### 4.2 Hashing-based Feature Reuse

In light of our analysis, we developed Hash3D, a solution that incorporates hashing techniques to optimize the SDS. Hash3D is fundamentally designed to minimize the repetitive computational burden typically associated with the diffusion model, effectively trading storage space for accelerated 3D optimization.

At the core of Hash3D is a hash table for storing and retrieving previously extracted features. When Hash3D samples a specific camera pose $\bm{c}$ and timestep $t$, it first checks the hash table for similar features. If a match is found, it’s reused directly in the diffusion model, significantly cutting down on computation. If there’s no match in the same hash bucket, the model performs standard inference, and the new feature is added to the hash table for future use.

Grid-based Hashing. For efficient indexing in our hash table, we use a *grid-based hashing function* with keys composed of camera poses $\bm{c}=(\theta,\phi,\rho)$ and timestep $t$. This function assigns each camera and timestep to a designated grid cell, streamlining data organization and retrieval.

Firstly, we define the size of our grid cells in both the spatial and temporal domains, denoted as $\Delta\theta,\Delta\phi,\Delta\rho$ and $\Delta t$ respectively. For each input key $[\theta,\phi,\rho,t]$, the hashing function calculates the indices of the corresponding grid cell. This is achieved by dividing each coordinate by its respective grid size

$$ $\displaystyle i=\left\lfloor\frac{\theta}{\Delta\theta}\right\rfloor,j=\left \lfloor\frac{\phi}{\Delta\phi}\right\rfloor,k=\left\lfloor\frac{\rho}{\Delta \rho}\right\rfloor,l=\left\lfloor\frac{t}{\Delta t}\right\rfloor$ (5) $$

Upon obtaining these indices, we combine them into a single hash code that uniquely identifies each bucket in the hash table. The hash function $\texttt{idx}=(i+N_{1}\cdot j+N_{2}\cdot k+N_{3}\cdot l)\mod n$ is used, where $N_{1},N_{2},N_{3}$ are large prime numbers [, ], and $n$ denotes the size of the hash table.

Through this hash function, keys that are close in terms of camera pose and timestep are likely to be hashed to the same bucket. This grid-based approach not only making the data retrieval faster but also maintains the spatial-temporal relationship inherent in the data, which is crucial for our method.

Collision Resolution.
When multiple keys are assigned to the same hash value, a collision occurs. We address these collisions using *separate chaining*.
In this context, each hash value idx is linked to a distinct queue, denoted as $q_{\texttt{idx}}$. To ensure the queue reflects the most recent data and remains manageable in size, it is limited to a maximum length $Q=3$. When this limit is reached, the oldest elements is removed to accommodate the new entry, ensuring the queue stays relevant to the evolving 3D representation.

Figure: Figure 4: Overall pipeline of our Hash3D. Given the sampled camera and time-step, we retrieve the intermediate diffusion feature from hash table. If no matching found, it performs a standard inference and stores the new feature in the hash table; otherwise, if a feature from a close-up view already exists, it is reused without re-calculation.
Refer to caption: x4.png

Feature Retrieval and Update.
Once the hash value idx is determined, we either retrieve existing data from the hash table or update it with new features. We set a hash probability $0<\eta<1$ to make sure the balanced behavior between retrieval and update. In other words, with probability $\eta$, we retrieve the feature; otherwise, it performs hash table updates.

For feature updates, following prior work [], we extract the feature $\mathbf{v}_{l-1}^{(U)}$, which is the input of last up-sampling layer in the U-net. Once extracted, we compute the hash code idx and append the data to the corresponding queue $q_{\texttt{idx}}$. The stored data includes input noisy latent $\bm{x}$, camera pose $\bm{c}$, timestep $t$, and extracted diffusion features $\mathbf{v}_{l-1}^{(U)}$.

For feature retrieval, we aggregate data from $q_{\texttt{idx}}$ through weighted averaging. This method considers the distance of each noisy input $\bm{x}_{i}$ from the current query point $\bm{x}$. The weighted average $\mathbf{v}$ for a given index is calculated as follows:

$$ $\displaystyle\mathbf{v}=\sum_{i=1}^{|q_{\texttt{idx}}|}W_{i}\mathbf{v}_{i}, \text{ where }W_{i}=\frac{e^{(-||\bm{x}-\bm{x}_{i}||_{2}^{2})}}{\sum_{i=1}^{|q _{\texttt{idx}}|}e^{(-||\bm{x}-\bm{x}_{i}||_{2}^{2})}}$ (6) $$

Here, $W_{i}$ is the weight assigned to $\mathbf{v}_{i}$ based on its distance from the query point, and $|q_{\texttt{idx}}|$ is the current length of the queue. An empty queue $|q_{\texttt{idx}}|$ indicates unsuccessful retrieval, necessitating feature update.

### 4.3 Adaptive Grid Hashing

In grid-based hashing, the selection of an appropriate grid size $\Delta\theta,\Delta\phi,\Delta\rho,\Delta t$ — plays a pivotal role. As illustrated in Section 4.1, we see three insights related to grid size. First, feature similarity is only maintained at a median grid size; overly large grids tend to produce artifacts in generated views. Second, it is suggested that ideal grid size differs across various objects. Third, even for a single object, optimal grid sizes vary for different views and time steps, indicating the necessity for adaptive grid sizing to ensure optimal hashing performance.

Learning to Adjust the Grid Size. To address these challenges, we propose to dynamically adjusting grid sizes. The objective is to maximize the average cosine similarity $\text{cos}(\cdot,\cdot)$ among features within each grid. In other words, only if the feature is similar enough, we can reuse it. Such problem is formulated as

$$ $\max_{\Delta\theta,\Delta\phi,\Delta\rho,\Delta t}\frac{1}{|q_{\texttt{idx}}|} \sum_{i,j}^{|q_{\texttt{idx}}|}\text{cos}(\mathbf{v}_{j},\mathbf{v}_{i}),\quad s .t.|q_{\texttt{idx}}|>0\quad[\text{Non-empty}]$ (7) $$

Given our hashing function is *non-differentiale*, we employ a brute-force approach. Namely, we evaluate $M$ predetermined potential grid sizes, each corresponding to a distinct hash table, and only use best one.

For each input $[\theta,\phi,\rho,t]$, we calculate the hash code $\{{\texttt{idx}}^{(m)}\}_{m=1}^{M}$ for $M$ times, and indexing in each bucket.
Feature vectors are updated accordingly, with new elements being appended to their respective bucket. We calculate the cosine similarity between the new and existing elements in the bucket, maintaining a running average $s_{\texttt{idx}^{(n)}}$ of these similarities

$$ $s_{\texttt{idx}^{(m)}}\leftarrow\gamma s_{\texttt{idx}^{(m)}}+(1-\gamma)\frac{ 1}{|q_{\texttt{idx}^{(m)}}|}\sum_{i=1}^{|q_{\texttt{idx}^{(m)}}|}\text{cos}( \mathbf{v}_{new},\mathbf{v}_{i})$ (8) $$

During retrieval, we hash across all $M$ grid sizes but only consider the grid with the highest average similarity for feature extraction.

Computational and Memory Efficiency. Despite employing a brute-force approach that involves hashing $M$ times for each input, our method maintains computational efficiency due to the low cost of hashing. It also maintains memory efficiency, as hash tables store only references to data. To prioritize speed, we deliberately avoid using neural networks for hashing function learning.

## 5 Experiment

In this section, we assess the effectiveness of our HS by integrating it with various 3D generative models, encompassing both image-to-3D and text-to-3D tasks.

### 5.1 Experimental Setup

Baselines. To validate the versatility of our method, we conducted extensive tests across a wide range of baseline text-to-3D and image-to-3D methods.

- •
Image-to-3D. Our approach builds upon techniques such as Zero-123+SDS  [], DreamGaussian [] and Magic123 []. For Zero-123+SDS, we have incorporated Instant-NGP [] and Gaussian Splatting [] as its representation. We call these two variants Zero-123 (NeRF) and Zero-123 (GS).
- •
Text-to-3D. Our tests also covered a range of methods, such as Dreamfusion [], Fantasia3D [], Latent-NeRF [], Magic3D [], and GaussianDreamer [].

For DreamGaussian and GaussianDreamer, we implement Hash3D on top of the official code. And for other methods, we use the reproduction from threestudio(^1^11https://github.com/threestudio-project/threestudio).

Implementation Details. We stick to the same hyper-parameter setup within their original implementations of these methods. For text-to-3D, we use the stable-diffusion-2-1(^2^22https://huggingface.co/stabilityai/stable-diffusion-2-1) as our 2D diffusion model. For image-to-3D, we employ the stable-zero123(^3^33https://huggingface.co/stabilityai/stable-zero123). We use a default hash probability setting of $\eta=0.1$. We use $M=3$ sets of grid sizes, with $\Delta\theta,\Delta\phi,\Delta t\in\{10,20,30\}$ and $\Delta\rho\in\{0.1,0.15,0.2\}$. We verify this hyper-parameter setup in the ablation study.

Dataset and Evaluation Metrics. To assess our method, we focus on evaluating the computational cost and visual quality achieved by implementing Hash3D.

- •
Image-to-3D. For the image-to-3D experiments, we leverage the Google Scanned Objects (GSO) dataset [] for evaluation [, ]. We focused on evaluating novel view synthesis (NVS) performance using established metrics such as PSNR, SSIM [], and LPIPS []. We selected 30 objects from the dataset. For each object, we generated a 256$\times$256 input image for 3D reconstruction. We then rendered 16 different views at a 30-degree elevation, varying azimuth angles, to compare the reconstructed models with their ground-truth. To ensure semantic consistency, we also calculated CLIP-similarity scores between the rendered views and the original input images.
- •
Text-to-3D. We generated 3D models from 50 different prompts, selected based on a prior study. To evaluate our methods, we focused on two primary metrics: mean$\pm$std CLIP-similarity [, , ] and the average generation time for each method. For assessing CLIP-similarity, we calculated the similarity between the input text prompt and 8 uniformly rendered views at elevation $\phi=0^{\circ}$ and azimuth $\theta=[0^{\circ},45^{\circ},90^{\circ},135^{\circ},180^{\circ},225^{\circ},27
0^{\circ},315^{\circ}]$. Additionally, we recorded and reported the generation time for each run.
- •
User Study. To evaluate the visual quality of generated 3D objects, we carried out a study involving 44 participants. They were shown 12 videos of 3D renderings, created using two methods: Zero-123 (NeRF) for images-to-3D, and Gaussian-Dreamer for text-to-3D. These renderings were made both with and without Hash3D. Participants were asked to rate the visual quality of each pair of renderings, distributing a total of 100 points between the two in each pair to indicate their perceived quality difference.
- •
Computational Cost. We report the running time for each experiment using a single RTX A5000. Besides, we report MACs in the tables. Given that feature retrieval is stochastic — implying that retrieval of features is not guaranteed with attempt in empty bucket — we provide the theoretical average MACs across all steps, pretending that all retrieval succeeded.

Figure: Figure 5: Qualitative Results using Hash3D along with Zero123 for image-to-3D generation. We mark the visual dissimilarity in yellow.
Refer to caption: x5.png

### 5.2 3D Generation Results

Image-to-3D Qualitative Results. Figure 5 demonstrates the outcomes of incorporating Hash3D into the Zero-123 framework to generate 3D objects. This integration not only preserves visual quality and consistency across views but also markedly decreases the processing time. In specific instances, Hash3D outperforms the baseline, as evidenced by the enhanced clarity of the dragon wings’ boundaries in row 1 and the more distinct taillights of the train in row 4. A similar level of visual fidelity is observed in Figure 1, where Hash3D is applied in conjunction with DreamGaussian, indicating that the integration effectively maintains quality while improving efficiency.

Image-to-3D Quantitative Results. For a detailed numerical analysis, refer to Table 1, which outlines the novel view synthesis performance, CLIP scores, running times for on top of all 4 baseline methods. Notably, For DreamGaussian and Zero-123(NeRF), we speed up the running time by $4\times$ and $3\times$ respectively. This reduction in running times is mainly due to the efficient feature retrieval and reuse mechanism employed by Hash3D. Additionally, our approach not only speeds up the process but also slightly improves performance. We believe this enhancement stems from the sharing of common features across different camera views, which reduces the inconsistencies found in independently sampled noise predictions, resulting in the smoother generation of 3D models.

**Table 1: Speed and performance comparison when integrated image-to-3D models with Hash3D. We report the original running time in their paper.**
| Method | Time$\downarrow$ | Speed$\uparrow$ | MACs$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CLIP-G/14$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DreamGaussian | 2m | - | 168.78G | 16.202$\pm$2.501 | 0.772$\pm$0.102 | 0.225$\pm$0.111 | 0.693$\pm$0.105 |
| + Hash3D | 30s | 4.0$\times$ | 154.76G | 16.356$\pm$2.533 | 0.776$\pm$0.103 | 0.223$\pm$0.113 | 0.694$\pm$0.104 |
| Zero-123(NeRF) | 20m | - | 168.78G | 17.773$\pm$3.074 | 0.787$\pm$0.101 | 0.198$\pm$0.097 | 0.662$\pm$0.0107 |
| + Hash3D | 7m | 3.3$\times$ | 154.76G | 17.961$\pm$3.034 | 0.789$\pm$0.095 | 0.196$\pm$0.0971 | 0.665$\pm$0.104 |
| Zero-123(GS) | 6m | - | 168.78G | 18.409$\pm$2.615 | 0.789$\pm$0.100 | 0.204$\pm$0.101 | 0.643$\pm$0.105 |
| + Hash3D | 3m | 2.0$\times$ | 154.76G | 18.616$\pm$2.898 | 0.793$\pm$0.099 | 0.204$\pm$0.099 | 0.632$\pm$0.106 |
| Magic123 | 120m | - | 847.38G | 18.718$\pm$2.446 | 0.803$\pm$0.093 | 0.169$\pm$0.092 | 0.718$\pm$0.099 |
| + Hash3D | 90m | 1.3$\times$ | 776.97G | 18.631$\pm$2.726 | 0.803$\pm$0.091 | 0.174$\pm$0.093 | 0.715$\pm$0.107 |

**Table 2: Speed and performance comparison between various text-to-3D baseline when integrated with Hash3D.**
| Method | Time$\downarrow$ | Speed$\uparrow$ | MACs$\downarrow$ | CLIP-G/14$\uparrow$ | CLIP-L/14$\uparrow$ | CLIP-B/32$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Dreamfusion | 1h 00m | - | 678.60G | 0.407$\pm$ 0.088 | 0.267$\pm$0.058 | 0.314 $\pm$0.049 |
| + Hash3D | 40m | 1.5$\times$ | 622.21G | 0.411$\pm$0.070 | 0.266$\pm$ 0.050 | 0.312$\pm$0.044 |
| Latent-NeRF | 30m | - | 678.60G | 0.406$\pm$0.033 | 0.254$\pm$0.039 | 0.306$\pm$0.037 |
| + Hash3D | 17m | 1.8$\times$ | 622.21G | 0.406$\pm$0.038 | 0.258$\pm$0.045 | 0.305$\pm$0.038 |
| SDS+GS | 1h 18m | - | 678.60G | 0.413$\pm$0.048 | 0.263$\pm$0.034 | 0.313$\pm$0.036 |
| + Hash3D | 40m | 1.9$\times$ | 622.21G | 0.402$\pm$0.062 | 0.252$\pm$0.041 | 0.306$\pm$0.036 |
| Magic3D | 1h 30m | - | 678.60G | 0.399$\pm$0.012 | 0.257$\pm$0.064 | 0.303$\pm$0.059 |
| + Hash3D | 1h | 1.5$\times$ | 622.21G | 0.393$\pm$0.011 | 0.250$\pm$0.054 | 0.304$\pm$0.052 |
| GaussianDreamer | 15m | - | 678.60G | 0.412$\pm$0.049 | 0.267$\pm$0.035 | 0.312$\pm$0.038 |
| + Hash3D | 10m | 1.5$\times$ | 622.21G | 0.416$\pm$0.057 | 0.271$\pm$0.036 | 0.312$\pm$0.037 |

Figure: Figure 6: Visual comparison for text-to-3D task, when applying Hash3D to DreamFusion [], SDS+GS and Fantasia3D [].
Refer to caption: x6.png

Text-to-3D Qualitative Results. In Figure 6, we present the results generated by our method, comparing Hash3D with DreamFusion [], SDS+GS, and Fantasia3D []. The comparison demonstrates that Hash3D maintains comparable visual quality to these established methods.

Text-to-3D Quantitative Results. Table 2 provides a detailed quantitative evaluation of Hash3D. Across various methods, Hash3D markedly decreases processing times, showcasing its adaptability in speeding up 3D generation. Significantly, this reduction in time comes with minimal impact on the CLIP score, effectively maintaining visual quality. Notably, with certain methods such as GaussianDreamer, Hash3D goes beyond maintaining quality; it subtly improves visual fidelity. This improvement suggests that Hash3D’s approach, which considers the relationship between nearby camera views, has the potential to enhance existing text-to-3D generation processes.

Figure: Figure 7: User preference study for Hash3D.
Refer to caption: x9.png

User preference study. As shown in Figure 7, Hash3D received an average preference score of 52.33/100 and 56.29/100 when compared to Zero-123 (NeRF) and Gaussian-Dreamer. These scores are consistent with previous results, indicating that Hash3D slightly enhances the visual quality of the generated objects.

### 5.3 Ablation Study and Analysis

In this section, we study several key components in our Hash3D framework.

Ablation 1: Hashing Features *vs.* Hashing Noise. Our Hash3D involves hashing intermediate features in the diffusion U-Net. Alternatively, we explored hashing the predicted *noise* estimation directly, leading to the development of a variant named Hash3D with noise (Hash3D w/n). This variant hashes and reuses the predicted score function directly. We applied this approach to the image-to-3D task using Zero123, and the results are detailed in Table 9. Interestingly, while Hash3D w/n demonstrates a reduction in processing time, it yields considerably poorer results in terms of CLIP scores. This outcome underscores the effectiveness of our initial choice to hash features rather than noise predictions.

Ablation 2: Influence of Hash Probability $\eta$. A crucial factor in our Hash3D is the feature retrieval probability $\eta$. To understand its impact, we conducted an ablation experiment with Dreamfusion, testing various $\eta$ values $\{0.01,0.05,0.1,0.3,0.5,0.7\}$.

The relationship between CLIP score, time, and different $\eta$ values is depicted in Figure 9. We observed that running time steadily decrease across all values. Interestingly, with smaller $\eta$ values (less than 0.3), Hash3D even improved the visual quality of the generated 3D models. We speculate this improvement results from the enhanced smoothness in predicted noises across different views, attributable to feature sharing via a grid-based hash table. However, when $\eta>0.3$, there was negligible impact on running time reduction. Figure 10 showcases the same trend in terms of visual quality. A moderately small $\eta$ effectively balances performance and efficiency. Consequently, we opted for $\eta=0.1$ for the experiments presented in our main paper.

Ablation 3: Adaptive Grid Size. In this study, we introduce a dynamic adjustment of the grid size for hashing, tailored to each individual sample. This adaptive approach, termed AdaptGrid, is evaluated against a baseline method that employs a constant grid size, within the context of Dreamfusion. As illustrated in Table 3, the AdaptGrid strategy surpasses the performance of the constant grid size method. Larger grid sizes tend to compromise the visual quality of generated 3D objects. Conversely, while smaller grid sizes preserve performance to a greater extent, they significantly reduce the likelihood of matching nearby features, resulting in increased computation time.

**Table 3: Ablation study on the Adaptive *v.s.* Constant Grid Size.**
| $\Delta\theta,\Delta\phi,\Delta\rho,\Delta t$ | (10, 10, 0.1, 10) | (20, 20, 0.15, 20) | (30, 30, 0.2, 30) | AdaptGrid (Ours) |
| --- | --- | --- | --- | --- |
| CLIP-G/14$\uparrow$ | 0.408$\pm$0.033 | 0.345$\pm$0.055 | 0.287$\pm$0.078 | 0.411$\pm$0.070 |
| Time$\downarrow$ | 48m | 38m | 32m | 40m |

Figure: Figure 8: Ablation study with different hash probability $\eta$.
Refer to caption: x10.png

Figure: Figure 10: Quantitative ablation study with different hash probability $\eta$
Refer to caption: x11.png

## 6 Conclusion

In this paper, we present Hash3D, a training-free technique that improves the efficiency of diffusion-based 3D generative modeling. Hash3D utilizes adaptive grid-based hashing to efficiently retrieve and reuse features from adjacent camera poses, to minimize redundant computations. As a result, Hash3D not only speeds up 3D model generation by $1.3\sim 4\times$ without the need for additional training, but it also improves the smoothness and consistency of the generated 3D models.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. typo in L089: $1.3 \to 1.5$ according to the abstract and conclusion in introduction.
2. The authors should provide the ablation `only discretize the diffusion timestep` vs. `only discretize the view direction` to ensure that the view direction hash table can contribute to the optimization acceleration.
3. Does Hash3D help the model converge faster? Since the authors claim they set the hash probability as $10$% for the feature query and we need to run the whole U-net when feature updating, the maximal acceleration should be $\sim \times 1.11$. Otherwise, the definition of hash probability in L265 is a typo.
4. The definition of noisy input $x$ is not clear. Also, the authors should provide the quantitative results of `w/o Content-based Aggregation`.
soundness: 2
strengths: 1. The motivation is clear, and the results are convincing.
2. The adaptive grid size adjustment helps Hash3D not have to design the grid size for every generation.
summary: Hash3D proposed a novel feature reusing strategy for score distillation sampling.
By hashing and reusing these feature maps across nearby timesteps and camera angles, Hash3D can speed up existing methods by $1.5 \sim 4 \times$.
Also, the feature reusing strategy can improve the visual quality of the generation.
The comprehensive experiments demonstrate the high-efficient acceleration.
weaknesses: 1. The technical contribution is a little insufficient. Hash3D seems to be a simple extension of DeepCache[a].
2. The visual quality improvement is too marginal. The variance is already much more significant than the improvement claimed in the paper.
3. The acceleration rate is similar to DeepCache. It is hard to tell if the technical contributions in Hash3D really help compared with only caching the diffusion feature based on denoising steps, i.e., DeepCache.


Reference:
[a] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Deepcache: Accelerating diffusion models for free." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Besides the questions mentioned in Weaknesses, I have another question. When forwarding the clean input to large noise levels (e.g., t>700), the noisy latent should be dominated by the given noise since it has low SNR. It seems that the authors did not use same noise for each iteration, so I want to know how is the feature similarity of using different noise when t is large?
soundness: 2
strengths: * The method works as a plug-and-play module for various optimization-based 3D generation methods, without the requirement of further training.
* The proposed module can largely accelerate the process of 3D generation without significantly compromising the visual quality.
* The adaptive grid hashing technique effectively improves robustness of the proposed method when working on different cases.
summary: This paper proposes a novel training-free acceleration module for optimization-based 3D generation tasks. In particular, the authors firstly analyse the calculation redundancy of the diffusion based optimization process, probing on which they propose to reuse similar previously-computed features to accelerate the current iteration, in a gird-based hashing manner. Furthermore, the authors utilize an adaptive hashing strategy to more efficiently retrieve features. The experimental results show the effectiveness of the proposed method on accelerating 3D generation.
weaknesses: * There exists inconsistency between the scenario which the analysises in section 3.1 are conducted on, and which this work actually addresses. The former is a purely 2D task while the latter is facing a 3D task with the targeted object evolving. Since the optimized 3D object keeps changing during the generation procedure (especially in the initial stage), could the hypotheses, temporal similarity and spatial similarity, still be held?
* The proposed method seems to introduce numerical error to the optimization process, because it uses (worse) prediction results from previous optimization steps to update the current step. It is evidenced by several noisy visual results (e.g., case 1,4 of SDS+3DGS group and case 2,4 of Fantasia3d group in Figure 6).
* The paper mentions the existence of other kinds of works for acceleration of 3D generation, but the experimental results lack comparison with them.
* While the results in Figure 3 have shown unaccepetable outcome when $\delta>5$, why the chosen window size candidates (\[10, 20, 30\]) are all larger than 5?

Reasons for current ratings:

While the submission creates a novel training-free acceleration module for optimization-based 3D generation tasks, the evaluations for the base hypotheses and comparison with baselines of this work are not conducted soundly enough. I think the authors should further polish this work in the above mentioned aspects.

---

confidence: 1
contribution: 2
initial_rating: 5
presentation: 2
questions: Although this paper presents a relatively novel idea for accelerating SDS methods, it lacks experiments involving advanced SDS techniques, and some aspects of the presentation are unclear. Therefore, I am giving it a rating of 5 and encourage the authors to address the issues and concerns highlighted in the weaknesses section.
soundness: 3
strengths: Here is the strengths of this paper:
- This paper introduces grid hashing to store diffusion features for acceleration, which is a novel idea to improve the efficiency of SDS-based methods. Although somewhat counterintuitive, the authors conduct extensive experiments to demonstrate its effectiveness.
- To enhance the impact of grid hashing, the paper proposes an adaptive method to adjust the grid size, further optimizing performance.
- The paper presents multiple experiments to evaluate the effectiveness of the proposed approach.
summary: This paper presents a training-free method, called Hash3D, to accelerate SDS-based methods. The authors begin by analyzing the hypothesis of feature redundancy in the denoising process of diffusion models across different timesteps and views of a 3D model. They provide an experiment to support this hypothesis. Since SDS relies on this process to generate guidance, the authors propose using grid hashing to store and reuse features when sampling similar timesteps and camera positions. To improve the robustness of this method for various objects, they introduce an adaptive approach to building the grid hash. Extensive results demonstrate the efficiency of this proposed method.
weaknesses: This paper presents a novel and general acceleration method for generation and evaluation in SDS-based models. However, I have several concerns from different perspectives:

**Experiments:** 

ln 409:  distinct “train taillights” in row 4. WHERE IS ROW 4?

Ln 445: In the second row of Fig. 6, the Hash3D result significantly degrades generation quality (e.g., the rabbit's face and the consistency of the horse). This raises concerns about the robustness of this algorithm. A similar issue occurs in the "Batman" experiment with Fantasia3D. The authors should provide an explanation for this degradation.

This paper only compares the proposed algorithm using traditional SDS. Since it claims to be a universal acceleration for SDS-based methods, it should also be evaluated with several advanced SDS methods. Some examples for consideration include:

- Zhu, Joseph, and Peiye Zhuang. "Hifa: High-fidelity text-to-3D with advanced diffusion guidance." ICLR 2024.
- Wang, Zhengyi, et al. "ProlificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation." NeurIPS 2024.
- Liang, Yixun, et al. "LucidDreamer: Towards high-fidelity text-to-3D generation via interval score matching." CVPR 2024.

The authors should select at least one advanced SDS method for evaluation (it is no matter if it is the listed papers) to substantiate the claim of universal acceleration and demonstrate its effectiveness in various applications.

**Present:**

ln 212: The explanation in Synthesizing Novel Views for Free is somewhat confusing. For instance, "reusing and interpolating scores from precomputed nearby cameras" leaves questions about what "scores" refers to and how "interpolating" generates a novel view.  Also, the notation at angles (θ, φ) = (10 ± δ, 90) is how many angles? the notations make the "angles" seens like 10 with several offsets and 90. however, in the following sentense, "... synthesized a **third** view" so it seens like you only generate 2 views, one in 10 and other in 90? The authors should clarify this paragraph for better understanding.

**Typo:**
- ln 034: per se -> per scene ?

- ln 181: double comma

- ln 308: non-differentiale -> non-differentiable

- ln 326: HS -> Hash3D?

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Refer to weaknesses for details.
- Clarification of the feature similarity analysis is needed.
- The proposed method complicates the optimization process by introducing many hyperparameters, and the acceleration rate is very limited while the generation quality is low.
- The authors need to justify how the proposed method can preserve fine details in complex surfaces.
soundness: 2
strengths: - The paper presents a plug-and-play, training-free acceleration method for diffusion-based text-to-3D and image-to-3D models. By leveraging the redundancy in diffusion models for nearby views and timesteps, it employs a hash table structure to enhance efficiency without sacrificing quality.
- Hash3D uses adaptive grid-based hashing for feature retrieval, which effectively reduces computational demands across different views and times.
- Experimental results demonstrate that Hash3D can speed up the generation process by 1.5 to 4 times while maintaining comparable quality.
summary: This paper tackles the challenge of 3D object generation using score distillation sampling (SDS) and proposes a method called Hash3D. This approach aims to speed up the optimization process by utilizing a hashing technique to reuse feature maps across nearby timesteps and camera angles. The experiments indicate that this method can accelerate optimization by 1.5 to 4 times without significantly compromising quality.
weaknesses: While the paper introduces a new method aimed at speeding up SDS-based 3D object generation, several significant concerns arise:

- The feature similarity analysis relies on the Zero-123 model, which may not be applicable to other diffusion models. On page 4 (L204), the authors assert that similar patterns are observable in models like Stable Diffusion (Rombach et al., 2022). However, the methodology and visualizations used in this analysis are unclear.
- The data presented in Fig. 3 does not convincingly support the assumption that diffusion models exhibit redundant features for images generated from similar camera positions and timesteps. The blurred images produced at a delta angle of 5 degrees suggest that simply retrieving similar features may lead to a loss of surface detail in the generated objects.
- The need to maintain a memory hash table introduces several hyperparameters, and the overall speed-up is marginal (e.g., 1.5x on top of DreamFusion). Furthermore, the quality of the generated objects appears low (see Fig. 6), especially compared to state-of-the-art methods that offer faster runtimes.
- There are more effective strategies for improving speed, such as enhanced geometry initialization, efficient representations, and optimization techniques. This method does not seem to present a practical solution for accelerating 3D generation.