Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation
Authors: Ruihan Gao
Kangle Deng
Gengshan Yang
Wenzhen Yuan
Jun-Yan Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D neural rendering
diffusion model
texture synthesis
multi-modal generation
Primary area: generative_models
TL;DR: We leverage tactile sensing to improve geometric details of generated 3D assets for text-to-3D and image-to-3D tasks.

Abstract: 3D generation methods have shown visually compelling results powered by diffusion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We condition the visual texture generation on high-resolution tactile normals and guide the patch-based tactile texture refinement with a customized TextureDreambooth. We further present a multi-part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities of vision and touch.

Tactile DreamFusion:
Exploiting Tactile Sensing for 3D Generation
Ruihan Gaol
Kangle Deng'
Gengshan Yang' Wenzhen Yuan Jun-Yan Zhu
CCamegie Mellon University
University of Illinois Urbana-Champaign
https://ruihaaggao. github i0/ 1o/Tacti1eorremfussioo
Abstract
3D generation methods have shown visually compelling results powered by diffu- sion image priors. However, they often fail to produce realistic geometric details, resulting in overly smooth surfaces or geometric details inaccurately baked in albedo maps. To address this, we introduce a new method that incorporates touch as an additional modality to improve the geometric details of generated 3D assets. We design a lightweight 3D texture field to synthesize visual and tactile textures, guided by 2D diffusion model priors on both visual and tactile domains. We con- dition the visual texture generation on high-resolution tactile normals and guide the batch tactile texture refinement with customized TextureDreammbont We further present a multi part generation pipeline that enables us to synthesize different textures across various regions. To our knowledge, we are the first to lever- age legolrosol tactile sensing to enhance geometric details for 3D generation tasks. We evaluate our method in both text-to-3D and image-to-3D settings. Our experiments demonstrate that our method provides customized and realistic fine geometric textures while maintaining accurate alignment between two modalities vision and touch.
Introduction
Generating high-fidelity 3D assets crucial for wide range of applications, from content creation in gaming and VR/AR to developing realistic simulations for robotics. Recently, a surge in emerging methods has enabled the creation of 3D assets from a single image [ short text prompt 2
Their success can be attributed to advances in generative models 4 5 and neural rendering 6 as
well as the availability extensive 2D and 3D datasets
Although existing methods can effectively capture the overall shape and visual appearance of objects, they often struggle with synthesizing fine-grained geometric details, such stochastic patterns or bumps from the material, as shown on the left side of Figure [ As a result, the final mesh output tends to be either overly smooth (e.g., the avocado in the top row) or has geometric details incorreetly baked into the albedo map (e.g., the beanie in the bottom row). But why is that?
We argue that there two major bottlenecks. First, high-resolution geometric data largely absent current 2D and 3D datasets. For image datasets like LAION 8, obtaining geometric details challenging due the limited camera resolution in capturing real-world objects. In the case of 3D asset datasets like Objaverse [9], although the assets often include visual textures, they rarely feature high-resolution geometric textures. Second, it is difficult for humans to precisely describe fine geometric textures in natural language, making text-based applications even more challenging.
To address these issues, propose exploring tactile sensing capture high-resolution texture data for 3D generation. Given a text prompt or an input image, we first generate a base mesh with an albedo map our 3D representation. We then capture detailed surface geometry of the target texture
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text to 3D
Tactile Sensing
Ours
Tactile Input
An avocado
A
phone case
avocado
avocado texture
Tactile Input
A beange
A toy flover Hoven texture
miffy bunny
beanie
Our method leverages tactile sensing to improve existing 3D generation pipelines. Left: Given text prompt, we first generate an image using SDXL [2] and then run Wonder3D [13] to generate mesh from the image. This process often results in mesh with overly smooth surface. Right: Our method takes a text prompt and several lactile patches and generates high-fidelity coherent visual and lactile textures that can be transferred different meshes. Our method can easily adapt image-to-3D tasks, as shown the rightmost column, with the reference image's thumbnail displayed at the bottom right corner Please visit our webpage for vidco results.
Figure 1:
high-resolution tactile sensor. We then convert these tactile data into normal maps and train a TextureDreambott on these normal maps. To refine the albedo map and ensure alignment between visual and tactile modalities, learn lightweight 3D texture field that co-optimizes the albedo and normal maps using 2D diffusion guidance across both domains.
using GelSight [T
Furthermore, we extend our approach to synthesize textures across multiple object regions by aggregating 2D diffusion- part segmentation in 3D label field.
Our experiments demonstrate that our method outperforms existing text-to-3D and image-to-3D methods in terms of qualitative comparison and user study. Our method ensures coherent high- resolution textures with precise alignment between appearance and geometry, as shown on the right side of Figure I Our code and datasets are available on our project website.
2 Related Work
3D generation. Following the success of text-to-image models
we have witnessed booming development of 3D generative models conditioned text images. Recent works have used 2D diffusion priors in 3D generation 2 18 3
14 15 16, 17
13 23,, with notable methods like DreamFusion (2) introducing Score Distillation Sampling (SDS) to optimize 3D representation using gradients from 2D diffusion models. Follow-up works have further extended SDS optimization using multi-view diffusion models, improving both 3D generation and single-view
19 20
21
22
reconstruction
25
26
27
trains large-scale transformers to generate 3D shapes in a feed-forward manner, requiring high-quality large-scale 3D asset datasets. While these models effectively capture global shapes, they often fail to accurately render fine-grained geometric details, which are either incorrectly baked in color or lost entirely.
Another line of research 31 32
33 34
Geometry representation in 3D generation. Researchers have explored various 3D representations
for generation, such as voxel grids [35 36, 37 point clouds neural surfaces 48
40) meshes 41
44
45
[9]
neural radiance fields (NeRF) 46
20
22
13 25 and
Gaussians 49 50
Among these, meshes support faster and more efficient rasterization rendering than volumetric rendering, Meshes also integrate seamlessly with graphics engines for downstream

---

Object
Normal Map
3D Height Map
strawberry canvas bag striped steel Figure 2: TouchTexture dataset. We collect tactile normal data from 18 daily objects featuring diverse tactile textures. To demonstrate the local geometric intricacies, we show the tactile normal map and 3D height map for each object. Please refer the supplement for the full set of our data.
avocado
corn
rubber
applications. In contrast, volume representations require separate post- post-procsssing to extract meshes, often resulting in loss of quality. Our approach uses meshes represent coarse geometry while embedding local geometric details into a 3D texture field of tactile normal.
3D texture generation and transfer. Simultaneously generating geometry and texture often results in blurry textures due to surface smoothing or averaging across views. To address this, many approaches propose subsequent stage that focuses on refining high-resolution textures [22] [3] or
treats texture generation as a separate problem entirely [1 52 53
54
55 56). Additionally. various
texture transfer methods enable the transfer of texture to new shapes using images [57] [88 59 60)
or other material assets [6]]. While most methods focus solely on visual appearance, our approach produces high-fidee high-fidelity geometric details. Closely related to our method, NeRF-Texture [62] also transfers textures with geometric variations but requires 100-200 images of the same object. In addition, their method models the mesostructure with signed distance and coarse normal direction at each projected vertex. Hence, can model sparse structures at a centimeter scale but not delicate patterns on the object's surface. In contrast, method leverages high-resolution tactile sensing the millimeter scale and provides accurate surface normal details, with richer details at fine scales.
Tactile sensing in 3D. Tactile data is useful in providing contact information and are widely used in 3D perception and robotics tasks. Early works start with reconstructing simple objects [63 64 665 66), and then evolve to more complicated scenes with latest 3D neural representations [67 68 well as
robotic in-hand reconstruction with multi-finger contacts (69 70). Vision- tactile sensing [T0
7I particular sensing mechanism based on the photometric stereo principle, provide high- resolution surface texture information like surface normals and thus can be useful for high-quality 3D synthesis and generation. Several recent works have also used it for 2D generation
[72 73 74] and
3D scene reconstruction [75] (77 76 In contrast, to our knowledge, our work is the first to use tactile
sensing for 3D generation.
Tactile Data Acquisition
to acquire high-resolution geometric details by touching the object's surface. The raw sensor data are then pre-processed to obtain high-frequency signals, as shown Figure 3
We use GelSight [0 11
GelSight tactile sensor. The GelSight sensor is a vision-basee tactile sensor that uses photometric stereo measure geometry at high spatial resolution at the contact surface. It can capture the fine details on the surface, such as bumps on avocados' skin and patterns of crochet yarns. In our work, the GelSight mini with sensing 240x320, equivalent to about 85 micrometers per pixel. The sensor captures an RGB image, which can be mapped to the surface normals and then used reconstruct surface height map.
of 2m x 25mm (h w) and pixel resolution of
Tactile data pre-processing and contact mask. We manually press the GelSight sensor against the object's surface to obtain tactile image over small area. The derived height map contains both a low-frequency component, representing the surface's 's global shape, and high-frequency geometric textures. We first apply high-pass filter the height map to extract texture information and then center-erop the masked region to a square patch to remove any non-contact area. Finally, we convert the masked height map patch back to normal map patch by computing its gradients.
Figure 2 shows our dataset TouchTexture collected from 18 daily objects with diverse tactile textures. Our dataset is available on our project website.
3

---

Height Map f(r
Normal Map 0s af On 8g
11
High-pass Filtering
Sensor Output
GelSight Mini
2D Texture
Depth Estimation
Synthesis
Figure 3: Tactile data capture. We collect one patch by pressing GelSight Mini on an object surface. We use Poisson integration estimate the contact depth from the sensor output, apply high-pass filtering extract the high-frequency texture information, and then the 2D texture synthesis method of Image Quilting 777 obtain an initial texture map. Finally, we convert the height map back normal map.
Visual Matching
Visual Guidance
Contrnhio
L
No
Danclea
Ulill
Miffy busnny with sooven besture
Albedo UV rap
Cenenalton
Render
"ha Muff bunniy Iopuct Teat
L
80 Alylabettddo Iediel
Render
Tactile Matching
Tactile Guidance
Image Culting
Lys
Texturc EreaenElachh
Render
Ingun Tuctile
UV map
Refenence Normal
Randered Narmal
Reelned Normall
Ly
Figure 4: Method overview. Given an imput image or a text prompt, our method generates a mesh with high- quality visual and normal texture. We first generate a base mesh with albedo texture using text- or image-to to 3D method. We use a 3D texture field with hash encoding to represent albedo and tactile normal textures and train it with loss functions we sample distinct camera views, Py visual rendering and for tactile rendering. train the texture field with a visual matching loss LvM. to ensure fidelity to the input mesh, and a visual guidance loss with cormal-conditione" ControlNet, Lva. to enhance photorealism and cross-modal alignment. We further apply tactile matching loss, LIM. and tactile guidance loss, LrG, using customized Texture Dreambooth, achieve high-quality geometric details aligned with the distribution of tactile input V* texture exemplars.
rendered images. To capture the scale differences between visual and tactile modalities,
texture refinement, we
4 Method
Generating 3D assets with high-resoluttin geometric details is challenging due to the difficulty of acquiring low-level texture details from text prompts or a single image. Therefore, we incorporate an additional input modality, tactile normal maps, enhance the geometric details 3D) asset generation. Figure Figure4sshows our overall pipeline. Our method takes an input image and a tactile normal patch as input and generates 3D asset with high-fidelity, color-aligned normal map. The input image be a real image or generated by a text-to-image model such as SDXL [12]. Below, we first describe how we generate the base mesh in Section 4.1 We then introduce our core texture refinement algorithm in Section 4.2 and extend it to objects with multiple materials in Section 4.3
4.1 Base Mesh Generation
Similar to prior works with stage-wise geometry sculpting and texture refinement 22 49
we first generate base mesh with albedo texture using text- image-to-3D method. We then unwrap UV map of the exported mesh M and project the vertex albedo onto an albedo UV map.
A 2D texture transfer baseline. To incorporate the tactile information, one could synthesize large texture map with an exemplar of the preprocessed normal patch using 2D texture synthesis algorithms such as image quilting [77] and use it as normal UV map of the object mesh. However,

---

this would result in inconsistencies between visual and tactile textures, especially when transition in visual color indicates geometric change, as shown in Figure
4.2 Texture Refinement
To address this issue, we jointly optimize the albedo and the normal tactile texture to ensure alignment.
Texture representation. For ease of optimization, represent the visual and tactile normal textures as a 3D texture field. In practice, we use a multi-resolution hash grid (78) 3()), whose input a 3D spatial coordinate p on the mesh:
(p)) = (c, nr),
(I)
where E R3 is the albedo and n, IR3 is the tactile normal defined in the tangent space the mesh surface.
Given the base mesh M, the texture field (()) camera pose P. and lighting condition L, we can render color image L Bxxx3 of the object using differentiable rasterizer R, nvdiffrast [79
(c( RMM, ((),, L),
where H and W represent image height and width, respectively.
Specifically, for each 3D point the mesh we composite the base normal D, from the mesh geometry and the tactile normal ny from the texture to get the shading normal n:
QIEN nr [t, ny t, ng] nr.
where ng is the global geometric normal defined by the base mesh, is the tangent vector, and QTEN Bitangent-Normal (TBN) Matrix for every surface point. Given the calculated shading
is the normal and a sampled camera pose P. we use a point light and simple diffuse shading model to produce the color image (c) following prior works 2
We can also obtain (P).. (P) and (P) by projecting the albedo, tactile normal, and shading normal onto a sample view P.
3
Learning 3D texture field. Given the scale discrepancy between vision and touch, sample camera views differently for two modalities. To supervise visual renderings İ and İ. sample camera poses Pv orbiting the base mesh looking at the object center with perspective projection. To supervise tactile renderings I,, we sample camera poses Pr close to mesh surfaces based on vertex normals to emulate the captured data using a real sensor. Specifically, we randomly sample vertex, place a camera along this vertex's 's normal direction, and set the camera to look at the sampled vertex with orthographic projection.
We initialize the neural texture field of albedo with a reconstruction loss of rendered images:
CvM (.() - ( (P))
(4)
where I, is rendered using the exported albedo UV map in Section 4.1 from the same camera Pv. Similarly, we initialize the tactile normal texture field with a Tactile Matching (TM) loss:
LIm = 1 - cos (1.(P...
(5)
where 1 is rendered using the tactile normal UV map from the image quilting results from a tactile camera pose Py. Here, we use image quilting (77) a patch-based texture synthesis algorithm, to synthesize reference texture map I, that roughly matches the physical scale of real materials and use for initialization.
To refine the visual texture, we use a diffusion guidance loss inspired by the multi-step denoising process in SDEdit [80]. NnstEet [81], and DreamGaussian [49]. Different from existing works, we compute the Visual Guidance loss LvG based on a normal-conditioned ControlNet [2] to ensure the refined visual texture is consistent with the tactile normal. Given the rendered color image le from Pv. we perturb it with random noise e(!) to get a noisy image x, and apply multi-step denoising process (-) using the 2D diffusion prior to obtaining refined image:
Is (Pt) fo v. (Pv)).
(6)
5

---

where is the input text prompt, and fa is a normal-conditioned ControlNet with the Stable Diffusion backbone. We denoise the image from timestep a completely clean image Lg. which different from the typical single-step SDS loss in Dreamfusion [2. The starting timestep / E (0, 1) gradually decreases from 0.5 to 0.3 as training iteration increases, balancing the noise reduction enhance details without disrupting the original content. This refined image is then used to optimize the texture
through the LI and LPIPS [83
loss:
LvG = ((P) - L(R))|, + CLMPP (Ic(P))..(R))
While refining the visual textures, we jointly optimize the tactile texture with Tactile Guidance loss Cre. Similar to Lva. we add random noise to a rendered tactile normal map I, (P)), generated from tactile camera pose PI, and then apply multi-step denoising process to obtain refined normal image Ig. To capture the distribution of tactile normals, we replace the standard diffusion prior with Texture DreamBooth, customized model created by fine-tuning the Stable Diffusion model (t) for each tactile texture using a Low-Rank Adapter (LoRA) (84) with DreamBooth (85). We leave the LoRAs training details in Appendix A.3
We use the output L to refine the tactile texture:
LG 1- - cos (ir(t))
(8)
Optimization. The overall loss function is:
L AvMCVM + ATMLTM + AvaLva + ATaLTo-
To initiate our texture grid, we start by only optimizing the visual matching loss LvM and tactile matching loss LIm for 150 iterations with Avm = 500 and ArM = 1. After that, we run optimization for another 50 iterations to refine the output guided by diffusion priors. We reduce \TM from 0.05, change Lvvr from per-pixel error to mean-color error to allow more flexibility in texture refinement, and add the visual guidance loss Lva and tactile guidance loss Lra with Ava = and Avo = 0.05.
4.3 Multi-Part Textures
Another advantage of our 3D texture field is its ability to easily define non-uniform textures in 3D, allowing different tactile textures to be assigned to distinct parts of an object. For instance, when generating 3D asset of "a cactus by incorporating two tactile inputs along with text prompt that specifies the texture assignment, e.g.. "cactus with texture A, pot with texture B".
pot", we can apply different textures to the cactus and the pot,
Part segmentation based on diffusion features. Our method can automatically segment object parts based on the text prompt without manual annotation. We leverage the internal attention maps of diffusion models to segment the rendered views (Pv) of the object. Specifically, we add a random noise of level to I, and apply one denoising step with the SD UNet. Inspired by DiffSeg [86) and other text-to-image methods [87 [88, we perform unsupervised part segmentation by aggregating and spatially clustering self-attention maps from multiple layers. To assign part labels, we aggregate cross-attention maps corresponding to different parts and match them with the unlabeled segmentation maps based on Bullback-Leible (KL) divergence. This results in a list of labeled segmentation masks M E {0. 1} HxW n E fI....,.N] .N. where n denotes one of the N object parts. For example, given the prompt "cactus pot," we aggregate cross-attention maps for "cactus" and "pot" from different layers, then assign each part segmented from the clustered self-attention maps to either cactus or pot according respective cross-attention maps. More details are included in Appendix A.1
3D part label field. Since the 2D part segmentations may contain noise and lack multi-view consistency, we integrate them from different viewpoints into a coherent 3D part label field. We achieve this by incorporating part label RNN into our texture field ((-). To supervise it, render part label maps ((P)) from sampled camera poses Py and compute the cross entropy loss:
L. = Len (S(P))
(10)
where S(Pv) is the part logits and ((L) E {0, 1) 10 HXWXN are labels concatenated from M

---

Input
corn
an American football
Generated
Albedo
Generated
Normal
Rendered Full Color
3D generation with a single texture. For each object, we show generated albedo (top), normal (middle), and full color (bottom) renderings from two viewpoints, Our method works for both text-to-3D (com and football) and image-to- 3D (potato and strawberry), generating realistic and coherent visual textures and geometric details. (We use roughness=0.5 when rendering color views Blender for Figures
Figure 5:
PoART Shape
carfeisspe
scalpture
straderry
coffee cup with texture
Figure Diverse textures with the same object. With additional texture cues from tactile data, we can synthesize diverse textures with the same coarse shape for customized designs.
Optimization with multi-part textures. To learn multi part textures, keep the same Lvs and Lvg loss functions while modifying LIM and Lıe to incorporate part-specific tactile supervision:
CIM Σ
cos (M
c 1, M o r:)
(11)
LiG Σ cos (Mr o M. M . or
(12)
where O denotes the Hadamard product, 1 is the rendered reference view using the n-th part's tactile data, I is the refined tactile normal generated by Texture Dreambooth trained on the n-th part texture, and M is the binary mask for the n-th part obtained from Š rendered from our learned 3D label field. We omit P, for clarity since all patches are rendered from the same tactile camera pose.
Experiment
We present comprehensive experiments to verify the efficacy of our method. We perform qualitative and quantitative comparisons with existing baselines and ablation studies on the major components.
Dataset. We have collected 18 diverse types of textures from daily objects in our Texcurexture dataset, five tactile patches per texture, and pair them with short description. Please see Appendix for the full list. We randomly sample one patch to initialize the tactile UV map with the image quilting algorithm (77) and use five patches to train the customized Texture DreamBooth, For base mesh generation given text image input, while any mesh generation method is applicable,
7

---

Input
Predicted Labels Generated Albedo
Generated Normal
Rendered Full Color
"a cactus
a pat"
"a cactus in pot, cactus with orange texture, pot with heat restatent rubber with heart-shaps pattern texture"
"s with whibe whibe lampskadic and woodest base, tampshade with canvas bag's bedbure, base with metal gridl bexture"
"A goot sculpture with gold goat and black base, goot fine striped steel texbure, base with smooth wood bexture"
Figure 7: Multi-part texture generation. Our method allows users to specify an object (via text or image) and its two parts assign different textures (color-coded text prompts correspond to text descriptions for two textures). We show paired results for the predicted label, albedo, normal, and full-color renderings. The zoom-in patches demonstrate the generated normal textures on different parts.
we use Wonder3D 13 in the main experiments but include results using InstantMesh 89 and RichDreamer 990 in Appendix |A.4 as well. Specifically, Wonder3D takes an image as input and outputs mesh with albedo stored as vertex colors. We then convert the vertex albedo into albedo UV map as described in Section 4.1 During optimization in Section (42) we refine the albedo and tactile textures with the "textured prompts", i.e., A [object name] with V Iexexurre, where V represents the text description corresponding to the selected tactile texture map.
3D generation with a single texture. Figure shows our 3D generation results for a single texture, showing albedo, normal, and full-color rendering from two viewpoints for each object. Our method works for both text input (the corn and the football) and image input (the potato and the strawberry). generating realistic, coherent, and high-resolution visual and geometry details. results of applying different textures to the same object (a coffee cup), with normal rendering on top and color rendering below. Our joint optimization produces coherent visual-tactile textures and smooth, natural shadows from geometric variations.
the
Multi-part texture generation. As introduced in Section 4.3 users can specify an object (via text or image) and assign different textures to two distinct parts. Figure 7 shows results for multi-part texture synthesis. The left column shows the text or image input as well as the tactile input. The image input can be either real or generated from a text prompt. The color-coded text prompts correspond to text descriptions for two textures. The right columns show the results of the albedo, normal, and full-color rendering. Our method effectively segments parts with our 3D label field. The joint optimization successfully applies the textures to each corresponding part designated by the user and generates an overall coherent visual appearance and tactile geometry.
Baselines To our knowledge, this work is the first to leverage high-resolution tactile sensing to enhance geometric details for 3D generation tasks. Thus, we compare our method with existing text-to-3D and image-to-3D methods. For text-to-3D, we compare with DreamCraft3D (22) and use the same textured prompt, i.c., a prompt with a text description of the target tactile texture, as input. DreamCraft3D focuses on geometry sculpting using neural field and DMTet (9]) followed by texture boosting through fine-tuning DreamBooth with augmented multi-view rendering, requiring 10x computational cost compared our method. For image-to-30 we compare with DreamGaussian [49] and Wonder3D [3] DreamGaussian employs fast generation using 3D Gaussian primitives and refines an albedo map Stage with a multi-step denoising MSE loss. Wonder3D generates paired multi-view RGB and normal images, using a geometry fusion process to generate a 3D result. The input images remain the same as ours for these two baselines. We use the official implementations for all baselines.
8

---

Tactile Input
s strawberry
avocado
Ours
Wo
Dream Gauasian
Figure 8: Bascline comparison. Compared to SOTA image-to-3D (Wonder3D DreamGaussian and text-to-3D (DreamCraft3D) baselines, our method produces significantly more plausible low-level geometry. For a fair comparison, we use the same input image for the first three rows.
Table I: Human perceptual study. For all paired comparisons, method preferred (2 50%) the baselines both texture appearance and geometric details.
Ours DreamGaussian Ours vs Wonder3D
Ours vs DreamCraft3D
Texture Geometry
85.43 + 4.76
86.36 + 3.93
61.54 + 4.43
92.85 + 3.47
88.07 + 3.80
84.20 + 4.17
Qualitative evaluation. Figure g shows qualitative results of method compared against three baselines. For each example, we show color and normal rendering with zoomed-in patches at the same location for detailed comparison. Our method achieves higher visual fidelity, more realistic details in normal space, and better color-geometry alignment than the baselines. In particular, our textures exhibit sharper details than those of Wonder3D, which tend to be overly smooth. In contrast to DreamGaussian and DreamCraft3D, our generated geometry details are more realistic and align well with the color appearance. In the avocado example, DreamCraft3D suffers from the "Janus problem", generating brown core on both sides.
Quantitative evaluation. We perform human perceptual study using Amazon Mechanical Turk (AMTurk). We set up a paired test, showing a reference prompt and two rendering results, one generated with method and the other generated with one of the baselines. We conduct two separate surveys to evaluate the texture appearance and geometric details, respectively. For texture appearance, render full-color RGB images and ask users "Which of the following views has more realistic textures?". For geometric details, we render shaded colorless images that only demonstrate the mesh geometry and ask users "Which of the following views has Please see Appendix A.5 for example screenshots the paired rendering. Each user has two practice rounds followed by 20 test rounds evaluate method against DreamGaussian, Wonder3D, and DreamCraft3D. All samples are randomly selected and permuted, and we collect 1, 1,000 responses.
realistic geometric details?".
Table shows the mean and standard deviation of users' preference score; our method is preferred over all baselines. While DreamCraft3D has a close performance regarding texture appearance, fails to obtain high-resolution geometric details that align well with color texture.
Ablation study. We perform an ablation study evaluate key aspects of our method. We render both front and back views of sampled object for each experiment. To illustrate details and cross- modal alignment, we present global full-color view on the left, global normal view on the right, and patch views of the full-color, albedo, and normal renderings in between. Figure shows results of ablating diffusion-base guidance losses for texture refinement, specifically the visual guidance Lvg and tactile guidance LTG.

---

Curs
Input
strandecry
Oura wio
Visial guscnnce
Figure 9: Ablation study regarding texture refinement. We ablate our method regarding the tactile guidance loss Lra and visual guidance loss Lva. Removing Lro results in fewer details of the generated tactile texture. Removing the visual guidance introduces misaligned visual and tactile normal textures. For example, bumps the normal map are misaligned with the locations white seeds in the albedo rendering, as shown in the zoomed- in patches. Our method encourages the generated color to be consistent with the tactile normal using ControlNet-gun guided visual refinement loss while also enhancing the details in the tactile texture.
Toctle
Impal
heat-resssstee
glove KESA
Jocill
processing
Figure 10: Ablation study regarding tactile preprocessing. Without tactile data preprocessing including high-pass filtering and contact area cropping, the generated geometric details tend to be flat and unrealistic.
Tadlie Input veven texture
bact ingill
Figure 11: Ablation study regarding tactile input. We remove the tactile input while keeping the refinement loss. Without the tactile information, the generated 3D assets fail capture fine- mne-grained geometric details.
Omitting LTG reduces tactile texture details, while omitting LvG introduces misalignment between visual and tactile normal textures; for instance, bumps in the normal map do not match the white seeds in the albedo rendering. Our method ensures color consistency with tactile normals via ControlNet- guided visual refinement and enhances tactile texture details.
We also study the efficacy of tactile data processing by comparing our method with the texture map synthesized using the original tactile data without preprocessing stated in Section 3 Figure TO shows that using the original data produces much more flattened textures since the low-frequency deformation of the gel pad due to uneven contact during the data collection would dominate the tactile signal and thus degrade the details of synthesized texture maps. Figure [] shows the results of removing tactile input and only optimizing the albedo map with "textured prompts" using LyM and Lvg. Without tactile input, the output mesh is overly smooth, demonstrating the insufficiency of using text prompts only for fine geometry texture generation.
Discussion
Recent methods in 3D generation often struggle with unrealistic geometric details. To address this, we have introduced a novel approach that incorporates tactile information. Our method synthesizes both visual and tactile textures using a 3D texture field. Additionally, we have introduced multi-part texturing pipeline for controllable region-wise texture generation. To our knowledge, this is the first use of high-resolution tactile sensing to improve 3D generation. Our method produces realistic, fine-grained geometric textures while maintaining accurate visual-tactile alignment.
Limitations. The quality of the generated coarse shape depends on the existing 3D generative models, which struggle with complex geometry. Additionally, slight seams may appear in our results due to UV unwrapping.
10

---

Acknowledgments and Disclosure of Funding
We thank Sheng-Yu Yu Wang, Nupur Kumari, Gaurav Parmar, Hung-Jui Huang, and Maxwell Jones for their helpful comments and discussion. We are also grateful to Arpit Agrawal and Sean Liu for proofreading the draft. The project is partially supported by the Amazon Faculty Research Award, Cisco Research, and the Packard Fellowship. Kangle Deng is supported by the Microsoft Research PhD Fellowship. Ruihan Gao is supported by the A * National Science Scholarship (Ph.D.).
References
[1] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In IEEE International Conference on Computer Vision (ICCV), 2023.
[2] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR), 2023.
[3] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Liu, and Tsung-Y. Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Compuder Vision and Puttern Recognition (CVPR), 2023.
[4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Conference on Neural Information Processing Systems (NeurIPS), 2014.
[5] Jascha Sohl- Dickstein, Eric Weiss. Niru Maheswaranathain, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning (ICML). 2015.
[6] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[7] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4):1-14, 2023.
[8] Christoph Schuhmann, Richard Veneu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-texi pairs. arXiv preprint arXiv:2111.021
2021.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[10] Wenzhen Yuan, Siyuan Dong, and Edward H Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 2017.
[11] Shaoxiong Wang, Yu She, Branden Romero, and Edward Adelson. Gelsight wedge: resolution 3d contact geometry with compact robot finger. In IEEE International Conference on Robotics and Automation (ICRA), 2021.
Mcasuring high-
[12] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.
[13] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingji Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[14] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[15] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar 2rXiv:2204.06
1(2):3, 2022.
[16] Minguk Kang. Jun-Ya Yan Zhu, Richard Zhang, Jaesik Park. Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[17] Jiahui Yu. Yuanzhong Xu. Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. In International Conference Machine Learning (ICML), 2022.
11

---

[18] Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In IEEE International Conference on Computer Vision (ICCV). 2023.
[19] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In IEEE International Conferencee on Computer Vision (ICCV), 2023.
[20] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Conference on Neural Information Processing Systems (NewrIPS), 2024.
[22] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreameraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. In International Conference on Learning Representations (ICLR), 2024.
[23] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[24] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.00 1.w.2332.0020,,220 2023.
[25] Yiehun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In International Conference on Learning Representations (ICLR), 2024.
[26] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
[27] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view viewcconditioned diffusion for 3d recon- struction. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[28] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+++ Fast single image 3d objects with consistent multi-view generation and 3d diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[29] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image consistent multi-view diffusion base model. arXiv preprint arXiv:2310. 10, 2023.
[30] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syne- dreamer: Generating multiview-consistent images from a single-view image. In International Conference on Learning Representattins (ICLR), 2024.
[31] Vicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lmm: Large reconstruction model for single image to 3d. In International Conference on Learning Reprexentations (ICLR), 2024.
[32] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse- view generation and large recon- struction model. In International Conference on Learning Representations (ICIR), 2024.
[33] Yinghao Xu, Hao Tan, Fujun Luan. Sai Bi, Peng Wang, Jiahao Li. Zifan Shi, Kalyan Sunkavalli, Gor- don Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
[34] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang. Adam Letts, Yangguang Li, Ding Liang, Christian Laforte. Varun Jampani, and Yan-Pe Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.0211 2024.
[35] Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. In Conference Neural Information Processing Systems (NeurIPS), 2018
[36] Xiaogang Wang, Marcelo H Ang, and Gim Hee Lee. Voxel-based network for shape completion by leveraging edge generation. In IEEE International Conference on Computer Vision (ICCV), 2021.
[37] Pedro O O Pinheiro, Joshua Rackers, Joseph Kleinhenz, Michael Maser, Omar Mahmood, Andrew Watkins, Stephen Ra, Vishnu Sresht, and Saced Saremi. 3d molecule generation by denoising voxel grids. 2024.
[38] Yongbin Sun, Yue Wang. Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autoregressively learned point cloud generation with self attention. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020.
12

---

[39] Zijie Wu, Yaonan Wang, Mingtao Feng, He Xie, and Ajmal Mian. Sketch and text guided diffusion model for colored point cloud generation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[40] Yoni Kasten, Ohad Rahamim, and Gal Chechik. Point cloud completion with pretrained text-to- text-to-image diffusion models. In Conference on Neural Information Processing Systems (NewrIPS), 2024.
[41] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky. and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. ACM SIGGRAPH Asia, 2022.
[42] Yiwei Ma, Xiaoqing Zhang, Xiaoshuai Sun, Jiayi Ji. Haowei Wang. Guannan Jiang. Weilin Zhuang, and Rongrong Ji. X-mesh: Towards fast and accurate text-driven 3d stylization via dynamic textual guidance. In IEEE International Conference on Computer Vision (ICCV), 2023.
[43] Paul Henderson. Vagia Tsiminaki, and Christoph H Lampert. Leveraging 2d data to learn textured 3d mesh generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[44] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Gen- erating 3d mesh models from single rgb images. In European Conference on Computer Vision (ECCV), 2018.
[45] Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2msh+++: Multi-view 3d mesh generation via deformation. In IEEE International Conference on Computer Vision (ICCV), 2019.
[46] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[47] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran 7ada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. In IEEE International Conference on Computer Vision (ICCV), 2023.
[48] Yen-Chi Cheng, Hsin- Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[49] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient content creation. International Conference on Learning Representations (ICLR). 2024.
[50] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang. Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision (ECCV), 2024.
[51] An-Chich Cheng, Xueting Li, Sifci Liu, and Xiaolong Wang. Tuvf: Learning generalizable texture uv radiance fields. In International Conference on Learning Representations (ICLR), 2024.
[52] Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. Neutex: Neural texture mapping for volumetric neural rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[53] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nicßner. Text2tex: Text-driven texture synthesis via diffusion models. In IEEE International Conference on Computer Vision 2023.
[54] Yawar Siddiqui, Justus Thies, Fangchang Ma. Qi Shan, Matthias Nießner, and Angela Dai. Texturify: Generating textures on 3d shape surfaces. In European Conference on Computer Vision (ECCV), 2022.
[55] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. texturing of 3d shapes. In ACM SIGGRAPH, 2023.
Texture: Text-guided
[56] Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun-Yan Zhu, Tinghui Zhou, and Mancesh Agrawala. Flashtex: Fast relightable mesh texturing with lightcontrolnct. In European Conference on Computer Vision (ECCV), 2024.
[57] Aymen Mir, Thiemo Alldieck, and Gerard Pons-Moll. Learning transfer texture from clothing images 3d humans. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[58] Tuanfeng Y Wang, Hao Su, Qixing Huang, Jingwei Huang, Leonidas J Guibas, and Niloy J Mitra. Unsupervised texture transfer from images model collections. ACM Transactions Graphics (TOG), 2016.
[59] Anita Hu, Nishkrit Desai, Hassan Abu Alhaija, Seung Wook Kim, and Maria Shugrina. Diffusion texture painting. In ACM SIGGRAPH. 2024.
[60] Hendrik Baatz, Jonathan Granskog. Marios Papas, Fabrice Rousselle, and Jan Novák. Nerf-tex: Neural reflectance field textures. In Eurographics Symposium on Rendering (EGSR), 2021.
[61] Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M. Seitz. Photoshape: Photorealistic materials for large-scale shape collections. ACM Transactions on Graphics (TOG),
13

---

[62] Yi-Hua Huang, Yan-Pei Cao. Yu-Kun Lai, Ying Shan, and Lin Gao. Nerf-texture: Texture synthesis with neural radiance fields. In ACM SIGGRAPH, 2023.
[63] Sudharshan Suresh, Zilin Si, Joshua G Mangelson, Wenzhen Yuan, and Michael Kaess, Shapemap 3-d: Efficient shape mapping through dense touch and vision. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[64] Mohamed Tahoun, Omar Tahri, Juan Antonio Corrales Ramón, and Youcef Mezouar. Visual-tactile fusion for 3d objects reconstruction from single depth view and single gripper touch for robotics tasks. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
[65] Edward Smith, David Meger, Luis Pineda, Roberto Calandra, Jitendra Malik, Adriana Romero Soriano, and Michal Drozdzal. Active 3d shape reconstruction from vision and touch. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[66] Shaoxiong Wang. Jiajun Wu, Xingyuan Sun, Wenzhen Yuan, William T Freeman, Joshua B Tenenbaum, and Edward H Adelson. shape perception from monocular vision, touch, and shape priors. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
[67] Aiden Swann, Matthew Strong, Won Kyung Do, Gadiel Sznaier Camps, Mac Schwager, and Monroe Kennedy III. Touch-gs: Visual-tactile supervised 3d gaussian splatting. In JEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
[68] Mauro Comi, Yijiong Lin, Alex Church, Alessio Tonioni, Laurence Aitchison, and Nathan F Lepora. Touchsdf: A deepsdf approach for 3d shape reconstruction using vision-basse tactile sensing. In Conference on Neural Information Processing Systems (NeurlPS) Workshop, 2023.
[69] Edward Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, and Michal Drozdzal. 3d shape reconstruction from vision and touch. In Conference on Neural Information P Processing Systems (NeurIPS), 2020.
[70] Sudharshan Suresh, Haozhi Qi. Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, et al. Neuralfeels with neural fields: Visuotactile perception for in-hand manipulation. Science Robotics, 996)eaad00628, 2024.
[71] Ellion Donlon, Siyuan Dong, Melody Liu, Jianhua Li, Edward Adelson, and Alberto Rodriguez. Gelslim:
A high-resolution, compact, robust, and calibrated tactile-sensing finger. In JEEE/RS International Conference on Intelligent Roboty and Systems (IROS), 2018.
[72] Yunzhu Li, Jun Yan Zhu, Russ Tedrake, and Antonio Torralba. Connecting touch and vision via cross-modal prediction. In IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
[73] Ruihan Gao, Wenzhen Yuan, and Jun- Yan Zhu. Controllable visual-tactile synthesis. In IEFE International Conference on Computer Vision (ICCV), 2023.
[74] Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, et al. Binding touch to everything: Learning unified multimodal tactile representations. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[75] Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, and Andrew Owens. Tactile-augmented radiance fields. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[76] Mauro Comi, Alessio Tonioni, Max Yang. Jonathan Tremblay, Valts Blukis, Yijiong Lin, Nathan Lepora. and Laurence Aitchison. Snap-it, tap-it, splat-it: Pactile-informee 3d gaussian splatting for reconstructing challenging surfaces. arXiv preprint arXiv:2403.2022 2024.
[77] Alexci A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. In ACM SIGGRAPH. Association for Computing Machinery, 2001.
[78] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM Transactions on Graphics (TOG), 41(4):1-15, 2022.
[79] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (TOG), 39(6), 2020.
[80] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic diffcrential cquations. In International Conference on Learning Representations (ICLR), 2022.
[81] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerf2nerf: Editing 3d scenes with instructions. In IEEE International Conference on Computer Vision (ICCV), 2023.
[82] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, Adding conditional control to text-to-image diffusion models, 2023.
14

---

[83]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable elfectiveness of deep features as perceptual metric. In CVPR, 2018.
[84] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[85]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Ktir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[86] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez- Franco, Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[87] Songwei Ge, Taesung Park, Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. In IEEE International Conference on Computer Vision (ICCV), 2023.
[88] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object- level shape variations with texl-to-image diffusion models. IEEE International Conference on Computer Vision (ICCV), 2023.
[89] Jiale Xu, Weihao Cheng. Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient mesh generation from single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024.
[90] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: generalizable normal-depth diffusion model for detail richness text-to-3d. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[91] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yos Yu Liu, and Sanja Fidler. hybrid representation for high-resolution 3d shape synthesis. Conference on Neural Information Processing Systems (NeurlPS), 2021.
Deep marching tetrahedra:
[92] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
15

---

Appendix
A Implementation Details and Additional Results
Please check out our webpage for video results and more examples.
A.1 Diffusion-based Multi-Part Segmentation
In Section wc have mentioned using attention maps of the diffusion process to segment the input image based on text prompts. We provide more details here. Assume input image and its corresponding prompt y = [y,, 3/2 "*, yr are given, where T is the number of tokens from the text encoder (T 77 for CLIP text encoder [92)). Among these, Vp: ..., Upy define N parts in the image x, where Pı .. . PN indicate the indexes of the token. For example, with the prompt "a cactus in pot", the second token y/2 (pi 2) "cactus" and the fifth token ys (12 5) "pot" correspond to two parts (N = 2) of the object in the image, and our method outputs the segmentation mask for each part. Specifically, we first perturb x using random noise e, with noise level of 4, which is empirically set as 0.2. We then use the Stable Diffusion UNet to denoise I; for one step. During the denoising process, we collect the aoossant and self-attention probability maps for each of the 16 layers.
Segment and create masks using self-attention layers. Following DiffSeg [6], we aggregate the 16 self-attention probability maps single map A, with the shape of 64 64 64 x 64, and run iterative attention merging [86] on upsampled A, to obtain a preliminary cluster probability map Ay E 2512x512xK where each iy
1, ... K, is probability map,
Σ A = k 1,2, K.
(13)
We can then obtain preliminary segmentation mask 3,
Vi,j,
S(i.j] arg max A [..
(14)
Aggregate labels with cross-attention layers. Similarly, we aggregate the 16 cross-attention maps and up-sample them to Ar with the shape of 512 512 x 77. To find the exact pixels corresponding to each part specified in the prompt input, we extract cross-attention maps associated with the N tokens 1pp."" , A. E 1512x512xN
Acli. Pm.
Vijin
Aelijiml
(15)
Ee,y A. ,',
Silimar to the self-attention probability maps Ay, here normalize the cross-attention maps over the spatial dimensions ensure Acl n] is probability map for each n, where Σx, Acli. j. 1.
Associate masks with labels. Note that contains K parts but in general K * N. Therefore, we need assign each of the K segments to one of the N parts specified in the prompt.
We calculate KL divergence matrix D IR KNN between Ay and Ac.
Dik n] DRa. (A, A.lm)). k= 1,2,..., K, n= 1,,,, 1.2,...,NN ..
(16)
We merge the preliminary mask S according to D to get the final segmentation S,
Vi,J, Sli.1l = arg min D[S(,,)),n)
(17)
An example of output masks is shown in Figure 12
A.2
Dataset Details
Figure 13 shows additional textures included in our TeuchTexture dataset, and Table 2 shows the list of text descriptions corresponding to each texture.
16

---

Reference Albedo
Mask A: cactus
Mask B: pot
Attention Aggregation
Diffusion UNet
Mask Merging
cactus in pot
Figure 12: Illustration of diffusion-based multi-parts segmentation. At cach iteration, we run forward pass of the diffusion model for the target albedo image, aggregate its self- and cross- attention maps, and compute distance to merge the masks N parts. show an example we extract masks corresponding two tokens "cactus" and "pot", shown as Mask A and Mask The segmentation masks are then used to supervise the label field training to enable multi-part synthesis.
cactus a pot", where
Object
Normal Map
3D Height Map
woven
orange
cantaloupe
potato
football
g glove
Object
Normal Map
3D Height Map
chopping
table tennis handle
table cloth
cleaning cloth
metal grid
cork
Figure 13: Additional materials in tactile normal dataset TouchTexture. We collect tactile normal data from 18 daily objects featuring diverse tactile textures. We show the tactile normal map and a 3D height map for each object. This library of factile samples covers wide range of diverse materials commonly found in daily life.
A.3 Training Details
Regarding the network, we use TCNN encoding with two base linear layers, which then branches out into three output layers, one linear layer followed by sigmoid activation for albedo output, one linear layer followed by tanh activation for tactile normal output, and one optional linear layer for label field. We follow DreamBooth [55 to train texture LoRAs with Stable Diffusion (SD) V1.4 for tactile guidance loss. We find empirically that SD V1.4 works better for generating tactile normal maps. We use ControlNet (v1.1 normalbac version) with SD V1.5 for the diffusion loss. We follow Wonder3D to generate the base mesh and train the texture field network using Adam optimizer with Ir = 0.01. We train all models on A6000 GPUs and each experiment takes about 10 mins and 20G RAM to run.
A.4 Flexible Base Mesh Generation
Given a text or image input, our method seamlessly integrates with wide range of mesh gener- ation approaches to enhance the 3D output with refined details. Here, we presents results using RichDreamer [90] for base mesh generation in Figure [4a] and results using InstantMesh [89 in Figure 14b] RichDreamer generates detailed and consistent 3D meshes from text prompts by incor- porating depth and normal priors during pre-training, while InstantMesh employs feed-forward image-to-3D approach with a transformer-based reconstruction model and FlexiCubes for efficient mesh production. Both methods provide solid base meshes, and our approach effectively adapts to each, refining the output meshes with intricate texture details for greater realism and fidelity.
17

---

Table List of objects in TouchTexture and their text descriptions.
Object Name
Description
avocado
"avocado skin's"
strawberry
"strawberry"
canvas bag
"canvas bag"
striped steel
"fine striped steel"
corn
"corn"
rubber
"heat-resistant rubber with heart-shape pattern"
woven
"woven crochet"
orange
"orange"
cantaloupe
"cantaloupe skin's"
potato
"potato"
football
"football"
glove
"thin rubber with grid pattern"
chopping board
'cutting board"
table tennis handle
"smooth wood"
table cloth
patteerned table cloth"
metal grid
"metal grid
cork cleaning cloth
"cork mat"
"cleaning cloth's"
A.5 Human Perceptual Study Details
We perform human perceptual study on Amazon Mechanical Turk (AMTurk). We set up paired test, showing a reference prompt and rendering results, one generated with our method and the other generated with one of the baselines. For each result, we include single view of the entire object with zoom-in patch in the red box show the details. We use directional light shooting in from the left side for all mesh renderings. We give text prompt, e.g., "an avocado", and ask the user select the result that better matches the prompt. We render shaded colorless images for geometry evaluation as shown in Figure 15a and render full-color images for texture evaluation as shown in Figure 15b. The estimated time for each test is about minutes, and we pay users a corresponding amount of compensation higher than the minimum wage in the country of the data collector. The online survey does not pose any explicit potential risks expected to be incurred by participants.
B Soceital Impacts
Our method of incorporating tactile information into 3D generation enhances the geometric details of the generated results. This advancement helps automatically create high-fidelity assets, which are highly applicable in game and film production. However, there is also potential risk of misuse, as 3D assets could used to generate fake content for misinformation. Despite this concern, we believe humans can currently distinguish our synthesized objects from real ones.
18

---

wooden desk and chair from an elementary school
statue angel,
Input
3d asset
Albedo
Normal
Full Color
Albedo
Normal
Full Color
Rich
Dreamer
Rich
Dreamer
+ Ours
(a) integrate RichDreamer, purely text-to-3D baseline, into our method.
Input
Albedo
Normal
Full Color
Albedo
Normal
Full Color
Instant Mesh
Instant Mesh + Ours
(b) We integrate InstantMesh, an image-to-3D haseline, into our method
Figure 14: Our method is flexible and compatible with diverse text-to-3D and image-to-3D pipelines.
fslowing viewe movemore ralkiic giomuty
Whec of the kolowing sloso have moo voledo tonuree?
Velhos
Mithodd U
(a) sample test page for geometry evaluation.
(b) sample test page texture evaluation.
Figure 15: AMTurk setup for evaluation. For each paired result, we include single view of the entire object with zoom-in patch in the red box to show the details. For each result, include single view of the entire object with zoom-in patch in the red box show the details. We use directional light shooting in from the left side mesh renderings. We give text prompt, e.g., avocado". and ask user select result that hetter matches the prompt.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we include the paper's contribution and scope in the abstract and introduc- tion section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss the limitations of the work in the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: Yes]
Justification: We describe the network architecture and implementation details the paper. We will also release code and dataset upon publication.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We describe the implementation details in the paper to ensure reprodueibility. The code and data will be released on project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/Cooee Satpess/ooodSummiiiiiiooooo.oo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
wwtpiiiigguddeeeCCCeooe /ooeeee/eooodebmbiiiaoe ssiocoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide essential experiment settings in the main paper and provide additional details and parameters in the supplementary document.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide error bars for all quantitative evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the computational requirement, memory usage, and time execu- tion in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, our research conforms well with the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss societal impact the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release data or models that have a high risk of misuse and thus pose no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the original paper/code package/dataset to our best efforts.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.ppaeesstt paperswi thcode. cww/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. The code and data will be released upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We use AMTurk for user study and provide required details in the supplemen- tary material.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: Yes. provide related details in the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "QixzpmdJUC", "reviewer": "Reviewer_zBVq", "summary": "This paper proposes a method for generating 3D assets with detailed geometry through inputs from a tactile sensor. More specifically, given a bump-map as input from a tactile sensor (just a small patch is enough), the method uses it as regularization while maximizing the likelihood using a normals-conditioned stable diffusion model. The albedo is also optimized alongside the normals, though limited results are shown. Additionally, using diffusion based segmentation maps, different parts of the image can be given different textures. While each component of the method itself is not too novel, the sum total is and the results are pretty good.", "questions": "Just reiterating what I already mentioned in the weaknesses: It would be great if the authors could provide the results of an experiment where the object has two very different textures but those textures are only learnt through the diffusion prior. For example, in the cactus pot case, this would correspond to optimizing L_{tactile} only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa). \n\nI believe this experiment would give us insights on how the diffusion prior itself would perform if a part of the image already has detailed normals (via L_{tactile})", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This is a novel and unexplored task and the proposed method provides the community with a good baseline to build upon.\n\n2) The results of the method are quite convincing. The ability to edit only certain parts of the image with desired textures is particularly nice.\n\n3) The paper is well written with each component explained pretty well.", "weaknesses": "1) There are no results on the albedo provided, it would be great if the authors could explain why they were omitted. I strongly urge them to include it in the rebuttal. \n\n2) I may be mistaken, but it seems this method works only for repetitive textures (though the diffusion model is able to change it through optimization). It would be great if the authors could provide the results of an experiment where the object has two very different textures but those textures are only learnt through the diffusion prior. For example, in the cactus pot case, this would correspond to optimizing L_{tactile} only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging and insightful comments. We’re pleased that you recognize our setup as “a novel and unexplored task.” Below, we address the individual comments.\n\n### **Add albedo rendering results**\nDue to space limit, we omitted the albedo renderings in the main paper. Thanks for pointing it out and we agree that adding albedo renderings helps to demonstrate the quality more clearly. In the Figure 1 of the attached pdf, we show two examples and will add full results in the revision. Notably, our albedo rendering looks natural and contains minimal baked-in lighting effects or geometric details. \n\n### **Show experiments of textures learned from the diffusion priors, i.e., optimizing $L_{tactile}$ only for the pot and let the diffusion prior decide what the normals for the cactus must look like (or vice versa)**\nThanks for your insightful suggestions. We add our experiment results in attached pdf’s Figure 2  for the example “cactus in the pot”, where we keep the text prompt and tactile input unchanged Figure 2(a) has no $L_{tactile}$ for either part; Figure 2(b) and (c) have $L_{tactile}$ for one part, “pot” and “cactus” respectively; Figure 2(d) has $L_{tactile}$ for both parts. \n\nAs shown in Figure 2, (b) and (c) contain clear texture for the part with $L_{tactile}$ and synthesize more details for the other parts compared to (a). However, without clear reference, the inferred texture lacks detailed patterns compared to the full results in (d). In short, the texture generated using only diffusion priors appears plausible but tends to look flatter and less detailed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvoTAN1XIz", "reviewer": "Reviewer_MUQX", "summary": "This submission addresses the long-standing challenge of enhancing geometric details in results produced by text-to-3D and image-to-3D pipelines. The approach introduces a novel method that leverages tactile normal modality to synthesize high-fidelity geometric details. Additionally, it employs attention maps during the diffusion process to segment input images based on text prompts, allowing for the synthesis of multiple textures across various regions. The results demonstrate that this method effectively recovers geometric details and ensures alignment between geometry and color.", "questions": "1. I am curious about your criteria for selecting 3D generation baselines. For image-to-3D tasks, to my knowledge, more advanced baselines such as 'InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models' and 'TripoSR: Fast 3D Object Reconstruction from a Single Image' offer superior quality and may yield stronger results. For text-to-3D tasks, 'RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D' is a noteworthy pure text-to-mesh method. \n\n2. The proposed method in the paper first refines normals and then optimizes colors based on the refined normals. Does this imply that the geometric details introduced by your method are not derived from the original colors (i.e. not very necessary to original colors) of the objects? Additionally, could the color optimization process potentially disrupt the alignment between the original colors and the refined colors, potentially failing to meet the initial requirements?\n\n3. I am curious about the generalization and diversity capabilities of the proposed method. Can it handle more complex objects? Furthermore, with more powerful baselines, can this method achieve higher quality results on more complex cases?", "limitations": "The authors acknowledge their limitations in the main paper and address potential social impacts in the supplementary materials. Regarding the first limitation, the implementation of new 3D generative models, such as 'Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image' and 'Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention', is recommended. For the second limitation, utilizing a more powerful computer graphics tool could be beneficial. Concerning the social impact, issues related to deepfakes and the potential for misinformation are noteworthy. The authors assert that humans can currently distinguish their synthesized objects from real ones, a claim with which I concur. Although it is of low priority, it would be preferable for the authors to include a comparison between a generated object and a real one; a single case would suffice.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The approach is innovative in utilizing tactile normal modality to enhance geometric details.\n2. The introduction of a newly collected tactile dataset, TouchTexture, is beneficial to the research community.", "weaknesses": "1. The paper claims compatibility of the proposed method with both text-to-3D and image-to-3D pipelines. However, DreamCraft3D, selected as a text-to-3D pipeline, requires both an image and a text caption as inputs. Although DreamCraft3D can be considered as a 'text-to-image, text & image-to-3D' process, it differs from a purely text-to-3D approach. Thus, the compatibility between the proposed method and purely text-to-3D pipelines remains questionable.\n\n2. The pipeline overview in Figure 4 indicates the need for a reference image and tactile input. The paper does not address how to select an appropriate tactile input for the reference image, nor how to ensure the tactile details are compatible with the object in the image.\n\n3. The paper presents an intriguing text-guided segmentation strategy that leverages attention maps during the diffusion process based on text prompts. However, lacking expertise in diffusion-based segmentation, I am unfamiliar with the efficiency and success rate of this method, but I am positively impressed by it.\n\n4. The generalization and diversity of the proposed method are also of concern. The objects presented in the results are not very complex, and the dataset comprises 16 popular categories. It remains unclear how well the method would perform on more complex objects, such as those in sci-fi or fantasy genres.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging comments and feeback. We answer each question as below:\n\n### **Is the proposed method compatible with a purely text-to-3D pipeline?**\nYes, our method is compatible with purely text-to-3D backbones, such as RichDreamer (Qiu, et al., 2024). In our method, we generate our base mesh by first generating an image from the input text prompt using SDXL (Podell, et al., 2023) and then running Wonder3D (Long, et al., 2023), which is a ‘text-to-image-to-3D’ pipeline. However, our method is also compatible with most pipelines that output colored meshes. As suggested, we used a state-of-the-art text-to-3D method RichDreamer, and integrated it into our method. Specifically, we generate the base mesh using RichDreamer, and finetune the albedo and normal UV map using our proposed tactile matching loss, diffusion-based refinement loss, and regularization loss. Example results of both RichDreamer and our full method are shown in Figure 4 in the attached PDF.  As shown in Figure 4, our method is able to generate 3D objects corresponding to the text prompt while adding high-fidelity geometric details from tactile inputs. \n\n### **More advanced image-to-3D baseline and more complex objects**\nSimilarly, we can integrate InstantMesh (Xu, et al., 2024), a more recent image-to-3D method, into our method. Figure 4 and Figure 5 in the attached PDF demonstrate the results for more complex objects using the new backbones of RichDreamer and InstantMesh. Our method enhances the geometric details since the tactile information provides guidance of finer scale than the resolution of the base mesh generated by the backbones. These results highlight the compatibility of our method with different backbones, and its generalizability to complex objects. We will include them in the revision. \n\n### **How to select tactile input and how to ensure the tactile details are compatible with the object?**\nThe tactile input can be selected either for realism or creativity. For generating realistic outputs, we would choose the tactile input similar to the object, e.g., using a tactile patch collected from real strawberry to generate a strawberry mesh. Otherwise, if we aim for creativity, we can choose any tactile texture we want, as the various coffee cups show in Figure 6 of the main paper.  Since the tactile details are of millimeter scale, they are added upon the coarse geometry as a normal UV map, which is compatible with the base mesh. To further ensure alignment between the color and tactile details, we jointly optimize the albedo and normal maps using the refinement loss.\n\n### **What are the efficiency and success rate of the attention-based segmentation?**\nSince we leverage the same diffusion model to compute the attention maps for segmentation, it is more memory efficient than using  additional components such as off-the-shelf segmentors like SAM. In terms of running time, a single-part experiment takes about 6 mins and a multi-part experiment takes about 8 mins.\n\nTo quantitatively evaluate our segmentor, we manually segment 3 meshes to obtain ground-truth segmentation masks. We then run our segmentor on 100 renderings for each mesh. In terms of the metric, we calculate the IoU between our predicted masks and the labeled masks and also compute the accuracy of predicted labels. Our diffusion-based segmentor reaches an average IoU of 0.588 and an accuracy of 83.7%, which provides sufficient segmentation for multi-part optimization. Note that our multi-part optimization can partially resolve inaccurate and inconsistent segmentation by aggregating the gradients from different views, so our 2D segmentation does not need to be perfect. We are happy to expand this evaluation in the revision. \n\n### **Clarification about the color and geometry optimization**\nThe coarse geometry is fixed after the base mesh generation while the local geometric details, defined by normal UV maps, are further refined using tactile information, and thus not derived from the original colors. The color is also optimized simultaneously with the normals using a diffusion-based refinement loss. The optimization could introduce changes of color to align visual and tactile modality, while we add the regularization term so that the refined color maintains consistent with reference color on a larger scale. We balance the loss weight and a single set of parameters works for all our experiments.\n\n### **Comparison with real objects**\nWe show one example result of using a real image and corresponding captured tactile signals as input in Figure 3 of the attached PDF. Although our method can recover the color and geometric details reasonably well, we think humans can still distinguish between the real and generated objects by comparing with the input real image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egag2Xl1hG", "reviewer": "Reviewer_ecYi", "summary": "This paper proposes a lightweight 3D texture field that ensures the consistency between visual and tactile textures while preserving photorealism. The experiments demonstrate that quantitative and qualitative results show good generation quality.", "questions": "See my previous sections.", "limitations": "I'm not confident about voting for accepting this paper because of the potential similarity to existing methods and the limited novelty. \nThere is more engineering effort than novelty. The novelty might be limited.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The authors pioneered the use of tactile sensing to enhance geometric details. \n2. The exposition is good. The paper is easy to understand.\n3. They created a TouchTexture dataset, comprising 16 everyday objects, contributing a new dataset to the community.", "weaknesses": "1. This paper suggests that existing methods struggle to achieve fine-grained geometric details. However, methods like Neuralangelo and PermutoSDF can recover highly detailed geometric information.\n\n2. The TouchTexture dataset presented in Figure 3 and the supporting materials seemingly do not capture the \"local geometric intricacies.\"\n\n3. There are only very limited objects been shown in the main paper and supplementary material. Are those objects cherry-picked? It would be great if more results on in-the-wild object can be provided to show the generalisation ability of the model.\n\n4. Ablation Study is an important part in paper and it would be more convincing with both quantitative and qualitative experiments, and cannot be simply summarized with only several sentences. If there are any figures and tables in the article, please indicate the specific table number or figure number in the analysis and analyze according to the specific visualization results.\n\n- [Neuralangelo: High-Fidelity Neural Surface Reconstruction. CVPR2023]\n- [PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices. CVPR2023]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your questions and comments. We would like to clarify that in this work, our main contribution is leveraging tactile sensing to enhance geometric details for 3D generation tasks. (1) We are the first to leverage tactile sensing to synthesize high-fidelity geometric details for 3D generation. (2) We present a new technique to learn high-quality 3D texture fields with aligned visual and geometry textures, with a multi-part extension that allows us to synthesize multiple textures across various regions. (3) Our experiments show that our method outperforms existing text-to-3D and image-to-3D methods.\n\n### **Comparison with the different setup of reconstruction pipeline**\nWe agree that methods like Neuralangelo (Li, et al., 2023) and PermutoSDF (Rosu, et al., 2023) can recover highly detailed geometric information. However, these methods focus on 3D reconstruction tasks and require sufficient data to achieve high-resolution reconstruction, e.g.,  “50 images” for an object and “~200-1000 images” for a larger scene. In our setup, we focus on a different task of text-driven 3D generation. We make use of a similar 3D representation, i.e., a multi-resolution 3D hash grid, as a texture field and strive to generate highly detailed 3D assets with minimal input, a text prompt / single-view image and a single tactile patch.\n\n### **Local geometric intricacies captured by Tactile data**\nThe GelSight mini sensor we use to collect the TouchTexture dataset has a sensing area of 21mm x 25mm. The normal maps and height maps in Figure 3 of the main paper and the supporting materials show the tactile data for a single touch, cropped to 18mm x 24mm in physical scale and 240 x 320 in pixels, which corresponds to a tiny patch as shown in Figure 2 of the main paper. We refer to the “submillimeter scale” geometry captured by the sensor as “local geometric intricacies”.\n\n### **More Diverse Results**\nWe show more complex and diverse results in Figure 4 and Figure 5 of the attached PDF. Our method is compatible with most pipelines that generate colored meshes, and can enhance the geometric details of their output. With different backbones such as RichDreamer (Qiu et al., 2024) and InstantMesh (Xu et al., 2024), our method is able to generate diverse, in-the-wild objects. These results showcase the generalizability of our method. We will include more examples in the revised draft.\n\n### **Ablation studies**\nIn the main paper, Figures 9,10, and 11 illustrate the ablation studies and the analysis is provided in Line 238-250. Directly applying a normal texture map to a base mesh without joint optimization may introduce unnatural appearance, as the additional normal texture could be conflicting with the original albedo map, as shown in the misaligned strawberry seeds in Figure 9. Figure 10 shows that using the raw tactile data without proposed preprocessing produces much more flattened textures. This is because low-frequency deformation of the gel pad would dominate the tactile signal, reduce the signal-to-noise ratio, and degrade the synthesized geometric details. We also ablate the tactile input, and the example results are shown in Figure 11. Removing tactile input produces overly smooth meshes, as a text prompt could not provide sufficient guidance to geometric details at the millimeter scale. Figure 2 in the attached PDF also shows the ablation of tactile loss on a part of the object, demonstrating that the diffusion prior can infer some geometric variation to a certain level but is not capable of generating high-fidelity regular and consistent textures. We are happy to conduct additional ablation studies if there are specific aspects the reviewers would like us to investigate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Sd7OTm0yy", "reviewer": "Reviewer_a9oZ", "summary": "This paper proposes to use the tactile sensing signal (height map and normal map) to improve the 3D generation quality, especially the geometric details. The authors use a 3D mesh generation guided by a normal-conditioned ControlNet to ensure the consistency between the  visual textures and the tactile textures. They also develop a multi-part editing pipeline to generate objects with different texture parts. Experiments and ablation studies demonstrate the effectiveness of the proposed method.", "questions": "1. What is the generation time of the multi-part geneation? \n2. Can the authors add an experiment of real image to 3D mesh with real corresponding tactile signals?", "limitations": "Limitaions have been addressed in the paper, which include complex geometry generation and slight seams.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The origniality of the paper is good. To my knowledge, this is the first work to use the tactile sensing for 3D generation. The generation quality is satisfying. The generated meshes include good geometric details, which align well with the input tactile signals. The paper is well written and structured. The visualizations clearly demonstrate the quality of the generated meshes.", "weaknesses": "The connection between the tactile sensing and the 3D generation is not strong and critical. The tactile signals are only used to generate the normal maps of certain textures. It may be replaced by simpler alternatives such as texture reterival from a normal map database. It needs further elaboration why this combination is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and valuable comments. We address the questions below:\n\n### **Using texture reterival from a normal map database**\n\nWe agree with the reviewer that our work can be potentially extended to other high-resolution geometrical texture data such as the one from a normal map database. In this work, we focus on the data acquired from tactile sensors, due to several reasons. First, the normal map database could be limited in terms of texture variation, e.g., different avocados, as it often contains a single or very few normal maps per object. Second, the normal map data is sometimes hand-crafted rather than captured from real data, making it less realistic. Third, when the user has a particular texture in mind to synthesize, e.g., customizing the plush toy with the fabric of their favorite sweater. In these regards, we believe high-resolution tactile sensors provide a quick and scalable way to capture the precise textures of real objects and, therefore, will be a common way to capture surface textures, in the near future.  \n\nMoreover, in the future, we can extend the work of using tactile feedback to model the physical properties, such as the hardness of the material, as part of the object rendering. We thank the reviewer for pointing this out and will add this discussion in the revision. \n\n### **Generation time of the multi-part generation**\nAs mentioned in the supplementary material, we train all models on A6000 GPUs and each experiment takes about 10 mins and 20G VRAM to run. To be specific, single-part experiment takes about 6 mins and multi-part experiment takes about 8 mins to run.\n\n### **Experiment of using real Image and real corresponding tactile as input**\nWe show one example result of using a real image and corresponding captured tactile signals as input in Figure 3 of the attached PDF. Our method can recover the color and geometric details reasonably well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their efforts and feedback. The reviewers note that we solve “a novel and unexplored task” (zBVq) with an “innovative approach in utilizing tactile normal modality to enhance geometric details” (MUQX), provide “convincing” (zBVq) and “satisfying” (a9oZ) results, release a TouchTexture dataset that is “beneficial to the research community”  (MUQX, ecYi), and deliver a “structured, well-written” (a9oZ) and “easy to understand” (ecYi) exposition.

The reviewers have suggested additional experiments that will better highlight the strengths and capabilities of our method. **We are happy to report that we have conducted most of the experiments, with favorable results, and we will include them in the revision.** Please see the attached PDF for visual results. Here, we first summarize the new experiments, then provide a more detailed analysis and address other comments and questions in separate threads. 

* **Visualizations of albedo renderings (Figure 1)**: As suggested by Reviewer zBVq, we visualize albedo renderings together with the normal and the full-color renderings to present our results. Notably, our albedo rendering looks natural and contains minimal baked-in lighting effects or geometric details. 
* **Ablation study of tactile matching loss $L_{tactile}$ on different parts of an object (Figure 2)**: Following Reviewer zBVq’s advice, we show the “cactus in the pot” example where only a part of the object has $L_{tactile}$. The texture learnt purely from diffusion prior tends to be flatter and less detailed compared to those learned with  $L_{tactile}$.
* **Generating mesh using real image and corresponding tactile signal (Figure 3)**: As suggested by Reviewer MUQX and a9oZ, we generate a mesh from real data input, and our method achieves reasonable reconstruction with color and geometric details.
* **Integrating our method with more recent baselines (Figure 4 and Figure 5)**: As suggested by Reviewer MUQX and ecYi, We integrate RichDreamer (Figure 4), a purely text-to-3D baseline, and InstantMesh (Figure 5), a recent image-to-3D baseline into our method, demonstrating our method’s compatibility with various pipelines. Our method, when integrated with these baselines, generalizes well to complex objects across different genres, and achieves better results compared to using the baselines alone.
* **Quantitative Evaluation of our diffusion-based segmentation method**: To answer Reviewer MUQX’s question, we quantitatively evaluate our segmentation method by manually annotating a test set of meshes. Our method achieves an accuracy of 83.7%, demonstrating its capability to provide sufficient segmentation for multi-part optimization.

Meta-review: This submission has garnered 1 accept, 2 weak accepts, and 1 borderline reject. Reviewers zBVq, MUQX, and a9oZ support the paper, recognizing its novelty and the convincing nature of its contributions. The authors have successfully addressed the concerns raised during the review process, which include issues of repetitive textures, adaptability to complex objects, and the role of tactile sensing. Reviewer ecYi expressed reservations, comparing the proposed text-driven 3D generation method to traditional 3D reconstruction approaches; however, the authors' rebuttal effectively clarified the distinctions between these methods. Given the supportive majority among the reviewers, the AC recommends acceptance of this manuscript.
Decision: Accept (poster)