RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation
Authors: Hanzhuo Huang
Qingyang Bao
Zekai Gu
Zhongshuo Du
Cheng Lin
Yuan Liu
Sibei Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Models
Primary area: generative models

Abstract: In this paper, we propose a 3D asset-referenced diffusion model for image generation, exploring how to integrate 3D assets into image diffusion models. Existing reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to single-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present a cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achieving precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled generation mechanism ensure the simultaneous generation of two spatially aligned but content-disentangled outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effectively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation.

Published as a conference paper at ICLR 2026
REFANY3D: 3D ASSET-REFERENCED DIFFUSION MODELS FOR IMAGE GENERATION
Hanzhuo Huang' Qingyang Bao"
Zekai Gu
Zhongshuo Du
Cheng Lin Yuan Liu
Sibei Yang*t
'ShanghaiTech University, "Sun Yat-sen University, UUniversity of Toronto, *The Hong Kong University of Science and Technology. *SynWorld, Maaaau University of Science and Technology Shanghanzhuo@shanggee tech, cn, quanlylust.he yangsb3 @mail sysu. edu. en,
ABSTRACT
In this paper, we propose a 3D asset-referenced diffusion model for image gen- eration, exploring how to integrate 3D assets into image diffusion models. Ex- isting reference-based image generation methods leverage large-scale pretrained diffusion models and demonstrate strong capability in generating diverse images conditioned on a single reference image. However, these methods are limited to tongle-image references and cannot leverage 3D assets, constraining their practical versatility. To address this gap, we present cross-domain diffusion model with dual-branch perception that leverages multi-view RGB images and point maps of 3D assets to jointly model their colors and canonical-space coordinates, achiev- ing precise consistency between generated images and the 3D references. Our spatially aligned dual-branch generation architecture and domain-decoupled gen- eration mechanism ensure the simultaneous generation of two spatially aligned but content-disentanglee outputs, RGB images and point maps, linking 2D image attributes with 3D asset attributes. Experiments show that our approach effec- tively uses 3D assets as references to produce images consistent with the given assets, opening new possibilities for combining diffusion models with 3D content creation. Project page: https:/ /tt///jddggnent//e github. io/ Re 3D.
INTRODUCTION
In recent years, text-to-image diffusion models (Ho et al., 2020; Ramesh et al., 2022; Rombach et al., 2021; Saharia et al., 2022; Labs, 2024) have made remarkable progress image synthesis, enabling the generation of high-quality and diverse images from textual prompts. However, relying solely on text prompts is often insufficient to capture fine-grained semantics and complex visual details (Witteveen & Andrews, 2022). This limitation is particularly pronounced in scenarios that require faithful preservation of a subject's 's identity, such as personalized content creation, advertis- ing, marketing, and artistic design. Although existing methods (Gal et al., 2022; Ruiz et al., 2023; Ye et al., 2023; Li et al., 2023; 2024b; Shi et al., 2024; Cai et al., 2025; Tan et al., 2025) have concen- trated on preserving object identity within 2D images,
prenttiiip generation conditioned on 3D assets remains underexplored and represents a promising direction for future research.
Current subject-driven generation methods preserve identity by leveraging either local features or global semantics from reference images. Some methods (Gal et al., 2022; Hu et al., 2022; Ruiz et al., 2023) fine-tune diffusion models or text embeddings separately for each subject to capture fine-grained details, which is computationally expensive. More recent approaches (Shi et al., 2024; Cai et al., 2025; Tan et al., 2025) improve efficiency by introducing attention mechanisms between generated and reference images, enabling the model to effectively learn reference- guided generation. Other approaches (Ye et al., 2023; Li et al., 2023) map images into the text space to encode global semantics, which is computationally efficient but tends oversimplify representations. Compress-
1 Corresponning Authors.

---

Published as a conference paper at ICLR 2026
WHH
8.4
3D asset
3D asset
in the garden
in a corner of cramped room
inside train
the desktop
the room
Figure I: Results of our RefAny3D. Given 3D asset, our method can generate high-quality and 3D asset-consistent images.
ing an image into a few text tokens results the loss of spatial information, making it difficult for the generated images to faithfully correspond to the fine-grained details of the reference.
While extensive subject-driven generation (Tan al., 2025; Kumari et al., 2025) has demonstrated single few 2D reference images maintain identity
that image generation conditioned consistency, 3D asset-referenced image generation remains unexplored. practical applications, creators often require the direct use of 3D assets, such meshes, references to visualize how an object would manifest across diverse scenes and environments. Such scenarios go beyond the scope of purely 2D references and give rise new problem: how leverage 3D assets conditioning signals to generate images that are not only identity-consisten, but also geometry-consisten and texture-consistent.
Although existing methods (Cai et al., 2025; Tan et al., 2025) have achieved impressive results with 2D reference images, they remain fundamentally limited when extended the task of generation conditioned on 3D assets. Unlike 2D settings, 3D asset-referenced generation requires not only semantic-lever identity preservation, but also strict consistency with the geometry and texture of the reference 3D asset. This goes beyond the capability of existing methods. Specifically, first, the con- sistency capability of current methods remains inadequate for 3D refee-eee generation, where the synthesized images are required to align precisely with the geometric structure and texture of the reference 3D asset. Second, approaches (Ye al., 2023; Tan et al., 2025) limited to a single ref- erence image are inherently unable to capture the full appearance of the object. Finally, in methods based on multi-image conditioning (Kumari et al., 2025; Zeng et al., 2024), or in straightforward extensions of existing approaches multiple inputs, the absence of 3D structural priors prevents consistent spatial correspondence across views, leading to viewpoint conflicts and cross-view in- consistencies. In addition, recent image editing models (Labs et al., 2025; Wu al., 2025) have shown strong instruction-following and image understanding capabilities. However, they remain insufficient for effectively addressing the challenge of 3D alimitationsse--reeeeeeee straightforward way to leverage such models is to manually select a viewpoint of the 3D asset, render into an image, and then apply editing instructions to the rendered view. The primary shortcoming of this approach is that the resulting images often suffer from foreground backkrruund inconsistency and may hallucinate non-existent content. Overall, the key challenge of 3D asset-referenced gener- ation lies in effectively leveraging the structural and textural priors of 3D assets achieve faithful, view-consistent, and detail- preservnn image synthesis.
generation. A
In this paper, we propose RefAny3D, a 3D asset-referenced and 3D structure-aware image genera- tion framework, which is designed to synthesize images with faithful fidelity and consistent align- ment to the 3D assets. The core idea is to construct 3D-aware generative framework that leverages the correspondence between normalized object coordinates (point maps) (Wang et al., 2019) and their associated RGB values, thereby ensuring consistent alignment with the 3D assets. This consis-
2

---

Published as a conference paper at ICLR 2026
tency stems from two key properties of point maps. First, while texture information alone may in- troduce ambiguities or repetitions across different views, point maps are uniquely tied to the object's geometry, enabling more reliable cross-view correspondence. Second, point maps continuous and invariant to object pose or position, making them easier to learn and more effective anchors for linking geometric structure with texture. Specifically, we formalize the generation process as modeling the joint distribution of RGB appearance and point maps. Conditioned on multi-view RGB images and point maps of the 3D asset, the framework trained to simultaneously generate photorealistic images of the object together with their corresponding point maps. To achieve this, dual- generation strategy that enables
we introduce spatially aligned, Nomain-decouppe the model to synthesize both RGB images and point maps in a unified manner. Unlike prior ap- proaches, our method explicitly leverages object coordinates to build structural awareness of the 3D object, while the point maps establish pixel-level correspondences across different views, which is otherwise difficult to achieve without such guidance. Consequently, our approach maintains faithful consistency of complex geometry and texture.
In summary, our main contributions are as follows: (I) We propose a 3D asset-referenced image gen- eration framework that ensures faithful alignment and consistency with the underlying 3D assets. (2) We design a spatially aligned, pomainn-decuppe model to jointly generate RGB images and point maps, thereby enhancing its 3D structural aware- ness. (3) We demonstrate that our approach achieves accurate preservation of the visual identity of 3D objects. Extensive qualitative and quantitative evaluations show that it consistently outperforms existing baselines on the 3D asset-referenced generation task, delivering superior fine-grained con- sistency and robust fidelity even for complex models with intricate geometric and textural details.
dual-branch generation strategy that enables the
2 RELATED WORK
Diffusion Models. Recently, diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021; Ramesh et al., 2022; Rombach et al., 2021) trained on large-scale datasets (Schuhmann et al., 2022; Byeon et al., 2022) have achieved significant breakthroughs in generating photorealistic and diverse visual content, excelling across a wide range of image generation tasks (Huang et al., 2023; 2025; Gu et al., 2025), including image editing (Batifol et al., 2025; Wu et al., 2025), controllable content generation (Zhang al., 2023), and subject-driven generation (Gal al., 2022; Ruiz et al., 2023; Ye et al., 2023). Pioneering works (Ramesh et al., 2022; Rombach et al., 2021) first showcased the strong generative and generalization capabilities of diffusion models trained on large-scale datasets. To further enhance their generative capability, Transformer (Vaswani et al., 2017; Zhu et al., 2025; Dai et al. 2025; Dai Yang. 2025; Wei et al., 2025; Tang et al., 2025a; Yang et al., 2025; Zheng et al., 2025; Tang et al., 2025b; Qian et al., 2025; Yang et al.; Zhang et al., 2025; Shi et al., 2025) architectures have been incorporated into diffusion models (Peebles & Xie, 2023), enabling greater scalability. More recent model (Labs, 2024) adopts flow-matching (Lipman et al., 2022) training in conjunction with MMDiT (Esser et al., 2024) architectures and large-scale datasets, achieving state- of-the-art performance in text-to- text-to-image generation. Despite these advances, text-to-image models still lack effective approaches for generating images conditioned on 3D assets as references.
Subject- -riven Generation. The goal of subject-driven generation is to capture the characteristics of a given reference subject, enabling the synthesis of realistic images of the subject across diverse scenes. Early methods (Gal et al., 2022; Hu et al., 2022; Ruiz et al., 2023) typically adapt the text embedding layer (Gal et al., 2022) or the model (Hu et al., 2022; Ruiz et al., 2023) using only a few reference images, while applying regularization (Ruiz et al., 2023) to maintain the model's general- ization capability. Because these methods require retraining and fine-tuning for each subject, they entail significant computational and time costs, limiting their practical applicability. To reduce train- ing overhead, some works (Ye et al., 2023; Li et al., 2023) learn an adapter from image space to text space, enabling direct encoding of images into text embeddings without per-subject training. How- ever, these approaches compress an image into only few text tokens, often limiting fine-grained fidelity to the reference. Recent approaches (Cai et al., 2025; Tan et al., 2025) concatenate the gen- erated and reference images into a unified token sequence and leverage shared attention to better capture fine- fine-grained correspondences, addressing the challenge of limited detail fidelity. Beyond smage-guided subject-driven generation, several works (Wu & Zheng, 2022; Wu et al., 2023; Wang et al., 2024a;b) investigate 3D-guider generation using single or few 3D exemplars. Wu & Zheng (2022) generate 3D shapes from a single reference 3D shape using multi-scale 3D representations;
3

---

Published as a conference paper at ICLR 2026
31 Assets
"A brown recliner wooden deck by the lake" Text
Aftiatiin
Praiaiaal Waghi
Shared
Multi-view conditions
Rebrease
Shared
Shared
Point Map
DIT Block xN
Overview of RefAny3D. Given a 3D asset, render multi-view inputs conditioning signals for the diffusion model and simultaneously generate the point map of the target RGB image. To ensure pixel-level consistency across different viewpoints, adopt shared positional encoding strategy. Moreover, to disentangle the RGB domain from the point map domain, we incorporate Domain-specifi LoRA and Text-agnostic Attention. Benefiting from this 3D-aware disentangle- ment design, our method is able to generate high-quality images that maintain strong consistency with the underlying 3D assets.
Figure 2:
Sin3DM (Wu al., 2023) learns diffusion model from single textured 3D shape; ThemeSta- tion (Wang et al., 2024a) produces theme-aware 3D assets from few exemplars; and Phidias (Wang al., 2024b) enables text-, image-, and 3D-conditioned content creation via erteencc-aaugmented diffusion. While these methods leverage 3D inputs, they primarily focus on 3D asset generation, rather than using provided 3D assets guide 2D subject-driven image generation. Despite their merits, these methods often lack precision when using 3D assets with complex textures as refer- ences. contrast, our approach effectively leverages 3D structural cues achieve more faithful and consistent reference generation.
Multi-modal Image Generation. Recent studies (Ke al., 2024; Long et al., 2024; Yang et al., 2024; Li et al., 2024a; Huang et al., 2024; Fu et al., 2024; He et al., 2024; Ye et al., 2024; Zhang al., 2024) have demonstrated that diffusion models are capable of generating not only high-fidelity RGB images but also diverse physical property images, such albedo, normal, roughness, and ir- radiance. Several recent multi-view generation works (Long et al., 2024; Li et al., 2024a; Huang et al. 2024) improve 3D reconstruction quality by jointly generating multi view normals and color images. Moreover, an increasing number of studies (Ke et al., 2024; He et al., 2024; Fu et al., 2024; Ye al,, 2024) leverage the multi-modal generation capability of diffusion models to perform dense prediction tasks. These methods typically condition on RGB images to predict pixel-aligned nor- mal depth maps. However, these multi-modal image generation methods either jointly generate multiple modalities but remain constrained to multi-view settings (Liu et al., 2023), or they simply translate one modality into another single modality (Ke et al., 2024; Gu al., 2025). In contrast, our approach simultaneously generates information across multiple modalities and is not restricted to object-centric multi-view settings.
METHOD
We propose 3D asset-conditioned image generation framework, which jointly models RGB image and point map distributions to create high-quality images for a 3D object. The overall generation pipeline is shown in Fig. 2.
Overview. Given a conditional 3D object y. we represent it as a set of multi-view RGB RGB-point map pairs {CI, Cr.)i 0, 1........ .N, where N is the total number of views. C₁, hax denotes the RGB images from the i-th viewpoint, and Cp, e xx denotes the corresponding rasterized 3D coordinates of the object. The RGB images and point maps pixel-wise aligned,
4

---

Published as a conference paper at ICLR 2026
jointly encoding the color and its associated 3D position. Formally, our objective is learn the conditional distribution given a reference 3D model and a text prompt
xply.
where 21 denotes the target RGB image, and denotes its corresponding point map. To accu- rately preserve the visual identity of 3D object, we adopt a dual-branch conditional generation framework that jointly generate the RGB image and point map, leveraging 3D spatial constraints to reinforce reference consistency (3.1). However, generating spatially aligned images from two domains introduces the challenge of texture bleeding. To address this, we propose domain decou- pling generation strategy (3.2). For training RefAny3D, we prepare an object pose-aligned datasets, which consists of images containing the objects of interest, their corresponding 3D object models, and the associated object poses (3.3).
3.1 SPATIALLY ALIGNED DUAL- BRANCH GENERATION
We simultaneously generate spatially aligned RGB images and point maps to provide precise 3D spatial information of the reference object for conditional image generation. To achieve conditional generation for the diffusion model, similar to prior works (Tan et al., 2025; Wang al., 2025), we concatenate the target tokens and condition tokens into a unified sequence. Furthermore, generate spatially-aligned cross-domain images, we employ shared positional embeddings.
Conditional Token Sequence. We encode the RGB images and point maps of the 3D model into latent conditional tokens using a pretrained VAE encoder, which are subsequently concatenated with the noisy target latent. To preserve the fidelity of the conditional features, set the timestep of the conditional tokens to 0 during the diffusion denoising process.
Shared Positional Embedding for Cross-Domain. To maintain spatial alignment between the RGB image and point map during generation, we apply shared positional encodings to tokens across both domains. This approach exploits inherent property of DiT, induced by positional encod- ing, to naturally assign higher attention scores to tokens with the same positional embeddings. To mitigate biases caused by inconsistent distances among conditional tokens, introduce unified positional shift term. Specifically, for a conditional token at spatial position (i,j), ) the positional encoding is set to (i w,)), where is the width of the target latent image. This shift guarantees that the conditional tokens and target tokens remain spatially disjoint.
3.2 DOMAIN DECOUPLING GENERATION
The core challenge of jointly generating point maps and RGB images lies in their inherent informa- tion asymmetry. A point map defines only the object's 3D geometry and pose, while the RGB image contains photorealistic details of the entire scene. In unified framework, this asymmetry often causes the point map, which lacks background information, to be affected by interference from the RGB branch and text prompt. Therefore, we introduce domain-specific LoRA and a text-agnostic attention to achieve accurate generation in both domains.
Domain-specific LORA. We decouple domain knowledge using a domain switcher and a dual LoRA structure. The domain switcher specifies the domain of each token to guide the generation of RGB images and point maps. Specifically, we associate each domain with a learnable embedding, which is then concatenated with the timestep embedding. To further decouple the learning of domain-specific knowledge, we also introduce two independent LoRA (Hu et al., 2022) modules, termed Reference- LoRA and Domain-LoRA. to separately learn the 3D object reference generation and point map domain generation. The Reference-Lofe is activated for all conditioning tokens to learn general appearance features, while the specialized Domain-LoRA, is activated only for point map tokens to leamn specific geometric information. This design enables the model to generate high-fidelity point maps and RGB images.
Text-agnostic Attention. To further suppress background information leakage into the point map, we introduce a text-agnostic attention mask in the point map branch. This design minimizes the in- fluence of text tokens the point map. as textual descriptions often contain substantial background information that is irrelevant to the point map. In contrast, the RGB tokens can attend to all tokens, allowing them fully exploit the information from both the text and the point map. This design
5

---

Published as a conference paper at ICLR 2026
Data Construction Pipeline
Grounding
Humyuand0
Poundation Poss
(a) Data construction pipeline
(b) Examples from the dataset
Figure 3: (a) Data construction pipeline. We first GroundingDINO (Liu et al., 2024) extract the objects of interest, then convert the images into 3D models using Hunyuan3D (Zhao et al., 2025), and finally apply FoundationPose (Wen et al., 2024) estimate the poses of the 3D models in the images. (b) Examples from the dataset.
ensures the point map generated as a geometric proxy, reducing the influence of background cor- ruption, while the RGB branch can fully utilize the accurate geometrie guidance to render detailed shapes and textures.
3.3 DATASETS
To train our model Specifically, this dataset is composed of images containing the objects of interest, their correspond- ing 3D assets, and the associated object poses, which are used to generate the corresponding point maps. However, existing public datasets do not provide all the required data. We build upon Sub- jects200k, subject-driven generation image dataset, and further enhance it by incorporating 3D assets along with object pose annotations for each image. The overall construction pipeline is illus- trated in Fig. 3. First, for each image, we use the object names provided in Subjects200k as prompts GroundingDINO (Liu al., 2024) extract the corresponding objects of interest. Next, convert each extracted object from the image into a 3D asset using Hunyuan3D (Zhao et al., 2025). Finally, taking the generated 3D asset as input, we estimate its pose using FoundationPose (Wen et al., 2024).
asset-reference generation, we require an object pose-aligned dataset.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Implementation details. We use 1-dv as our base model. Following Tan et al. (2025), we train the model with the Prodigy optimizer (Li et al., 2023). Our model is trained for 30k steps 8 H800 GPUs. To enable classifier-free guidance (Ho & Salimans, 2022), we randomly drop the text and the reference multi-view images with probability of 0.1 each. The number of views in the multi-view images is set to 8.
Baselines. We adopt Textual Inversion (Gal al., 2022), DreamBooth (Ruiz et al., 2023), IP- Adapter (Ye et al., 2023), DSD (Cai et 2025), and OminiControl (Tan et al., 2025) as our baseline methods. For personalized text-to-image generation methods that require training (e.g.. Textual Inversion and DreamBooth), we use multi-view images of each 3D asset as training data, fine-tune the model accordingly, and then sample from the customized model obtain generated images. For methods that do not require additional training (e.g.. IP-Adapter, DSD, and OminiControl), we use their official pre-trained models. Since these models do not support multiple images as input conditions, select single view of each 3D asset input to generate images for comparison with our approach.
Metrics.
We employ both foundational vision models and LVLMs to comprehensively evaluate ID consistency, texture consistency, and aesthetic quality of the generated images and 3D assets. Specifically, measure semantic consistency by computing CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) feature similarities between the generated images and the multi-view renderings of the 3D assets. In addition, compute the CL.IP text-image similarity score for

---

Published as a conference paper at ICLR 2026
3D Assets
Ours
OminiControl
DSD
Idaptrr
DreamBouth
Textual Inversion
Stpi TES
Figure 4: Qualitative comparison with other methods. Our approach achieves superior geometric and texture consistency compared to alternative methods.
GPT-eval
CLIP Img/Max.1
DINO
GIM
Mothod
Texturet
Geometrict
Aesthetict
Overalit
Img/Avg.t
Text
Avg."
Max.t
Count
Textual Inversion
2.894
4.421
6.263
4.526
0.827
0.878
0,323
0.548
0.653
3359.880 895
DreamBooth
5.368
6,684
6.894
63315
0.867
0.912
0.328
0.695
0.809
3483.
IP-Adupter
3.833
5.278
5.167
4.759
0.863
0.913
0.312
0.652
0,760
3137. 167
DSD
4.842
6,473
2,105
6.140
0.832
0.884
0.329
0.644
0.761
3568 737
OminiContral
5.631
6.578
6.893
6.367
0,855
0.901
0.332
0.665
0.783
3474.21
6.315
7.368
7.687
7.123
0.923
0.340
0.720
0.843
3901.316
Table 1: Quantitative results comparing our method against the baselines, with the best scores high- lighted in bold and the second-best underlined. Our approach achieves the best performance across all evaluation metrics, including both GPT-based measures and baseline model metrics. In particu- lar, our method yields substantial gains on the GPT-eval Texture and Geometric scores, as well as the GIM metric, which are particularly indicative fine- grne-graiedd geometric and textural fidelity. These results demonstrate the effectiveness of our framework in faithfully preserving 3D asset consistency and generating high-quality outputs beyond existing baselines.
each generated image and its corresponding text prompt. We assess texture consistency by using GIM (Shen et al., 2024) to count the number of matched keypoints between the generated images and the multi-view images. Furthermore, we leverage GPT evaluation by providing both the gener- ated and multi-view images to GPT-5 to obtain scores on textual consistency, geometric consistency, and aesthetic quality, thereby enabling more comprehensive assessment of the generation results. We then compute the average these three scores as an overall score.
4.2 COMPARISONS
Qualitative results. As shown in Fig. 4, we present qualitative comparisons between our method and baseline methods. Our method achieves superior geometric and texture consistency with the 3D assets compared to others. In the first row of Fig. 4, our method accurately captures the undulating geometric surface of the chair cushion, whereas other methods fail to reproduce these fine structures. For 3D assets with complex textures (second row), our method faithfully reproduces the characters and illustrations on the vase, while other methods fail to maintain such consistency. Moreover, un- like DreamBooth (Ruiz et al., 2023) and Textual Inversion (Gal et al., 2022), our method requires no additional training on the reference 3D asset to produce consistent results, highlighting its practical advantage. Fig. 5 further shows the point map results generated by our method, demonstrating that can simultaneously generate the foreground object and its corresponding point map to indicate relative coordinate relationships. In addition, our method can be integrated with multi-view to 3D generation models (Zhao et al., 2025), enabling the synthesis of images conditioned on multiple reference views. Fig. illustrates the effectiveness of our approach multi view inputs. By mod- ifying the prompt or adjusting the 3D asset's local coordinate system, the generated images exhibit diverse perspectives, as illustrated in Fig. 9.
7

---

Published as a conference paper at ICLR 2026
sota..."
crday
Agper
Figure 5: Qualitative results with different 3D assets as references. Our method takes a given 3D mesh input and generates both RGB images and point maps in unified manner. By enforcing pixel-level spatial alignment between the point maps and RGB outputs, the framework ensures con- sistent geometry-texture correspondence across views. Moreover, the incorporation of point maps enhances the model's s 3D structural awareness, thereby improving the fidelity and consistency of image generation with respect to the reference 3D assets.
Method
Faithfulf
ID1
Aesthetict
Rank
Textual Inversion
2,182
3.053
3,526
5.158
DreamBooth
4.315
2.526
IP Adupter
1.982
2.947
2.053
5.737
4.158
OminiControl
3,909
4.263
4.526
3.158
4,632
Muhiisteww Images
Sn the
"on the belcory
the beach"
Table 2: Quantitative results of the user study. We evaluate 3D consistency (Faithful), identity preservation (ID), aesthetic quality, and overall ranking (Rank).
Figure 6: Qualitative results on multi-view im- ages. Our method can integrated into existing multi-view image-to-3D generation pipelines.
Quantitative Results. The quantitative evaluation results comparing our method with other base- lines are shown in Table 1. We report the evaluation results from both large-scale vision-language models (LVLMss) and vision foundation models. For LVLM evaluation, we employ GPT-5 assess texture consistency, geometric consistency, and aesthetic quality, and additionally report an overall score as their average. Specifically, we concatenate the generated image and the multi-view im- ages of the 3D asset into 3x3 grid and prompt GPT-5 to rate the generated image on each metric from to 10, where higher scores indicate better consistency and quality. Our method achieves best performance across all GPT metrics compared other baselines, with notably superior results on texture and geometric consistency. For evaluation using vision foundation models, we adopt CLIP (Radford et al., 2021) and DINO (Caron al., 2021) image encoders compute feature similarities as a measure of semantic consistency. Since the reference consists of multiple images, compute both the average and the maximum similarity between the generated image and the multi-view references to obtain a more comprehensive assessment. To reduce background effects on the CLIP scores, we remove the background from all images before comparing them with the reference renderings. Our method outperforms all other approaches on CLIP and DINO metrics except AP-Atttter (Ye et al., 2023).
To further capture fine-gratd inc-grained correspondences. we employ GIM (Shen et al., 2024), a state-of-the- art image matching method, compute the number of matched keypoints between the generated image and the multi-view references, thereby quantifying the correspondence with the real 3D asset details. Our method also outperfor ms all competing baselines on the vision foundation model met- rics. As shown in Table 2, we also conduct user study evaluate generation quality along four dimensions: Faithfulness, Identity, Aesthetic Quality, and Overall Rank. Faithfulness measures the 3D consistency between the generated image and the reference 3D asset. while Identity evaluates whether the generated object preserves the identity of the 3D asset. Aesthetic Quality reflects the visual appeal of the generated image, and Overall Rank asks participants provide holistic com-
8

---

Published as a conference paper at ICLR 2026
Irput Medi
(3) PI
(b Wio Shared PE
(c) WHO Tent-egaeste Attentoon
(c) WHO Donate segeiff LaRA
Ablation studies on different components of our method: (a) full model; (b) without Shared Positional Embedding for Cross-Domaies (c) without Text-agnostic Attention; (d) without Domain-specific specific LoRA.
Figure 7:
Domain
3D
Ours
Render-and-edit
w/o Pointmap
6view
4view
Figure Comparisons of ablation studies and the editing-bace based baseline.
parison across methods. The results indicate that method is competitive across all metrics, with particularly strong performance in Faithfulness and Identity consistency.
4.3 DISCUSSIONS
In this section, we further design a set of experiments to validate the effectiveness of our proposed
components.
Without Shared Positional Embedding for Cross-Domain. As shown Fig. (b), we remove the specially designed positional embeddings and train the model with token sequences arranged in their natural order, then compare the results against our full model. This comparison demonstrates the necessity of sharing positional embeddings between the RGB and point map under the same viewpoint. Without shared positional embeddings, the network lacks positional priors and strug- gles to learn accurate pixel-level correspondences between the point maps and RGB images. The resulting misalignment leads to degraded geometric consistency with the reference 3D asset. For example, the top of the "backpack" and the overall contour of the "griffin" fail remain consistent with the reference.
Without Domain- Decouppig Generation. To evaluate the effectiveness of our proposed cross- domain decoupling generation strategy, we trained two models without the Domain -specffc LoRA and Text-agnostic Attention modules and tested their performance. As shown in Fig. 7 (b) and (c), the generated point maps and RGB images exhibit color blending, particularly in the background re- gions the point maps. When Text-agnostic Attention removed, the point maps are influenced by the input text, which often contains rich background semantics, causing the point map background to align with that of the RGB branch. In addition, without Domain-specifie LoRA, although the influence of text semantics mitigated, single LoRA is overburdened with simultancously gen- erating two domains and learning reference consistency, which still results background artifacts and further degrades the overall generation quality.

---

Published as a conference paper at ICLR 2026
8
(a) Input Assets
(b) Generated Images
(n) Toput Mesh
(b) Genentted images from diffcrent views.
Figure 9: An example of controllable generation of object images from different viewpoints.
Limitation non-rigid objects. While our method achieves high fidelity to the input 3D assets, does not account for physical interactions in the scene.
Figure 10:
Without Pointmap Generation. To assess the role of pointmap prediction, we trained model without the pointmap. As shown in Fig. 8, removing this branch eliminates explicit 3D cues, making training unstable and resulting in poor 3D consistency and mismatch with the reference asset.
Number of Conditional Views. To evaluate the effect of the number views on the results, we trained models with 6 and 4 views and compared them with our view model. As shown in Fig. 8, the results show that our method still works effectively with fewer views, and performance consis- tently improves the number of views increases.
Comparison with Editing-basee Methods. straightforward approach to generating images 3D objects is to first render the 3D model and then apply editing model to modify the rendered image. However, this requires manual selection of viewpoints, object placement, and renderer quality. Poor choices can lead foreground-backgroume mismatches unrealistic floating objects. Furthermore, the method is limited to a single visible viewpoint. For novel viewpoints, the editing model may hallucinate nonexistent parts, leading inconsistencies with the 3D model. We tested using Qwen- Image-Edit-2509 (Wu et al., 2025), and the results are shown in Fig. 8. For the rear of the tank, the model hallucinates parts that are inconsistent with the 3D asset. For the helmet, the editing model clearly produces a mismatch between the rendered foreground and the real background, with the object unrealistically floating in midair.
CONCLUSION
In this paper, We propose RefAny3D new 3D asset-referenced image generation framework that possesses 3D awareness and can synthesize high-quality images with precise consistency to the given references. Our key idea is to leverage normalized object coordinates (point maps) as structural anchors, jointly modeling RGB appearance and point maps to achieve reliable geometry-texture correspondence. To this end, we design a spatially aligned, domain-decoupled dual-branch strat- egy that enables simultaneous generation of RGB images and point maps, thereby enhancing the model's 's structural awareness. Experiments demonstrate the effectiveness of our approach, showing that it delivers superior fine-grained consistency and robust fidelity compared on the 3D asset-referenced generation task. Limitations. Although RefAny3D enables 3D asset- referenced generation with strong geometric and texture consistency, is less effective in handling non-rigid object references due to dataset limitations. As demonstrated in Fig. 10. deformable assets such as ropes and cushions retain an rigidity. In addition, conditioning the diffusion model on an ex- tended number of viewpoints introduces significant computational and time overhead. Nevertheless, this limitation could be alleviated in the future by employing more efficient attention optimization strategies to improve computational efficiency.
existing baselines
Acknowledgments: This work is supported in part by the National Natural Science Foundation of China under Grant No.62576365.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on developing and evaluating image generation models. We acknowledge that such models carry potential ethical risks, particularly in relation to the generation of synthetic images that could be misused for the creation of deceptive or misleading content. In addition, the outputs of generative models may inadvertently infringe upon copyright or intellectual property rights if they resemble existing works too closely. To mitigate these risks, our experiments were conducted solely for research purposes, and all generated examples in this paper are used exclusively for scientific illustration. We emphasize that our work does not intend to promote harmful applications, and further safeguards, such as watermarking, usage policies, and responsible release practices, should be considered in future deployment of such systems.
REFERENCES
Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dock- horn, Jack English, Zion English, Patrick Esser, Sumith Kulal, et al. Flux. 1 kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv e-prints, pp. arXiv-2506, 2025.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https: htt.gg/bb...//kaaaaaaaaeaaeee kaaam/kkkaabaa///// Poyo-datasse 2022.
Shengqu Cai, Eric Ryan Chan, Yunzhi Zhang, Leonidas Guibas, Jiajun Wu, and Gordon Wetzstein. Diffusion self-distillation for zero-shot customized image generation. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, pp. 18434-18 18443, 2025.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Qiyuan Dai and Sibei Yang. Free on the fly: Enhancing flexibility in test-time adaptation with online em. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9538-9548, 2025.
Qiyuan Dai, Hanzhuo Huang, Yu Wu, and Sibei Yang. Adaptive part learning for fine-grained generalized category discovery: A plug-and and and-play enhancement. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 25444,25453, 2025.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma. Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision, pp. 241-258. Springer, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208 01618, 2022.
Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et Diffusion shader: 3d-aware video diffusion for versatile video generation control. In Proceedings of the Special Interest Group Computer Graphics and Interactive Techniques Conference Conference Pupers, pp. 1-12, 2025.
Jing He, Haodong Li, Wei Yin, Yixum Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion- based visual foundation model for high-quality dense prediction. arXiv preprint 40Xiv:2409.18//1 2024.

---

Published as a conference paper at ICLR 2026
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-fre
diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 13:6840-6852 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2:3, 2022.
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: 7ero- shot text-to-video generator with IIm director and Idm animator. Advances in Neural Information Processing Systems, 36:26135-26158 2023.
Hanzhuo Huang, Yuan Liu, Ge Zheng, Jiepeng Wang, Zhiyang Dou, and Sibei Yang. Mytokenflow: High-quality 4d content generation using multiview token flow. arXiv preprint arXiv:2502.1169) 2025.
Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, and Lu Sheng. Mv-adapter: Multi-view consistent image generation made easy. arXiv preprint arXiv:2412.03632. 2024.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9492-
9502, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Nupur Kumari, Xi Yin, Jun-Yan Yan Zhu, Ishan Misra, and Samaneh Azadi. Generating multi-image synthetic data for text-to-image customization. arXiv preprint arX1v:2502.01720, 2025.
Black Forest Labs. Flux. https: http:/twww..cccccccctooeeeeeeeee
2024.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. URL httph:t////// //axxv///vvvv.// org/ abs / 22..6.50.555 15742.
Dongxu Li, Jumnan Li, and Steven Hoi. Blip-diffusion: Pre- tre-traned subject representation for con- trollable text-to-image tmae generation and editing. Advances in Neural Information Processing Sys- tems, 36:30146-3011 2023.
Peng Li, Yuan Liu, Xiaoxiao Long, Feihu Zhang, Cheng Lin. Mengfei Li, Xingqun Qi, Shanghang Zhang, Wei Xue, Wenhan Luo, et al. Era3d: High-resolution multiview diffusion using effi- cient row-wise attention. Advances in Neural Information Processing Systems, 37:55975-5666 2024a.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho- tomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 8640-8650, 2024b.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European conference on computer vision, pp. 38-55. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisene images from a single-view image. arXiv preprint 0rXf::2300.03453, 2023.
Xiaoxiao Long. Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingje Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9970-9980, 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Jiaye Qian, Ge Zheng, Yuchen Zhu, and Sibei Yang. Intervene-all-patts: Unified mitigation Ivim hallucinations across alignment formats. arXiv preprint arXiv:2511.17254 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systemx, 35:36479-36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-2523 25794,, 2022.
Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. arXiv preprint arXiv:2402.11095, 2024.
Cheng Shi, Yizhou Yu, and Sibei Yang. Vision function layer in multimodal llms. arXiv preprint arXiv:2509.24791, 2025.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8543-8552, 2024.
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Min- imal and universal control for diffusion transformer. 2025.
Jiajin Tang, Zhengxuan Wei, Ge Zheng, and Sibei Yang. Closed-loop transfer for weakly-supervisse affordance grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9530-9533, 2025a.
13

---

Published as a conference paper at ICLR 2026
Jiajin Tang, Zhengxuan Wei, Yuchen Zhu, Cheng Shi, Guanbin Li, Liang Lin, and Sibei Yang. Sim- detr: Unlock detr for temporal sentence grounding. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 22060-26622 22771, 2025b,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Haoxuan Wang, Jinlong Peng, Qingdong He, Hao Yang, Ying Jin, Jiafu Wu, Xiaobin Hu, Yanjie Pan, Zhenye Gan, Mingmin Chi, et al. Unicombine: Unified multi-conditional combination with diffusion transformer. arXiv preprint vxXiv:2503.0927,, 2025.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2642- 2651,2019
Zhenwei Wang, Tengfei Wang, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Themestation: Generating theme-aware assets from few exemplars. Acm siggraph 2024 conference papers, pp. 1-1 2024a.
Zhenwei Wang, Tengfei Wang, Zexin He, Gerhard Hancke, Ziwei Liu, and Rynson WH Lau. Phidias: A generative model for creating 3d content from text, image, and 3d conditions with reference-augmentted diffusion. arXiv preprint 2rXiv:2499./1400 2024b.
Zhengxuan Wei, Jiajin Tang, and Sibei Yang. Augmenting moment retrieval: Zero-dependency two- stage learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3401-3412, 2025.
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 17868-17879, 2024.
Sam Witteveen and Martin Andrews. Investigating prompt engineering in diffusion models. arXiv preprint arXiv:2211. 15462, 2022.
Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng-ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508.02332 2025.
Rundi Wu and Changxi Zheng. Learning to generate 3d shapes from single example. arXiv preprint arxiv:2288.029. 2022.
Rundi Wu, Ruoshi Liu, Carl Vondrick, and Changxi Zheng. Sin3dm: Learning a diffusion model from single 3d textured shape. arXiv preprint arXiv:2305. 15399, 2023.
Bin Yang, Yulin Zhang, Hong-Yu Yu Zhou, and Sibei Yang. No more sibling rivalry: Debiasing human- object interaction detection. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pp. 22707-22717. 2025.
Honghui Yang. Di Huang, Wei Yin, Chunhua Shen, Haifeng Liu, Xiaofei He, Binbin Lin, Wanli Ouyang, and Tong He. Depth any video with scalable synthetic data. arXiv preprint arXiv:2410.108/5. 2024.
Sibei Yang, Ge Zheng, Jiajin Tang, Jiaye Qian, Hanzhuo Huang, and Cheng Shi. Discovering com- positional hallucinations in Ivlms. In The nhith Annual Conference on Neural Information Processing Systems.
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo. Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43(6):1-18, 2024.
Hu Ye, Jun Zhang, Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.067. 2023.
14

---

Published as a conference paper at ICLR 2026
Yu Zeng, Vishal M Patel, Haochen Wang, Xun Huang, Thug-Chun Wang. Ming- Yu Liu, and Yogesh Balaji. Jedi: Joint-image diffusion models for finetuning free personalized text-to-image genera- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6786-6795, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
Yulin Zhang, Cheng Shi, Yang Wang, and Sibei Yang. Eyes wide open: Ego proactive video-llm for streaming video. arXiv preprint arXiv:2510. 14560, 2025.
Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, et al. Dreammat: High-quality pbr material generation with geometry-and light-aware diffusion models. ACM Transactions on Graphics (TOG), 43(4); 1-18. 2024.
Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d assets generation. arXiv preprint (rXiv:2501.1200: 2025.
Ge Zheng, Jiaye Qian, Jiajin Tang, and Sibei Yang. Why Ivlms are more prone to hallucinations in longer responses: The role of context. In Proceedings of the IEEE/CVF International Conference 2025.
on Computer Vision, pp.
Yuchen Zhu, Cheng Shi, Dingyou Wang, Jiajin Tang, Zhengxuan Wei, Yu Wu, Guanbin Li, and Sibei Yang. Rethinking query-based transformer for continual image segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4595 4606, 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASETS DETAILS
To train our 3D asset-reference generation model, require a dataset of pose-aligned objects. We construct this dataset using Subjects200k, a large-scale subject-driven image collection. Our process begins by filtering for high-quality images, selecting those with the highest image quality scores pro- vided in the dataset. Next, we remove the backgrounds from these images to prepare them for mesh extraction with the Hun Yuan 3D model (Zhao et al., 2025). For each image, we leverage the object names supplied by Subjects200k as prompts for GroundingDINO (Liu et al., 2024), which gener- ates bounding boxes to accurately localize the objects of interest. Then we use Segment Anything Model (SAM) (Kirillov et al., 2023) to obtain the objects within the bounding box. The foreground generated by the SAM model may not be very good, so we set a threshold for the mask area ratio, and discard the results that exceed it. With the obtained foregrounds, we place all these objects into Hunyuan3D to first generate a white model. Since Hunyuan3D- P-Pinn is too slow in unfolding UVs, first reduce the number of faces of the objects and then Open3D's UV-unwrapping script to handle this task, which significantly improves the generation speed. For pose estimation, the inputs consist of RGB images of the target object, depth maps estimated from these images using Depth Pro, object masks that segment the target from the background, and reference 3D mesh model. The masked depth maps are calibrated to match the canonical scale of the mesh ensuring consistency between the observed geometry and the model. With the calibrated depth and object mask, Foun- dationPose employed to estimate the object's 6D) pose relative to the camera. From the aligned model, a scene pointmap is rendered, where each pixel encodes the 3D coordinates of the visible sur- face point in the camera frame. The outputs include the estimated 6D pose, the rendered pointmaps (both standalone and overlaid on the input image), and consolidated RGB-D samples with pose an- notations. This filtering method excludes failure cases such as incorrect orientations, mismatched viewpoints, object localization errors. To further ensure that the generated 3D mesh truly matches the appearance of the object in the 2D image, we place the 3D asset at the estimated pose and com- pute the LPIPS between the rendered object and the corresponding image region, retaining only samples with an LPIPS value below 0.3. This step filters out texture mismatches or reconstruction artifacts in the 3D assets. These two complementary checks, Mask IoU for geometric and pose align- ment and LPIPS for texture and appearance fidelity, form a reliable data filtering pipeline. After this filtering step, we use the estimated pose together with the 3D mesh, we additionally rendered videos showing the object rotating around its axis as well as the corresponding rotating pointmaps. Last, the remaining valid samples were consolidated and organized into the training dataset.
A.2 IMPLEMENTATION DETAILS
The training is conducted resolution of 512 x 512, with the LoRA rank fixed to 16. Multi- view conditioning is realized by uniformly sampling N = viewpoints at equal angular intervals, ensuring balanced geometric coverage. The model is trained on 8 NVIDIA H800 GPUs for 30k steps (approximately eight days), starting from the Flux-dev pretrained checkpoint. To incorporate multi- view conditions, we adopt the MMDiT architecture, which enables effective fusion of multimodal signals. To address the asymmetry between point maps and RGB images, we introduce Domain- specific LoRA and Texttagnostic Attention. Specifically, domain knowledge is decoupled via domain switcher and dual-LoRA structure: a Reference-LoRA learns general appearance features across all tokens, while a Domain-LoRA is activated only for point map tokens capture geometric information. In parallel, text-agnostic attention mask suppresses the influence of background information from text tokens on the point map branch, ensuring that point maps serve as purely geometric proxies, while RGB tokens can fully exploit both semantic and geometric cues. Our framework thus comprises two coordinated branches, one generating geometry-oriented point maps and the other producing photorealistic RGB images, achieving consistent and high-fidelity results across both domains.
A.3 THE USE OF LARGE LANGUAGE MODELS
In the process of preparing this paper, employed large language models (LLMs) to polish the writing. Specifically, LLMS were used to improve the clarity, fluency, and coherence of our expres- sions without altering the substantive content arguments. All core ideas, analyses, and conclusions
16

---

Published as a conference paper at ICLR 2026
MIN
Figure 11: Qualitative results with different 3D assets references.
were developed independently by the authors, while the LLM served solely as a language refinement tool ensure readability and academic style.
A.4 SUPPLEMENTARY QUALITATIVE RESULTS
We present additional experimental results using 3D assets references, shown in Figures 11 and 12. These examples further demonstrate the consistency with the 3D references and the high quality of our results.
17

---

Published as a conference paper at ICLR 2026
Figure 12: Qualitative results with different 3D assets references.
18


Reviews:
[{"review_id": "1fPoAX91WB", "reviewer": "Reviewer_fJ67", "summary": "The paper introduces RefAny3D, a 3D-asset–referenced diffusion framework that jointly generates an RGB image and its point map conditioned on a rendered multi-view set of a reference 3D asset plus text. Core ideas include: (1) a spatially aligned dual-branch architecture with shared positional encodings so RGB and point-map tokens stay pixel-aligned; (2) Domain-specific LoRA (Domain-LoRA for point-map tokens and Reference-LoRA for appearance/reference tokens) and a text-agnostic attention design to reduce cross-domain interference; and (3) a new pose-aligned dataset constructed by extracting objects with GroundingDINO/SAM, generating meshes with Hunyuan3D, estimating 6D pose with FoundationPose, and rendering point maps aligned to the images. Experiments show improved geometric/texture consistency versus baselines (DreamBooth, IP-Adapter, DSD, OminiControl), with comprehensive ablations and qualitative/quantitative comparisons.", "questions": "More details of the dataset construction pipeline should be provided, such as data filtering and robust checking.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The pipeline that pairs real images with mesh assets (via Hunyuan3D + FoundationPose) to obtain aligned point maps is practically useful for research on 3D-aware image diffusion.\n\n2. Domain-LoRA and Reference-LoRA help the network simultaneously generate point maps and RGB, improving stability and disentanglement.\n\n3. The paper is easy to follow, with sound motivation and diagrams.\n\n4. Ablations and comparisons are extensive; qualitative results show crisper textures and better geometric adherence than baselines.", "weaknesses": "1. Potential supervision noise from image-to-3D. The dataset relies on image-to-3D generators, which may not perfectly preserve reference fidelity, injecting bias into the training signal. The paper should quantify how frequently generator artifacts or pose errors degrade the learned 3D-conditioned diffusion, and propose mitigation.\n\n2. It remains unclear how much of the gain comes from generating point maps versus simply conditioning on multi-view images; a rigorous comparison against a “no-point-map” generator (or a latent geometric proxy) would clarify necessity.\n\n3. No user study. Given the goal (faithful, identity-preserving generation consistent with a 3D asset), a human preference study would strengthen claims about perceptual quality and faithfulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEbLr8UwX1", "reviewer": "Reviewer_CgtG", "summary": "This paper studies how to synthesize photorealistic images of a specific 3D object given the 3D object model, a straightforward and trivial problem given the current literature. The 3D object is first rendered to multi-view RGB images and point maps encoding the per-view geometry. Then, the stylized image is generated simply via conditioning on the rendered images and point maps.", "questions": "I only have one simple question, why is this method novel?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Product-wise, this task is very useful for commercializing generative models for rendering different photos of specific object products, mostly like a diffusion model shader.", "weaknesses": "There is too little technical contribution in this paper. Although the pipeline works, it is a straightforward engineering pipeline. I believe this is very practical for industry and product applications, but far below the standard of an ICLR paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nztegi5uKD", "reviewer": "Reviewer_RqVJ", "summary": "This paper proposes RefAny3D, a 3D asset-referenced diffusion framework for image generation. The model jointly synthesizes RGB images and point maps through a spatially aligned dual-branch architecture to ensure geometry–texture consistency. It further introduces domain-specific LoRA and text-agnostic attention to decouple visual and structural domains. Experiments show clear improvements over 2D reference-based baselines in geometric fidelity and visual quality.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly structured and well-written, with smooth logical flow from motivation to methodology and results.\n2. The work introduces a novel 3D asset-referenced diffusion framework that bridges the gap between 2D reference-based generation and 3D-aware synthesis.\n3. The experimental section is comprehensive, including qualitative, quantitative, and ablation studies that convincingly demonstrate the superiority of the method.", "weaknesses": "1. Motivation. The paper employs 3D assets as conditioning inputs to ensure geometry–texture consistency; however, generating 2D images does not inherently require multi-view conditioning. The authors should clarify the motivation for introducing 3D asset-based conditioning in this context.\n2. Task Definition. Given that the 3D asset is already available, there exist simpler approaches to achieve similar results—for example, rendering the desired viewpoint as a conditioning image and feeding it into an inpainting or editing model. The authors should justify why their proposed framework is necessary compared to these straightforward alternatives.\n3. Controllability. The proposed method lacks explicit control over the camera viewpoint of the generated image, which seems inconsistent with the premise of conditioning on a 3D asset. The authors are encouraged to discuss this limitation and explain how viewpoint control could be incorporated.\n4. Missing References. The Subject-Driven Generation section in the related work only covers a group of image-guided generation methods, where several related 3D-guided generation methods are missed, such as:\n- Wu R, Liu R, Vondrick C, et al. Sin3dm: Learning a diffusion model from a single 3d textured shape\n- Wu R, Zheng C. Learning to generate 3d shapes from a single example\n- Wang Z, Wang T, Hancke G, et al. Themestation: Generating theme-aware 3d assets from few exemplars\n- Wang Z, Wang T, He Z, et al. Phidias: A generative model for creating 3d content from text, image, and 3d conditions with reference-augmented diffusion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RzMoKgm2Cp", "reviewer": "Reviewer_tm5m", "summary": "This paper presents RefAny3D, a diffusion-based framework for image generation conditioned directly on 3D assets, enabling consistent alignment between generated images and 3D geometry.\nUsing multi-view renderings of the 3D asset along with their point maps as conditioning inputs, RefAny3D introduces a dual-branch diffusion architecture that jointly predicts RGB images and point maps, thereby establishing pixel-level correspondences between texture and geometry.\nThe model employs Shared Positional Encoding (SPE) for spatial correspondence between rgb image and point map, and combines Domain-specific LoRA with Text-agnostic Attention to decouple texture and geometry domains.\nFor training, the authors construct a pose-aligned dataset derived from Subjects200k using Grounding DINO, Hunyuan3D and Foundation Pose.\nExperimental results demonstrate consistent improvements over existing baselines on CLIP, DINO, GIM metrics, and GPT-based evaluations, particularly in maintaining geometric and textural consistency.", "questions": "The paper sets the number of reference views to N = 8.\nHave the authors tested other values (e.g., 4, 12, 16)?\nHow sensitive is the performance to the number and distribution of viewpoints?\nUnderstanding this sensitivity would clarify the scalability and generalization of the approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Novelty lies in using 3D assets as direct conditioning for diffusion models, addressing a key gap in 2D reference-based generation by ensuring geometric consistency.\n2. The dual-branch architecture is well-motivated and technically sound . Claims are strongly supported by comprehensive experiments and ablations.\n3. The paper is clearly written, well-organized, and easy to follow. The proposed architecture is well-illustrated, and component functions are explicitly defined .\n4. This work offers an effective and natural solution to the major challenge of spatial consistency in reference-based generation. The joint modeling of RGB and point maps significantly enhances 3D awareness, representing an important contribution to the field.", "weaknesses": "1. Missing \"Render-and-Edit\" Baseline: The introduction mentions a straightforward \"render-and-edit\" alternative, noting its foreground-background inconsistency issues . However, this intuitive baseline was not included in the experimental comparisons . Empirically demonstrating RefAny3D's superiority over this method would significantly strengthen the paper's claims.\n2. Dataset Transparency and Reproducibility: While the data construction pipeline is detailed , key statistics about the final dataset (e.g., scale, category coverage) and its release status are omitted. Providing these details is crucial for transparency and enabling reproducibility.\n3. Lack of Failure Analysis: The paper admits poor performance on non-rigid objects due to dataset limitations  but provides no qualitative examples or analysis. It is unclear if this is purely a data issue or an architectural limitation (e.g., rigid spatial alignment). Including such an analysis would make the work more complete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)