EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Authors: Zhuofan Zong
Dongzhi Jiang
Bingqi Ma
Guanglu Song
Hao Shao
Dazhong Shen
Yu Liu
Hongsheng Li
Venue: ICML 2025 poster (2025)
Keywords: personalized image generation
diffusion model
multimodal large language models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Significant achievements in personalization of diffusion models have been witnessed. Conventional tuning-free methods mostly encode multiple reference images by averaging or concatenating their image embeddings as the injection condition, but such an image-independent operation cannot perform interaction among images to capture consistent visual elements within multiple references. Although tuning-based approaches can effectively extract consistent elements within multiple images through the training process, it necessitates test-time finetuning for each distinct image group. This paper introduces EasyRef, a plug-and-play adaption method that empowers diffusion models to condition consistent visual elements (e.g., style and human facial identity, etc.) across multiple reference images under instruction controls. To effectively exploit consistent visual elements within multiple images, we leverage the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM), prompting it to capture consistent visual elements based on the instruction. Besides, injecting the MLLM's representations into the diffusion process through adapters can easily generalize to unseen domains. To mitigate computational costs and enhance fine-grained detail preservation, we introduce an efficient reference aggregation strategy and a progressive training scheme. Finally, we introduce MRBench, a new multi-reference image generation benchmark. Experimental results demonstrate EasyRef surpasses both tuning-free and tuning-based methods, achieving superior aesthetic quality and robust zero-shot generalization across diverse domains.

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Zhuofan Zong" 12
Dongzhi Jiang'
Binggi Ma
Guanglu Song?
Hao Shao
Dazhong Shen
Yu Liu
Hongsheng Li 145
Style Consistency
Human Facial Identity Consistency
sketch portroit mon'
man is smillng, Vincent van gogh style"
"Several fish are swimming the water" references
5 references
"Toylor Swift is riding horse"
Reference images
Character Identity Consistency
"A girl in school uniform plays girl school guitar uniform sits on bench with other musical instruments
music room with drum sets and sunflowers nearby"
Reference images
Reference images
references
Figure 1. EasyRef adopts a single multimodal LLM to capture consistent visual elements such as style, character identity, and human facial identity across various reference images and generates customization results with diffusion model.
Abstract
adaption method that empowers diffusion models condition consistent visual elements (e.g., style and human facial identity, etc.) across multiple reference images under instruction controls. To ef- fectively exploit consistent visual elements within multiple images, we leverage the multi-image comprehension and instruction-followinn capa- bilities the multimodal large language model (MILMM), prompting it capture consistent visual elements based on the instruction. Besides, inject- ing the MILM's representations into the diffusion process through adapters easily generalize to unseen domains. To mitigate computational costs and enhance fine-grained detail preservation, we introduce an efficient reference aggregation strat- egy and progressive training scheme. Finally, we introduce MRBench, new multi-reference image generation benchmark. Experimental sults demonstrate EasyRef surpasses both tuning- free and tuning-based methods, achieving supe- rior aesthetic quality and robust zero-shot gener- alization across diverse domains.
Significant achievements in personalization of dif- fusion models have been witnessed. Conventional tuning-fite free methods mostly encode multiple refer- ence images by averaging or concatenating their image embeddings as the injection condition, but such an mage-indeepeneet operation cannot per- form interaction among images to capture consis- tent visual elements within multiple references. Although tuning-based approaches extract consistent elements within multiple im- ages through the training process, it necessitates test-time finetuning for each distinct image group. This paper introduces RasyRef, a plug-and-play
effectively
CCUHK MMLab 'SenseTime Research 'Najjing Univer- sity of Aeronautics and Astronautics "Shanghai Al Labora- tory *CPII under InnoHK. Correspondence Hongsheng <hsli@ ec ww.cubk.edu.hkk
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
1. Introduction
Significant achievements in diffusion models (Rombach
Generation
al., 2022; Podell al., 2023; Betker et al., 2023; Esser et al., 2024; Ramesh et al., 2022; 2021; Saharia et al., 2022) have been witnessed because of their remarkable abilities to create visually stunning images. To improve the precision and controllability of diffusion models, researchers have been exploring personalized generation conditioned on a small number reference images, categorized into test-time tuning-basse baann methods (Hu et al., 2021; Ruiz et al., 2023; Gal et al., 2022) and tuning-free methods (Ye et al., 2023; Wang et al., 2024e;a; Zhang al., 2023; Li et al., 2025).
Avenge crbciling
Refeneecce Traages
X Aftribute Confatios
Srjject
Figure 2. Attrihute confusion and subject hallucination issues of the embedding averaging operation.
of reference images with arbitrary aspect ratios. To mitigate the computational demands imposed by the long context of multi-image inputs, we propose querying the MILM and aggregating reference representations through reference tokens the final layer of the MLLM architecture. Addi- tionally, to address the limitations of MILMS in capturing fine-grained visual details, we employ a progressive training strategy to enhance the MLLM's s capacity for fine-grained detail and human facial identity preservation. Unlike pre- vious methods that rely on sophisticated and task-specific feature encoders (Radford et al., 2021; Oquab et al., 2023; Deng et al., 2019), we demonstrate that the single MILM in EasyRef can effectively extract diverse consistent ref- erence representations, including style, character identity, object identity, and human facial identity, from an arbitrary group of reference images under instruction controls, while also exhibiting strong generalization ability. Finally, we in- troduce a multi reltiirreferncc consistent generation benchmark (MRBench) for multi-reference consistent image generation to evaluate our work and guide future research. Compared to prevalent tuning-free free JP-Adapter and tuning- based Low- Rank Adaptation (LoRA). EasyRef achieves superior aes- thetic quality and reference consistency performances across diverse domains and demonstrates robust generalization.
Despite the promise of tuning-free methods, they have sev- eral limitations. First, encoders tailored for specific refer- ence elements, such as style or facial identity, often rely on complex, task-specific architectures and specialized training processes. Second, most methods (Ye et al., 2023; Wang et al., 2024a; Qi al., 2024) are limited to training with single reference image and fail to fully encode consistent vi- sual representations from multiple references. For instance, as illustrated in Figure 2, IP-Adapter (Ye et al., 2023) with average embeddings encounters two issues: (1) Attribute confusion arises when reference images have overlapping subjects (e.g., a dog partially or fully covering a chair), causing the averaged features to incorrectly blend attributes (e.g., the dog acquiring the chair's color and vice versa). (2) Subject hallucination occurs when positional discrepancies between reference subjects (e.g., a dog positioned in front of a chair versus seated on it) mislead the fusion method, resulting in the diffusion model erroneously generating an extra dog-shaped object on the chair. Although tuning-based methods can extract consistent elements within multiple im- ages through the training process, necessitates finetuning for each distinct image group.
In summary, our contributions are threefold: (1) We in- troduce EasyRef, a plug-and-play method that empowers diffusion models to condition various consistency across multiple reference images under instruction controls. (2) We propose an efficient reference aggregation strategy and a progressive training scheme to mitigate computational costs and enhance the MLLM's fine-grained perceptual abilities. (3) A novel MRBench is proposed for evaluating diffusion models in multi-reference consistent generation scenarios.
This paper introduces EasyRef, a plug-and-plat adaption method that empowers diffusion models to condition consis- tency (e.g.. style, character identity, human facial identity, etc.) across multiple reference images under instruction controls. Conventional methods encode consistent elements across reference images through averaging or concatena- tion (Shi et al., 2024), but these image-indee inngee-ndeeendent opera- tions fail to capture desired visual elements through effec- tive image interaction under explicit controls. To alleviate this issuc, we leverage the multi-image comprehension and instruction-follovinng capabilities of the multimodal large language model (MLLM) (Liu al., 2024b;a; Li et al., 2023; Chen et al., 2024), prompting it to capture consistent visual elements based on the instruction. Besides, injecting the MLLM's representations into the diffusion process through adapters can easily generalize to unseen domains, mining the consistent visual elements within unseen data. FasyRef also inherits the MLLM's ability to process arbitrary number
2. EasyRet Ref
2.1. Methodology
As illustrated in Figure 3, EasyRef comprises four key com- ponents: (1) a pretrained diffusion model for conditional image generation, (2) pretrained multimodal large lan- guage model (MLLM) for encoding a set of reference im- ages and the instruction, (3) a condition projector that maps the representations from the MLLM into the latent space of diffusion model, and (4) trainable adapters for integrating
2

---

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Multimodal LLM
Diffusion Model
Generated Image
LLM Final Layer
Projector
Reference Tokens
Image Condition
Large Language Model
Text Condition
U-Net
Cross Attention
Vision Encoder
Tokenizer
Text Encoders
U-Net
TAB
Visualize a scene that closely resembles human face of provided images, capturing details in this prompt: A portrait woman with blonde hair and blue eyes.
A portrait of a woman with blonde hair and blue eyes.
Reference Images
Instruction
User Prompt
Noise
Figure 3. Overview of EasyRef with Stable Diffusion XL. EasyRef extracts consistent visual eliments from multiple reference images and the prompt via MLLM, injecting the condition representations into the diffusion model through cross-atiention layers. We only plot cross-attention layer for simplicity.
image conditioning embedding into the diffusion process.
to achieve efficient inference. However, all parameters of LLM must be trained to interpret these newly added tokens. To enhance training efficiency, we append Frer to the con- text sequence Fi-1 at the final layer of LLM, keeping all previous L.M layers frozen during alignment pretraining:
Reference Representation Encoding. Existing mainstream approaches (Ye et al., 2023; Qi al., 2024; Wang et al., 2024a;e) mostly average or concatenate the image embed- dings of all reference images as the condition. These image- independent operations cannot effectively capture consistent visual elements among reference images. It also fails to en- code reference information under explicit instruction control and causes the spatial misalignment issue as presented in Figure 2. To alleviate this issue, we propose to leverage the multi-image comprehension and instruction-followii capa- bilities of the MLLM to encode multi-reference inputs based on the instruction. We adopt the state-of-the-art Qwen2-VL. 2B our MLLM in this work. The MLLM consists of 1-layer large language model (LL.) and vision encoder capable of handling images with arbitrary resolutions. The input image is initially converted into visual tokens with the vision encoder. Then we employ an instruction and integrate all images into the instruction, which explicitly encourages the MILM focus on desired consistent visual elements within the reference images. These multimodal input tokens are subsequently processed by the L.M.
F Concat F,
Then we employ bi-directional self-attention facilitate the propagation of representations across the reference im- ages in the final layer, followed by multi-layer perception network (MLP):
F (LP(BiiAttennion(())))
(2)
where we omit the residual addition operations for simplicity. Next, we split Fy into the updated representations F, and the encapsulated reference tokens F
Fr, F Split(F")).
Finally, we project F' rer through a trainable MLP condition projector obtain the final conditioning vector c;:
C; MIP((F).),
(4)
Efficient Reference Aggregation. Increasing the number of reference images inevitably raises the number of visual to- kens in the L.M. This extended context length substantially elevates the computational cost for the diffusion model. We propose to encapsulate the reference representations into N' learnable reference tokens F E DDD in the LLM
Reference Representation Injection. The text conditions are injected into the pretrained diffusion model through cross-attention layers. Following IP-Adapter, we introduce a new cross-attention layer into each cross-attention layer of the U-Net. Given the latent features X, text conditions
3

---

RasyRef: Oeni-Generrlizee Group Image Reference for Diffusion Models via Multimodal LLM
is applied the conditioning images. We keep the original aspect ratio for each target image.
Table 1. Comparison of FasyRef against other counterparts.
Consistency
Multiple referenees
Method
Instraction
Tuniing-fce
eacoding
Training Supervision. We use the same training objective as the original stable diffusion model:
LoRA (Hu al. 2021)
x
IP-Adapter (Ye et al,, 2023)
x
x
Kosmos-Gi G (Pan
2023)
EasRRef
MoMA (Song et 2025)
X
x
L = ee |/e ex Ct ca O)I2,
(6)
where c₁ and c2 denote the text condition and image condi- tion, respectively.
Ct. and image conditions C₁2 the injected features X are computed by the cross-attention layer as follows:
QK?
QK?
2.3. Discussion
X Softmax
V
Softmax
V, (5)
Vd
vd
Comparison with Other Methods. We compare EasyRef with LoRA, IP-Adapter (Ye et al., 2023), and other image personalization methods that adopt MLLMs in Table 1. First, the task differs significantly. Compared to Kosmos-G (Pan et al., 2023) and MoMA (Song et al., 2025), both EasyRef and LoRA possess the capability to encode consistent visual elements, such as style, subject, and human face, within group of reference images. While Kosmos-G primarily focuses on the combination of different elements from mul- tiple images, MoMA utilizes a MLLM to extract the subject feature of the single reference for subject-driven generation. Second, the reference aggregation paradigm is different. Kosmos-G employs an elaborate encoder-decoder Align- erNet, while MoMA utilizes learnable tokens and subject- aware masked attention to bridge the MLLM to the diffusion U-Net. In contrast, we demonstrate that simply using ref- erence tokens in the final layer of the MLLM can provide sufficient reference information and efficiently inject condi- tions into the U-Net. Additionally, we propose a progressive training scheme for the MLLM to enable the extraction of fine- ane-ggrinedd details (e.g.. human facial details).
where K = c, We and V = c, W. Both We and W, are
newly added trainable parameters.
2.2. Progressive Training Scheme
Alignment Pretraining. To facilitate the adaption of MLLM's visual signals to the diffusion model, we construct a large-scale dataset containing 13M high-quality image- text pairs, including LB (Schuhmann et al., 2022) and other internal datasets for the alignment pretraining. During the pretraining phase, we only optimize the final layer and reference tokens of the MLLM along with the newly added adapters and condition projector while preserv- ing the capabilities of the initial MILMM and diffusion model. The model is pretrained for 300k iterations. We center crop 1024 x 1024 pixels of the input image.
Single-reference Finetuning. Following alignment pre- training, the MILM is trainable and subjected to single- reference finetuning. Specifically,, we unfreeze the vision encoder and all layers of the MLLM to enhance its capac- ity for fine-grained visual perception at the second stage. We additionally incorporate trainable Low-Rank Adaption (LoRA) layers to attention layers of the frozen U-Net. Build- ing upon the aforementioned pretraining dataset, we aug- ment the training data with 4M real-world human images from LAION-5B, utilizing cropped face regions as condi- tioning inputs for better human facial identity preservation. Thanks to the efficient aggregation design and alignment pretraining, we only train the model for 80k iterations.
3. Multi-Reference Generation Benchmark
Data Source. Our data source encompasses two parts: (1) We collect images from several large-scale publicly available datasets, including LAION-2B (Schuhmann et al., 2022), COYO-700M (Byeon et al., 2022), and DataComp- IB (Gadre al., 2024). (2) We also constructed a tag list that includes celebrity names, character names, styles, and subjects, and collected filtered images from diverse sources based on this list. Images sharing the same tag are set into the same group. Subsequently, these images were used to train LoRA models for each group and we incorporated high-quality images generated by these models into our training data.
Multi-reference Finetuning. The third stage enables the MLLM to accurately comprehend the consistent visual el- ements across multiple image references under instruction controls. Training is performed on a curated dataset com- prising image groups, where each group contains multiple images of the same topic (e.g., a Tesla Cybertruck, etc.) with varying aspect ratios. During training, one image from each group is randomly selected as the optimization target, while the remaining ones serve as the conditioning inputs. Data augmentation, including random shuffling and truncation,
Dataset Construction. To generate aligned text captions of the images, synthetic captions generated by Qwen2-VL- 7B using the instruction "Give a brief, concise and precise caption for this image." " were adopted for each sample. To achieve controllable reference encoding, we annotate the
4

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Style
Human face
Character
Object
Live subject
STOP
STOP
STC
Figure Example images for cach group our proposed MRBench, We only present images for each group.
manually annotate a subset of the collected samples. The valid images are labeled as positive, while the images be filtered are labeled negative. We then train CLIP-based binary classifier to filter these negative instances. Finally, we perform deduplication to eliminate redundant images.
40
Percent
Data Clustering. We construct facial identity groups and character identity groups based on a predefined tag list. For images containing human faces or characters, we use Co- DETR (Zong al., 2023b) detect face bounding boxes and character body boxes. Face embeddings and character embeddings are then extracted using ArcFace (Deng et al., 2019) and CLIP, respectively. For each facial or character identity group, we calculate the cosine similarity between the embeddings of the current group and those of newly collected samples. The similarity scores for each new sam- plc are summed, and the group with the highest score is assigned to the sample. Samples with low similarity scores are discarded to ensure relevance. Groups for style and sub- ject (e.g., common objects and animals) are constructed in a similar manner. We use CLIP to extract style embeddings and DINOv2 (Oquab et al., 2023) to extract subject embed- dings. Unlike the previous strategy, a new group created if collected sample does not match any existing group. Be- sides, do not only assign the group with highest score the sample and sample can be assigned to multiple groups. For instance, an image of a Bernese Mountain Dog can be- long to both the Bernese Mountain Dog Group and the Dog Group. We find that subjects belonging to the same category but not the same instance can grouped together. How- ever, this does not significantly impact subject-driven perfor- mance. Multi-image subject-driven generation requires that the main subjects across multiple reference images represent the same instance, and the generated outputs should contain
20
10
9
12
15
18
21
24
Number images per group
Figure 5. Distribution of our curated datasct.
consistent visual elements of each group using Qwen2-VL 72B. Then we explicitly ask the MLLM to capture desired references by incorporating the annotated consistent ele- ments into instructions during training. The resulting dataset comprises 3,918.984 images organized into 994,275 groups. We set the maximum and minimum group sizes as 16 and 2, respectively, ensuring balanced group size distribution within the dataset. Figure 5 illustrates the group distribution of our curated dataset.
Data Filtering. We also employ a series of efforts for data cleaning. Initially, images with low resolution, poor aesthetic scores, extreme aspect ratios were excluded. Then we filter out image-text pairs with low CLIP image- text similarity scores using CLIP ViT-L/14 (Radford et al., 2021). Since the aforementioned filtering methods may not effectively identify special patterns such as image collages, high-quality images with dense text, watermarks, etc., we
5

---

RasyRef: Omni-Gencralize Group Image Reference for Diffusion Models via Multinodal LLM
Trage Prompt
EogyBe (Dora)
image varichion rriicion
5D UnGLEP
Varantlle b ofusiinn
cardindly
Open
T2T Adgptoe style
Uni Gentroi
Sea
ContreBart Shaftic
Contrellert Reference only
Adayhar
BB
Figure 6. Comparisons of EasyRef with other counterparts in various single -mage reference scenarios. The same image prompts in (Ye et al., 2023) are used for clear comparisons.
the same instance. When multiple reference images depict subjects from the same category but not the same instance, using a target image with subject from the same category is still a reasonable training approach.
Dataset Samples. As shown in Figure 4, we provide some samples of MRBench. To ensure the benchmark's 's high quality, we manually collect, verify, and integrate selected images into the MRBench. The synthetic captions are gen- erated using DWLn22VL-7B. and detailed descriptions are removed from the captions to prevent content leakage.
Benchmark Splits. The collected image-text pairs are di- vided into the training dataset, the held-in evaluation set, and the held-out evaluation set. We first sample 50 image groups to construct the held-out evaluation set to evaluate the model performance on unseen data. The number of images in each set varies. There are a total of 300 images in the held-out evaluation set. To avoid data contamination, we use the CLIP image embeddings of the held-out images to retrieve and filter out similar images from the remaining method with multi-reference gener-
Evaluation Protocol. For each group of evaluation data, each image can be chosen as the target image and others are regarded as the reference images. When evaluating the held- in and held-out sets, we use the reference images and caption of the target image to generate two images for each group. Then we employ conventional metrics, including CLIP-I, CLIP-T, and DINO-I, to measure the alignment between generated images and the corresponding target images or prompts. We mainly consider CLIP-I and DINO-I for the Image-image alignment, which computes the similarities of image embeddings from CLIP ViT-L/14 and DINOv2- Small (Oquab et al., 2023). For the image-text alignment, we adopt the CLIPScore (Hessel et al., 2021).
data. To compare ation approaches that require finetuning (e.g., LoRA), we randomly selected 300 groups from the remaining 994,215 groups to form a test set of 1434 samples. Unlike the held- out set, only a randomly selected image serves as the target image in each group. All reference images of the held-in in split and other 993,915 groups construct the training set. To improve the aesthetic quality of generated images, only images with aesthetic scores higher than 5.5 can be used as the training target images. There are 2.117.435 valid training target images in the training set, with an average of 2.1 images per group.
Comparison with Other Benchmarks. The Dream- Bench (Ruiz et al., 2023) benchmark is a pioneering dataset for evaluating multi-reference generation. However, there are several key differences between it and MRBench. First, MRBench covers a more diverse range of categories, span- ning five distinct types, while the DreamBench focuses solely on subject-driven generation for objects and live sub- jects pets. Second, MRBench comprises 300 meticu- lously curated images, compared to only 158 images in the DreamBench. Third, the evaluation protocols differ signifi- cantly. In MRBench, each image, along with its caption, can be selected as the target, while the remaining images in the
Benchmark Statisties. We categorize the consistent visual elements of each group in the benchmark into five cate- gories: style, human faces, characters, objects, and live subjects/pets. Specifically, the held-out dataset includes 25 style groups, 10 human face groups, 5 character groups. 5 object groups, and 5 groups of live subjects.
6

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Reference Images
BasyRef
IP A-Aoopeer
Reference Images
EasyRef
IP-Adepeer
persce saking Bifeel Tower
ren and dhee avelickiigg crush oge sheme"
"Ancthronu salkaan
"boeld Tway?"
cha oile cits
actrity teach
werton
"Aderbble alemiial parple
kay sity grord sarfece
stylized pertruit veterceler styla"
wizard
Figure 7. Visualization of generated samples with various mefti-reference inputs, Best viewed by zooming in. Note that the base diffusion model cannot generate these reference images of celebrities and characters with text prompts.
group serve as the conditioning inputs. Visual consistency is then evaluated between the generated images and the single target image. In contrast, the DreamBench treats all images as references and evaluates the average similarity between the generated and reference images.
Table 2. Performance comparisons on COCO validation set. Meth- ods with * use CLIP embeddings and tend achieve higher scores of CLiP-based metries due to its preference.
Method
CLIP-
CLIP-T
DINO-11
Training from scratch
Open unCLIP (Ramesh et 2022)
4. Experiments
Kandinsky-2- 1-1 (Arseniy ei al., 2023)
Vensatile Diftusion (Xu 2023)
0.830
0.855
0.587
0,599
Flurtuning
4.1. Implementation Details
Image Varistions SD unCLIP
0.548
0.810
D.584
Training. We build our EasyRef with the established Sta- ble Diffusion XL (Podell et al., 2023) model, utilizing the state-of-the-art Qwen2-VL-2B (Wang et al., 2024d) as the MLLM. The resolution of an input image with arbitrary aspect ratio processed by the MLLM can not exceed 336 x 336. We introduce 64 reference tokens in the MLLM. We also employ a drop probability of 0.1 for both text and im- age prompts independently, and a joint drop probability of 0.1 for simultaneous removal of both modalities. We simply treat a square black image as the empty image condition if the image condition is dropped. For the implementation of LoRA comparison, we fine-tuned the model using the reference images and employed a LoRA rank of 32.
Adapters
Uni-Control Cottrrooolle (Zhao et al., 20144 (Gilobal Control)
0.736
D.506
T21-Adapter (Mou et 2024) (Style)
0.548
0.485
ControlNet Shuffle (Zhang et al., 2023)
0.616
0.421
IP-Adapter (Ye al., 2023)
0.828
0.588
SDDpter--DDD (Ye et al., 2023)
0.836
0.617
0.650
EasyRef
0.876
0,621
0,873
4.2. Quantitative and Qualitative Results
Single-image Reference. We quantitatively compare our method with other counterparts in single-reference scenarios using the COCO 2017 validation dataset (Lin et al., 2014), which comprises 5000 image-text pairs. We use the check- point trained by single-reference finetuning. As shown in Table 2, EasyRef consistently outperforms other methods in both CLIP-T and DINO-I metrics, demonstrating superior alignment performance. For instance, our model signifi- cantly surpasses the IP-Adapter-SDXL by 0.223 DINO-I score. Note that IP-Adapter utilizes CLIP image embed- dings for conditioning, its generated images may exhibit a bias towards CLIP's preference, potentially increasing scores when evaluated using Cased metrics. We fur- ther conduct qualitative comparisons using some reference
Evaluation. During inference, we leverage DDIM (Song et al., 2020) sampler with 30 steps and guidance scale (Ho & Salimans, 2022) of 7.5. As the original IP-Adapter does not support multi-image references, erage of the CLIP embeddings as the conditioning input. The reference images presented in our visualizations are excluded from the training data. We present more experi- mental results in the Appendix A.2.
employed the
7

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
Reference Images
Reference Images
mountuik in background
"Sunglasses on
"A stuffed onimal in the snow"
"A toy surrounded by sunflowers'
the whoot Field"
"A with the background"
Pigure 8. Visualization of generated samples on Dream
Table Evaluation comparisons on MRBench. "failed" means
Table Quantitative comparisons on the DreamBench.
LoRA fails to generalize to the unseen held-out set.
Method
CLIP-I
CLIP-T DINO-1 f
Method
CLIP-I CLIP-T DINO-I
Real Images
0.885
0.774
Held-in split
Tuning-based
LoRA (Hu al., 2021)
0.831
0.715
0.654
Textual Inversion (Gal et al., 2022)
0.780
0.255
0.569
IP- DP-AdapeerSDDX (Ye el ul., 2023)
0,768
0.632
0,527
DreamBooth (Ruiz et al., 2023)
0.803
0.308
0.668
EasyRef
0.856
0.739
0.675
Taming-free
Held-out split
BLIP-Diffusion (Li et al., 2024a)
0.779
0,300
0.594
LoRA (Hu et al., 2021)
failed
failed
failed
Re-Imagen (Chen 2022)
0.740
0.270
0.600
IP-Adapter-SDXX (Ye et al., 2023)
0.813
0.646
0.611
IP-Adapter (Ye al., 2023)
0.793
0.330
0.612
EasyRef
0.838
0.715
0.653
SSR-Enooder (Zhang el al., 2024)
0.308
0.612
X-ECLIPSE (Patel et al., 2024)
0.307
0.613
Kosmos- (Pan et al., 2023)
0.822
0,250
0.618
images that encompass various consistent elements. As presented in Figure 6, our method achieves better aesthetic quality and consistency with the original image prompts.
MoMA (Song et al., 2025)
0.803
0.348
0.618
EasyRef Ref
0.807
0.302
0.651
with IP-Adapter and LoRA in terms of reference consistency and aesthetic quality. The human evaluation is conducted on our proposed MRBench. Human evaluators were pre- sented with pairwise image comparisons, one generated by EasyRef and the other by competing model, under blind conditions to ensure fairness. As illustrated in Figure EasyRef outperforms other models in both image-reference alignment and visual aesthetics in user study. This demon- strates EasyRef's 's capacity to generate high-fidelig images that conform to the provided reference images.
Multi-image References. We first compare our method with IP-Adapter and the tuning-based LoRA on the MR- Bench in Table On the held-in split, the tuning-free EasyRef consistently achieves better performances than the tuning-based approach LoRA. In the zero-shot setting, the results demonstrate our method surpass the IP-Adapter with embedding averaging in alignment with the reference im- ages and user prompt. We also present the qualitative visu- alizations in Figure?. 7.
Then we present the single-entity subject-driven genera- tion performance comparisons on the DreamBench (Ruiz et al., 2023). We follow the original evaluation settings of DreamBooth. As presented in Table 4, EasyRef yields a comparable CLIP-I score to the tuning-based DreamBooth method. Moreover, it surpasses other tuning-free methods in DINO-I, which is the metric preferred by DreamBooth for evaluating subject preservation. We present the gener- ated results in Figure 8. These experiments demonstrate our framework is capable of fully mining consistent visual ele- ments among multiple reference images while maintaining strong generalization ability.
4.3. Ablation Study
Scaling the Number of Reference Images. Figure 10 il- lustrates EasyRef's performance across varying inference lengths. The model exhibits slightly robust performance across varying numbers of references when the number of reference images is within the training constraint. Specifi- cally, the performances continue to increase the number of references increases within the training constraint. How- ever, performance degrades when the number of references exceeds this constraint. This due to the limited number of groups with more than 16 images during training and the long-context finetuning may be inadequate. Moreover, the
Human Evaluation. We systematically evaluate RasyRef
8

---

RasyRef: Omni-Generllized Group Image Reference for Diffusion Models via Multimodal LLM
Aesthetic Quality
Reference Consistency
References
EasyRef
w/ instruction
w/o prompt
100
Winrate [%6
60
40
"A bride and groom kissing, surrounded by lemon
yellow white flowers."
20
Figure 11. Impact of the multimodal instruction design.
IP-Adapter
LoRA
Figure 9. Comparisons of human preference evaluation on our MR- Bench. EasyRef can surpass other methods across the aesthetic quality and reference alignment.
Ref References
EasyRef
w/o stage
w/o stage
C1P-2
DONO-T I
Lundtry
"A cute panda sitting in a garden.'
019-
0.31
0.50
0.00
0.50
0.00
Trakin Crop SD
0.50
Tetnng Smap
6/0
16 28
56
112
16 26
X
Inference Group Size
partnait of woman."
Figure 10. Evaluation of inference group size scaling. We ran- domly select 112 reference images and target image-text pair with the same topic. "Latency" in the figure is measured in seconds per image.
Figure 12. Effect of the progressive training scheme. "stage 1" and "stage 2" denote the alignment pretraining stage and single- reference finetuning stage, respectively.
5. Conclusion
This paper presents EasyRef, a plug-and-pla adaption method that empowers diffusion models to condition con- sistent visual elements (e.g., style and human facial iden- tity, etc.) across multiple reference images under instruc- tion controls. Our approach can effectively capture con- sistent visual elements within multiple reference images and the text prompt through an multi-image comprehension and instruction- instruction-following followin paradigm, while simultaneously maintaining strong generalization capabilities due to the integration of adapter- based injection. The proposed effi- cient reference aggregation strategy and progressive training scheme further enhance computational efficiency and fine- grained detail preservation. Through extensive evaluation on popular benchmarks and our newly introduced MR MRBench, EasyRef has demonstrably surpassed both tuning-free free and tuning-base based approaches, in terms aesthetic quality and zero-shot generalization across diverse domains.
inference efficiency of EasyRef further evaluated and we find it still maintains acceptable efficiency with 56 reference images due to the effecient token aggregation design.
Multimodal Instruction Input. An ablation study was con- ducted to investigate the design of multimodal input to the LLM. As shown in Figure 11, the inclusion of instructions improves generation performance. The instruction lever- ages the MI.LM's Instruction-following ability to enable it attend desired contents within the reference images un- der explicit controls. Furthermore, incorporating the image prompt can exploit the cext-understanding capacity of the MILM and enhance text-image alignment.
Progressive Training Scheme. We visualize the impact of each stage on model's ability to capture fine-grained vi- sual details Figure 12. For some reference contents, such as the pixel art style, EasyRef without alignment pretraining or single-reference finetuning maintains comparable per- formance. For reference images involving human facial identity preservation (e.g., Taylor Swift), we find significant alignment improvements when adopting all training phases. We also investigate and visualize the effect of our training scheme in the Appendix A.2.
A Acknowledgements
This study was supported in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd., CUHK-led Inno- Centre under the InnoHK initiative the Innovation and Technology Commission of the Hong Kong SAR Govern-

---

RasyRef: Omni-Generallzed Group Image Reference for Diffusion Models via Multimodal LLM
ment, and in part by NSFC-RGC Project N. CCHHK449/224. Hongsheng Li is Pl of CPII under the InnoHK. This work was supported in part by the National Natural Science Foun- dation of China (Grant No. 62406141).
large language models. arXiv preprini arXiv:339./17102 2023.
Gadre, S. Y., Ihharco, G., Fang. A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang. J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Impact Statement
This paper presents work whose goal advance the field of Computer Vision and Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 00611000.000 2022.
References
Guo, Z., Wu, Y., Zhuowei, C., Zhang, P., He, Q., et al. Pulid: Pure and lightning id customization via contrastive alignment. Advances in neural information processing systems, 37:36777-36804 2024.
Arseniy, S., Anton, R., Aleksandr, N., Vladimir, A., Igor, P., Andrey, K., and Denis, D. kandinsky 2.1, 2023.
Bai, J., Bai, S., Yang. S., Wang, S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
He, Z. Sun, B., Juefei-Xu, F., Ma, H., Ramchandani, A., Cheung, V., Shah, S., Kalia, A., Subramanyam, H., Zareian, A., et al. Imagine yourself: Tuning-free personal- ized image generation. arXiv preprint arXiv:2409.1338 13346, 2024.
Betker, J., Goh, G., Jing, I.., Brooks, T., Wang, J., Li, I.., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., al. Improving image generation with better captions. Computer Sci- ence. htlps://cdn. openai. com/papers/dullle-3 pdf, 2(3):
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.,, and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning. arXiv preprint arXi::2104.0877 104.08718, 2021.
8, 2023.
Byeon, M., Park, B., Kim, H., Lee, S., Back, W., and Kim, S. Coyo-700m: Image-text pair dataset. https: //tp://ggttt///////e com/ com/kaaaobrain/ doyo-datasst,,0000
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207 2022.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems,
Chen, W., Hu, H., Saharia, C., and Cohen, W. W. Re- imagen: Retrieval-augmented text-to-image generator. arXiv preprint ar arXiv:2209 14491, 2022.
2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (0o::/20060685,, 2021.
Chen, Z., WU, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Huang, J., Dong, X., Song. W., Chong, Z., Tang, Z., Zhou, J., Cheng, Y., Chen, L.., Li, H.,, Yan, Y., al. Consistentid: Portrait generation with multimodal fine-graine identity preserving. arXiv preprint arXiv:2404. 16771, 2024a.
Deng, J., Guo, J., Xue, N., and Zafeiriou, S. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-46999 2019.
Huang, Y., Xie, L., Wang, X., Yuan, Z., Cun, X., Ge, Y..
Zhou, J., Dong, C., Huang, R., Zhang, R., et al. Smartedit: image editing with multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8362-8371, 2021, 2024b.
Exploring complex instruction-
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Jiang, D., Song, G., Wu, X., Zhang, R., Shen, D., Zong, Z., Liu, Y., and Li, H. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint arXiv:2404 03653, 2024.
Fu, T.-J., Hu, W., Du, X., Wang, W. Y., Yang. Y., and Gan, Z. Guiding instruction- based image editing via multimodal
10

---

EasyRef: Omni-Generallzeed Group Image Reference for Diffusion Models via Multimodal LLM
Jiang, D., Guo, Z., Zhang, R., Zong, Z., Li, H., Zhuo, L., Yan, S., Heng, P.-A., and Li, H. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. XrXiv preprint arXiv:2505.00700: 2025.
Liu, H., Li, C., Li, Li, Li, B., Zhang, Y., Shen, S., and Lee, J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1931-1941, 2023.
Lu, 7., Zhou, A., Ren, H., Wang, K., Shi, W., Pan, J., Zhan, M., and Li, H. Mathgenie: Generating syn- thetic data with question tack-translation for enhancing mathematical reasoning of llms, 2024a. URL https:
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
/ //arxiv. org/ abs/////////////
Lu, Z., Zhou, A., Wang, K., Ren, H., Shi, W., Pan, J., Zhan, M., and Li, H. Mathcoder2: Better math reasoning from continued pretraining model-translated mathe- matical code, 2024b. URL
Li, D., J., and Hoi, S. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36, 2024a.
abs / 2410.00000000 08196.
Li, J., Liu, X., Zong, 7.,, Zhao, W., Zhang, M., and Song, J. Graph attention based proposal 3d convnets for action detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 4626-4633, 2020.
Lu, Z., Zhou, A., Wang. K., Ren, H., Shi, W., Pan, J., Zhan, M., and Li, H. Step-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning, 2024c. URL https: / aat.i.iiiiivv//.ii//...aaaaaaava org/ abs /200:.0/00000000
Li, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-ing image pre- teeeirrining with frozen image encoders and large language models. In International conference on machine learning, pp. 19730 19742. PMLR, 2023.
Lu, Z., Yang, Y., Ren, H., Hou, H., Xiao, H., Wang, K., Shi, W., Zhou, A., Zhan, M., and Li, H. Webgen-bench: Evaluating Ilms on generating interactive and functional websites from scratch, 2025. URL https: org/ abs 2500. 03733.
Li, M.. Yang. T., Kuang, H., Wu. J., Wang, Z., Xiao, X., and Chen, C. Controlnet++:: Improving conditional con- trols with efficient consistency feedback. In European Conference on Computer Vision, pp. 129-147. Springer, 2025.
Ma. B., Zong. Z., Song, G., Li, H., and Liu, Y. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406. /1831, 2024.
Li, W., Xu, X., Liu, J., and Xiao, X. Unimo-g: Unified im- age generation through multimodal conditional diffusion. arXiv preprint 24t1:22001.33388 2024b.
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 4296-4304, 2024.
Li, Z., Cao, M., Wang. X., Qi, Z., Cheng, M.-M., and Shan, Y. Photomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8640-8650, 2024c.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec. M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J.,, Perona, P., Ra- manan, D.,, Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Pan, X., Dong, L., Huang, S., Peng, Z., Chen, W., and Wei, F. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992. 2023.
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L.,, Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07573 2023.
Patel, M., Jung, S., Baral, C., and Yang, Y. X-eclipse: Multi-concept personalized text-to-image diffusion mod- els by leveraging clip latent space. arXiv preprint arXiv:2402.05 2024.

---

EasyRef: Omni-Generallzeed Group Image Reference for Diffusion Models via Multimodal LLM
Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Advances in Neural Information Processing Systems, 35: 25278-25294 2022.
Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L., Liu, Y., and Li, H. Visual cot: Unleashing chain- of-thought reasoning in multi-modal language models. arXiv preprint WxXiv:2403.66990 2024a.
Qi. T., Fang, S., Wu, Y., Xie, H., Liu, J., Chen, L.,, He, Q.. and Zhang, Y. Deadiff: An efficient stylization diffusion model with disentangled representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8693-8702, 2024.
Shao, H., Wang, S., Zhou, Y., Song, G., He, D., Qin, S., Zong, Z. Ma, B., Liu, Y., and Li, H. Vividface: A diffusion-based hybrid framework for high-fidelity video face swapping. arXiv preprint arXiv:2412.11275 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Shen, D., Song. G., Zhang, Y., Ma, B., Li, L., Jiang, D., Zong, Z.,, and Liu, Y. Adt: Tuning diffusion models with adversarial supervision. arXiv preprint arXiv:2504.11423, 2025.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831, Pmlr, 2021.
Shi, J., Xiong, W., Lin, Z., and Jung, H. J. Instantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8543- 8552, 2024.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 1(2):3, 2022.
Song, J., Meng. C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Ren, H., Zhan, M., Wu, Z., and Li, H. Empowering character-level text infilling by climinating sub-tokens. arXiv preprint arXiv:2405.17000
Song, K., Zhu, Y., Liu, B., Yan, Q.. Elgammal, A., and Yang. X. Moma: Multimodal IIm adapter for fast personalized image generation. In European Conference on Computer Vision, pp. 117-132. Springer, 2025.
2024a.
Ren, H., Zhan, M., Wu, 7., Zhou, A., Pan, J., and Li, H. Reflectioncoder: Learning from reflection sequence for enhanced one-off code generation. arXiv preprint vrXiv:2405.17055 2024b.
Tong, S., Liu, Z.,, Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pallern recognition, pp. 10684-10695, 2022.
pp.
Wang. H., Spinelli, M., Wang, Q., Bai, X., Qin, Z., and Chen, A. Instantstyle: Free lunch towards style- preserving in text-to-image generation. arXiv preprint arXiv:240..02733, 2024a.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510, 2023.
Wang, H., Xing, P., Huang, R., Ai, H., Wang, Q. and Bai, X. Instantstyle-pluu Style transfer with content- preserving in text-to-image generation. arXiv preprint arXi::4407.00888 2024b.
Saharia, C., Chan, W., Saxena, S., Li, I.., Whang. J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic exxtto-image dif- fusion models with deep language understanding. Ad-
Wang, K., Ren, H., Zhou, A., Lu, 7.,, Luo, S., Shi, W., Zhang, R., Song, L., Zhan, M., and Li, H. Mathcoder: Seam- less code integration in Ilms for enhanced mathematical reasoning. arXiv preprini arXiv:2310.03731 2023.
vances in neural information processing systems, 35: 36479-36494, 2022.
Wang, K., Pan, J., Shi, W., Lu, Z., Ren, H., Zhou, A., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024c. URL https: / /opperevv ew, net/ fo rumm?? d=QWTCcxMpPA.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open sarge-scale dataset for training next generation image-text models.
12

---

EasyRef: Omni-Gennrrlizeed Group Image Reference for Diffusion Models via Multimodal LLM
Wang, K., Pan, J., Wei, L., Zhou, A., Shi, W., Lu, Z., Xiao, H., Yang, Y., Ren, H., Zhan, M., and Li, H. Mathcoder- vl: Bridging vision and code for enhanced multimodal mathematical reasoning, 2025. URL https: / arxiv. org/ htgg/bb//t /albs//5500 2505. 10557.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672. 2023.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409. 121.099..1191... 2024d.
Zhang, Y., Song, Y., Liu, J., Wang. R., Yu, J., Tang, H., Li, H., Tang, X., Hu, Y., Pan, H., et al. Ssr-encoder: En- coding selective subject representation for subject-driven generation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8069-8078, 2024.
Wang. Q. Bai, X., Wang. H., Qin. Z., Chen, A., Li, H., Tang, X., and Hu, Y. Instantid: Zero-shot adentity-presenvve generation in seconds. arXiv preprint arXiv:2401.07519, 2024e.
Zhao, S., Chen, D., Chen, Y.-... Bao, J., Hao, S., Yuan, L., and Wong, K.-Y. K. Uni-controlnet: All-in-one control text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, X., Fu, S., Huang, Q. He, W., and Jiang, H. Ms- diffusion: Multi-subject zero-shot image personalization with layout guidance. arXiv preprint arXiv:2406.07209 2024f.
Zhou, A., Wang, K., Lu, Z., Shi, W., Luo, S., Qin, Z., Lu, S., Jia, A., Song, L.. Zhan, M., and Li, H. Solv- ing challenging math word problems using gpt-4 code interpreter with code-based self- verffietii,n,tion,, 2023. URL https: / /awx/viivvvv////////////////aavv www//r///aaa.ooooo /aabbs//2330 2308 07921.
Xiao, H., Wang, G., Chai, Y., Lu, Z., Lin, W., He, H., Fan, L., Bian, L., Hu, R., Liu, L.,, et al. Ui-genie: A self- improving approach for iteratively boosting mallm-based mobile gui agents. arXiv preprint 2XX5:2505.21190 2025a.
Zhuo, L.,, Du, R., Xiao, H., Li, Y., Liu, D., Huang, R., Liu, W., Zhu, X., Wang, F.-Y., Ma, Z., al. Lumina- next: Making lumina-12x stronger and faster with next-dit. Advances in Neural Information Processing Systems, 37: 131278-131315 131315, 2024.
Xiao, H., Xie, Y., Tan, G., Chen, Y., Hu, R., Wang, K., Zhou, A., Li, H., Shao, H., Lu, X., Gao, P., Wen, Y., Chen, X., Ren, S., and Li, H. Adaptive markup language generation for contextually- contextually-grounde grounded visual document understanding, 2025b. URL https: /ttss////xxvvvvv org/ abs/2505. 05446.
Zong, Z., Cao. Q., and Leng, B. Renet: Reverse feature pyramid and cross-scale shift network for object detection. Proceedings of the 29th ACM International Conference on Multimedia, pp. 5637-5645, 2021.
Xu, X., Wang, Z., Zhang, G., Wang, K., and Shi, H. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7754-7765, 2023.
Zong, Z., Li, K., Song, G., Wang, Y., Qiao, Y., Leng, B.,, and Liu, Y. Self-slimmed vision transformer. In European Conference Computer Vision, pp. 432-448. Springer, 2022.
Xue, Z., Liang, J., Song, G., Zong, Z., Chen, L., Liu, Y., and Luo, P. Large-batch optimization for dense visual predictions: Training faster r-cnn in 4.2 minutes. Ad- vances in Neural Information Processing Systems, 35: 18694-18706, 2022.
Zong, 7., Jiang, D., Song, G., Xue, 7., Su, J., Li, H., and Liu, Y. Temporal enhanced training of milti-view 3d object detector via historical object prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3781-3790, 2023a.
Xue, 7.,, Song, G., Guo, Q., Liu, B., Zong, Z., Liu, Y., and Luo, P. Raphael: Text-to-image generation via large mix- ture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
Zong, Z., Song, G., and Liu, Y. Detrs with collabora- tive hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2023b.
Xue, Z., Wu, J., Gao, Y., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.078/8 2025.
Zong. Z. Ma, B., Shen, D.,, Song, G., Shao, H., Jiang, D., Li, H., and Liu, Y. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404. 13046, 2024.
13

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
A. Appendix
Reference Images
Control
EasyRef
IP-Adapter
"A smiling woman"
"A car"
Figure 13. Comparison between EasyRef and IP-Adapter-SDDX. with additional structure controls.
A.1. Related Work
Image Personalization. Image customization in text-to-image diffusion model applications (Xue et al., 2024; Jiang et al., 2025; Shao al., 2024b; Xue et al., 2025; Zhuo et al., 2024; Shen et al., 2025) aims to generate images that align with textual descriptions while incorporating specific visual attributes or references. Image personalization approaches can be categorized into tuning-fiit frre methods (Wang et al., 2024e; Zhang et al., 2023; Li et al., 2025; Jiang et al., 2024; He et al., 2024; Qi et al., 2024; Li et al., 2024c; Shi et al., 2024; Wang et al., 2024f; Patel et al.. 2024; Zhang et al., 2024; Li et al., 2024b; Fu et al., 2023; Huang et al., 2024b) and tuning-based methods (Hu et al., 2021; Ruiz et al., 2023; Gal et al., 2022; Kumari et al., 2023). Tuning-free approaches typically extract visual representations, such as style and subject, from the reference image and inject these into the diffusion model. IP-Adapter (Ye et al., 2023) enhances image prompting capabilities through a decoupled cross-attention mechanism. Building upon IP-Adereer, InstantStyle (Wang et al., 2024a;b) injects CLIP (Radford et al., 2021) style embeddings into style-specific blocks. Both IP-Adapter-Faas (Ye et al., 2023) and InstantID (Wang et al., 2024e) employ additional face encoders (Deng al., 2019) to improve human facial identity preservation. A limitation of tuming-fire methods is that they are trained with singlo-reference input, failing to fully exploit the consistent elements within multiple reference images. Tuning- based approaches, such as LoRA (Hu et al., 2021) and DreamBooth (Ruiz et al., 2023), finetuned the diffusion model using limited set of images. Although uuning-based methods are capable of multi-inan inlti-iimae references, a key limitation is they necessitate specific finetuning for each distinct image group. In this work, we extend tuning-free methods to accommodate multiple reference images and the text prompt like tuning-bacc based methods while maintaining robust generalization capabilities.
Multimodal Large Language Models. Multimodal large language models (MLLMs) (Liu et al., 2024a; Zong et al., 2024; Shao et al., 2024a; Chen et al., 2024; Wang et al., 2024c; 2025; Xiao et al., 2025b;a) have achieved impressive performance on open-world tasks, surpassing both traditional unimodal and multimodal approaches (Li et al., 2020; Zong et al., 2021; 2022; 2023a; Xue et al., 2022). Conventionally, MLLMS are constructed by integrating pretrained large language model (LLM) (Lu et al., 2025; Wang et al., 2023; Lu et al., 2024b; Ren et al., 2024b; Lu et al., 2024a: Ren et al., 2024a; Zhou et al., 2023; Lu al., 2024c) with encoders for additional modalities, such as vision. Pioneering works like LLaVA (Liu et al.,
14

---

RasyRef: Omni-Ceneralized Group Image Reference for Diffusion Models via Multimodal LLM
Table 5. Ablation of reference token design.
Table Ablation of reference representation aggregation.
Number
Position
CLIP-I
CLIP-T
DINO-I
Average token number
Method
CLIP-1
CLIP-T
DINO-I
32 tokens
0.818
0.699
0.630
128 tokens
0.832
0.711
Average
144
0.825
0.694
0.619
64 tokens
-2
0.836
0.710
0.655
64 tokens
-3
0.833
0.708
0.656
Concatenation
797
0.832
0.700
0.624
64 tokens
0.838
0.715
0.653
EasyRef
64
0.838
0.715
0.653
G
Reference Image
Stage? 1
Stage 2
Figure 14. Single-reference generation visualizations. "Stage 1" and "Stage 2" refer to the alignment pretraining stage and single-reference finetuning stage, respectively.
2024b) and BLIP-2 (Li et al., 2023) consistently projected the vision representation from pretrained CLIP vision encoder into the LLM for multimodal comprehension. Qwen- VL (Bai et al., 2023) collected massive multimodal tuning data and adopted elaborate training strategy for better optimization. The
designs, such as SPHINX (Lin et al., 2023), MoF (Tong et al., 2024), and MoVA (Zong et al., 2024), were explored to enhance the visual capabilities of MLLMs. Furthermore, models like LLaVA-NeXT (Liu et al., 2024a) and DW (Wang et al., 2024d) sought to enable the processing images with arbitrary resolutions. LI-DiT (Ma et al., 2024) investigated how to effectively unleash the MILM's prompt encoding capabilities for diffusion models. In this paper, we are the first leverage the multi-image comprehension and instruction-followee following capabilities the MLLM to jointly encode consistent representations of multiple reference images and the text prompt.
A.2. More Experiments
Compatibility with ControlNet. As shown in Figure 13, our EasyRef fully compatible with the popular controllable tool, ControlNet (Zhang et al., 2023). Compared to the IP-Adapter, EasyRef can generate high- high-fidelity, high-quality, and more consistent results when processing multiple reference images with additional structure controls.
Reference Token Design. We first ablate the number of reference tokens on the MRBench held-out split. The results in Table show that too many or few tokens can hurt the performance. Hence, we choose 64 tokens to achieve the best trade-off between accuracy and efficiency. Furthermore, we observed comparable performances across various insertion
15

---

RasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
IP-Adapter FaceID
Consistent
Reference Images
EasyRef
InstantID
PuLID
ID
wearing Christmas hat"
"A man holding saber"
"A womon is swimming in the pool
chair"
man is giving speech"
"A man wearing traditional Chinese clothing"
Figure 15. Qualitative comparison on human facial personalized generation.
positions (e.g., the final, second to last, and third to last layers of the LLM) for the reference tokens. Consequently, propose to insert the reference tokens into the final layer for optimal computational efficiency.
Reference aggregation design. In this experiment, we compare our reference token aggregation paradigm with embedding averaging and embedding concatenation. Specifically, we average or concatenate the vision encoder's representations of reference images. As shown in Table 6, averaging the multi-reference representations leads performance degradation and the concatenation can increase the reference token number by more than x1x. Therefore, utilizing the multi-image comprehension and instruction-folllwwe following capability of MLLMs enhance the inference efficiency and performance.
Fine-grained Detail Preservation. Figure 14 illustrates how the progressive training scheme enhances the fine-grained detail extraction capabilities of MILMSs Compared to the model trained solely in the first stage, the model further finetuned the second stage demonstrates significantly improved preservation of fine-grained details, including logos, text, layouts, and other intricate elements. However, some text details of the reference image are not preserved due the limited text rendering capability of the base Stable Diffusion XI..
Human Facial Identity Preservation. We also compare our EasyRef with other state-of-the-art specialist models (Ye et al., 2023; Wang et al., 2024e; Huang et al., 2024a; Guo et al., 2024) for human facial personalized generation in Figure 15, We observe that our method generally achieves high-quality generation, promising generation diversity, and strong facial identity fidelity.
A.3. Preliminary
Denoising Diffusion Probabilistic Models (Ho et al., 2020) (DDPMs) are trained by maximizing the log- liglliiei of the training data, given a data distribution (xx)). The training process involves a forward diffusion process that gradually adds Gaussian noise to the data over timesteps:
16

---

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
(xx:T](0)) = II a(xx|xe-1):
qxx ( Nx(xi va (1-)))
Here, x, represents the noisy data at timestep and ar is schedule parameter controlling the noise level each timestep.
The core of DDPM training lies in learning parameterized model to approximate the reverse process:
po(Xo:T) = p(xT) II po (ixe)
(9)
pe p N (x ue (Xe, t), of
(10)
This model learns to progressively remove noise from given noisy sample Xt, recovering the original data Xo.
Finally, with appropriate parameterization, the simplified per-timestep loss function becomes:
L.
xx))
20? || Ex( +)13
(11)
where represents the Gaussian noise added during the forward process and Ex t) is the model's prediction of this noise. Minimizing this loss function effectively trains the DDPM denoise and generate high-quality samples.
17


Reviews:
[{"review_id": "1HrcnMbaGU", "reviewer": "Reviewer_sCdX", "summary": "In the area of image personalization, tuning-free methods fail to capture consistent visual elements across multiple references, and tuning based methods require finetuning for new groups. In response, to learn a effective and efficient subject representation across a group of references, this paper proposes EasyRef, an approach that leverages MLLM's ability to capture the visual features across multiple images. In the representation learning process, a reference aggregation strategy and a progressive training scheme have been designed. To evaluate the performance, a paired image benchmark, MRBench, is collected.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sCdX,\n\nThanks for appreciating our work and your advice. We will address your concerns below.\n\n**Q1. Can EasyRef capture intricate details?**\n\n   We provide visualizations on the DreamBench benchmark (https://github.com/anonymous-projectuser/image), demonstrating our method's ability to preserve intricate texture details. We also show a failure case in the last row. EasyRef fails to accurately preserve the textual details, primarily due to the inherent limitations in text rendering capabilities of the SDXL base model.\n\n**Q2. The collection method does not guarantee that only the same instance will be grouped together.**\n\n   1. We acknowledge that subjects belonging to the same category but not the same instance can be grouped together. However, this does not significantly impact subject-driven performance. Multi-image subject-driven generation requires that the main subjects across multiple reference images represent the same instance, and the generated outputs should contain the same instance. When multiple reference images depict subjects from the same category but not the same instance, using a target image with a subject from the same category is still a reasonable training approach.\n   2. To further enhance subject-driven performance, we have also constructed image groups using a stricter data filtering strategy (described in line 797 of our paper) by applying a higher DINO score threshold. Additionally, as discussed in line 250, we collected high-quality images and used them to train subject LoRA models. These LoRA models were then used for generating additional training data.\n   3. We randomly sampled 200 images from the training dataset and found that the proportion of such data is small (11 out of 200).\n\n**Q3. Ablating the design of the reference aggregation.**\n\n   We have already conducted ablation studies on the aggregation designs, as shown in Table 5 and Table 6 of our paper. Additionally, we provide more ablations in the table below:\n\n| Design | CLIP-I | CLIP-T | DINO-I |\n| ------ | ------ | ------ | ------ |\n| EasyRef | 0.833 | 0.709 | 0.614 |\n| + causal attention | 0.826 | 0.708 | 0.610 |\n| + inserting tokens before the first LLM layer | 0.836 | 0.710 | 0.615 |\n| + new cross-attention layers | 0.816 | 0.712 | 0.601 |\n\nFrom these results, we observe (1) using bidirectional attention in the final layer enables better interaction among reference tokens, (2) inserting reference tokens before the first layer of the LLM increases training costs but does not yield significant performance gains, and (3) the adopted decoupled cross-attention mechanism is more effective and efficient compared to adding new cross-attention layers.\n\n**Q4. The improvement of EasyRef seems marginal on DreamBench.**\n\n   Please refer to the Q4 of rebuttal for Reviewer 35bv.\n  \n**Q5. It would be more helpful if other failure examples could be provided.**\n\n   We have included two additional failure cases for analysis in https://github.com/anonymous-projectuser/image:\n   1. Attribute Confusion: When the regions of the dog and the chair in the reference images significantly overlap (e.g., the dog partially or fully covers the chair), simply averaging the features from these reference images can result in attribute confusion. For instance, the generated dog might inherit the chair's color, while the chair adopts the dog's color, leading to inconsistent generation results.\n   2. Subject Hallucination: When the dog appears in front of the chair in one reference image but is seated on the chair in another, the simple fusion method may be misled by the positional discrepancy. This can result in subject hallucination, where the diffusion model generates an additional dog-shaped object on the chair.\n\n**Q6. EasyRef should be compared with more strong baselines.**\n\n   We will cite and discuss these state-of-the-art methods, such as CustomDiffusion [1], MS-Diffusion [2], and λ-ECLIPSE [3] in the final version. Their performance scores will be incorporated into Table 3.\n\n**Q7. In Fig. 1, the task is not very clearly showed.**\n\n   We will adjust the teaser figure and modify its caption to make the task clear.\n\n**Q8. During the alignment pretraining, are the MLLM and Diffusion trained together or is the MLLM the only model pretrained?**\n\n   We insert newly initialized reference tokens into the final layer of the LLM for aggregation. This makes the alignment pretraining process efficient, as we only train the final layer of the LLM, the reference tokens, the newly added cross-attention adapters in the U-Net, and the condition projector. The rest of the model remains frozen.\n\n[1] Multi-Concept Customization of Text-to-Image Diffusion\n\n[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance\n\n[3] λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space", "claims_and_evidence": "- The paper claims that the proposed representation learning method can accurately extract the consistent details from multiple references and requires no optimization or test-time finetuning. While this claim is mostly correct (proved by Tab. 2 and 3), it is still questionable whether it can capture intricate details of subjects having complex textures (e.g., objects from DreamBench). Also the paired-data collection method (Sec. A.3) does not guarantee that only the same instance will be grouped together.\n- The proposed method is very effective on capturing human faces and style, as demonstrated by the results in Fig. 5 and 6.", "methods_and_evaluation_criteria": "- One of the main technical contributions, progressive training scheme is proved effective in Sec. 4.3; however, there should be more experiments ablating the design of the reference aggregation.\n- In Tab. 3, the improvement of EasyRef seems marginal over the other baselines on DreamBench.", "theoretical_claims": "The equations (5-9) look reasonable and correct to me.", "experimental_designs_or_analyses": "- (Minor) Fig. 2 shows the spatial misalignment issue of the embedding averaging, which provides valuable insights for similar works. However, this could be resolved by segmenting and cropping around the object. It would be more helpful if analysis/examples of other failure scenarios could be provided.\n- EasyRef should be compared with more strong baselines in image customization, such as CustomDiffusion, MS-Diffusion, etc. Now the visual results are mostly about styles and human faces, and objects with complex textures (e.g., the rigid objects from DreamBench) are missing. More visual comparisons should be shown.\n- The current pipeline of data clustering (Sec. A.3) is limited since it cannot ensure that only images of the same instance are grouped (e.g., images of the same category but different instance may also be included). This limitation will constrain the model's ability in detail preservation. Including object-centric video data may be one way to further improve identity preservation.", "relation_to_broader_scientific_literature": "In image personalization, most previous methods focus on improving the diffusion model itself. However, in this paper, we leverage MLLMs as feature extractors and introduce a carefully designed representation learning stage to capture visual features, offering new insights to the community.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "- In Fig. 1, the task is not very clearly showed. It may lead to misunderstanding that the images on the right are generated by MLLMs. Maybe mentioned that Diffusion models are used for the customization.\n- (Strength) In the second stage, the vision encoder of the MLLM is also trained, which is a reasonable setup to improve the quality and capacity of the learned representations.", "other_comments_or_suggestions": "My concerns are mainly about the performance on identity preservation with complex objects; but the idea is solid and novel.", "questions_for_authors": "- During the alignment pretraining, are the MLLM and Diffusion trained together or is the MLLM the only model pretrained?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "isprLVNhNf", "reviewer": "Reviewer_35bv", "summary": "This paper proposes to use a vision language model (VLM) to encode subjects in reference images, and convert them to soft tokens to personalize diffusion models. It can take objects, animals and human faces as subjects.\n\nUPDATE after author response:\n\nPer my request, the authors provided extra evaluation data on human face similarities compared with PuLID, InstantID and ConsistentID. It signals an impressive message that EasyRef outperform these SOTA face encoders. However, I'm unable to reproduce such results. Instead, my own runs of the EasyRef repo led to different conclusions: EasyRef performs poorly on human faces. I actually didn't require EasyRef to outperform SOTA methods on human faces, and just wished the authors to provide a reference data point. However, the authors didn't provide honest answers. In this regard, I decide to lower my rating to reject.\n\nUpdate 2:\nThe authors provided sample images generated by their most recent model checkpoints. Seems the subject similarity indeed improves substantially compared to the (outdated) online demo. Therefore, I decide to revert my rating to a 3, as an appreciation of the tremendous efforts paid by the authors. Nonetheless, the compositionality/editability of the updated samples seems to be lower than the SOTA face encoders. Moreover, I still think the writing of this paper should be improved substantially, in particular, highlight novel architectural design choices and training recipies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 35bv,\n\nThanks for your comments. We will address your concerns below.\n\n**Q1. What are the \"Inconsistent Result\" shown in figure 2? There's no spatial information encoded in subject embeddings.**\n\n   1. The essence of the \"Inconsistent Result\" lies in the insufficient understanding of multiple reference features. Traditional methods fail to distinguish whether these features correspond to multiple subjects or different representations of the same subject. As a result, features corresponding to different spatial positions of the same subject are uniformly decoded, leading to the repetition of multiple subjects in the output.\n   2. Some subject-driven methods utilize subject-specific segmentation masks to exclude background and spatial information from the reference images. However, using a mask limits the generalizability of the framework. For instance, it becomes unsuitable for style and position-aware reference. General-purpose methods, such as IP-Adapter, avoid relying on masks, which can lead to such inconsistencies. Similarly, EasyRef does not elaborately preprocess input images, as it prioritizes generality over being limited to subject-driven tasks. \n   3. Additionally, using masks increases the complexity of the pipeline by requiring an extra segmentation model. Furthermore, inaccurate masks can degrade performance. For users, manually annotating masks during inference adds to the overall cost and reduces usability.\n   4. In cases where multiple subjects occlude each other (e.g., as discussed in Q5 of the rebuttal for Reviewer sCdX), using masks or bounding boxes may result in inaccurate subject embeddings.\n\n**Q2. The novelty of this paper.**\n\n   Please refer to the Q1 of rebuttal for Reviewer nUGh.\n\n**Q3. EasyRef uses significantly more training data compared to MOMA.**\n\n   1. MOMA is specifically designed for subject-driven generation task using a single reference image, whereas EasyRef is a general framework for multi-image consistent generation. EasyRef is capable of preserving various consistent elements, including subjects (e.g., common animals or objects), characters, styles, and human faces. Given their distinct design objectives and pipelines, directly comparing the data scales of the two approaches is neither fair nor meaningful.\n   2. Furthermore, our multi-image fine-tuning leverages approximately 350K clean subject-driven target images, which is comparable in scale to the data used by MOMA.\n\n**Q4. EasyRef has almost identical performance as MOMA on DreamBench.**\n   1. The DINO score is preferred by DreamBooth because its self-supervised training objective encourages the model to preserve fine-grained subject features. This makes it uniquely well-suited, compared to CLIP scores, for evaluating subject fidelity in generated images.\n   2. As shown in Table 3, EasyRef surpasses MOMA by 1.8% in DINO score. This performance gain is not trivial since MOMA improves the DINO score of IP-Adapter from 61.2% to 61.8% on the DreamBench. \n\n**Q5. It's not necessary to include baseline methods that are too weak and non-SOTA.**\n\n   We will carefully exclude some weak baselines in the final version.\n\n**Q6. Comparing EasyRef with SOTA facial reference methods and face similarities are not evaluated.**\n\n   We conducted qualitative comparisons (visualizations in https://github.com/anonymous-projectuser/image) showing EasyRef’s superior facial fidelity and quantitative evaluations using the face similarity metric adopted by PuLID on the MRBench: \n\n| Method       | Face Sim |\n| ------------ | -------- |\n| PuLID        | 0.632    |\n| ConsistentID | 0.652    |\n| InstantID    | 0.657    |\n| EasyRef      | 0.673    |\n\n   EasyRef outperforms SOTA methods in identity preservation, and these comparisons will be emphasized in the revised manuscript.\n\n**Q7. This paper follows the terminology of MOMA to call the VLM as multimodal LLM (MLLM), it's more accurate to refer to it as VLM.**\n\nWe use \"MLLM\" to align with its growing adoption in the community for architectures combining vision encoders, projectors, and LLMs (e.g., LLaVA). While terms like \"LMM\", \"VLM\" or \"LVLM\" are also commonly used, our choice reflects current conventions rather than solely following MOMA.\n\n**Q8. The LLM of the model is quite weak.**\n\nPlease refer to the Q4 of rebuttal for Reviewer nUGh.\n\n**Q9.  Both MOMA and EasyRef don't use complicated language capabilities.**\n\nOur synthetic captions (typically 1–3 sentences) are more detailed than conventional single-sentence prompts and require strong language comprehension capability to encode. Unlike MOMA, which only focuses on subject categories, EasyRef explicitly controls the extraction of consistent elements like face, style, and subject through instructions, better utilizing the model’s instruction-following capabilities.\n\n**Q10.  The first 3 pages are written disproportionately.**\n\nWe will streamline the opening sections and expand Sections 2.2 and 2.3 in the revised manuscript.", "claims_and_evidence": "1. The claim that \"Conventional methods encode consistent elements across reference images through averaging or concatenation (Shi et al., 2024),...  fail to capture desired visual elements through effective image interaction under explicit controls\" seem to be inaccurate. The supporting example is \"the IP-Adapter (Ye et al., 2023) generates an inconsistent image when the spatial locations of the target subject vary across the reference images\" (Figure 2). What are the \"Inconsistent Result\" shown in figure 2? Different poses/views of the same person? Is this really inconsistent? Moreover, I don't think there's much issue with averaging or concatenation the subject embeddings. There's no spatial information encoded in subject embeddings, so the inconsistency across result images shouldn't be caused by averaging or concatenation.", "methods_and_evaluation_criteria": "1. The method seems to be a straightforward scaling-up of MOMA (ECCV 2024). MOMA also uses a VLM (referred to as a \"MLLM\" in the MOMA and this paper) to encode subject characteristics and map them to subject embeddings. Moreover, MOMA also has a \"Diffusion Learning Stage\" that corresponds to the \"Alignment pretraining stage\". There are some minor differences on the architecture though, but I think the biggest difference is on the scale of the dataset (MOMA used 282K vs. EasyRef used 13M, 46x scaling up!) \n2. Taking the drastically different data scales into consideration, it's not an apple-to-apple comparison to directly compare MOMA with EasyRef. An EasyRef model trained with similar data sizes would be more supportive of its architectural advantages, if there are any. Regardless of the data scale difference, EasyRef has almost identical performance as MOMA on DreamBench.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Many baseline methods are too weak and non-SOTA. It's not necessary or useful to include them. For example in Fig 5., \"SD image variations\", \"SD unCLIP\", \"Kandinsky 2.1\", \"Open unCLIP\" etc. \n2. When comparing with facial images, the SOTA methods are InstantID, consistentID, PuLID, etc. IP-Adapter FaceID is a weak baseline. The authors only compared with InstantID in the appendix. However, face similarities are not evaluated, which is one of the most important metrics on face embeddings.", "relation_to_broader_scientific_literature": "It's a simple and straightforward extension of MOMA (ECCV 2024). The biggest difference is it uses 46x larger training data, and encodes human faces in addition to objects and animals. There are some minor differences (e.g. adding a self-attention layer at the end of VLM to mix prompt tokens with the subject embeddings), but I don't think they are essential difference and don't have much impact on the overall performance.", "essential_references_not_discussed": "A few SOTA face encoders are not mentioned, including PuLID and ConsistentID.", "other_strengths_and_weaknesses": "Since EasyRef is just a simple and straightforward extension of MOMA (ECCV 2024) with 46x larger training data, I don't feel it has sufficient technical novelties to warrant publication in ICML.", "other_comments_or_suggestions": "1. Although this paper follows the terminology of MOMA (ECCV 2024) to call the VLM as \"multimodal LLM\" (MLLM), personally I think it's more accurate to refer to it as \"Vision language model\" (VLM), because the LLM part of the model (in this paper, Qwen2-VL-2B) is quite weak compared with the commonly used MLLMs. Moreover, both MOMA and EasyRef don't use complicated language capabilities of the model. Therefore I think calling them VLMs would be more appropriate.\n2. The first 3 pages are written disproportionately. The abstract and a cover image takes one page and the introduction takes another, followed by some diffusion equations which everyone in the field is familiar with. These parts can be greatly condensed. The most important part, Section 2.2 and 2.3, are written briefly. The saved space should be used for more technical details and discussions.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rV1ygRm7f5", "reviewer": "Reviewer_nUGh", "summary": "This paper proposes EasyRef, a plug-and-play method for diffusion models to generate consistent images from multiple references under instruction controls. It uses a multimodal large language model (MLLM) to capture consistent visual elements and introduces an efficient aggregation strategy and progressive training scheme to enhance performance and reduce computational costs. Additionally, a new benchmark (MRBench) is introduced for evaluating multi-reference consistent image generation. Experiments show that EasyRef outperforms existing methods in terms of consistency and generalization ability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nUGh,\n\nThank you for appreciating our approach. We will address your concerns below.\n\n**Q1. The novelty of this paper.**\n\n   While we acknowledge that our project does not introduce groundbreaking new architectures, we wish to emphasize that it offers several important conclusions and methods that contribute novel insights to the community that are untouched by prior works. These conclusions and methods include identifying key limitations, developing effective data construction strategies, optimizing architecture and training for efficiency, and creating a new benchmark. Together, they provide a practical blueprint for building and evaluating a simple yet effective pipeline for multi-image consistent generation. We want to highlight three parts of differences.\n   1. Although some methods also utilize MLLMs and diffusion models, as acknowledged by Reviewer sCdX and nUGh, our design objectives, architecture and training optimizations, and tasks differ fundamentally. Other approaches leveraging MLLMs primarily use them for feature extraction in explicitly defined tasks (e.g., single-subject customization with a single reference image). In contrast, we demonstrate that MLLMs possess strong task instruction and multi-image comprehension capabilities, enabling them to extract consistent information based on instructions and multi-image contexts. Our method emphasizes the flexible understanding and extraction of consistent elements across multiple references, guided by instruction-based control.\n   2. We demonstrate that EasyRef with a simple architecture, can serve as a general-purpose model for multi-image customization and effectively cover various types of reference customization, such as subject, style, character, and face, rather than being limited to single-subject preservation or single-image subject-driven tasks. Consequently, EasyRef avoids the need for elaborate image preprocessing, such as subject-specific masking, making the process simpler and applicable to a broader range of scenarios.\n   3. By leveraging the unique capabilities of MLLMs combined with data construction methods and efficient designs, we achieve strong performance on the general multi-image consistent generation task. Our work focuses on building a comprehensive pipeline, including data construction, optimization of architecture and training strategies, and benchmark creation, rather than proposing an entirely new architecture.\n\n   Together, we believe these underscore the impact and utility of our work, providing practical insights and methodological advancements that benefit the broader research community.\n\n**Q2. Additional computational complexity introduced by using an MLLM to process multiple reference images and text prompts.**\n\n   We analyzed the model's performance and inference efficiency when varying the number of reference images, as shown in Figure 8. The table below presents the inference latency. The \"baseline\" refers to the standard SDXL text-to-image generation without image references. Our results show that incorporating an MLLM with multiple reference images only increases latency by 10%-20%.\n\n| Number of references | baseline | 1 | 2 | 4 | 8 |\n| - | - | - | - | - | - |\n| Latency | 4.31s | 4.78s | 4.87s | 4.96s | 5.11s |  \n\n**Q3. Customized generation when the multiple reference images contain the same two or more distinct targets.**\n\n   We provide visualizations of customized generation using two subjects as references at https://github.com/anonymous-projectuser/image. As shown in the examples, the generated cat and dog exhibit high fidelity to their respective reference subjects. EasyRef leverages the instruction-following and multi-image comprehension capabilities of MLLM to flexibly identify and preserve consistent elements (e.g., two subjects) without the need for complex image preprocessing.\n\n**Q4. The impact of different LLMs and different diffusion models.**\n\n   We compare several EasyRef variants on MRBench, as shown in the table below. First, we observe that using a stronger base MLLM or diffusion model improves performance. Second, larger MLLMs significantly increase model complexity and latency. Therefore, we select Qwen2-VL-2B and SDXL to achieve the best trade-off between performance and efficiency.\n   \n| MLLM | Diffusion Model | CLIP-I | CLIP-T | DINO-I | Latency |\n| - | - | - | - | - | - |\n| Qwen2-VL-2B | SD1.5 | 0.821 | 0.705 | 0.607 | 3.84s |\n| Qwen2-VL-7B | SDXL | 0.836 | 0.714 | 0.620 | 6.02s |\n| Qwen2-VL-2B | SDXL | 0.833 | 0.709 | 0.614 | 4.96s |", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper's contributions are closely related to the broader scientific literature on diffusion models and multimodal large language models (MLLMs). It builds on recent advancements in using MLLMs and diffusion models for image generation, addressing limitations of existing methods in handling multiple reference images and fine-grained details. The introduction of an efficient aggregation strategy and a new benchmark further aligns with ongoing efforts to enhance computational efficiency and standardize evaluation in this field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. Strengths: The paper is well-organized and clearly motivated, focusing on addressing the limitations of existing methods in handling multiple reference images and fine-grained details. The proposed EasyRef framework and the Multi-Reference Generation Benchmark are both meaningful contributions. The experimental results are also promising.\n\n2. Weaknesses: While it is reasonable to extract consistent visual elements from multiple reference images and text prompts via an MLLM, the EasyRef architecture lacks novelty. Additionally, the authors only discuss the performance gains compared to previous methods but do not analyze the increased computational complexity introduced by using an MLLM to process multiple reference images and text prompts.", "other_comments_or_suggestions": "Some figures in the paper, such as Figure 3, need significant refinement in terms of aesthetics. Issues such as oversized fonts are currently present.", "questions_for_authors": "1. The visualizations presented by the authors currently focus on generating results from multiple reference images of the same single target. I am curious whether EasyRef can achieve customized generation when the multiple reference images contain the same two or more distinct targets.\n2. The ablation studies in this paper seem somewhat incomplete. The authors does not explore the impact of different LLMs and different diffusion models on the proposed EasyRef framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dsjZvDnqI", "reviewer": "Reviewer_gzGG", "summary": "Leveraging the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM), this paper studies the personalization of diffusion models. It utilizes MLLM to capture the visual elements based on the provided reference images and instruction. The proposed framework could achieve style customization, character customization, and person id customization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gzGG,\n\nThanks for your advice. We will address your concerns below.\n\n**Q1. Missing references and discussions.**\n\n   Thank you for raising this point. We have discussed the differences between these MLLM-based frameworks and EasyRef in the Q1 of rebuttal for Reviewer nUGh. We will cite and discuss these image editing and generation methods with MLLMs in the Related Work Section of our revised manuscript. \n\n**Q2. The novelty of this paper.**\n\n   Please refer to the Q1 of rebuttal for Reviewer nUGh.\n\n**Q3. Multi-reference consistent generation capability of other MLLM-based frameworks.**\n\n   1. Multi-image consistent generation is not achievable simply by supporting multi-image input in a framework; it requires architecture optimization, novel training algorithms, and collaborative integration with data construction to achieve the desired results.\n   2. Kosmos-G is another MLLM-based framework that can accept multi-image input. However, it focuses more on the composition of distinct elements and lacks the capability to extract consistent elements and generate consistent images based on those elements. For example, in its official setting for benchmarking DreamBench, which contains multiple images per instance, Kosmos-G uses only a single reference image. In addition, as shown in our comparison with EasyRef on the multi-image consistent generation task (https://github.com/anonymous-projectuser/image), visualizations demonstrate that EasyRef produces higher-fidelity and more consistent results.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "no issues.", "relation_to_broader_scientific_literature": "The proposed method and models could be used to achieve controllable image generation.", "essential_references_not_discussed": "This paper proposes to use MLLM to achieve controllable image generation. However, there are some methods [1,2] also using MLLM to achieve image editing. Although the tasks are different to some extent, both share the similar ideas. In addition, [3] using MLLM to achieve image generation should also be discussed.\n\n[1] GUIDING INSTRUCTION-BASED IMAGE EDITING VIA MULTIMODAL LARGE LANGUAGE MODELS? ICLR'24\n\n[2] SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models. CVPR'24\n\n[3] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion. ACL'24", "other_strengths_and_weaknesses": "Strengths: \n\nThis paper builds a multi-reference image generation benchmark, which is helpful for the community.\n\nThe proposed method achieves better performance than previous works.\n\nWeaknesses:\n\n- The idea is not new. \n\nIn Tab.1, the paper makes comparisons between the proposed method and some previous methods. While Tab.1 shows the proposed method can support more functions, the difference between it and KOSMOS-G / MoMA regarding architecture and motivation is slight. For example, regarding the \"multiple reference images\", I believe such capability could be achieved by using a MLLM which supports the input of multiple reference images. The proposed method achieves better performance, which I believe makes sense, but the novelty is limited. \n\n- In addition, there are some additional similar works not discussed [1,2,3].\n\n[1] GUIDING INSTRUCTION-BASED IMAGE EDITING VIA MULTIMODAL LARGE LANGUAGE MODELS? ICLR'24\n\n[2] SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models. CVPR'24\n\n[3] UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion. ACL'24", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper reveived two Accepts, one Weak Accept, and one Weak Reject. Reviewers acknowledged the novelty/contribution of this paper, the contribution of the benchmark dataset, and promising experimental results. Especially, Reviewer 35bv run the demo code of the proposed EasyRef method and confirmed its effectiveness. Negative points majorly raised for the discussion of novelty, while authors provided detailed rebuttal and all reviewers acknowledged that their concerns had been addressed except that Reviewer gzGG was still not very satisfactory with it but was not against acceptance of the paper. Reviewer gzGG mentioned three papers which used MLLM for image editing. Authors listed detailed differences of their mehtod compared to these three existing methods. While the tasks are different (image generation vs. image editing), the compared difference seems reasonable. Therefore, considering that all reviewers acknowledged the contribution of the paper and accepted the rebuttal, except the last concern which was almost addressed, AC is happy to accept the paper for publication. Authors are required to update the rebuttal and discussion contents to the camera-ready version of the paper to improve it so as to address the raised concerns in the final paper.
Decision: Accept (poster)