Event-Customized Image Generation
Authors: Zhen Wang
Yilei JIANG
Dong Zheng
Jun Xiao
Long Chen
Venue: ICML 2025 poster (2025)
Keywords: Customized Image Generation
Diffusion Model
Primary area: applications->computer_vision

Abstract: Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity (i.e., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient ''exactly same'' reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation. Given a single reference image, we define the ''event'' as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.

Event-Customizze Image Generation
Zhen Wang'
Yilei Jiang'
Dong Zheng'
Jun Xiao
Long Chen
ape robot
Spiderman Batman
?
<>> sleeping
<>>>>>
deghouse
girl, hat
giz wearing a hat scarf
tiger lion
cat orange
(a) The subject customization
skeleton statue monkey baok
① Wolverine Spiderman Deadpool MacBook
Spiderman <V>
panda <V>
1
<
cat
monkey
monkey
cat
<V>>
(b) The action interaction customization
(c) The event customization
Figure Customized Image Generation. (a) Generating customized images with given subjects new contexts. (b) Generating customized images with co-existing basic action or interaction in given images. (c) Generating customized images for complex events with various target entities. Different colors and numbers show associations between reference entities and corresponding target prompts.
Abstract
tion to more complex scenes, propose new task: event-customized image generation. Given a single reference image, we define the "event" as all specific actions, poses, relations, or inter- actions between different entities in the scene. This task aims accurately capturing the com- plex event and generating customized images with various target entities. To solve this task, we pro- posed novel training-free free method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: Entity switching path: it applies cross-attention guidance and regulation for target entity gener- ation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event gen- eration. We further collected two new evaluation benchmarks. Extensive experiments have demon- strated the effectiveness of FreeEvent.
Customized image generation has raised signif- icant attention due to its creativity and novelty. With impressive progress achieved in subject cus- tomization, some pioneer works further explored the customization of action and interaction be- yond entity (e.g., human and object) appearance. However, these approaches only focus basic actions and interactions between two entities, and their effects are limited by insufficient "exactly same" reference images. To extend the customiza-
'Zhejiang University, Hangzhou, China "The Hong Kong Uni- versity of Science and Technology, Hong Kong, China. Work was done when Zhen Wang visited HKUST. Correspondence Long Chen <longchen @ hst.
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Event-Customized Image Generation
1. Introduction
2024b;a; Jía et al., 2024) tend represent them by learning corresponding identifier tokens, which can be further used for generating new images. However, for each action, or interaction, these training- training-based processes typically require a set of reference images (e-g-. 10 images) paired with corresponding textual descriptions across different entities. Unfortunately, each action or interaction is highly unique and distinctive, i.e., gathering images that depict the exact same action or interaction is challenging. As shown in Figure 1(b), there are still significant differences in the same action (e.g., handstand between different reference images, which thus compromises the accuracy of learned tokens, leading to inconsistencies in action between generated images. This insufficient data issue for identical action or interaction has severely limited the practicality and generalizability of these methods.
Recently, large-scale pre-trained diffusion models (Dhari- wal & Nichol, 2021; Nichol et al., 2022; Ramesh et al., 2022; Rombach et al.. 2022: Saharia et al., 2022) have demonstrated remarkable success in generating diverse and photorealistic images from text prompts. Leveraging these unparalleled creative capabilities, a novel application customized image generation (Gal et al., 2023a; Ruiz et al., 2023; Chen et al., 2024a) - has gained increasing attention for generating user-specified concepts. Significant progress has already been made in subject-customized image gen- eration (Ye et al., 2023; Chen et al., 2024c). As shown in Figure I(a), given set of user- sser-provided subject images, isting methods can accurately capture the unique appearance features of each subject (e.g., corgi) with a special identi- fier token, enabling creative rendering in new and diverse scenarios. Moreover, they can seamlessly integrate multiple subjects into cohesive compositions, preserving distinctive characteristics while adapting them to novel contexts.
To address these limitations and extend customized im- age generation to more complex scenes, we propose a new and meaningful task: event-customized image generation. Given a single reference image, we define the "event" as all actions and poses of each single entity, and their relations and interactions between different entities. As shown in Figure (c)), event customization aims to accurately capture the complex and diverse event from the reference image to generate target images with various combinations of target entities. Since it only needs one single reference image, the event customization also eliminates the need for collecting "exactly same" reference images.
Beyond the appearance of different entities (i.e., humans, animals, and objects) in the images, pioneering approaches have been developed to customize the user-specified ac- tions (Huang et al., 2024a), interactive relations (Huang et al., 2024b) and poses (Jia et al., 2024) between the en- tities. As shown in Figure 1(b), these methods attempt to capture the single-entity action (e.g-, handstand) or inter- actions (e.g., back to back) between two entities that co-exist in the given reference images and transfer them to the synthesis of action- or interaction-speciffi images with new entities.
To solve this challenging task, we proposed a novel training- free event customization method, denoted as FeeeEvent. Based on the two main components of the reference image, i.e., entity and event, FreeEvent decomposes the event cus- tomization into two parts: 1) Switching the entities in the reference image to target entities. 2) Transferring the event from the reference image to the target image. Following this idea, alongside the general denoising process of diffusion generation, we designed two extra paths: entity switching path and event transferring path. Specifically, entity switch- ing path guides the localized layout of each target entity for entity generation. Event transferring path further ex- tracts the event information from the reference image and then injects it into the denoising process to generate the specific event. Through this direct guidance and injection, FreeEvent offers significant advantage over existing meth- ods by eliminating the need for tine-consuming training. Furthermore, as shown in Figure I(c), FreeFivet can also serve as a plug-and-play framework to combine with subject customization methods, generating creative images with both user-specified events and subjects.
However, for real-world scenes that typically involve multiple entities with more complex interactions (e.g., Fig- ure I(c), row 3: three humans are discussing in front of computer with different poses), these works (Huang et al., 2024b;a; Jia et al., 2024) still face notable limitations. 1) Simplified Cus- tomization. Current action customization (Huang et al., 2024a) focuses solely on the basic actions of a single person. Similarly, interaction customizations (Huang et al., 2024b; Jia et al., 2024) are limited to basic interactive relations poses between just two entities. There is lack of exploration into more complex and diverse actions or interactions that involve multiple humans, animals, and objects. Additionally, while these methods typically perform well when generating images with the same type of entity (e.g., all monkeys or all cats), they struggle when faced with more diverse and complex entities and their combinations. These narrow focuses and limitation on entity generation have strictly limited their abilities to customize more complex and diverse scenes with creative content. 2) Insufficient Data. To capture specific actions or interactions, existing methods (Huang et al.,
Moreover, as pioneering effort in this direction, we also collected two evaluation benchmarks from the existing dataset (i.e., SWiG (Pratt et al., 2020) and HICO-DET (Chao
2

---

Event-Customized Image Generation
et al., 2015)) and the internet for event-customized image generation, dubbed SWiG-Event and Real-Event, respec- tively. Both benchmarks include reference images featuring diverse events and entities, along with manually crafted tar- get prompts. Extensive experiments demonstrate that our approach achieves state-of-the-ar performance, enabling more complex and creative customization with enhanced practicality and generalizability.
and training large-scale multimodal encoders (Gal al., 2023b; Li et al., 2024). 2) Multi-subjecct composition, in- cluding cross-attention modification (Tewel et al., 2023), constrained model fine-tuning (Kumari et al., 2023). lay- out guidance (Liu et al., 2023), and gradient fusion of each subject (Gu et al., 2024). In conclusion, these works are tai- lored to capture the appearance of the entities in the image, without considering the customization of actions or poses.
In summary, we make several contributions in this paper:
Action and Interaction Customization. They aim to gen- erate customized images with co-existing actions or interac- tions in user -soriprovided reference images. Re (Huang et al., 2024b) first proposes to customize specific interac- tive relations by optimizing the learnable relation tokens. ADI (Huang et al., 2024a) makes progress in customiz- ing specific actions for a single subject. And a following work (Jia et al., 2024) further extends it to learning inter- active poses between two individuals. However, all these works only focus on simplified customization of some basic actions and interactions, and their effect is strictly limited by the insufficient data of reference images. In contrast, our proposed event customization only requires one reference image, and our training-free framework can achieve effec- tive customization of complex events with various creative target entities.
1) We propose the novel event-customized image genera- tion task, which extends customized image generation to more complex scenes in real-world applications. 2) We pro- pose FreeEvent, the first training- free method for event cus- tomization, which can be further combined with subject cus- tomization methods for more creative and generalizable cus- tomizations. 3) We collect two evaluation benchmarks for event-customized image generation, and FreeEvent achieves outstanding performance compared with existing methods.
2. Related Work
Text-to-Image Diffusion Generation. Diffusion mod- els (Ho al., 2020; Nichol & Dhariwal, 2021; Song al., 2021) have emerged leading approach for image syn- thesis. The -mtae diffusion models (Nichol et al., 2022; Ramesh et al., 2022; Saharia et al., 2022) further inject user-provided text descriptions into the diffusion process via pre-trained text encoders. After trained on large-scale text- image pairs, they have shown great success in text-to-image generation. Different from these models that operate the diffusion process on pixel space, the latent diffusion models (LDMs) (Rombach et al., 2022) propose to perform it on la- tent space with enhanced computational efficiency. Besides, existing works (Hertz et al., 2023; Tumanyan et al., 2023; Cao et al., 2023; Alaluf et al., 2024) have discovered the spa- tial feature and attention maps in LDMs contain localized semantic information of the image and the layout correspon- dence between textual conditions. As a result, these features and attention maps have been utilized to control the layout, structure, and appearance in text-to- text-to-image generation. This can be achieved either through a plug-and-play feature in- jection (Tumanyan et al., 2023; Xu et al., 2024; Lin et al., 2024) or by computing specific diffusion guidance (Epstein et al., 2023; Mo al., 2024) for generation. In this paper, we utilize the pre-trained LDM Stable Diffusion (Rombach et al., 2022) as our base model.
3. Methods
3.1. Preliminary
Latent Diffusson Model (LDM). Generally, LDMs include pretrained autoencoder and a denoising network. Given an image x, the encoder & maps the image into the latent code in = {z), where the forward process is applied to sample NN, I) to it to obtain 2 vārzo + VI - cgc from time step t ~ [1, T] with predefined noise schedule a While the backward process iteratively removes the added noise on 2t to obtain 20. and decodes it back to image with the decoder x = D(zo). Specifically, the diffusion model is trained by predicting the added noise e conditioned time step t and possible conditions like text prompt P. The training objective is formulated as:
Guassian noise
CLDM = Eg--e(3),PP--W10.. [llc- [[Ic-co<stte P) (1)
where co is the denoising network.
Diffusion Guidance. The diffusion guidance modifies the sampling process (Ho et al., 2020) with additional score functions to guide it with more specific controls like object layout (Xie et al., 2023; Mo et al., 2024) and attributes (Ep- stein et al., 2023; Bansal et al., 2023). We express it as
Subject Customization. This task aims to generate cus- tomized images of user-specified subjects. Current main- stream subject customization works mainly focus on 1) Single subject customization, including learning specific identifier tokens (Gal et al., 2023a), finetuning the text-to- image diffusion model (Ruiz et al., 2023; 2024), introduc- ing layer-wise wise learnable embeddings (Voynov et al., 2023)
é c L P) g(z;; 4, P),
(2)
where g is the energy function and s is a parameter that controls the guidance strength.
3

---

Event-Customized Image Generation
(b) Event Transferring Path
Encoder
lG
forward process
Ee
ZT
zt
Entity Switching
Event Transferring
Decoder
Zt
G
t
zt-1
G
2
x (T-1)
6
x (t-1)
P: skeleton, statue, monkey, book
P: skeleton, statue, monkey, book
(a) Entity Switching Path
Denoising Process
cross-attention guidance
prross-attentin regulation
spatial feature injection
self-attention injection
The overview of pipeline. Given the reference image, the event customization is overall general diffusion denoising process with two extra paths. 1) The entity switching path guides the generation of each target entity through cross-attention guidance and regulation 2) The event transferring path injects the spatial features and self-attention maps from the reference image to the denoising process. The final =f is then transformed back to target image 7G by the decoder.
Figure 2:
3.2. Task Definition: Event-customized Generation
of event customization, denoted as the entity switching path and the event transferring path, respectively. Generally, as shown in Figure 2, the generation of J starts by randomly initializing the latent 24 ~ (0, I), and iteratively denoise it to z During this denoising process, the entity switch- ing path guides the generation of each target entity through cross-attention guidance and regulation based on the target prompt P and reference entity masks M. The event trans- ferring path extracts the spatial features and self-attention maps from the reference image 1", and then injects them to the denoising process. The final =f is then transformed back to the target image 1 by the decoder.
In this section, first formally define the event-customized image generation task. Given a reference image 7 involves N reference entities E = {R,,.. Rr we define the "event" as the specific actions and poses each single ref- erence entity, and the relations and interactions between different reference entities. Together we have the entity masks M- {m...... m... where m, is the mask of its corresponding entity R4 The event-customized image gen- eration task aims to capture the reference event, and further generate a target image le under the same event but with diverse and novel target entities F.C = (G.,..., GN) in the target prompt P {wg...., WN) ). where w; the de- scription of the target entity GG and each target entity G; should keep the same action pose with its corresponding reference entity R, As the example shown in Figure 2, given the reference image with four reference entities (e.g., aims
U-Net Architecture The Stable Diffusion (Rombach et al., 2022) utilizes the U-Net architecture (Ronneberger et al., 2015) for fg. which contains an encoder and a decoder, where each consists of several basic encoder/decoder blocks, and each encoder/decoder block further contains several en- coder/decoder layers. Specifically, as shown in Figure ((a), each U-Net encoder/decoder layer consists of a residual module, self-attentio module, and a cross-attention mod- ule. For block layer I, and timestep 4, the residual mod- ule produces the spatial feature of the image as f. The self-attention module produces the self-attention map as
three people and one object), the event-customizatte to capture the complex reference event and generate the target image with novel combination of different target entities (e.g., skeleton, statue, monkey, book).
3.3. Approach
SA = Softmax( Q.K." where Q₂ and K2 are query and key features projected from the visual features. For text-to- image generation, the cross-attention module further pro- duces the cross aross-attention map between the text prompt P and the image CA Softmax( Q.K.T where Q, is the reference event (i.e., transferring the event from the refer- ence image to the target image). Following this idea, we design two extra paths for the diffusion denoising process
Overview. We now introduce the proposed training-free event customization framework FreeEvent. Specifically, we decompose the event-customized image generation into two parts, 1) generating target entities (i.e., switching each ref- erence entity to target entity), and 2) generating the same
vi
query features projected from the visual features, and K. is the key features projected from the textual embedding of

---

Event-Customized Image Generation
latent =
Residual bock
self- attention
cross- attention
2 c- = - -
V2
Σ (CA, m,.)
(4)
f
SAK
XEN
E
where 7j is the guidance scale and or v V11-á Ad- ditionally, avoid the appearance leakage between each target entity, further regulate the cross-attention map of each token within its corresponding area. Specifically, for cross-attention maps CA calculated at timestep during the denoising process, we have:
SAG
Residual block
self- attention
cross- attention
Residual block
self- attention
cross- Attention
CA
(a) U-Net Layer
(c) Event Transferring
CA = my C CA.
(5)
CA
where CA is the cross-attention map of token wi..
lskeleton
statue
monkey
book
Event Transferring Path. This path aims to extract the specific reference event from the reference image 1 in- cluding the action, pose, relation, or interactions between each reference entity, and transferring them to the target im- age _ Meanwhile, from the perspective of image spatial information, the event is essentially the structural, semantic layout, and shape details of the image. Thus, based on the observation that the spatial features and self-attention maps can be utilized to control the image layout and structure (Tu- manyan al., 2023; Xu al., 2024; Lin et al., 2024), perform spatial feature and self-attention map injection to achieve the event transferring.
M O CA
M
8(M, CA
CA
Lakeleton
statue
monkkey
book
(b) Entity Switching
Figure 3: (a) The architecture of the U-Net layer. (b) The process of cross-attention guidance and regulation. (c) The process of spatial feature and self-attention injection.
Specifically, as shown in Figure 2(b) we first get the latent code of the reference image z E(T),, and at each time step during the denoising process, we obtain 2f via the diffusion forward process. We then input z into the U- Net to extract the spatial features and self-attention maps of the reference image f and SAR Parallelly, for the denoising process, we input =f together with the target prompt P into the U-Net, and calculate the spatial features and self-attention maps for the generated image as: fG and SA G. Then, as shown in Figure 3(c), we perform the injection by directly replacing corresponding spatial features and self-attention maps:
Entity Switching Path. This path aims on generating tar- get entities E = {G1,..., Gr) in 1 by switching each refenrece entity R; to G; based on the target prompt P and reference entity masks M. And the key is to ensure each target entity G; is generated at the same location as their corresponding reference entity R2 and avoid the appear- ance leakage between different entities. Inspired by prior works (Hertz et al., 2023; Chen al., 2024b) that utilize the noss-attention maps control the layout of text-to- image generation, we apply the cross-attention guidance and regulation to achieve the entity switching.
As shown in Figure 2(a), at the timestep t of the denoising process, we first obtain the latent for entity switching as 2/ 2 we then input 2/4 together with the target prompt P into the U-Net, and calculate the cross-attention maps CA Then, we introduce energy function to bias the cross-attention of each token W/ as (cf., Figure 3(b)):
g - PR and SA + SA"
(6)
Highlights. By applying cross-attention guidance and regu- lation on each text token, our attention-guide -uided entity switch- ing can also be used to generate target entities of user- specified subjects, iee, represented by specific identifier tokens. Thus, our framework can be easily combined with subject customization methods to generate creative images with both customized events and subjects.
g( (CCA, m:) = (1 CAA1 miy
(3)
CA.
where CAAA is the cross cross-attention map of token w/- Opti- mizing this function encourages the cross-attention maps of each target entity G; to obtain higher values inside the corresponding area specified by m,, which further guides the localized layout of each target entity. We calculate the gradient of this guidance via backpropagation to update
4. Experiments
4.1. Experimental Setup
Evaluation Benchmarks. In order to provide sufficient and suitable conditions for both quantitative and qualitative
5

---

Event-Customized Image Generation
Image Retrieval
Verb Detection
Image Similarity
Model
CLIP-T
FID
R@I it
R@5t
@@ 10t
T-11
T-5t
T-10t
CLIP-It
DreamSimt
DINOt
ControlNet
10.64
26.12
36.82
10.66
23.98
31.28
0.6009
0.3714
0.2599
0.2198
70.45
MIGC
10.90
27.00
37.64
10.62
26.14
35.04
0.6456
0.3772
0.2467
0.2145
49.81
BoxDiff
8.60
22.48
32.08
5.58
14.52
19.42
0.5838
0.3135
0.2099
0.2153
68.49
FreeEvent 41.12
63.02
72.74
34.10
62.04
71.82
0.7044
0.6282
0.5116
0.2238
29.05
Table 1: Performance of our model and state-of-art conditional text-to-image generation models on SWiG-Event. For image retrieval, the R@k represents that among the top-k images with the highest similarity to the target image, its corresponding reference image is included. For verb detection, the T-K represents the top-k detection accuracy.
comparisons on this new task, we collect two new bench- marks'. 1) For quantitative evaluation, we present SWiG- Event, a benchmark derived from SWiG (Pratt et al., 2020) dataset, which comprises 5,000 samples with various events and entities, i.e., 50 kinds of different actions, poses, and interactions, where each kind of event has 100 reference images, and each reference image contains I to 4 entities with labeled bounding boxes and nouns. 2) For qualita- tive evaluation, we present EvelnE which comprises 30 high-quality reference images from HICO-DET (Chao et al., 2015) and the internet with wide range of events and enti- ties (e.g., animal, human, object, and their combinations). We further employ Grounded-S AAM (Kirillov et al., 2023; Ren et al., 2024) to extract the mask of each entity.
reference entities. Additionally, ControlNet takes the se- mantic map merged from the masks as the layout condition. MIGC and Box take the bounding boxes with labeled entity nouns as the layout condition'.
Evaluation. Our evaluations follow the principle of whether generated images are aligned or similar with their reference images, and we apply multiple metrics to evaluate the cus- tomization quality of 5,000 target images from different perspectives. 1) Global image similarity: image retrieval performance and similarity scores. We retrieved each target image for its corresponding reference image based on the CLIP score across all the 100 reference images that have the same reference event class. Specifically, we extracted the im- age feature of each image through a pre-trained CLIP (Rad- ford et al., 2021) visual encoder and calculated the cosine similarities for image retrieval. Besides, we also used the CLIP-I, DreamSim (Fu et al., 2023), and DINO (Oquab et al.) scores evaluate the image alignment of generated images with their reference images. 2) Event similarity: verb detection performance. We utilized the verb detection model GSRTR (Cho et al., 2021) which was trained on the SWIG dataset to detect the verb class of each generated image, and then calculated the detection accuracy based on the annotations of the reference images (i.e., whether the generated images and their reference images have the same verb class). 3) Entity similarity: CLIP-T (Radford et al., 2021) score. We use the CLIP-T score to evaluate the text alignment of the generated images with text prompts. 4) Standard image generation metric. For more comprehen- sive comparison, we used the FID (Heusel et al., 2017) score to evaluate the overall quality of generated images.
Baselines. We compared several kinds of SOTA baselines. For conditioned text-to-image generation baselines, we com- pared with training- rriseng-baseed method ControlNet (Zhang et al., 2023), MIGC (Zhou al., 2024), and training-free method BoxDiff (Xie et al., 2023). For localized editing base- lines, we compared with training-free methods PnP (Tu- manyan et al., 2023) and MAG-Edit (Mao et al., 2024). For customization baselines, we compared with training- based methods Dreambooth (Ruiz et al., 2023) and ReVer- sion (Huang et al., 2024b).
Implementation Details. We use Stable Diffusion v2-1- base base model for all methods. Images generated with resolution of 512x512 on a NVIDIA A100 GPU!.
4.2. Quantitative Comparisons
We compare FreeEvent with state-of-the-art conditional text-to-image generation baselines ControlNet (Zhang et al., 2023), MIGC (Zhou et al., 2024), and BoxDiff (Xie et al., 2023) on the SWiG-Event.
Results. As shown in Table we can observe: 1) FreeEvent has better retrieval performance than both ControlNet, MIGC, and Dixf This demonstrates that the target im- ages generated by FreeEvent better preserve the overall char- acteristics of the reference event and entity. 2) FreeEvent also achieves the best verb detection performance, which in- dicates our method can better preserve the interaction seman- ties of the generated images. 3) FreeEvent further achieves superior performance over baselines across all similarity scores and standard image generation metrics, indicating our method can generate images with better qualities and
Setting. Each reference image in SWiG-Event contains ref- erence entities together with labeled event class, bounding boxes, nouns, and their corresponding masks. Specifically, we construct the target prompt as list of reference entity nouns, i.e., we ask all the methods to reproduce the event of the reference image with the same reference event and same
I Due to limited space, more details/results are in the Appendix.
6

---

Event-Customized Image Generation
Reference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
1 Spiderman @ Batman
① robot @ bird @ apple
① Spiderman @ orange @ strawberry
1 bear @ Spiderman x panther
① Tarzan tiger x lion
Spiderman 2 robot bear monkey Figure 4: Comparision Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
alignment with both the reference images and texts. These results all demonstrate the effectiveness of FreeEvent for event customization.
the semantic map and BoxDiff takes the labeled bounding boxes the layout conditions. MAG-Edit takes the refer- ence entity masks for localized editing. Dreambooth and Re learn event-specific identifier tokens for text-to- image generation.
4.3. Qualitative Comparisons
We compare FreeEvent with wide range of SOTA baselines on the Real-Event', including conditioned text-to-image generation method ControlNet (Zhang et al., 2023) and BixDiff (Xie et al., 2023), localized image editing method PnP (Tumanyan et al., 2023) and MAG-Edit (Mao et al., 2024), image customization methods Dreambooth (Ruiz al., 2023) and Version (Huang al., 2024b).
Results. As shown in Figure 4, we can observe: 1) Con- ditional eext-to-image generation models ControlNet and Diff only maintain the rough layout of each entity and struggle to capture the detailed action, pose, inter- action between different entities. And they both failed to match the generated entity with the desired target prompt. 2) For localized image editing methods PnP and MAG- Edit, while they can capture the reference event, they both struggle to accurately generate the target entities, and suffer from severe appearance leakage between each target entity
Setting. For each reference image in Real-Event manu- ally constructed target prompts with various combinations of different target entities. Specifically, ControlNet takes

---

Event-Customized Image Generation
Reference Image
Original Output
(a) Ablation on two paths
(b) Ablation target prompt background
w/o guidance
w/o regulation
w/o injection
verb
style
Target prompt:
ape robot
boxing
on the beach
cartoon
Target prompt:
tiger 2 lion meat
are playing
the snow
Chinese painting
Target prompt: ① skeleton 2 statue
monkey
④ book
are discussing
in the library
old photo
Figure 5: Ablations of the proposed paths and the target prompt. The "guidance" and "regulation" denote the cross- attention guidance and cross-attention regulation in entity switching path, respectively, The "injection" denotes the event transferring path.
(e.g., orange and strawberry in row three, t tie and lion in row five), and sometimes even failed to edit and output the original content. 3) The cubject-customizatio model Dreambooth and the castiooncustomiizaionn model ReVersion both failed to generate satisfying results. As discussed before, these training-based methods require mul- tiple reference images and are unable to Iearn the specific event when facing only one reference image. 4) Obviously, our FreeEvent successfully achieves the customization of various complex events with novel combinations of target entities. Meanwhile, the ControlNet and the localized image editing models tend to generate the target entities strictly matching the mask of their corresponding reference entities (e.g., bird rd in row two), which appears very incongruous. On the contrary, the entities generated by FreeEvent not only match the layout of the reference entity but also keep it harmonious. After all, while we use the reference entity mask to guide the generation of each target entity, the cross- attention guidance focuses directing the overall layout of each target entity and doesn't 't restrict detailed appearance, allowing for more diverse generation of target e
cross-attention guidance results the failure of target en- tities generation (e.g., the ape, the meat), and removing cross-attention regulation leads to the appearance leakage between entities (e.g., the tiger and lion, the skeleton and statue). 2) After removing the event transferring path, al- though the target entities can be generated, the reference events are completely lost (i.e., the pose, action, relations, and interactions between each entity). These results all corroborates the effect of two paths event customization.
Influence of Different Target Prompts. Noteably, in our paper, the target prompt only contains the nouns of the target entities, we then run the ablations to analyze the influence of different descriptions (i.e., verb, background, style) in the target prompt for event customization.
Results. From Figure 5(b) we can observe: 1) Adding verb description leads to a certain degree of negative impact on entity appearance (e.g., the head of the ape, the face of the monkey) since these verbs may not be aligned with the model. Besides, accurately describing events in plex scenes can be challenging for users. Therefore, since FreeEvent can already achieve precise extraction and trans- fer the reference events, users do not need to describe the specific events in the target prompt, which further demon- strates FreeEvent's practicality. FreeEvent can accurately generate extra contents for the background and style. Al- though there may be some detailed changes in the entity's appearance compared to the original output, these do not affect the entity's 's characteristics or the event. This also
4.4. Ablations
Effectiveness of Entity Switching Path and Event Trans- ferring Path. We first run ablations to verify the effect of two proposed paths during event customization.
Results. As the results are shown Figure 5(a), we can observe: 1) For the entity switching path, removing the
8

---

Event-Customized Image Generation
Subjects
VO>
<V>>
< V2>
< <V3>> < V4>
V5>
V6
< V
Event-Subject Customization
Reference Image
VO>
V5>
VT>
V3>
Reference Image
1 v < rl> v2> Otypewriter
Reference Image
< V4>
2 bear on <V6>
monkey @ astronaut on <V7>
Reference Image Dold man Dold woman V6 >
Figure 6: Results of Event-Subject Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Oure ControlNet Box PnP MAGEdi DreamBooth ReVersion
models. They are asked to select at least one and up to three images that they believe demonstrate the best results in event customization, taking into account the generation effects of the events and entities, as well as the overall coherence of the images. We prepared 50 trials and asked the experts give their judgments. The target image which got more than six votes is regarded as human judgment.
HJ
50
25
7
26
23
8
Table 2: Results of the user studies on the Real-Event.
demonstrates FreeFventtss 's strong generalization capability.
Combination of Event and Subject Customization. We further validate the ability of our framework to combine with subject customization methods generate target entities with user-specified subjects, i.e., represented by identifier tokens. We took the Break-A-Scene model (Avrahami et al., 2023) to Iearn identifier tokens for subjects and replaced the Stable Diffusion models in Figure 2 with the fine-tuned one.
Results. As shown in Table 2, FreeEvent achieves better performance on human judgments (HJ) compared with all the baseline models.
5. Conclusion
Results. As shown in Figure 6, FreeEvent can effectively generate various given subjects in specific events. Specifi- cally, FreeEvent enables the flexible generation of a wide range of subject concepts (e.g., humans, regular objects, and backgrounds) and their combinations. These results demonstrated the great potential of our framework for Event- Subject customization.
In this paper, we proposed a new task: Ceett-Customized Image Generation. It focuses on the customization of com- plex events with various target entities. Meanwhile, we pro- posed the first training-free customiustoiizaiion framework FreeEvent. To facilitate this new task, we also collected two evaluation benchmarks from existing datasets and the internet, dubbed SWiG-Event and Real-Event, respectively. We validate the effectiveness of FreeEvent with extensive comparative and ablative experiments. Moving forward, we are going to I) extend the event customization into other modalities, e.g., video generation; 2) explore advanced tech- niques for the finer combination of different customization works, e,g,, subject, event, and style customizations.
4.5. User Study
Setting, We conducted user studies on Real-Event to further evaluate the effectiveness of FreeEvent. Specifically, we invited 10 experts and gave them reference image, tar- get prompt, and seven target images generated by different

---

Event-Customized Image Generation
Acknowledgements
In The Twelfth International Conference on Learning Representations, 2024a.
This work was supported by the National Key Research & Development Project of China 1144YFB3332900) Key R&D Program of Zhejiang (2025C01 128), an Fundamental Research Funds for the Central Universities. Long Chen was supported by the Hong Kong SAR RGC Early Career Scheme (26208924), the National Natural Science Foun- dation of China Young Scholar Fund (62402408), Huawei Gift Fund, and the HKUST Sports Science and Technology Research Grant (SSTRG24FG04).
Chen, M., Laina, I., and Vedaldi, A. Training-fres layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5343-5353, 2024b.
Chen, X., Huang. L.,, Liu, Y., Shen, Y., Zhao, D., and Zhao, H. Anydoor: Zero-shot object-level image customization. In Proceedings of the IEEF/CV Conference on Com- puter Vision and Pattern Recognition, pp. 6593-6602, 2024c.
Impact Statement
Cho, J., Yoon, Y., Lee, H., and Kwak, S. Grounded situation recognition with transformers. In British Machine Vision Conference (BMVC), 2021.
Since FreeEvent can seamlessly integrate with subject cus- tomization methods to generate target entities based on user- specified subjects, this capability also raises the same con- cerns about the potential misuse of pretrained SD models for malicious applications (e-g.. Deepfakes) involving real human figures. To address this, is essential to implement robust safeguards and ethical guidelines, similar to the se- curity measures and NSFW content detection mechanisms already present in existing diffusion models.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 14:8780-8794, 2021.
Epstein, D., Jabri, A., Poole, B., Efros, A., and Holynski, A. Diffusion self- leff-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36: 16222-16239, 2023.
References
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., and Isola, Dreamsim: Learning new dimensions of human visual similarity using synthetic data. Advances Neural Information Processing Systems, 36:50742- 50768, 2023.
Alaluf, Y., Garibi, D., Patashnik, O., Averbuch-Eloo H., and Cohen-Or, D. Cross-image attention for zero-shot ap- pearance transfer. ACM SIGGRAPH 2024 Conference Pupers, pp. 1-12, 2024.
Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. Break-a-scene: Extracting multiple con- cepts from single image. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-, 2023.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-or, D. An image is worth word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Confer- ence on Learning Representations, 2023a.
Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein, T. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 843- 852, 2023.
Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42(4):1-13, 2023b.
Gu, J., Wang. Y., Zhao, N., Fu, T.-J., Xiong, W., Liu, Q. Zhang, Z., Zhang, H., Zhang, J., Jung, H., et al. Pho- toswap: Personalized subject swapping in images. Ad- vances in Neural Information Processing Systems, 36: 35202-35217 2023.
Cao, M., Wang, X., Qi, 7., Shan, Y., Qie, X., and Zheng, Y. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2,660,,2570,-22 2023.
Gu, Y., Wang, X., Wu, J. Z., Shi, Y., Chen, Y., Fan, Z., Xiao, W., Zhao, R., Chang, S., Wu, W., et al. Mix- of-show: Decentralized low-rank adaptation for multi- concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Chao, Y.W.., Wang, Z., He, Y., Wang, J., and Deng, J. Hico: A benchmark for recognizing human-object interactions in images. In Proceedings of the IEEE international conference on computer vision, pp. 1017-1025, 2015.
Chen, H., Zhang, Y., WU, S., Wang, X., Duan, X., Zhou, Y., disentan- gled tuning for subject-driven text-to-image generation.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Hoopp-to-promp image editing with cross- atts-atttntionn control. In ICLR, 2023.
and Zhu, W. Disenbooth:
10

---

Event-Customized Image Generation
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Mo, S., Mu, F., Lin, K. H., Liu, Y., Guan, B., Li, Y., and Zhou, B. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7465-7475, 2024.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-688 66511,6661110 2020.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Huang, S., Gong, B., Feng, Y., Chen, X., Fu, Y., Liu, Y., and Wang, D. Learning disentangled identifiers for action- customized text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7797-7806, 2024a.
Nichol, A. Q. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text- ext-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Huang, Z., Wu, T., Jiang, Y., Chan, K. C.,, and Liu, Z. Re- version: Diffusion- relation inversion from images. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024b.
Oquab, M., Darcet. T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F,, El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research.
Jia, X., Isobe, T., Li, X., Wang, Q., Mu, J., Zhou, D., Lu, H.. Tian, L., Sirasao, A., Barsoum, E., et al. Customizing text-to-image generation with inverted interaction. In ACM Multimedia 2024, 2024.
Pratt, S., Yatskar, M., Weihs, L., Farhadi, A., and Kembhavi, A. Grounded situation recognition. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part IV 16, pp. 314 332. Springer, 2020.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 4015-4026, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1931- 1941, 2023.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06//55, 1(2):3 2022.
Li, D., Li, J., and Hoi, S. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36, 2024.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 2024.
Lin, K. H., Mo, S., Klingher, B., Mu, F., and Zhou, B. Ctrl- x: Controlling structure and appearance for text-to-image generation without guidance. Advances in Neural Infor- mation Processing Systems, 12-18999,,, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Liu, Z.,, Feng, R., Zhu, K., Zhang, Y., Zheng, K., Liu, Y., Zhao, D., Zhou, J., and Cao, Y. Cones: concept neurons in diffusson models for customized generation. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 21548-21566, 2023.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted Intervention-MICC
Mao, Q., Chen, L.,, Gu, Y., Fang, Z., and Shou, M. Z. Mag- edit: Localized image editing in complex scenarios via based attention-adjuue guidance. In Proceedings of the 32nd ACM International Conference on Multime- dia, pp. 6842-6850 2024.
2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pp. 234 241. Springer, 2015.

---

Event-Customized Image Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-22510, 2023.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3836-38860 3847, 2023.
Zhou, D., Li, Y., Ma, F., Zhang, X., and Yang. Y. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6818 6828, 2024.
Ruiz, N., Li, Y., Jampani, V., Wei, W., Hou, T., Pritch, Y., Wadhwa, N., Rubinstein, M., and Aberman, K. Hyper- dreambooth: Hypemetworks for fast personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6527-6536, 2024.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E. L., Ghasemipour, K., Gomtijo Lopes, R., Karagol Ayan, B., Salimans, T., al. Photorealistic text-to-imaae dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-36494, 2022.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
Tewel, Y., Gal, R., Chechik, G., and Atzmon, Y. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1- 11, 2023.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, 1921-1930, 2023.
Voynov, A., Chu, Q., Cohen-Or, D., and Aberman, K. p+: Extended textual conditioning in text-to-image genera- tion. arXiv preprint ar arXiv:2303.09522. 2023.
Wang, X., Fu, S., Huang, Q., He, W., and Jiang, H. Ms- diffusion: Multi-subject zero-shot image personalization with layout guidance. In ICLR, 2025.
Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W. Zheng, Y., and Shou, M. Z. Boxdiff: text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7452-7461, 2023.
Xu, S., Huang, Y., Pan, J., Ma, Z., and Chai, J. Inversion- free image editing with natural language. In Conference Computer Vision and Pattern Recognition 2024, 2024.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-imen texa-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.
12

---

Event-Customized Image Generation
Reference Image
ControlNet
MIGC BoxDiff
Image Retrieval
bounding boxes and masks
FreeEvent
event class: encouraging nouns: boy, woman
Target Prompt: boy, woman
100 reference images with event class: encouraging
(a) SWiG-Event Sample
(b) Retrieval- based Evaluation
Figure 7: (a) The SWiG-Event sample. (b) The process of quantitative evaluation and image retrieval.
Appendix
The Appendix is organized as follows:
In Sec. A, we show more implementation details.
In Sec. B, we show more details of the SWiG-Eyent benchmark and the process of quantitative evaluation and image retrieval.
In Sec. C, we show quantitative results for path effectiveness.
In Sec. D, we show the results for attribute generation during event customization.
In Sec. E, we show more results for event-subject customization comparisions.
In Sec. F, we provide more discussion about the hyperparameter setting.
In Sec. G, we provide the discussion of our work's limitations.
In Sec. H, show more qualitative comparison results of event customization the Real-Event.
A. Implementation Details.
The denoising process was set with 50 steps. For entity switching path, for all blocks and layers containing the cross- attention module, we apply the cross-attention guidance during the first 10 steps. And apply the cross-attention regulation during the whole 50 steps. For event transferring path, perform spatial feature injection for block and layer at {decoder block 1 :[layer 1]} during the whole 50 steps. And perform self-attention injection for blocks and layers at (decoder block 1 :layer 1, 2) decoder block 2 :layer 0, 1, 2]. decoder block :layer 0, 1, 2]} during the first 25 steps. We set the classifier-frea guidance scale to 15.0.
B. Details of SWiG-Event and process of image retrieval.
As shown in Figure 7(a), each SWiG-Event sample consists of reference image with labeled bounding boxes and masks for each reference entity, the nouns of each reference entity, and the event class. As shown in Figure 7(b), we constructed the target prompt list of reference entity nouns. The ControlNet takes the semantic map merged from the masks as the layout condition, and Biff takes the bounding boxes with labeled entity nouns as the layout condition.
To compare the image retrieval performance, we retrieved the target image for its corresponding reference image across all the 100 reference images that have the same reference event class.
13

---

Event-Customized Image Generation
Model
CLIP-1
DINO
DreamSimf
CLIP-TY
FreeEvent
0.5771
0.2865
0.3877
0.3445
FreeEvent w/o guidance
0.5140
0.1058
0.2079
0.2979
FreeEvent w/o regulation
0.5459
0.1297
0.3011
0.3205
FreeEvent w/o injection 0.4945
0.0825
0.2694
0.3311
Table 3: Quantitative results for ablation study of two paths Real-Event.
Spiderman @ red apple
1 Spiderman @ green apple
1 Spiderman 2 crystal apple
Reference Image
① old lady @ cake
1 blonde lady @ cake
1 noble lady @ cake
Figure 8: The results of attribute generation during event customization.
C. Quantitative results for Path Effectiveness.
We ran ablations of two paths on Real-Event. We evaluated the CLIP-I, DreamSim, and DINO scores for image similarity between reference images and the generated images. We also evaluated the CLIP-T score for image-text similarity between text prompts and the generated images. Table demonstrates the effectiveness of two paths.
D. Attribute Generation Results.
In this paper, we didn't explicitly model the attributes during generation. However, as the results are shown in Figure 8, since we can generate extra content for background and style by giving corresponding text descriptions, we thus tried to model the attributes by giving extra adjectives to the target prompt as an easy and natural exploration. Meanwhile, to ensure the accurate generation of the attributes, applied the cooss-attention guidance and regulation each attribute using the mask of the entity they describe. As the results shown in Figure 8, our method successfully addresses the attributes of the corresponding entity (e.g., colors, materials, and ages). After all, while the attribute part is not the primary focus of our work, our approach shows potential and effectiveness in addressing it, and we would be happy to conduct further research in our future work.
E. More Event-Subject Customization Comparisons.
In this section, provide more event-subject customization comparisons with exisitng subject swapping and multi-subject customization methods, including Anydoor (Chen et al., 2024c), PhotoSwap (Gu et al., 2023) and MS-Diffuison (Wang et al., 2025). As shown in Figure 9, Freefivent outperforms all other methods.
We need to clarify that the primary focus of this paper is event customization, while event-subject combined customization only potential capability of FreeEvent, rather than a key aspect we intend to emphasize or compare with existing methods. Moreover, FreeEvent serves as a plug-and-play and-play framework for event-subject combined customization, making unsuitable
14

---

Event-Customized Image Generation
Subjects
VO
V/2>
< 13>> V4>
<V5>>
< V6
Reference Image
Target Prompt
Ours
Anydoor
PhotoSwap
MS-Diffusion
VO>
< V3>
① V/>
2 < V3>
D<ra> D<v>> DC v2> Otypewriter
< Vita bear <V8>>
Figure 9: Comparisions of Event-Subject Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
for direct compare with subject customization methods as their settings and applicable scenarios differ. We will provide further discussions and results event-subject customization in future work.
F. Discussion about hyperparameter Setting.
As the first work on event customization, our goal is to enable FreeEvent to perform high-quality event customization across diverse reference images using unified set of hyperparameters (see Appendix Sec A). This default setting ensures faithful event transferring while allowing flexibility in target entity generation.
However, when there is large shape discrepancy between the target entity and the reference entity, the layout information from the reference entity may undesirably affect the appearance of the target entity. This reflects fundamental trade-off between event transferring and entity switching: prioritizing accurate event customization based on the reference image may lead to some compromise in the generation of the target entity. As the example shown in Figure 10, when the shape differences are significant (horse vs. dinosaur), the default setup may result in suboptimal generation (Target Image 1).
straightforward solution to adjust the parameters the event transferring and entity switching paths. Specifically,
15

---

Event-Customized Image Generation
Reference Image
Target Prompt
Target Image1
Target Image2
knight
dinosaur
Cross-attention Guidance
Cross Cross-attention Regulation
Spatial Feature Injection
Self-attention
Injection
Target Image1
10 steps
50 steps
50 steps
25 steps
Target Image2
15 steps
50 steps
30 steps
15 steps
Figure 10: Comparisions of Event Customization with different hyperparameters settings. Different colors and numbers
show the associations between reference entities and their corresponding target prompts.
enhancing the entity switching emphasize the generation of the dinosaur, while slightly reducing the strength of event transferring mitigate layout constraints from the reference entity.
As the shown in Figure 10, provide updated result (Target Image 2) by increasing the number of cross-attention guidance steps (from 10 to 15) and reducing the number of injection steps to 60% of the original. This rebalancing enables more suitable trade-off for this case, resulting in more prominent dinosaur features (e.g., shorter front claws and upright back legs) while still preserving the core event structure.
This case study also demonstrates a practical and accessible way for users adjust the trade-off between event transferring and entity switching according to their own customization needs. Looking ahead, we plan explore more flexible and general solutions, such adaptive parameter scheduling during generation further improve the controllability and diversity of target entities while maintaining event fidelity.
explicit entity switching mechanisms,
G. Limitation.
The main limitation of FreeEvent lies the complexity events and the number entities. The customization effect may be compromised when there are too many entities in an image, especially if they are too small. As the first work this direction, we hope our method can unveil new possibilities for more complex customization and the generation of greater number of richer, more diverse entities. Additionally, since model is built pretrained Stable Diffusion (SD) models, our performance depends on the generative capabilities of SD. This can lead to suboptimal results for entities that the current SD struggles with, such as human faces and hands.
H. More Qualitative Comparision Results.
We show more comparisons on Real-Event in Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15. Specifically, we list them by the order of entity numbers. And we use different combinations of target entities for the same reference image generate diverse target images.
(The figures are in the next pages.)
16

---

Event-Customized Image Generation
Roference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
warrior sword
1 girl 2 carrot
Jedi @ lightsaber
ape robot
① Spiderman @ Batman
ape robot, cartoon
+ Spiderman @ Batman
monkey 2 otter
① tiger 2 otter
① Spiderman batman
Figure 11: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
17

---

Event-Customized Image Generation
Reference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
① robot ? monkey
1 Superman Ironman
① policeman 2 female soldier
orange
① old lady W cake
El
ATSAILS
① player
W baseball
old 2 tennis ball
① woman @ sheep
Egyptian 2 camel
Spiderman D
Figure 12: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
18

---

Event-Customized Image Generation
Reference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
1 book @ sheep
1 bear 2 cake
1 flower
2 backpack
Spiderman
knight 2 dinosaur
Egyptian camel
! monkey 2 horse
princess 2 unicorn
soldier tank
Spiderman
Figure 13: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
19

---

Event-Customized Image Generation
Reference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
bear ? fox K polar bear
i
tiger lion
! bear @ Spiderman 3 panther
cat dog 3 orange
tiger lion @
① dog @ dog K apple
① monkey / lemon orange
robot cake donut
① Spiderman ? orange x
strawberry
orange strawberry
Figure 14: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
20

---

Event-Customized Image Generation
Reference Image
Ours
ControlNet
BoxDiff
PnP
MAG-Edit
DreamBooth
ReVersion
1 robot @ bird < apple
① Spiderman 2 cat @ tiger
Batman 2 wolf 3 bear .aartonn
Tarzan tiger X lion
bear tiger
robot tiger
skeleton 2 starue @ monkey 4 book
Wolverine Spiderman Deadpool MacBook
① Spiderman 2 robot x bear 4 monkey
① Spiderman robot ② bear 4 monkey
Figure 15: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
21


Reviews:
[{"review_id": "dO5XCe4OLM", "reviewer": "Reviewer_yvsP", "summary": "This paper introduces FreeEvent, a diffusion-based image generation technique designed to address the Event-Customized image synthesis problem identified in this study. The authors define this problem by analyzing the progress and limitations of existing controllable image generation methods, particularly highlighting two key challenges: (a) overly simplified customization and (b) the reliance on multiple reference images, which is especially impractical in event-customized generation scenarios.\n\nTo overcome these challenges, FreeEvent incorporates two novel pathways in addition to the standard diffusion denoising process: the Entity Switching Path and the Event Transferring Path. Experimental results on two benchmark datasets demonstrate the effectiveness of FreeEvent, showcasing its superiority over existing methods. Furthermore, an ablation study confirms the contributions of the two proposed pathways to the overall performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Link for additional results: https://anonymous.4open.science/r/FreeEvent-EB1D/README.md\n\n\n\n\n\n## Q1: The defination of ``event\"\n\n\n**A1:** In early NLP tasks, event was defined “an occurrence of an activity that happens at a particular time and place” [1]. Later, in visual scene analysis, works as GSR [2] concretized activities and situations by representing them through entities’ roles (nouns), locations (bounding boxes), and their associated verbs (actions, poses, interactions).\n\nIn our work, considering the context and limitations of existing customization works, where current methods independently address only simple actions or interactions, we use \"event\" to extend these concepts to a more general setting. This allows us to cover a broader range of visual scenarios that include diverse actions, poses, relations, and interactions. At the same time, following prior research on event definitions, we maintain entity-centric settings by explicitly considering their roles (text prompts for entities) and locations (masks). Meanwhile, this also provide a straightforward and adaptable way to measure an event, i.e., by the number of entities. This is also consistent with existing methods for measuring activities and situations, which often rely on the number of involved entities and their bounding boxes.\n\n\nAfter all, we want to emphasize that our definition of “event” aligns with existing research while refining and specifying it within the context of customization. As the first work in this direction, our goal is to generalize customization beyond isolated actions, poses, or interactions and unify them under a broader and more structured definition. We will refine our descriptions, particularly regarding the use of terms like “ALL” in our revision.\n\n\n[1] Event Extraction: A Survey. 2022\n[2] Grounded situation recognition with transformers. 2021\n\n\n## Q2: About Event Transferring Path\n\n\n**A2:** First, we want to clarify that this is not a shift or reduction in the definition of “event”. Instead, we are analyzing the event customization from the perspective of visual spatial information, while subject customization is more related to visual appearance. Specifically, the actions, poses, and interactions of entities in a reference image are closely tied to its spatial information. Actions and poses for each entity often correlate with fine-grained shape details (e.g., limb positioning, body orientation), while interactions between different entities are more linked to the semantic layout (e.g., the relative locations, physical contact). This perspective provides a novel approach for implementing event customization, as it allows us to leverage spatial representations to effectively capture and transfer event-related details.\n\nBesides, the usage of spatial features and self-attention maps are empirically based on common obervactions that they are highly related to image structure and layout informations. And it has also been widely acknowledged by existing diffusion works.\n\nAfter all, the event transferring path is designed based on the analysis of events and the reasonable application of existing findings. We will refine our descriptions in the revision to ensure clarity, and avoid terms like “essentially” to prevent potential misunderstandings.\n\n\n\n\n\n## Q3: The evaluation metrics\n\n\n**A3:** First, directly evaluating the alignment of complex events remains a challenging and open problem. As introduced in Sec 4.2 and Appendix (Sec B), our quantitative experiments are designed to **reproduce** the reference image while maintaining the same reference event and entities. Thus, our evlautions follow the principle of **whether generated images are matched/aligned/similar with their reference images** from different perspectives: 1) Global image similarity: the retrieval results and the CLIP-I score. 2) Entity similarity: the CLIP-T score. 3) Event similarity: the verb detection results. Together, these metrics give a comprehensive evaluation of the event customization quality based on our specific settings. Additionally, we also evluated FID score to ensure the overall quality of generated images. \n\n\n\n\n\n\n## Q4: More benchmark methods\n\n\n**A4:** Since checkpoint for MIGC++ is now not available, we incorporated MIGC for a more comprehensive quantitative comparison. We also evluated two more metrics for global image similarity suggested by reviewer t7sW (DINO score and DreamSim). As shown in Table R1,R3 and Figure R1 in the link. FreeEvent succesfully outperforms MIGC.\n\nSince Anydoor is designed for image customization, we compared it with FreeEvent on the event-subject customization setting. As shown in Figure R2 in the link. FreeEvent outperforms Anydoor and other subject customization method. Besides, we need to clarify that **event-subject combined customization** is only a potential capability of FreeEvent, rather than a key aspect we intend to emphasize or compare with existing methods.", "claims_and_evidence": "The experimental results validate the capability of the proposed FreeEvent in customizing the identities of multiple instances in the reference image, as well as their interactions, which are represented by attributes such as pose, action, and relationships. However, whether such customization of identities and interactions fully aligns with the concept of \"event\" remains somewhat subjective. Additionally, the adequacy of the evaluation metrics employed in the experimental section to effectively measure the quality of event customization is open to question. For further details, please refer to the following section.", "methods_and_evaluation_criteria": "This section highlights the primary concerns of this study, which will be elaborated as follows:\n\n1) As stated in the abstract, the authors define an \"event\" as **ALL** specific actions, poses, relations, or interactions between different entities in the scene. How do the authors ensure that **ALL** such attributes, including the most subtle and nuanced ones, are adequately considered in the customization process? I strongly encourage the authors to provide a clearer and more measurable definition of the concept of \"event\" to enhance its interpretability and reproducibility.\n\n2) Furthermore, in the \"Event Transferring Path\" described in Section 3.3, the definition of \"event\" shifts to \"essentially the structural, semantic layout, and shape details of the image,\" which are said to be captured by \"spatial features and self-attention maps\" as suggested by existing studies (so convenient isn't it?). This apparent inconsistency in the definition of \"event\" is confusing. Earlier, the term \"event\" encompassed abundant attributes such as actions, poses, relations, and interactions, yet in the technical implementation, it is reduced to \"structural, semantic layout, and shape details of the image.\" Given the elevated expectations set by the authors, this simplification raises serious concerns. Specifically: Why are \"actions, poses, relations, and interactions among instances\" considered equivalent to \"structural, semantic layout, and shape details of the image\"? Why can \"structural, semantic layout, and shape details of the image\" be adequately represented by spatial features and self-attention maps? These crucial questions must be explicitly addressed and thoroughly discussed to clarify the proposed method's rationale.\n\n3) The authors employ three evaluation metrics to measure the effectiveness of the proposed method. However, it remains unclear whether these metrics can accurately reflect the capability of event customization. Instead of merely introducing the implementation details of these metrics, I encourage the authors to establish a clear connection between the chosen metrics and specific aspects of the method's performance. For example, How does the CLIP score relate to the quality of event customization? Does higher CLIP score necessarily reflect that the 'event' is better transferred? A more detailed discussion linking the evaluation metrics to the core objectives of the study would significantly strengthen the validity of the experimental results. Also, if the definition of 'event' is '**ALL** specific actions, poses, relations, or interactions', why not use more straightforward evaluation metrics to measure the alignment of these concrete aspects?", "theoretical_claims": "This is not applicable, as no novel theoretical claims have been proposed in this study.", "experimental_designs_or_analyses": "I have two primary concerns regarding the experimental designs:\n\n1) The rationale behind the selection of evaluation metrics: As described in the \"Methods and Evaluation Criteria\" section, the justification for choosing the specific metrics is insufficiently detailed. A more thorough explanation is required to establish the relevance and appropriateness of these metrics in assessing the proposed method.\n\n2) The limitations of the benchmark methods used for comparison: The methods included for benchmarking are somewhat restricted and already outdated, as most were proposed in 2023. I strongly encourage the authors to incorporate more recent and relevant methods formally published in 2024 and beyond. For instance, AnyDoor and MIGC (CVPR 2024), as well as MIGC++ (TPAMI 2025), could provide a more comprehensive and up-to-date comparison.", "relation_to_broader_scientific_literature": "As discussed by the authors, this study primarily addresses the following two limitations: 1) restricted and overly simplistic interactions among instances, and 2) the dependency on multiple reference images.", "essential_references_not_discussed": "To the best of my knowledge, there is no such an essential reference missing, although the benchmark methods are sowehow out-of-date.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "Line 407-408, doesn't -> does not", "questions_for_authors": "Please refer to Methods And Evaluation Criteria for my primary concerns regarding this work. I would like to raising my rating if my conerns are well-addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Smj7tXr6Kd", "reviewer": "Reviewer_t7sW", "summary": "In the area of customized image generation, existing methods face the limitations of simplified customizations and insufficient data. To address these challenges, this paper defines a novel task, event-customized image generation, covering complex layout, actions, interactions between more than two objects. The training free method consists of two paths: the entity switching path and the event transferring path, via manipulation of cross attention, self attention and spatial features. The authors conclude that the proposed method can be a plug-and-play module for other models and is able to support more complex scenes. They also collect two benchmarks for the evaluation of this new task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Link for additional results: https://anonymous.4open.science/r/FreeEvent-EB1D/README.md\n\n\n\n\n\n## Q1: About story \n\n\n**A1:** Our intention in introducing subject customization and action customization is to provide background on the broader customization task and naturally introduce event customization. Our main focus is event customization, while event-subject combined customization is only a potential capability of FreeEvent rather than a key aspect we aim to emphasize or compare with existing methods. We acknowledge your concern and will refine our writing to clarify the relationships between these tasks, ensuring a clearer distinction between FreeEvent’s primary capabilities and its potential abilities.\n\n\n## Q2: More baselines and metrics\n\n**A2:** Since GLIGEN and LayoutDiffusion were proposed in 2023, considering reviewer yvsP’s comments and limited rebuttal time, we incorporated a more recent layout-guided baselines MIGC. Additionally, we reported the DINO score and DreamSim. As shown in Table R1,R3 and Figure R1 in the link, FreeEvent successfully outperforms all methods.\n\n\n\n\n\n## Q3: Design better metrics\n\n**A3:** We need to first emphasize that evaluating the accuracy of complex events is a challenging and open problem. Similar to other customization tasks (e.g., subject or action customization), existing works also face evaluation difficulties and primarily rely on similarity metrics or user studies. Moreover, designing new metrics requires extensive effort, experimentation, and validation, which go beyond the scope of this work. Regarding metrics like AP, they are not suitable in our case, as we do not impose strict constraints on the exact shape or layout of the generated target entities at pixel level.\n\nIn our experiments, we tried to evaluate event customization from multiple perspectives: 1) Global image similarity: retrieval results and CLIP-I score. 2) Entity similarity: CLIP-T score. 3) Event similarity: verb detection results. Together with user study, we believe these metrics provide a comprehensive evaluation of the event customization quality. As the first work for this direction, we hope our evaluation and metrics serve as a valuable starting point, and we will explore more suitable metrics in future work.\n\n\n## Q4: About subject customization\n\n\n**A4:** We provided more event-subject customization comparisons in Figure R2 in the link, and FreeEvent outperforms all other methods. \n\nHowever, we need to clarify that the primary focus of this paper is **event customization**, while **event-subject combined customization** is only a potential capability of FreeEvent, rather than a key aspect we intend to emphasize or compare with existing methods. Moreover, FreeEvent serves as a plug-and-play framework for event-subject combined customization, making it unsuitable for direct compare with subject customization methods as their settings and applicable scenarios differ.\n\nAs mentioned in Q1, we will refine our writing to better clarify the relationships between these tasks and provide further discussions and results on event-subject customization in future work.\n\n\n## Q5: Quantitative results for ablation study\n\n\n**A5:** We ran ablations of two paths on Real-Event. We evaluated the image and text similarities with reference image and text prompt, respectively. Table R2 in the link demonstrates the effectiveness of two paths.\n\n\n## Q6: Choice of layers\n\n\n**A6:** Our choices follow widely acknowledged empirical insights from diffusion models:\n\n1) During object generation, early denoising steps determine layout and position, while later steps refine appearance.\n\nSo we only apply attention guidance in early steps, which also saves the inference time.\n\n2) Spatial features and attention maps from decoder layers encode localized layout information.\n3) Injecting features in deeper layers can better preserve the structure but risk appearance leakage.\n\nSo we only inject spatial features at first decoder layer but attention maps at all decoder blocks.\n\n\n\n\n## Q7: Does FreeEvent works with other models\n\n\n\n**A7:** For Unet-based models, the insights from Q6 can be easily transfered, making FreeEvent easily integrable into SD-XL. Specifically, for 50-step DDIM sampling in SD-XL v1.0: 1) spatial feature injection: {decoder block 1}. 2) Self-attention injection: {decoder block 1,2,3}. 3) cross-attention guidance in first 10 steps. Figure R3 in the link shows various results.\n\nFor DiT-based models, due to their distinct architecture from UNet, direct injection of FreeEvent is challenging. However, the core ideas of entity switching and event transferring can still be adapted using similar insights:\n1) Leveraging cross-attention on text tokens to guide target object generation.\n2) Modifying attention on visual tokens to control structure generation.\n\nAs the first work in this direction, we hope FreeEvent serves as an effective benchmark and look forward to exploring its potential on DiT models in future work.", "claims_and_evidence": "- In the teaser, abstract and introduction, the authors first list the tasks of subject customization and action customization, and then indicate that the proposed task/method can address the challenges of the older tasks. However, in most part of the paper, subject customization is not mentioned (except that in Fig. 6 it is briefly discussed), and FreeEvent seems more like a pose/layout-guided class-conditioned generation. The story needs to be refined by either removing the subject customization or improving FreeEvent to naturally support subject personalization.\n- In introduction, the authors claim that \"gathering images that depict the exact same action or interaction is challenging\", which is a very reasonable constraint for the previous method to learn a specific action.\n- At the end of Sec. 3.3, the paper claims that the \"framework can be easily combined with subject customization methods\"; however, it is limited to UNet architectures (based on SD v2.1), so it is questionable whether this method can be extended to DiT-based methods (e.g., FLUX, SD3, etc.).", "methods_and_evaluation_criteria": "- In Tab. 1, to evaluate the image alignment of the generated images with the references, CLIP-I is used. Why not use other image-based metrics such as DINO score and DreamSim?\n- To demonstrate the advancement of the proposed model, more layout-guided T2I methods should be included for comparison, such as GLIGEN and LayoutDiffusion (Tab. 1).\n- In this new task, current metrics cannot effectively measure the accuracy of the poses/interactions. Better metrics should be designed for this task to evaluate the layout/pose preservation (e.g., AP).\n- Subject customization is ignored in most part of the paper. There should be more discussion and results shown other than Fig. 6 and Fig. 10: 1) comparison with existing methods, such as PhotoSwap and MS-Diffusion; 2) use reference images with more complex textures, such as DreamBench.", "theoretical_claims": "Equation 3), 4), 5) look correct and reasonable to me.", "experimental_designs_or_analyses": "- The issues of the experiments mainly lie in the metrics design and baseline selection. Please refer to \"Methods And Evaluation Criteria\" above.\n- To demonstrate that the proposed method can be easily integrated into other models, there should be more discussion on injecting FreeEvent into other models, such as SD-XL.\n- The current ablation study only shows visual results (Fig. 5), which cannot reflect the general performance. More quantitative results will. help.", "relation_to_broader_scientific_literature": "In previous papers of image customization or conditioned T2I, the use case of replacing the entities from an image with specific classes/concepts is usually overlooked. In response, this paper defines a new task for this missing scenario.", "essential_references_not_discussed": "Please see the methods mentioned in \"Methods And Evaluation Criteria\".", "other_strengths_and_weaknesses": "- Although the chosen metrics cannot fully reflect the effectiveness of the methods, the paper includes a user study in Sec. 4.5, where human preference demonstrate the quality of FreeEvent.\n- The method section is well written and easy to understand.\n- The left part of Fig. 2a is a little hard to understand (the noise maps), may need to be revised.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "- Does the proposed method work for architectures other than UNet? E.g., DiT?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3uG23GmXw", "reviewer": "Reviewer_SH2p", "summary": "This paper introduces a new task called event customization, which aims to generate new images that maintain the same event depicted in a reference image. An event contains specific actions, poses, relationships, and interactions between different entities within a scene. To address this task, a training-free FreeEvent is proposed that integrates an entity switching path and an event transferring path into the diffusion denoising process. Additionally, two benchmarks are proposed to evaluate the proposed method both qualitatively and quantitatively.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Link for additional results: https://anonymous.4open.science/r/FreeEvent-EB1D/README.md\n\n\n\n\n\n## Q1: Additional baseline methods\n\n\n**A1:** Considering both reviewer yvsP and t7sW’s comments and the limited rebuttal time, we have incorporated a more recent baselines MIGC (2024) for a more comprehensive comparison. As shown in Table R1,R3 and Figure R1 in the link, our FreeEvent succesfully outperforms it.\n\n\n\n\n## Q2: Compare with image inpainting methods\n\n\n**A2:** We compared our method with the text-based image inpainting approach Blended Latent Diffusion on Real-Event. As shown in Table R3 and Figure R1 in the link, FreeEvent consistently outperformed it.  \n\nWhile image inpainting methods can generate target entities in specified regions, they primarily focus on independently adding or replacing objects at the entity level, often overlooking interactions between them. As a result, directly using masks and text prompts for inpainting can disrupt the event details in the reference image, altering entity poses, actions, and interactions.\n\nAdditionally, Sec. 4.3 presented a comparison with MAG-Edit, a localized image editing method that utilizes reference entity masks for region-specific modifications. Our results demonstrated that FreeEvent also surpasses MAG-Edit in preserving event details and interactions.\n\n\n## Q3: About Novelty\n\n\n**A3:** Thanks for your concerns. We want to first emphasize that we made three folds of contributions in this paper: 1) The new and meaningful event-customized image generation task. 2) The first training-free method for event customization. 3) Two evaluation benchmarks for event-customized image generation. Specifically, for our training-free method FreeEvent, we provide more discussion below.\n\n- **Motivation.** Based on the two main components of the reference image, i.e., entity and event, we proposed to decompose the event customization into two parts: 1) Switching the entities in the reference image to target entities. 2) Transferring the event from the reference image to the target image. Inspired by the observation that the spatial features and attention maps have been utilized to control the layout, structure, and appearance in text-to-image generation, we further designed the two corresponding paths to address the two parts. While these observations have been widely recognized in previous works, we are the first to integrate them to address this new task in a training-free manner. This approach demonstrates a thoughtful analysis of the task and a strategic application of existing technologies. \n\n- **Improvements.** We also made several specific improvements to better address the event customization task. 1) For entity switching, besides the cross-attention guidance, we further regulate the cross-attention map of each entity to avoid the appearance leakage between each target entity. 2) For event transferring, in contrast to previous works [A, B] that perform DDIM inversion on reference images, we directly use forward diffusion. This further reduces the appearance leakage from the reference image and saves the inversion cost and additional model inference time. \n\nWhile FreeEvent does incorporate some existing methods, its design is rooted in a thoughtful analysis of the new task and a strategic application of existing insights. Furthermore, we also introduced specific improvements, enabling it to address this new task more effectively and efficiently. FreeEvent has proved its effectiveness and efficiency in a wide range of experiments, beating existing controllable generation, image editing, and customization works. As the first work in this direction, we hope our method can unveil new possibilities for more complex customization, meanwhile serving as a challenging baseline for future works.\n\n[A] Plug-and-play diffusion features for text-driven image-to-image translation. CVPR, 2023.\n\n[B] Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. ICCV, 2023.\n\n\n\n## Q4: Reference and target concepts with different shape\n\n\n**A4:** Many samples in our paper show the performance of FreeEvent when handling target concepts that has different shape with the one in reference image.\n\n1) Figure 4, row 2, dog -> bird.\n2) Figure 4, row 4, dog -> Spiderman.\n3) Figure 4, row 6, human -> bear.\n4) Figure 11, row 8, woman -> otter.\n5) Figure 11, row 9, man -> tiger.\n6) Figure 13, row 7, human -> monkey.\n7) Figure 15, row 6, human -> robot.\n\nAs the above results shown, ControlNet and the localized image editing models tend to generate the target entities strictly matching the shape of their corresponding reference entities, which appears incongruous. On the contrary, the entities generated by FreeEvent not only match the layout of the reference entity but also keep it harmonious with different shapes, allowing for more diverse generation of target entities.", "claims_and_evidence": "The claims are clear and easy to understand.", "methods_and_evaluation_criteria": "The FreeEvent is reasonable. The proposed benchmarks are diverse and adequate for evaluation, and the metrics used are suitable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Extensive experiments have been conducted. However, there are some concerns:\n\n1. In Table 1, only DreamBooth and BoxDiff are compared. It is recommended to include additional methods to ensure a more comprehensive evaluation.\n2. It would be beneficial to compare with text/reference-based image inpainting tasks, as these methods can also generate target concepts in specified regions.", "relation_to_broader_scientific_literature": "While the task of event customization is new, the key techniques have been explored in recent methods. For example, the entity switching path is similar to that in [1], and the event transferring path resembles the approach used in [2].\n\n[1] Chefer, Hila, et al. \"Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.\"  \n[2] Cao, Mingdeng, et al. \"Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.\"", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n1. The proposed event customization task is interesting and new.\n2. The proposed method is straightforward and easy to follow.\n\nWeaknesses\n\n1. The novelty may be limited, as discussed in the \"Relation to Broader Scientific Literature\" section.\n2. In the figures in the paper, the target concepts usually share a similar shape to the one in the reference image. How well does the proposed method perform if they have different shapes, for example, a human -> a tiger?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "My main concerns have been listed in the weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxK2hAdQuF", "reviewer": "Reviewer_SWq5", "summary": "This paper introduces event-customized image generation, a new task that extends customized image generation to complex scenes which includes detailed actions, poses, relationships, and interactions between entities. To tackle this new task, it proposes FreeEvent, a training-free method that enhances the diffusion denoising process through two key paths: the entity switching path for precise entity control, and the event transferring path to guide event generation. It further collects two benchmarks for this task. Experimental results on the two benchmarks demonstrate the advantage of FreeEvent.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Link for additional results: https://anonymous.4open.science/r/FreeEvent-EB1D/README.md\n\n\n\n\n\n## Q1: Adding prompt for event\n\n\n**A1:** We have analyzed the impact of incorporating an explicit “event” description in the ablation studies (Appendix, Section C). Specifically, as shown in Figure 8, adding verb descriptions to the prompt negatively affects the appearance of entities in the generated images. The primary reason is that these descriptions may not align well with the pretrained diffusion model. Moreover, for practical application, accurately describing events in complex scenes can be challenging for users. Since our method already enables precise extraction and transfer of reference events, users are not required to explicitly specify events in the target prompt. This further highlights the practicality of FreeEvent.\n\nApart from verb description, we also analyzed the influence of adding the background, style and attribute description. As shown in Section C and Section E, our method can accurately generate these additional elements without affecting the event. Theses results all demonstrates FreeEvent’s strong generalization capability.\n\n\n\n## Q2: Setting of the user study \n\n\n**A2:** Thanks for the suggestion. We have adjusted the setting of “choose three targets” to “select at least one and up to three” and conducted a new user study. This updated setting better reflects expert judgment and highlights the differences between different methods. We also included two more baselines suggested by other reviewers. Table R3 in the link shows that our approach achieves the best performance in human judgments (HJ).\n\n\n\n\n## Q3: Distribution of SWiG-Event \n\n**A3:** The distribution of entity numbers (1, 2, 3, 4) in SWiG-Event is (20%, 30%, 30%, 20%). As the first benchmark for event customization, we aimed for an overall balanced distribution while making slight adjustments—specifically, increasing the proportion of two- and three-entity cases. This decision ensures that the benchmark remains well-rounded: single-entity events may be too simplistic, while four-entity events could be overly challenging. By striking this balance, SWiG-Event serves as a well-designed benchmark for effectively evaluating model capability.", "claims_and_evidence": "This paper introduces a new task, accompanied with two datasets and a newly proposed method. However, more details on the datasets should be provided. For example, authors should provide some necessary statistics on the two datasets to better prove the coverage and diversity, and describe how the two newly collected datasets differ from their original counterparts.", "methods_and_evaluation_criteria": "The methodology includes an entity switching path to guide the target entity generation and an event transferring path for event generation. The design appears to be reasonable. Regarding to the evaluation criteria, the authors use both qualitative and quantitative measurements such as CLIP scores. Besides that, they also employ subjective user study. The evaluation appears to be comprehensive.", "theoretical_claims": "There is no theoretical claim or proofs. However, the model designs seem reasonable.", "experimental_designs_or_analyses": "The experimental designs and analyses seem to be comprehensive and of good soundness and validity.", "relation_to_broader_scientific_literature": "This paper proposes the task of event customized image generation, which considers the control factors in image generation task more comprehensively than any other related studies. It paves a new way towards the image generation by the control of more complicated semantics and knowledge.", "essential_references_not_discussed": "The reference information seems adequate.", "other_strengths_and_weaknesses": "Strength:\n\n1 This paper is well-written and easy to follow.\n\n2 The motivation for proposing the Event-Customized Image Generation task is strong and clear, with a thorough analysis of the limitations of existing methods. The new benchmarks also enhanced the foundation of this research.\n\n3 The proposed FreeEvent is well-designed by decomposing event generation into two key components, ensuring a reasonable and effective solution. As a training-free approach, it is also computationally efficient for real-world applications.\n\n4 The FreeEvent demonstrated its effectiveness through extensive experiments over various baseline methods. Moreover, it produced several inspirational results, such as its combination with subject customization and background images. These findings further highlight the potential of the proposed task and method for broader applications.\n\nWeakness:\n\n1 It seems that the target prompt only contains the entities, what about adding the prompt of the ``event”? For example, using the event class to describe the event more detailly, will this lead to better results? The author should provide more analysis and clarifications.\n\n2 The setting of the user study seems somewhat unreasonable. As the authors state, “every expert was asked to choose three targets.” However, based on the visualization results presented in the paper, many samples do not contain three clearly preferable images. This setting may weaken the credibility of the results. The authors should consider improving the evaluation or statistical method to more accurately reflect the user study’s findings.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "The paper claims that “each reference image in SWiG-Event contains 1 to 4 entities”. While SWiG-Event ensures diversity in event types (i.e., 50 kinds), what is the specific distribution of entity numbers in SWiG-Event? Is it evenly distributed to fairly evaluate the model’s capability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes FreeEvent, a diffusion-based method for event-customized image synthesis. The core innovation is a disentanglement-recomposition framework enabling fine-grained event adaptation with minimal references. After the rebuttal, most of the concerns have been addressed, and reviewers raised their scores to accept or weakly accept. The work establishes a new paradigm for event-aware generation, warranting acceptance.
Decision: Accept (poster)