Event-Customized Image Generation

## Abstract

Abstract Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity ( i . e ., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient “exactly same” reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation . Given a single reference image, we define the “event” as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent . Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.

## 1 Introduction

Recently, large-scale pre-trained diffusion models (Dhariwal & Nichol, ; Nichol et al., ; Ramesh et al., ; Rombach et al., ; Saharia et al., ) have demonstrated remarkable success in generating diverse and photorealistic images from text prompts. Leveraging these unparalleled creative capabilities, a novel application — customized image generation (Gal et al., ; Ruiz et al., ; Chen et al., ) — has gained increasing attention for generating user-specified concepts. Significant progress has already been made in subject-customized image generation (Ye et al., ; Chen et al., ). As shown in Figure 1(a), given a set of user-provided subject images, existing methods can accurately capture the unique appearance features of each subject (e.g., corgi) with a special identifier token, enabling creative rendering in new and diverse scenarios. Moreover, they can seamlessly integrate multiple subjects into cohesive compositions, preserving their distinctive characteristics while adapting them to novel contexts.

Beyond the appearance of different entities (i.e., humans, animals, and objects) in the images, pioneering approaches have been developed to customize the user-specified actions (Huang et al., ), interactive relations (Huang et al., ) and poses (Jia et al., ) between the entities. As shown in Figure 1(b), these methods attempt to capture the single-entity action (e.g., handstand) or interactions (e.g., back to back) between two entities that co-exist in the given reference images and transfer them to the synthesis of action- or interaction-specific images with new entities.

However, for real-world scenes that typically involve multiple entities with more complex interactions (e.g., Figure 1(c), row three: three humans are discussing in front of a computer with different poses), these works (Huang et al., ; ; Jia et al., ) still face notable limitations. 1) Simplified Customization. Current action customization (Huang et al., ) focuses solely on the basic actions of a single person. Similarly, interaction customizations (Huang et al., ; Jia et al., ) are limited to basic interactive relations or poses between just two entities. There is a lack of exploration into more complex and diverse actions or interactions that involve multiple humans, animals, and objects. Additionally, while these methods typically perform well when generating images with the same type of entity (e.g., all monkeys or all cats), they struggle when faced with more diverse and complex entities and their combinations. These narrow focuses and limitation on entity generation have strictly limited their abilities to customize more complex and diverse scenes with creative content. 2) Insufficient Data. To capture specific actions or interactions, existing methods (Huang et al., ; ; Jia et al., ) tend to represent them by learning corresponding identifier tokens, which can be further used for generating new images. However, for each action, or interaction, these training-based processes typically require a set of reference images (e.g., 10 images) paired with corresponding textual descriptions across different entities. Unfortunately, each action or interaction is highly unique and distinctive, i.e., gathering images that depict the exact same action or interaction is challenging. As shown in Figure 1(b), there are still significant differences in the same action (e.g., handstand) between different reference images, which thus compromises the accuracy of learned tokens, leading to inconsistencies in action between generated images. This insufficient data issue for identical action or interaction has severely limited the practicality and generalizability of these methods.

To address these limitations and extend customized image generation to more complex scenes, we propose a new and meaningful task: event-customized image generation. Given a single reference image, we define the “event” as all actions and poses of each single entity, and their relations and interactions between different entities. As shown in Figure 1(c), event customization aims to accurately capture the complex and diverse event from the reference image to generate target images with various combinations of target entities. Since it only needs one single reference image, the event customization also eliminates the need for collecting “exactly same” reference images.

To solve this challenging task, we proposed a novel *training-free* event customization method, denoted as FreeEvent. Based on the two main components of the reference image, i.e., entity and event, FreeEvent decomposes the event customization into two parts: 1) Switching the entities in the reference image to target entities. 2) Transferring the event from the reference image to the target image. Following this idea, alongside the general denoising process of diffusion generation, we designed two extra paths: entity switching path and event transferring path. Specifically, entity switching path guides the localized layout of each target entity for entity generation. Event transferring path further extracts the event information from the reference image and then injects it into the denoising process to generate the specific event. Through this direct guidance and injection, FreeEvent offers a significant advantage over existing methods by eliminating the need for time-consuming training. Furthermore, as shown in Figure 1(c), FreeEvent can also serve as a plug-and-play framework to combine with subject customization methods, generating creative images with both user-specified events and subjects.

Moreover, as a pioneering effort in this direction, we also collected two evaluation benchmarks from the existing dataset (i.e., SWiG (Pratt et al., ) and HICO-DET (Chao et al., )) and the internet for event-customized image generation, dubbed SWiG-Event and Real-Event, respectively. Both benchmarks include reference images featuring diverse events and entities, along with manually crafted target prompts. Extensive experiments demonstrate that our approach achieves state-of-the-art performance, enabling more complex and creative customization with enhanced practicality and generalizability.

In summary, we make several contributions in this paper: 1) We propose the novel event-customized image generation task, which extends customized image generation to more complex scenes in real-world applications. 2) We propose FreeEvent, the first training-free method for event customization, which can be further combined with subject customization methods for more creative and generalizable customizations. 3) We collect two evaluation benchmarks for event-customized image generation, and our FreeEvent achieves outstanding performance compared with existing methods.

## 2 Related Work

Text-to-Image Diffusion Generation. Diffusion models (Ho et al., ; Nichol & Dhariwal, ; Song et al., ) have emerged as a leading approach for image synthesis. The text-to-image diffusion models (Nichol et al., ; Ramesh et al., ; Saharia et al., ) further inject user-provided text descriptions into the diffusion process via pre-trained text
encoders. After trained on large-scale text-image pairs, they have shown great success in text-to-image generation. Different from these models that operate the diffusion process on pixel space, the latent diffusion models (LDMs) (Rombach et al., ) propose to perform it on latent space with enhanced computational efficiency. Besides, existing works (Hertz et al., ; Tumanyan et al., ; Cao et al., ; Alaluf et al., ) have discovered the spatial feature and attention maps in LDMs contain localized semantic information of the image and the layout correspondence between textual conditions. As a result, these features and attention maps have been utilized to control the layout, structure, and appearance in text-to-image generation. This can be achieved either through a plug-and-play feature injection (Tumanyan et al., ; Xu et al., ; Lin et al., ) or by computing specific diffusion guidance (Epstein et al., ; Mo et al., ) for generation. In this paper, we utilize the pre-trained LDM Stable Diffusion (Rombach et al., ) as our base model.

Subject Customization. This task aims to generate customized images of user-specified subjects. Current mainstream subject customization works mainly focus on 1) Single subject customization, including learning specific identifier tokens (Gal et al., ), finetuning the text-to-image diffusion model (Ruiz et al., ; ), introducing layer-wise learnable embeddings (Voynov et al., ) and training large-scale multimodal encoders (Gal et al., ; Li et al., ). 2) Multi-subject composition, including cross-attention modification (Tewel et al., ), constrained model fine-tuning (Kumari et al., ), layout guidance (Liu et al., ), and gradient fusion of each subject (Gu et al., ). In conclusion, these works are all tailored to capture the appearance of the entities in the image, without considering the customization of actions or poses.

Action and Interaction Customization. They aim to generate customized images with co-existing actions or interactions in user-provided reference images. ReVersion (Huang et al., ) first proposes to customize specific interactive relations by optimizing the learnable relation tokens. ADI (Huang et al., ) makes progress in customizing specific actions for a single subject. And a following work (Jia et al., ) further extends it to learning interactive poses between two individuals. However, all these works only focus on simplified customization of some basic actions and interactions, and their effect is strictly limited by the insufficient data of reference images. In contrast, our proposed event customization only requires one reference image, and our training-free framework FreeEvent can achieve effective customization of complex events with various creative target entities.

## 3 Methods

### 3.1 Preliminary

Latent Diffusion Model. Generally, the LDMs include a pretrained autoencoder and a denoising network. Given an image $x$, the encoder $\mathcal{E}$ maps the image into the latent code $z_{0}=\mathcal{E}(x)$, where the forward process is applied to sample Guassian noise $\mathbf{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$ to it to obtain $z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$ from time step $t\sim\left[1,T\right]$ with a predefined noise schedule $\bar{\alpha}$. While the backward process iteratively removes the added noise on $z_{t}$ to obtain $z_{0}$, and decodes it back to image with the decoder $x=\mathcal{D}(z_{0})$. Specifically, the diffusion model is trained by predicting the added noise $\mathbf{\epsilon}$ conditioned on time step $t$ and possible conditions like text prompt $\mathrm{P}$. The training objective is formulated as

$$ $\mathcal{L}_{\mathrm{LDM}}=\mathbb{E}_{z\sim\mathcal{E}(x),P,\epsilon\sim \mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t};t,P)\|_{2}^{2} \right].$ (1) $$

where $\epsilon_{\theta}$ is the denoising network.

Diffusion Guidance. The diffusion guidance modifies the sampling process (Ho et al., ) with additional score functions to guide it with more specific controls like object layout (Xie et al., ; Mo et al., ) and attributes (Epstein et al., ; Bansal et al., ). We express it as

$$ $\hat{\epsilon}_{t}=\epsilon_{\theta}(z_{t};t,P)-s\cdot\mathbf{g}(z_{t};t,P),$ (2) $$

where $\mathbf{g}$ is the energy function and $s$ is a parameter that controls the guidance strength.

### 3.2 Task Definition: Event-customized Image Generation

In this section, we first formally define the event-customized image generation task. Given a reference image $I^{R}$ involves $N$ reference entities $\mathrm{E}^{R}=\{R_{1},\dots,R_{N}\}$, we define the “event" as the specific actions and poses of each single reference entity, and the relations
and interactions between different reference entities. Together we have the entity masks $\mathrm{M}=\{m_{1},\dots,m_{N}\}$, where $m_{i}$ is the mask of its corresponding entity $R_{i}$. The event-customized image generation task aims to capture the reference event, and further generate a target image $I^{G}$ under the same event but with diverse and novel target entities $\mathrm{E}^{G}=\{G_{1},\dots,G_{N}\}$ in the target prompt $\mathrm{P}=\{w_{0},\dots,w_{N}\}$, where $w_{i}$ is the description of the target entity $G_{i}$, and each target entity $G_{i}$ should keep the same action or pose with its corresponding reference entity $R_{i}$. As the example shown in Figure 2, given the reference image with four reference entities (e.g., three people and one object), the event-customization aims to capture the complex reference event and generate the target image with a novel combination of different target entities (e.g., skeleton, statue, monkey, book).

### 3.3 Approach

Overview. We now introduce the proposed training-free event customization framework FreeEvent. Specifically, we decompose the event-customized image generation into two parts, 1) generating target entities (i.e., switching each reference entity to target entity), and 2) generating the same reference event (i.e., transferring the event from the reference image to the target image). Following this idea, we design two extra paths for the diffusion denoising process of event customization, denoted as the entity switching path and the event transferring path, respectively. Generally, as shown in Figure 2, the generation of $I^{G}$ starts by randomly initializing the latent $z^{G}_{T}\sim\mathcal{N}(0,\mathbf{I})$, and iteratively denoise it to $z^{G}_{0}$. During this denoising process, the entity switching path guides the generation of each target entity through cross-attention guidance and regulation based on the target prompt $\mathrm{P}$ and reference entity masks $\mathrm{M}$. The event transferring path extracts the spatial features and self-attention maps from the reference image $I^{R}$, and then injects them to the denoising process. The final $z^{G}_{0}$ is then transformed back to the target image $I^{G}$ by the decoder.

Figure: Figure 2: The overview of pipeline. Given the reference image, the event customization is overall a general diffusion denoising process with two extra paths. 1) The entity switching path guides the generation of each target entity through cross-attention guidance and regulation 2) The event transferring path injects the spatial features and self-attention maps from the reference image to the denoising process. The final $z^{G}_{0}$ is then transformed back to target image $I^{G}$ by the decoder.
Refer to caption: x2.png

U-Net Architecture The Stable Diffusion (Rombach et al., ) utilizes the U-Net architecture (Ronneberger et al., ) for $\epsilon_{\theta}$, which contains an encoder and a decoder, where each consists of several basic encoder/decoder blocks, and each encoder/decoder block further contains several encoder/decoder layers. Specifically, as shown in Figure 3(a), each U-Net encoder/decoder layer consists of a residual module, a self-attention module, and a cross-attention module. For block $b$, layer $l$, and timestep $t$, the residual module produces the spatial feature of the image as $\mathbf{f}$. The self-attention module produces the self-attention map as $\mathbf{SA}=\mathrm{Softmax}(\frac{\mathbf{Q}_{s}\mathbf{K}_{s}^{\mathrm{T}}}{
\sqrt{d}})$, where $\mathbf{Q}_{s}$ and $\mathbf{K}_{s}$ are query and key features projected from the visual features. For text-to-image generation, the cross-attention module further produces the cross-attention map between the text prompt $\mathrm{P}$ and the image as $\mathbf{CA}=\mathrm{Softmax}(\frac{\mathbf{Q}_{c}\mathbf{K}_{c}^{\mathrm{T}}}{
\sqrt{d}})$, where $\mathbf{Q}_{c}$ is the query features projected from the visual features, and $\mathbf{K}_{c}$ is the key features projected from the textual embedding of $\mathrm{P}$.

Entity Switching Path. This path aims on generating target entities $\mathrm{E}^{G}=\{{G}_{1},\dots,G_{N}\}$ in $I^{G}$ by switching each refenrece entity ${R}_{i}$ to ${G}_{i}$ based on the target prompt $\mathrm{P}$ and reference entity masks $\mathrm{M}$. And the key is to ensure each target entity ${G}_{i}$ is generated at the same location as their corresponding reference entity ${R}_{i}$ and avoid the appearance leakage between different entities. Inspired by prior works (Hertz et al., ; Chen et al., ) that utilize the cross-attention maps to control the layout of text-to-image generation, we apply the cross-attention guidance and regulation to achieve the entity switching.

As shown in Figure 2(a), at the timestep $t$ of the denoising process, we first obtain the latent for entity switching as $z^{A}_{t}=z^{G}_{t}$, we then input $z^{A}_{t}$ together with the target prompt $\mathrm{P}$ into the U-Net, and calculate the cross-attention maps as $\mathbf{{CA}}^{A}$. As shown in Figure 3(b), we then introduce an energy function to bias the cross-attention of each token $w_{i}$ as:

$$ $\mathbf{g}(\mathrm{{CA}}^{A}_{i},m_{i})=(1-\frac{\mathrm{{CA}}^{A}_{i}*m_{i}}{ \mathrm{{CA}}^{A}_{i}})^{2}$ (3) $$

where $\mathrm{{CA}}^{A}_{i}$ is the cross-attention map of token $w_{i}$. Optimizing this function encourages the cross-attention maps of each target entity ${G}_{i}$ to obtain higher values inside the corresponding area specified by $m_{i}$, and further guide the localized layout of each target entity. We calculate the gradient of this guidance via backpropagation to update the latent $z^{G}_{t}$:

$$ $z^{G}_{t}=z^{A}_{t}-\sigma^{2}_{t}\eta\bigtriangledown_{z^{A}_{t}}\sum_{i\in N }\mathbf{g}(\mathrm{{CA}}^{A}_{i},m_{i})$ (4) $$

where $\eta$ is the guidance scale and $\sigma_{t}=\sqrt{(1-\bar{\alpha_{t}}/\bar{\alpha_{t}}}$. Additionally, to avoid the appearance leakage between each target entity, we further regulate the cross-attention map of each token within its corresponding area. Specifically, for cross-attention maps $\mathbf{{CA}}^{G}$ calculated at timestep $t$ during the denoising process, we have:

$$ $\mathrm{{CA}}^{G}_{i}=m_{i}\odot\mathrm{{CA}}^{G}_{i}$ (5) $$

where $\mathrm{{CA}}^{G}_{i}$ is the cross-attention map of token $w_{i}$.

Figure: Figure 3: (a) The architecture of the U-Net layer. (b) The process of cross-attention guidance and regulation. (c) The process of spatial feature and self-attention injection.
Refer to caption: x3.png

Event Transferring Path. This path aims to extract the specific reference event from the reference image $I^{R}$, including the action, pose, relation, or interactions between each reference entity, and transferring them to the target image $I^{G}$. Meanwhile, from the perspective of image spatial information, the event is essentially the structural, semantic layout, and shape details of the image. Thus, based on the observation that the spatial features and self-attention maps can be utilized to control the image layout and structure (Tumanyan et al., ; Xu et al., ; Lin et al., ), we perform spatial feature and self-attention map injection to achieve the event transferring.

Specifically, as shown in Figure 2(b) we first get the latent code of the reference image $z^{R}_{0}=\mathcal{E}(I^{R})$, and at each time step $t$ during the denoising process, we obtain $z^{R}_{t}$ via the diffusion forward process. We then input $z^{R}_{t}$ into the U-Net to extract the spatial features and self-attention maps of the reference image as $\mathbf{f}^{R}$ and $\mathbf{{SA}}^{R}$. Parallelly, for the denoising process, we input $z^{G}_{t}$ together with the target prompt $\mathrm{P}$ into the U-Net, and calculate the spatial features and self-attention maps for the generated image as: $\mathbf{f}^{G}$ and $\mathbf{{SA}}^{G}$. Then, as shown in Figure 3(c), we perform the injection by directly replacing corresponding spatial features and self-attention maps:

$$ $\mathbf{f}^{G}\leftarrow\mathbf{f}^{R}\quad\textrm{and}\quad\mathbf{{SA}}^{G} \leftarrow\mathbf{{SA}}^{R}.$ (6) $$

*Highlights.* By applying cross-attention guidance and regulation on each text token, our attention-guided entity switching can also be used to generate target entities of user-specified subjects, i.e., represented by specific identifier tokens. Thus, our framework can be easily combined with subject customization methods to generate creative images with both customized events and subjects.

## 4 Experiments

### 4.1 Experimental Setup

Evaluation Benchmarks. In order to provide sufficient and suitable conditions for both quantitative and qualitative comparisons on this new task, we collect two new benchmarks(^1^11Due to the limited space, more details are left in the Appendix.). 1) For quantitative evaluation, we present *SWiG-Event*, a benchmark derived from the SWiG (Pratt et al., ) dataset, which comprises 5,000 samples with various events and entities, i.e., 50 kinds of different actions, poses, and interactions, where each kind of event has 100 reference images, and each reference image contains 1 to 4 entities with labeled bounding boxes and nouns. 2) For qualitative evaluation, we present *Real-Event*, which comprises 30 high-quality reference images from the HICO-DET (Chao et al., ) and the internet with a wide range of events and entities (e.g., animal, human, object, and their combinations). We further employ the Grounded-SAM (Kirillov et al., ; Ren et al., ) to extract the mask of each entity.

Baselines. To evaluate the effectiveness of our method, we compared it with several kinds of state-of-the-art baselines. For conditioned text-to-image generation baselines, we compared with the training-based method ControlNet (Zhang et al., ) and the training-free method BoxDiff (Xie et al., ). For localized editing baselines, we compared with training-free methods PnP (Tumanyan et al., ) and MAG-Edit (Mao et al., ). For customization baselines, we compared with training-based methods Dreambooth (Ruiz et al., ) and ReVersion (Huang et al., ).

Implementation Details. We use Stable Diffusion v2-1-base as the base model for all methods, and images are generated with a resolution of 512x512 on a NVIDIA A100 GPU1.

### 4.2 Quantitative Comparisons

In this subsection, we compare our method with conditional text-to-image generation baselines ControlNet (Zhang et al., ) and BoxDiff (Xie et al., ) on the SWiG-Event benchmark.

Retrieval-based Evaluation. Each reference image in SWiG-Event contains reference entities together with labeled event class, bounding boxes, nouns, and their corresponding masks. Specifically, we construct the target prompt as a list of reference entity nouns, i.e., we ask all the methods to *reproduce* the event of the reference image with the same reference event and same reference entities. Additionally, ControlNet takes the semantic map merged from the masks as the layout condition, and BoxDiff takes the bounding boxes with labeled entity nouns as the layout condition1. To evaluate the customization quality of 5,000 target images, we retrieved each target image for its corresponding reference image across all the 100 reference images that have the same reference event class. Specifically, we extracted the image feature of each image through a pre-trained CLIP (Radford et al., ) visual encoder and calculated the cosine similarities for image retrieval.

Results. As shown in Table 1, our FreeEvent achieves superior retrieval performance over both ControlNet and BoxDiff. Specifically, the Recall@1 of FreeEvent is significantly higher than that of the baselines, and it even surpasses the Recall@10 of the baselines. This demonstrates that the target images generated by FreeEvent better preserve the characteristics of the reference event and entity, showcasing a stronger capability for event customization.

**Table 1: Performance of our model and state-of-art conditional text-to-image generation models on SWiG-Event. The Recall@k represents that among the top-k images with the highest similarity to the target image, its corresponding reference image is included.**
| Model | Recall@1 | Recall@5 | Recall@10 |
| --- | --- | --- | --- |
| ControlNet (Zhang et al., ) | 10.64 | 26.12 | 36.82 |
| BoxDiff (Xie et al., ) | 8.60 | 22.48 | 32.08 |
| Free-Event (ours) | 41.12 | 63.02 | 72.74 |

Figure: Figure 4: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Refer to caption: x4.png

### 4.3 Qualitative Comparisons

We compare FreeEvent with a wide range of state-of-the-art baselines on the Real-Event benchmark, including the conditioned text-to-image generation methods ControlNet (Zhang et al., ) and BoxDiff (Xie et al., ), the localized image editing methods PnP (Tumanyan et al., ) and MAG-Edit (Mao et al., ), the image customization methods Dreambooth (Ruiz et al., ) and ReVersion (Huang et al., ).

Setting. For each reference image in Real-Event, we manually constructed target prompts with various combinations of different target entities. Specifically, ControlNet takes the semantic map and BoxDiff takes the labeled bounding boxes as the layout conditions. MAG-Edit takes the reference entity masks for localized editing. Dreambooth and ReVersion learn event-specific identifier tokens for text-to-image generation.

Results. As shown in Figure 4, we can observe: 1) Conditional text-to-image generation models ControlNet and BoxDiff can only maintain the rough layout of each entity and struggle to capture the detailed action, pose, or interaction between different entities. And they both failed to match the generated entity with the desired target prompt. 2) For localized image editing methods PnP and MAG-Edit, while they can capture the reference event, they both struggle to accurately generate the target entities, and suffer from severe appearance leakage between each target entity (e.g., orange and strawberry in row four, tiger and lion in row six), and sometimes even failed to edit and output the original content. 3) The subject-customization model Dreambooth and the relation-customization model ReVersion both failed to generate satisfying results. As discussed before, these training-based methods require multiple reference images and are unable to learn the specific event when facing only one reference image. 4) Obviously, our FreeEvent successfully achieves the customization of various complex events with novel combinations of target entities. Meanwhile, the ControlNet and the localized image editing models tend to generate the target entities strictly matching the mask of their corresponding reference entities (e.g., bird in row three), which appears very incongruous. On the contrary, the entities generated by FreeEvent not only match the layout of the reference entity but also keep it harmonious. After all, while we use the reference entity mask to guide the generation of each target entity, the cross-attention guidance focuses on directing the overall layout of each target entity and does not restrict their detailed appearance, allowing for a more diverse generation of target entities1.

### 4.4 Ablations

Effectiveness of Entity Switching Path and Event Transferring Path. We first run ablations to verify the effect of two proposed paths during event customization.

*Results.* As the results are shown in Figure 5(a), we can observe: 1) For the entity switching path, removing the cross-attention guidance results in the failure of target entities generation (e.g., the ape, the meat), and removing cross-attention regulation leads to the appearance leakage between entities (e.g., the tiger and lion, the skeleton and statue). 2) After removing the event transferring path, although the target entities can be generated, the reference events are completely lost (i.e., the pose, action, relations, and interactions between each entity). These results all corroborates the effect of two paths in event customization.

Influence of Different Target Prompts. Noteablly, in our paper, the target prompt only contains the nouns of the target entities, we then run the ablations to analyze the influence of different descriptions (i.e., verb, background, style) in the target prompt for event customization.

*Results.* From Figure 5(b) we can observe: 1) Adding *verb* description leads to a certain degree of negative impact on entity appearance (e.g., the head of the ape, the face of the monkey) since these verbs may not be aligned with the model. Besides, accurately describing events in complex scenes can be challenging for users. Therefore, since FreeEvent can already achieve precise extraction and transfer of the reference events, users do not need to describe the specific events in the target prompt, which further demonstrates FreeEvent’s practicality. 2) FreeEvent can accurately generate extra contents for the *background* and *style*. Although there may be some detailed changes in the entity’s appearance compared to the original output, these do not affect the entity’s characteristics or the event. This also demonstrates FreeEvent’s strong generalization capability.

Figure: Figure 5: Ablations of the proposed paths and the target prompt. The “guidance" and “regulation" denote the cross-attention guidance and cross-attention regulation in the entity switching path, respectively. The “injection" denotes the event transferring path.
Refer to caption: x5.png

Combination of Event and Subject Customization. We further validate the ability of our framework to combine with subject customization methods to generate target entities with user-specified subjects, i.e., represented by identifier tokens. For an easy exploration, we took DreamBooth (Ruiz et al., ) to learn identifier tokens for each subject and replaced the Stable Diffusion models in Figure 2 with the fine-tuned one.

*Results.* As shown in Figure 6, FreeEvent can effectively generate various given subjects in specific events, showing great potential for Event-Subject customization.

Figure: Figure 6: Results of Event-Subject Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Refer to caption: x6.png

### 4.5 User Study

Setting. We conducted user studies on Real-Event to further evaluate the effectiveness of FreeEvent. Specifically, we invited 10 experts and gave them a
reference image, a target prompt, and seven target images generated by different models. They are asked to choose the three target images that they believe demonstrate the best results in event customization, taking into account the generation effects of the events and entities, as well as the overall coherence of the images. We prepared 30 trials and asked the experts to give their judgments. The target image which got more than six votes is regarded as human judgment.

Results. As shown in Table 2, FreeEvent achieves better
performance on human judgments (HJ) compared with all the baseline models.

**Table 2: Results of the user studies on the Real-Event. “HJ" denotes the count of human judgment.**
| Model | Ours | ControlNet | BoxDiff | PnP | MAG-Edit | DreamBooth | ReVersion |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HJ | 30 | 12 | 2 | 19 | 9 | 1 | 0 |

## 5 Conclusion

In this paper, we proposed a new image generation task: Event-Customized Image Generation. It focuses on the customization of complex events with various target entities. Meanwhile, we proposed the first training-free event-customization framework FreeEvent. To facilitate this new task, we also collected two evaluation benchmarks from existing datasets and the internet, dubbed SWiG-Event and Real-Event, respectively. We validate the effectiveness of FreeEvent with extensive comparative and ablative experiments. Moving forward, we are going to 1) extend the event customization into other modalities, e.g., video generation; 2) explore advanced techniques for the finer combination of different customization works, e.g., subject, event, and style customizations.

Limitations. The main limitation of FreeEvent lies in the complexity of events and the number of entities. The customization effect may be compromised when there are too many entities in an image, especially if they are too small. As the first work in this direction, we hope our method can unveil new possibilities for more complex customization and the generation of a greater number of richer, more diverse entities. Additionally, since our model is built on pretrained Stable Diffusion (SD) models, our performance depends on the generative capabilities of SD. This can lead to suboptimal results for entities that the current SD struggles with, such as human faces and hands.

Potential Negative Societal Impacts. Since FreeEvent can seamlessly integrate with subject customization methods to generate target entities based on user-specified subjects, this capability also raises the same concerns about the potential misuse of pretrained SD models for malicious applications (e.g., Deepfakes) involving real human figures. To address this, it is essential to implement robust safeguards and ethical guidelines, similar to the security measures and NSFW content detection mechanisms already present in existing diffusion models.

## Appendix

The Appendix is organized as follows:

- $\bullet$
In Sec. A, we show more implementation details.
- $\bullet$
In Sec. B, we show more details of the SWiG-Event benchmark and the process of retrieval-based quantitative comparisons.
- $\bullet$
In Sec. C, we show more qualitative comparison results of event customization on the Real-Event.
.

## Appendix A Implementation Details.

The denoising process was set with 50 steps. For entity switching path, for all blocks and layers containing the cross-attention module, we apply the cross-attention guidance during the first 10 steps. And apply the cross-attention regulation during the whole 50 steps. For event transferring path, we perform spatial feature injection for block and layer at $\left\{\mathrm{decoder\,block\,1:}[\mathrm{layer}\,1\right]\}$ during the whole 50 steps. And perform self-attention injection for blocks and layers at $\left\{\mathrm{decoder\,block\,1:}[\mathrm{layer}\,1,2],\mathrm{decoder\,block
\,2:}[\mathrm{layer}\,0,1,2],\mathrm{decoder\,block\,3:}[\mathrm{layer}\,0,1,2
]\right\}$ during the first 25 steps. We set the classifier-free guidance scale to 15.0.

## Appendix B Details of SWiG-Event and retrieval-based comparisons.

As shown in Figure 7(a), each SWiG-Event sample consists of a reference image with labeled bounding boxes and masks for each reference entity, the nouns of each reference entity, and the event class. As shown in Figure 7(b), we constructed the target prompt as a list of reference entity nouns. The ControlNet takes the semantic map merged from the masks as the layout condition, and BoxDiff takes the bounding boxes with labeled entity nouns as the layout condition. We then retrieved the target image for its corresponding reference image across all the 100 reference images that have the same reference event class.

Figure: Figure 7: (a) The SWiG-Event sample. (b) The process of the retrieval-based comparisons.
Refer to caption: x7.png

## Appendix C More Qualitative Comparision Results.

We show more comparisons on Real-Event in Figure 8, Figure 9 and Figure 10.

Figure: Figure 8: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Refer to caption: x8.png

Figure: Figure 9: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Refer to caption: x9.png

Figure: Figure 10: Comparision of Event Customization. Different colors and numbers show the associations between reference entities and their corresponding target prompts.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Can the author list some specific applications in reality of the new-proposed task setting to convince me of its value?
In some cases, the interaction semantic and poses perservation requirement is conflicted, as mentioned in weaknesses. Can the author explain the priority of such requirements?
2, Can the proposed approach enable more flexible content generation, like specify the background image? Can the entity-subject customization ability be expand to more diverse subject customization other than only some celebrities?
3. Can the evaluation setting of retrieval-based experiment be more clearly explained with some visualized examples?
4. The author should provide more experiments to validate the effectiveness of approach, like report the performance on interaction detection task.
soundness: 3
strengths: - The proposed training-free approach is easy to adopt and can generate satisfying results.
- The paper is clear and easy to follow.
summary: This paper proposes a new task called Event-Customized Image Generation, which aims to not only control subjects, but also customize all specific actions, poses, relations, or interactions between different entities in the scene. Then it designs a training-free approach, by alternating the cross-attention within U-Net to eanble target subject generation, and utilizing origin spatial feature and self-attention to enale entity transfer. Experiments on two data-sets validated the effectiveness.
weaknesses: 1. The task setting of event-customized image generation raises following concerns: The poses of each entity, as well as the overall spatial configuration of generated image is restricted to maintain identical with reference image, which hinders the diversity of generated images. Further, even if the poses are successfully maintained, the interaction semantic may be influenced. In the example of “skeleton, statue” in Figure(1), when the laptop is changed to a book, the interaction semantic between human and object is changed. Is it against the proposed definition of task setting? Further, can this approach be integrated with other components to enable generation on specific backgroud image? 
2. For Method design : (1) the spatial features and self-attention maps of reference images are adopted to inject event information, how to ensure such direct injection can prevent the leakage of subject information of reference image. (2) The auther claims that by equipping with subject-customized image generation approaches, it can generate entity-subject customized images by injecting target concept identifier tokens, can this approach be integrated with more advanced customization approaches to enable more flexible customization of regular concepts rather than just some celebrities visualized in Figure 6. 
3. In Experiment, (1) The task setting for quantitative evaluation is confusing. How to reproduce the reference image if all the entities within the image stay the same pose as input condition. Providing one visualized example would be helpful to understand it. (2) Quantitative evaluation only adopts a retrieval-based experiment, which is not convicing enough, and the user study for qualitative evaluation only adopts 30 samples. (3) Further, the interaction semantic of generated image is not verified across all the experiments. Considering that HICO-DET and SWIG both contains annotations of interaction, the generated images should also evaluate  corresponding interaction detection performance.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: see weakkness
soundness: 3
strengths: -	The method seems correct
-	The writing and organization seem clear
-	The efforts to formalize a new benchmark and task, although from existing datasets
summary: This paper proposed a new task of extending the customized image generation to more complex scenes for general real-world applications.  The event incorporates specific actions, poses, relations and interactions between different entities in the scene. Free-event the method is proposed to solve the task by adding two extra paths along the diffusion denoising process, entity switching path, and event transferring paths. In addition, for evaluation, the paper proposed two more benchmarks.
weaknesses: -	The major issue to argue in this paper is the usage of existing technical contributions for this new task. No theoretical justifications and no further big novel ideas. Whether the technical contributions are limited is worth discussion. The experimental results proved the evaluations. The new task and datasets are also worth reporting.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. The definition of the "event-customized image generation" is somehow not clear. "Given a single reference image, we define the event as all actions and poses of each single entity, and their relations and interaction between different entities." The entity part is fully addressed, however, how about the attribute part? Is there any quantitative definition for how complex can leads to an event?
2. The training free framework and the focus on entity and attribute is somehow similar to a prior work, Imageanything[A]. Please give clear discussion and comparison with this work regarding motivation, methodology, and frameworks.
[A] Lyu Y, Zheng X, Wang L. Image anything: Towards reasoning-coherent and training-free multi-modal image generation[J]. arXiv preprint arXiv:2401.17664, 2024.
3. What kind of similarity metric is used to assess the methods in Tab.1?
4. Could the author include more metrics to assess the proposed FreeEvent and the existing methods, such as FID, CLIP score?
soundness: 2
strengths: 1. Clear limitation analysis of existing works, i.e., simplified customization and insufficient data.
2. Good motivations for addressing the proposed task.
summary: This paper introduces a new task, event-customized image generation, which aims at accurately capturing the complex task and generate customized images with various target entities. Meanwhile, a training-free event customization method, FreeEvent is proposed to solve the event-customized image generation task. The FreeEvent consists of two paths along side the general diffusion denoising process, i.e., the entity switching path and the event transferring path. The entity switching path applies cross-attention guidance and regulation for target entity generation while the event transferring path injects the spatial feature and self-attention maps from the reference image to the target image for event generation.
weaknesses: 1. Unclear Definition of "Event-Customized Image Generation": The paper’s definition of "event-customized image generation" lacks clarity, especially regarding the complexity and scope of events. Although the paper explains entity interactions, it does not address attributes adequately. Additionally, there is no quantitative measure for defining the complexity level that qualifies as an event, leaving ambiguity in how "event" is operationalized.

2. Lack of Comparison with Related Work: The training-free framework and emphasis on entity and attribute handling appear to align closely with prior work, specifically the ImageAnything framework [A]. The paper fails to compare itself with ImageAnything in terms of motivation, methodology, and structural framework, missing an opportunity to clarify its novelty and improvements over similar approaches.

3. Limited Information on Similarity Metrics: The similarity metric used for assessments in Table 1 is not specified, leaving readers uncertain about the criteria for evaluation. Without this information, the results may be hard to interpret, limiting the reproducibility and transparency of the evaluation.

4. Insufficient Performance Metrics: The paper could enhance its assessment by including standard image generation metrics, such as FID (Fréchet Inception Distance) and CLIP scores, for a more comprehensive comparison. Relying on a limited set of metrics may not provide a well-rounded evaluation, which could affect the perceived robustness of the proposed method.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - Clarity and Readability: The writing quality of the manuscript is commendable, making the intent and message of the paper easy to comprehend.
- Resource-Efficient Methodology: The proposed method is training-free, which is notably advantageous in terms of computational resource requirements, making it accessible and feasible even in resource-constrained environments.
summary: This paper presents FreeEvent, a novel approach to customized image generation, targeting complex event-specific scenes rather than just entity appearances or basic interactions. FreeEvent tackles event-customized image generation, where an "event" includes detailed actions, poses, and relationships between entities. FreeEvent introduces two innovative pathways: the Entity Switching Path for entity-specific guidance and the Event Transferring Path for spatial feature and attention transfer.
weaknesses: - Overclaim: The manuscript introduces the 'event-customized image generation task' as a novel contribution. However, this task appears to have been previously addressed in the work titled "Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation" presented at CVPR 2024.
- Lack of Methodological Novelty: The two pathways proposed in the paper appear to primarily combine existing methods. The Entity Switching Path employs a strategy similar to Attend-and-Excite for controlling content generation in specific locations, while the Event Transferring Path largely follows approaches resembling MasaCtrl. These methods are widely established and have been extensively discussed across numerous publications, which may limit the perceived innovation in the paper’s methodology.
- Suboptimal Qualitative Results: The qualitative results presented show room for improvement. Specifically, I noticed that the retention of person identifiers is weak in Figures 1c and 6, which raises questions about the underlying cause; the authors should clarify this aspect further. Additionally, the prompt “P: skeleton, statue, monkey, book” appears four times throughout the paper. Reducing the repetition of this sample would likely enhance the diversity and impact of the results shown.

Meta-review: This work obtains two positive and two negative scores. After checking the paper, the AC is still concerned about the limited technical novelty of the proposed application. 

As the author said, "While these observations have been widely recognized in previous works, we are the first to integrate them to address this new task in a training-free manner. This approach demonstrates a thoughtful analysis of the task and a strategic application of existing technologies." This indicates that this work seems to be an extension of the existing techniques for the new proposed task. 

Moreover, the definition of event-customized is unclear. 

Even though the AC acknowledges this paper's well-written and sufficient experiments, this work is not enough for top conference papers.

Decision: Reject