Interactive-Action Image Generation via Synthetic Physical Priors

Authors: Xuanyi Liu, Pengliang Ji, Siwei Ma, Yizhi Wang
Primary area: generative models
Keywords: synthetic data, interactive action, generative model

Abstract:
While diffusion-based text-to-image generation has made notable advancements, generating accurate images containing interactive actions remains a challenge due to the lack of inherent physical and spatial priors. To address this problem, we propose a novel pipeline that synthesizes a dataset enriched with physical priors using a graphics engine, combined with a captioning technique. Building on the dataset, we introduce a distillation-structured fine-tuning method, where a teacher network assists in inverting the semantics of interactive actions, leveraging the synthesized priors effectively. This fine-tuning method disentangles the synthetic data features while mitigating random misalignment during the fine-tuning process. Extensive experiments demonstrate that our method not only achieves state-of-the-art results but also highlights the synthetic data's potential to be applied more broadly in enhancing the generation of interactive action images.

Decision: reject
Reviewer scores: 6, 8, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730463374964

thread_type:
official_review

<thread_object>
creator: reviewer_w9rr

<content>
summary:
This paper addresses the issue of text-to-image generation models struggling to accurately depict interactive actions (e.g., "a person picks up a cup"). To address this, the authors generate a synthetic dataset enriched with physical constraints using a graphics engine. They train the model through a teacher-student network with distillation learning, enhancing the model's ability to capture spatial and interactive relationships. The approach reduces alignment issues between subjects and objects, improving realism in generated images.

strengths:
* The distillation-based method is a straightforward yet reasonable approach.
* The authors address a challenging task effectively.
* They propose a new metric for evaluation.

weaknesses:
* Limited Interaction Complexity: The paper focuses on single-object manipulation, which seems somewhat limited in scope.
* The qualitative results seem somewhat lacking. It seems this issue should be addressed through the proposed distillation method.

questions:
* In the paper, they compare performance with ReVersion fine-tuned on their dataset, which is fair; however, the original ReVersion paper also demonstrates strong performance. For a more comprehensive comparison, the authors should include additional experimental results using the original (non-fine-tuned) ReVersion model.

* The ablation study shows that 'and' improves semantic understanding, while 'or' enhances visual representation. What is the reason for this conflict?

* Can the model perform in more complex interactions, such as multi-object manipulation or tool use (e.g., hammering a nail, tightening a screw with a screwdriver)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600152977

thread_type:
official_review

<thread_object>
creator: reviewer_esr5

<content>
summary:
The paper presents a method to enhance diffusion-based text-to-image generation for interactive actions by synthesizing a dataset with physical priors using a graphics engine. It introduces a distillation-based fine-tuning approach that improves understanding of interactive actions and reduces misalignment. Experiments show that this method achieves state-of-the-art results and expands the use of synthetic data in generating interactive images.

strengths:
1. This paper propose a novel method that synthesizes a dataset enriched with physical priors using a graphics engine.
2. This paper introduce a distillation-structured fine-tuning method using SD.
3. Qualitative results are good compared to the baselines.

weaknesses:
1. It is challenging to evaluate this as a new approach. However, the authors propose a new metric, the IAS score, and evidence is needed to demonstrate that this metric in the evaluation.
2. There is a lack of quantitative analysis for the ablation studies. (such as, preposition word ..)
3. The authors released the code in the supplementary materials. Is it possible to view more samples of the dataset or generated results?
4. I am curious about if SD-XL or other models were used instead of SD1.5.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621650992

thread_type:
official_review

<thread_object>
creator: reviewer_wdtc

<content>
summary:
This paper proposes a method to improve the capacity of text-to-image diffusion models to generate interactions by fine-tuning the model on synthetic data. To address the domain gap between the synthetic data and real images, the author developed a distillation approach to preserve the performance of the mode in generating the realistic style while still learning the interactions from the synthetic data.

strengths:
This paper studies the interesting and important problem that visual generative models lack the proper knowledge about the physical world and are not able to properly generate physical-plausible actions. The authors propose reasonable fine-tuning-based methods and collect synthetic data from a simulator to address this problem. The authors show that the existing methods do not work well due to the synthetic-real domain gap. Ablation studies are performed to help understand the function of each component.

weaknesses:
My major concern about this paper is its motivation to improve the ability of text-to-image models to generate human/animal-object interactions. With the improved performance of the base generation, I have been seeing this as less and less of a problem. In many of the examples shown in the papers where SD1.5 fails to generate correct interactions, the latest models, like SD3 and Flux, can do pretty well. That being said, I do believe that there are many actions that cannot be accurately described by the language. However, I don't see current methods as a solution to that, as there are cases where the learned motions are not clearly matched with the training images. For example, in the third row of the teaser image, the robot arm picks up the objects with two hands, while the generated images mostly contain a subject holding an object with one hand. Also, I also have the following concerns:

1. There are some conflicts between the different losses, i.e., the distillation loss would conflict with learning the interaction through the synthetic data, which is also verified by the ablation study in Figure 8 (c). The optimal trade-off between two losses may not be the same for different interactions.

2. The issue with overfitting the synthetic data has been studied, and the function of the proposed distillation loss is quite similar to the class-specific prior preservation loss. Although the authors claim that real data is not common, robot manipulation datasets like BridgeDatav2 and DROID datasets do exist. 

3. There exists another gap that the interaction between the robot arm and objects is quite different from how humans interact with objects - the robot arm typically does not have a similar structure compared with human hands. Instead, one could just collect more human-object interaction data. And there do exist such datasets, like Epic Kitchen, Hands23, and Ego4D.

questions:
I do not have additional questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695855052

thread_type:
official_review

<thread_object>
creator: reviewer_gbhf

<content>
summary:
This paper first points out that existing diffusion models tend to struggle with verbs that describe relationships between objects. To address this, the authors propose a method for fine-tuning these diffusion models to better capture action-related information. They generate synthetic data representing actions using robot simulation model, SAPIEN, and train a special token following a textual inversion framework. To account for the domain gap between synthetic and real images, they employ a distillation learning approach for fine-tuning. The proposed method shows improved results compared to previous studies on the evaluation benchmark designed by the authors.

strengths:
- The use of robot simulation to generate datasets for certain actions is an interesting and innovative approach.
- The paper clearly explains the challenges encountered during the development and the method is well described without ambiguity.
- The authors introduce new metric for evaluating the proposed task, and clearly present comparison results with previous studies.

weaknesses:
- The proposed method is limited to simple actions that can be performed in robot simulations, such as picking up or pulling something. More complex actions like tickling or shaving cannot be represented in this way, making it difficult to train on broader action concepts.
- The use of distillation learning increases the training load compared to previous studies.

questions:
Regarding the weakness mentioned above, how can a dataset be generated for more complex actions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730818024093

thread_type:
official_review

<thread_object>
creator: reviewer_ro2y

<content>
summary:
This work focuses on the interactive actions image generation based on pretrained text-to-image diffusion models, where a customized synthetic dataset with interactive actions is provided to finetune the pretrained model to improve the spatial and interactive action aware diffusion models.The finetune stage is akin to distillation procedure to address the problem of disentanglement and text-visiual misalignment. The experimental results demonstrate the effectiveness, especially the interactive actions between objects, of the proposed model.

strengths:
- The paper is easy to follow. This work clearly identifies the weakness of generating interactive actions in current t2i models and proposes a effective solution to address the limitation.
- The method design is well-motivated, including the usage of curated dataset and the specific distillation training framework to take advantage of both pretrained prior and the new learned knowledge.

weaknesses:
- While this work demonstrates the effectiveness on several interactive actions, it is unclear how well the method generalizes to a wider range of more complex interactions. Moreover, is the model capable of dealing with actions related to more than two objects.
- The motivation of IAScore is clear, however, its reliance on detection introduces potential errors. A more detailed analysis of the reliability and limitations of this metric would make it more credible.

questions:
Please refer to the weakness part for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732351365816

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
Dear Reviewers, ACs, SACs, and PCs,

Thanks for your insightful feedback on our manuscript: Interactive-Action Image Generation via Synthetic Physical Priors.

We are encouraged that you find our soundness is good (ro2Y, gBhf, Esr5), the presentation is good (ro2Y, gBhf, Esr5), the contribution is good (ro2Y, gBhf, Esr5).

We have **uploaded a new version** of our manuscript containing more experiments **(Figure 1,6,7,9,10,11)**. 

We summarize the main actions taken during the rebuttal:
1) We conduct new experiments to show that our method could handle a more complex action.
2) Additional experiments of more ReVersion results.
3) Additional experiments of our method on real robot dataset DROID.
4) Additional experiments of Stable Diffusion 3 and FLUX.
5) More explanation of IAScore.
6) Clarify the motivation of our paper and several confusions or misunderstandings regarding our paper.

Within this short rebuttal period, we did our best to reply to your questions, and we will incorporate your feedbacks in the final version. In the upcoming interaction period, if you have any questions towards our responses, please feel free to let us know.

Sincerely,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The objective of this work is to improve the interactive actions in image generation using physical priors. A customized synthetic dataset with interactive actions is provided to finetune the pretrained model to improve the spatial and interactive action aware diffusion models. To account for the domain gap between synthetic and real images, they employ a distillation learning approach for fine-tuning. Experimental results show that the method can improve the interactions between objects.