Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a modification of stable diffusion to better reflect per-object attributes in a sentence to the corresponding objects in the image and prevent missing objects.
For example, feeding "a red car and a white sheep" into stable diffusion produces and image with a red car and a red sheep.
For experiments, the authors introduce a new dataset for evaluation.
strength_and_weaknesses: Strengths

(+) The proposed method does not require training samples.

(+) This paper introduces a benchmark dataset to evaluate attribute binding.

(+) The experiments visualizes the causes of incorrect attribute binding (Figure 8).

(+) Sec. 2.1 provides thorough background.

Weaknesses

(-) Missing analysis: How do the attention maps from NPs and sentence evolve throughout the timetsteps?

(-) The contributions are limited.
- The attributes are limited to colors.
- The authors do not provide comparisons in the scene graph setting.
- Evaluation of the compositionality is limited to user study.
- Preventing objects from missing is not evaluated.

clarity,_quality,_novelty_and_reproducibility: (+) The human evaluation protocol is clearly described.

(-) Abstract and intro
- we observe that attribution-binding and compositional capabilities are still considered major challenging issues. <- The readers cannot guess what attribution-binding and compositional capability from abstract.
- Abstract has peripheral aspects (e.g., cross-attention) instead of the key component (tokenizing sentences).

(-) What is "functionality between attention maps and token semantics"?

(-) t and t_i are confusing.

(-) Eq. 4 maintains the image layout  <- What is the original image layout? Maybe the image layout from a full sentence embedding?

(-) Assuming above guess is correct, why does Mt remain unchanged although the sentence embedding and the token embeddings are different?

(-) c denotes hierarchies but NPs in a same hierarchy have different cs in Figure 3.

typo
- downsample and upsampling blocks -> downsampling and upsampling blocks

summary_of_the_review: I value the colors being correct in the resulting images. However, contribution of this paper is limited as mentioned in weaknesses. I consider the bar for ICLR is much higher.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to improve compositional text-to-image in large, pre-trained diffusion models without additional data or training requirements. To achieve this, language parsers are used to extract high-level information from a given caption (e.g. noun phrases) and use those to obtain additional text embeddings. These text embeddings are then used in combination with the original ones to scale the outputs of cross-attention layers. Experiments show that the approach can improve attribute binding and compositionality in generated images.
strength_and_weaknesses: Previous work has shown that the cross-attention layers have a lot of influence on the final content of generated images and, therefore, it makes sense to look into those more closely in this respect. The proposed approach does not need any additional training data or training of the model, making it easy to use in practice. The qualitative evaluation shows some improvement over the baseline model (Stable Diffusion) and human user studies indicate that the generated images are indeed better.

Some questions and concerns:
- I am not sure about the human user study; the authors say that they generate images from both approaches (their approach and the baseline Stable Diffusion model), then remove 20% of the most similar images, and only then run the user study; I understand that a lot of the images are similar since the same seed is used, but given that the performance of both approaches is probably similar on those 20% of the images this will affect the final results; in reality, the performance of the two models may be much closer to each other than the human user study indicates
- Table 1, CC-500, comparison to SD seems that SD is actually better, but the text seems to indicate otherwise?!
- Does it scale to longer captions with many constituents?
- Also, looking at real-life use people seem to append a lot of words for specific styles, how would behavior like that affect the performance? Captions like *a photograph of an astronaut riding a horse portrait pixiv art soft lines fantasy anime smooth illustration murata range 8 k studio ghibli sharp focus wlop kyoto animation style art viewer anime art close sharp detailed pixiv kawaii concept art high quality cute sunset exile intricate hd elegant 4 k cute face artgerm madhouse stunning ufotable valley alphonse mucha detailed background anime portrait yoshinari yoh artgem detailed face japanese anime cinematic studio lighting pink hue white background thick lines large eyes*
clarity,_quality,_novelty_and_reproducibility: The paper is well written, although I believe the notation of the cross-attention descriptions can be improved (e.g. $l$ and $d$ in Fig 2 do not seem to align with equations) and described more intuitively.
The approach itself seems novel to me.
summary_of_the_review: Overall I think the approach is novel, may increase the performance in some specifc settings, and does not need any additional training or data. I'm not entirely sure about the evaluation with the human user study and how well the approach will scale to more complex/longer captions or the kinds of captions that users seem to use in reality.
However, I see value in the approach and its simplicity and easy applicability are advantageous.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new diffusion-based T2I models to improve the compositional skills. The linguistic structures are incorporated with the diffusion guidance process based on the controllable properties of manipulating cross-attention layers in diffusion-based T2I models. The model is built based on the stable diffusion, and the proposed cross-attention design requires no additional training samples. The model can achieve SOTA compositional skills in qualitative and quantitative results.
strength_and_weaknesses: Strength:
This paper proposes to manipulate the cross-attention representations based on linguistic insights. And the compositional performance can be obviously improved.

Weakness:

1. As shown in Figs. 4 and 5, the proposed diffusion model's effects are mainly observed with the color attributes. The authors should provide the comparison with more types of attributes to demonstrate the improvement of compositional skills.

2. Why the quantitative comparisons between Stable Diffusion and Composable Diffusion are not conducted with the metrics of IS, FID, and R-prec?

3. The authors have claimed that their method can also take the scene graphs as the inputs. They should also compare the results with existing scene-graph to image approaches. And the visual cases in Fig. 6 have some errors. For example, there is not beach in the middle column of Fig. 6, while beach exists in the scene graph.

clarity,_quality,_novelty_and_reproducibility: The quality of this paper is good, with clear originality.
summary_of_the_review: I think the proposed linguistic insights strategy is a useful trick for T2I diffusion models. However, the experimental results are not convincing enough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for improving the attribute binding and compositional ability of denoising diffusion probabilistic models for image generation. The authors utilize the fact  that the attribute-object relation pairs can be obtained as text spans for free from the parsing tree of the sentence used for conditioning image generation. Using this free source of information, the authors propose  a method to combine the structured representations of prompts with the diffusion guidance process. Specifically, using the text spans from the parsing tree, the method obtains a cross-attention output that is a weighted average of the attention map of the full prompt multiplied by the value tensor of the full prompt and the value tensors of individual noun-pairs obtained by the parsing tree. To evaluate the method, the authors create two datasets. The first dataset, Attribute Binding Contrast set (ABC-6K) evaluates attribute binding ability of the method, by creating prompts with common and uncommon colors binding to objects in the COCO dataset. The second dataset, Concept Conjunction 500 (CC-500), tests the model’s ability to compose together objects that rarely appear together in reality. Pairwise comparisons in human evaluation study are used to compare the method to the baseline model (Stable diffusion) and to Composable diffusion for the CC-500 dataset. The proposed approach, StructureDiffusion, is preferred by users over Stable Diffusion on ABC-6K, while it performs worse on CC-500. However, StructureDiffusion outperforms Composable Diffusion on CC-500. FID scores on COCO are comparable to original Stable Diffusion. 
strength_and_weaknesses: Strengths: 
1. The proposed idea of using the free supervision from structure in the prompts is interesting and worth investigating. 
2. The problems of attribute-binding and composition in image generation are important and retain scope for improvement. 
3. The authors propose two well-crafted datasets for evaluating the attribute-binding and compositional ability of image generation models and perform a comprehensive comparison of the proposed approach with baseline using a human study.
4. The paper is well written and organized, which makes it easy to follow and reproducible. 

Weaknesses/suggestions:
1. It was not clear to me how human evaluators assess “image fidelity” in table 1. Please clarify this in the rebuttal.   
2. It was not clear to me why the authors claim that  “[On CC-500] our method outperforms Stable Diffusion by around 5-8% in terms of both image-text alignment and image fidelity”, when the reported numbers in Table 1 show that StructureDiffusion wins against Stable Diffusion (SD) only 31.8% of the time, while SD wins 38.9%. These numbers make me conclude that SD is actually superior to StructureDiffusion for compositionality. Please clarify this in the rebuttal.  
3. If I am interpreting the results in Table 1 correctly, then  StructureDiffusion is 20% better than SD in alignment, but 8% worse in compositionality. These results are not discussed in detail in the paper, which could help us understand how the proposed approach is helping in these tasks. 
4. No qualitative experiments are performed to show how the weighed cross-attention maps affect image formation (see the Hertz et al. paper for examples, which is used as an inspiration for the proposed approach) during the diffusion process. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The proposed idea is original, but it is not clear to me how effective it is in solving the problems of attribute binding and especially compositionality. 
summary_of_the_review: I am on the fence about how effective the proposed approach is in solving the problems of attribute binding and especially compositionality. It is also not clear to me why and how the proposed approach is working in different cases. So currently, I consider the paper to be borderline, but I am happy to re-evaluate after the rebuttal stage if I get clarification on the issues raised above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a method for improving attribute binding in text-to-image diffusion models, which is applied to improve the public Stable Diffusion model. The paper also introduces two benchmark datasets to evaluate their method.

Overall all reviewers appreciated the novelty of the problem statement, the fact that the method requires no new training data, and the introduction of the benchmark datasets used for evaluation. There were several issues raised by reviewers that were mostly addressed by revisions to the paper during the rebuttal period, including results on attributes other than colors, results with scene graphs, and automatic evaluation via GLIP. The primary remaining issues were issues with the user study (raised by Reviewer ZtBV) and the sentiment that the proposed method only offered relatively small improvements over the baseline model. Reviewer 5ZP1 was also concerned that the authors had made too many edits to the paper during the rebuttal period.

The AC met with Reviewers 5ZP1 and ZtBV to discuss this paper in more depth. During the discussion, we concluded that the revisions made by the authors during the rebuttal period were not a concern, and indeed had strengthened the paper. Both reviewers praised the overall idea behind the paper and the fact that the proposed method does not require any additional training data. The main weakness of the paper remained its relatively marginal improvements over the baseline.

In addition, Reviewer ZtBV raised an important concern about the user study in Table 1. The paper states that the authors “filter out 20% of the most similar pairs [of images]” before performing the user study. First, this omits the important detail of how “similar image pairs” was identified. Second, under the assumption that similar images would have likely resulted in ties (as the authors argue in their response), this exaggerates the difference between the two methods in Table 1. This could have been remedied either by (a) running the user study on all image pairs, even similar ones; or (b) add an extra column to Table 1 (Win / Lose / Tie / Similar Image) to give a more accurate sense of the relative performance of the models.

Overall, the AC feels that the paper’s novelty outweighs its relatively small improvements over the baseline method. The proposed method is easy to adapt to other models, and more importantly contributes an interesting set of new ideas that may spark interesting conversations and follow up work in the community. Reviewers 5ZP1 and ZtBV agreed with this viewpoint, and agreed that accepting the paper will benefit ICLR.

The authors are strongly encouraged to take the feedback from reviewers into account when preparing the camera-ready version of the paper, especially the issues with the user study in Table 1 discussed above.

justification_for_why_not_higher_score: The proposed method is novel and interesting, but gives fairly small gains over the baseline.
justification_for_why_not_lower_score: The paper addresses an important issue with text-to-image diffusion models, proposes an easy idea to improve it, and performs extensive experiments to validate their approach. The paper addresses an interesting problem, and has no glaring flaws or errors.
Decision: Accept: poster