The Male CEO and the Female Assistant: Gender Biases in Text-To-Image Generation of Dual Subjects
Authors: Yixin Wan, Kai-Wei Chang


Reviewer paper summaries: This paper investigates the gender bias issue in a new setting where bias does not occur in single-person image generation but in paired images. The authors create nearly 2K prompts to test gender bias in this setting. They also provide a new metric which they term SS in this role.
Summary of strengths: - The research topic is important as it will indirectly influence many people in the real world.
- The overall setting is interesting.
Summary of weaknesses: The biggest problem for this work is that it is too shallow to understand why this happens. Given a black-box API-based model with nearly 2K prompts, the authors only report results for that. I am not very sure of the scientific contribution of this work. Several good future directions or alternative problem formulation methods are: The T2I model would generate several objects, each object having several possible features. And you want to observe whether one feature you observe (gender feature for humans, in this case) is linearly independent of another feature (occupation, etc.) or not. Based on that, you could choose to i) design more stable statistical tests for that and improve your SS metrics calculation method, ii) observe the trend when adding more 'persons' in generating images, iii) take more T2I models with different architectures to see whether this is a special case for certain models, and iv) most importantly, take an open-source model and trace how these biases are generated and passed with your inputs (prompts). I feel the writing of this work is good and clarifies it very well, but TBH, I feel the setting is still somewhat shallow.
Comments, suggestions and typos: I want to ask an honest question about the setting: For figures, you report bias scores with different occupations. And you mentioned that
> a fair model should be equally likely to generate individuals with either gender traits, regardless of their occupation or power levels.

In this narrative, you want the probability of the model to be uniformly distributed. I understand for some cases (like boss and secretary), there is an implicit power difference in these occupations, which we want to make sure is balanced. However, for some jobs like mechanics, there are actually more men doing these jobs. The pretraining data (if sampled approximately unbiased) is reflecting the world's real scenario. I wonder what the real or good 'fair' results for that would be.
Overall assessment scores: 2.5