the models. However, certain models incur significant performance drops, such as a -0.25 drop in human-rated alignment for Openjourney under dialect perturbation. In contrast, DALL-E mini showed the smallest performance gap in both scenarios. Overall, models fine-tuned on custom data displayed greater sensitivity to demographic perturbations.
10. Robustness. Similar to fairness, about half of the models showed performance drops in human-rated alignment metrics when typos were introduced (https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_robustness). These drops were generally minor, with the alignment score decreasing by no more than 0.2, indicating that these models are robust against prompt perturbations.
11. Multilinguality. Translating the MS-COCO prompts into Hindi, Chinese, and Spanish resulted in decreased image-text alignment for the vast majority of models (https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_chinese, https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_hindi, https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_spanish). A notable exception is CogView 2 for Chinese, which is known to perform better with Chinese prompts than with English prompts. DALL-E 2, the top model for human-rated image-text alignment (4.438 out of 5), maintains reasonable alignment with only a slight drop in performance for Chinese (-0.536) and Spanish (-0.162) prompts but struggles with Hindi prompts (-2.640). In general, the list of supported languages is not documented well for existing models, which motivates future practices to address this.
12. Efficiency. Among diffusion models, the vanilla Stable Diffusion has a denoised runtime of 2 seconds (https://crfm.stanford.edu/heim/v1.1.0/?group=heim_efficiency(scenarios). Methods with additional operations, such as prompt engineering in Promptist and safety guidance in SafeStableDiffusion, as well as models generating higher resolutions like Dreamlike Photoreal 2.0, exhibit slightly slower performance. Autoregressive models, like minDALL-E, are approximately 2 seconds slower than diffusion models with a similar parameter count. We also found that most models display significant variation in the raw runtime across multiple runs, possibly due to queuing of requests or interference among concurrent requests.
13. Overall trends in aspects. Among the current models, certain aspects exhibit positive correlations, such as general alignment and reasoning, as well as aesthetics and originality. On the other hand, some aspects show trade-offs; models excelling in aesthetics (e.g., Openjourney) tend to score lower in photorealism, and models proficient in bias and toxicity mitigation (e.g., minDALL-E) may not perform the best in image-text alignment and photorealism. Overall, several aspects deserve attention. Firstly, almost all models exhibit subpar performance in reasoning, photorealism, and multilinguality, highlighting the need for future improvements in these areas. Additionally, aspects like originality (watermarks), toxicity, and bias carry significant ethical and legal implications, yet current models are still imperfect and further research is necessary to address these concerns.
14. Prompt engineering. Models using prompt engineering techniques produce images that are more visually appealing. Promptist + Stable Diffusion v1-4 outperforms Stable Diffusion in terms of human-rated aesthetics score while achieving a comparable image-text alignment score (https://crfm.stanford.edu/heim/v1.1.0/?group=heim_quality scenarios).
15. Art styles. According to human raters, Openjourney (fine-tuned on artistic images generated by Midjourney) creates the most aesthetically pleasing images across the various art styles (https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_artstyles). It is followed by Dreamlike Photoreal 2.0 and DALL-E 2. DALL-E 2 achieves the highest human-rated alignment score. Dreamlike Photoreal 2.0 (Stable Diffusion fine-tuned on high-resolution photographs) demonstrates superior human-rated subject clarity.
16. Correlation between human and automated metrics. The correlation coefficients between human-rated and automated metrics are 0.42 for alignment (CLIPScore vs human-rated alignment), 0.59 for image quality (FID vs human-rated photorealism), and 0.39 for aesthetics (LAION aesthetics vs human-rated aesthetics) (https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_fid, https://crfm.stanford.edu/heim/v1.1.0/?group=mscoco_base). The overall correlation is weak, particularly for aesthetics. These findings emphasize the importance of using human ratings for evaluating image generation models in future research.
17. Diffusion vs autoregressive models. Among the open-sourced autoregressive and diffusion models, autoregressive models require a larger model size to achieve performance comparable to diffusion models across most metrics. Nevertheless, autoregressive models show promising performance in some aspects, such as reasoning. Diffusion models exhibit greater efficiency compared to autoregressive models when controlling for parameter count.