[D] Quality of Image benchmarking is missing
Latency benchmarking measures how fast the system can generate a picture. It doesn’t tell you if the output is good, useful, looks right, or matches what was put into it (the product/input). At this point there are no real image quality benchmarks that cover: - How closely the generated image matches the product/garment being represented - The quality of the visuals and if they have any visual problems (artifacts) - How well the generated image retains its identity - How structurally accurate the generated image is - How well the generated image preserves the color, pattern and/or texture of the original image - How similar the generated image is to the reference image - Whether humans or visual language models (VLMs) judge the generated image to be high quality.
We measure how quickly the model generates trash, however there is nothing that measures whether the model generates quality images. Quality benchmarking should be done along with latency benchmarking.