SEAGULL: No-reference Image Quality Assessment for Regions of Interest
  via Vision-Language Instruction Tuning
Authors: name: Zewen Chen; name: Juan Wang; name: Wen Wang; name: Sunhan Xu; name: Hang Xiong; name: Yun Zeng; name: Jian Guo; name: Shuxun Wang; name: Chunfeng Yuan; name: Bing Li; name: Weiming Hu
Published: 2024-11-15 13:07:22

Abstract: Existing Image Quality Assessment (IQA) methods achieve remarkable success in
analyzing quality for overall image, but few works explore quality analysis for
Regions of Interest (ROIs). The quality analysis of ROIs can provide
fine-grained guidance for image quality improvement and is crucial for
scenarios focusing on region-level quality. This paper proposes a novel
network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a
Large vision-Language model. SEAGULL incorporates a vision-language model
(VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a
meticulously designed Mask-based Feature Extractor (MFE) to extract global and
local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs.
Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and
SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises
about 100w synthetic distortion images with 33 million ROIs for pre-training to
improve the model's ability of regional quality perception, and SEAGULL-3k
contains about 3k authentic distortion ROIs to enhance the model's ability to
perceive real world distortions. After pre-training on SEAGULL-100w and
fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained
ROI quality assessment. Code and datasets are publicly available at the
https://github.com/chencn2020/Seagull.