Inject Semantic Concepts into Image Tagging for Open-Set Recognition
Authors: name: Xinyu Huang; name: Yi-Jie Huang; name: Youcai Zhang; name: Weiwei Tian; name: Rui Feng; name: Yuejie Zhang; name: Yanchun Xie; name: Yaqian Li; name: Lei Zhang
Published: 2023-10-23 08:13:33

Abstract: In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
fundamental image recognition model with strong open-set recognition
capabilities, by injecting semantic concepts into image tagging training
framework. Previous approaches are either image tagging models constrained by
limited semantics, or vision-language models with shallow interaction for
suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
image-text alignment and image-tagging within a unified fine-grained
interaction framework based on image-tags-text triplets. This design enables
RAM++ not only excel in identifying predefined categories, but also
significantly augment the recognition ability in open-set categories. Moreover,
RAM++ employs large language models~(LLMs) to generate diverse visual tag
descriptions, pioneering the integration of LLM's knowledge into image tagging
training. This approach empowers RAM++ to integrate visual description concepts
for open-set recognition during inference. Evaluations on comprehensive image
recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
(SOTA) fundamental image recognition models on most aspects. Specifically, for
predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
enhancements over CLIP on OpenImages and ImageNet. For open-set categories
beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
and RAM respectively on OpenImages. For diverse human-object interaction
phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
Code, datasets and pre-trained models are available at
https://github.com/xinyu1205/recognize-anything.