The All-Seeing Project: Towards Panoptic Visual Recognition and
  Understanding of the Open World
Authors: name: Weiyun Wang; name: Min Shi; name: Qingyun Li; name: Wenhai Wang; name: Zhenhang Huang; name: Linjie Xing; name: Zhe Chen; name: Hao Li; name: Xizhou Zhu; name: Zhiguo Cao; name: Yushi Chen; name: Tong Lu; name: Jifeng Dai; name: Yu Qiao
Published: 2023-08-03 17:59:47

Abstract: We present the All-Seeing (AS) project: a large-scale data and model for
recognizing and understanding everything in the open world. Using a scalable
data engine that incorporates human feedback and efficient models in the loop,
we create a new dataset (AS-1B) with over 1 billion regions annotated with
semantic tags, question-answering pairs, and detailed captions. It covers a
wide range of 3.5 million common and rare concepts in the real world, and has
132.2 billion tokens that describe the concepts and their attributes.
Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified
framework for panoptic visual recognition and understanding. The model is
trained with open-ended language prompts and locations, which allows it to
generalize to various vision and language tasks with remarkable zero-shot
performance, including region-text retrieval, region recognition, captioning,
and question-answering. We hope that this project can serve as a foundation for
vision-language artificial general intelligence research. Models and the
dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo
can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.