The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
Authors: Weiyun Wang
Min Shi
Qingyun Li
Wenhai Wang
Zhenhang Huang
Linjie Xing
Zhe Chen
Hao Li
Xizhou Zhu
Zhiguo Cao
Yushi Chen
Tong Lu
Jifeng Dai
Yu Qiao
Venue: ICLR 2024 poster (2024)
Keywords: vision-language
open-world
region-text
object recognition
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world.

Abstract: We present the All-Seeing (AS) project: a large-scale dataset and model for recognizing and understanding everything in the open world.
Using a scalable data engine that incorporates human feedback and efficient models in the loop, we create a new dataset (AS-1B) with over 1.2 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world and has 132.2 billion tokens that describe the concepts and their attributes. Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified framework for panoptic visual recognition and understanding. The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including both region- and image-level retrieval, region recognition, captioning, and question-answering. We hope that this project can serve as a foundation for vision-language artificial general intelligence research. Code is available at https://github.com/OpenGVLab/all-seeing.

Published as a conference paper at ICLR 2024
THE ALL-SEEING ProJECT: TOWARDS PANOPTIC VI- SUAL RECOGNITION AND UNDERSTANDING OF THE OPEN WORLD
Weiyun Wang 1.2, Min Shi Qingyun Li Wenhai Wang"), Zhenhang Huang" Linjie Xing" Zhe Chen Hao Li Xizhou Zhu Zhiguo Cao Yushi Chen', Tong Lu Jifeng D Yu Qiao
'OpenGVLab. Shanghai Al Laboratory, FFudan University,
Huuzhong University of Science and Technology, "Harbin Institute of Technology, Nanjing University, "The Chinese University of Hong Kong
7 TSenseTime Research, *Tsinghua University
ABSTRACT
We present the All-Sceing (AS)' project: a large-scale dataset and model for recognizing and understanding everything in the open world. Using a scalable data engine that incorporates human feedback and efficient models in the loop, we create a new dataset (AS-1B) with over 1.2 billion regions annotated with semantic tags, question-anssering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world and has 132.2 billion tokens that describe the concepts and their attributes. Leveraging this new dataset, we develop the All-Seeing model (ASM), unified framework for panoptic visual recognition and understanding. The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including both region- and image-level retrieval, region recognition, captioning, and prestion-annweringg We hope that this project can serve as foundation for vision-language artificial general intelligence research. Code is available at https: / ////////// gighub...... com/ Open//aaa alll seeing.
INTRODUCTION
Creating artificial general intelligence (AGI) systems that match human intelligence and excel in any task across domains is the ultimate goal of artificial intelligence. Recent advancements in Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities in user-tailored natural language processing (NLP) tasks, suggesting new avenues for achieving AGI. However, as shown in Fig. 1a, most popular LLMs (OpenAl, 2022; Touvron et al., 2023; Chiang al., 2023) are limited to processing language information and lack the ability to understand the visual world. Although there have been some recent developments (Zhu al., 2023b; Liu al., 2023a; Li et al., 2023a; Dai et al., 2023) in open-world visual understanding, they are primarily focused on understanding images as whole, rather than recognizing and comprehending individual instances within the scene (see Fig. lb). This goes against the nature of the human visual system, as described by the feature integration theory (Treisman contexts in certain regions to achieve high-level understanding and recognition, rather than amalyzing all information simultaneously.
Gelade, 1980), which suggests that we attentively gather visual features and
To achieve instance-level visual understanding like humans, there are two major challenges follows: (1) The scarcity of open-world instance-text pair data. As listed in Table 1, existing datasets, such as Visual Genome (Krishna et al., 2017), have limitations in terms of data scale. Laion-5B (Schuhmann et al., 2022) only contains web-crawled image-text pairs without location information, and SA- IB (Kirillov al., 2023) lacks semantic information. (2) The lack of spatial information modeling in most existing models. These models mainly focus on whole-image understanding. In this work, we
"Equal contribution. This work is done when Weiyun Wang, Min Shi, and Qingyun Li interns Shanghai Laboratory. 'Corresponding to Jifeng Dai
"All-Sceing" is derived from "The All-Seeing Eye", which means having complete knowledge, awareness, insight into all aspects of existence.

---

Published as a conference paper at ICLR 2024
Web-Scale Language Datasets e.g, CC, Wikipedia, WebText2, BookCorpus, etc.
Web-Scale Image- Text Datasets e.g,, LAION-400M/SB, DataComp, etc.
The AS-1B Dataset:
1B Region-Text Pairs
3.5M Concepts
132B Tokens
Visual Large
See-nn
Large Language Model
Language Model
Model
Output
Output
Output
Prompt
Prompt
Image
Prompt
Image w/ Loc.
(a) LLMs possess extensive world knowledge and demonstrate im- pressive reasoning capabilities, but lack the ability receive and com- prehend visual information.
(b) VLLMs can process both text and images, but they can only cap- ture the holistic visual information of the whole image and understand it based on LLMs.
(c) Our ASM can comprehensively recognize and understand the ob- jects or concepts in regions of in- terest while maintaining the capa- bilities of VLLMs and LLMs.
Overview and comparison of our All-Seeing project with other popular large foun- dation models. To address the limitations of LLMs in understanding visual inputs and VLLMs in effectively leveraging region-aware information, we propose (1) large-scale dataset AS-1B which consists of over 1 billion region-text pairs, 3.5 million open-world concepts, and 100 billion tokens of region-related question-answerte and caption; and (2) the All-Seeing model (ASM), which capable of recognizing and understanding context in arbitrary regions.
Figure 1:
propose the All-Seeing (AS) project for open- spendworld panoptic visual recognition and understanding, driven by the goal of creating a vision system that mimics human cognition. The term "panoptic" refers to including everything visible in one view (Kirillov et al., 2019). The AS project addresses the challenges from both the data and model perspectives.
From the data aspect, propose the All-Seeing 1B (AS-1B) dataset, consisting of over 1.2 billion region annotations in various formats, such as semantic tags, question-answerti pairs, and detailed captions (refer to Fig. 2). AS-1B dataset is made possible by a scalable semi-automatic data engine, which significantly lowers the previously unaffordable expense of manually annotating massive loop, iteratively
amount of open-world semantics. The data engine operates in a tanta-human-mooee refining data quality. Initially, diverse models, including large language models (LLMs) (Chiang al., 2023), detection (Wang et al., 2023c; Fang et al., 2023a), captioning (Li et al., 2022a), and visual question answering (VQA) models (Liu et al., 2023a; Zhu et al., 2023b; Liu et al., 2023b), are employed as "annotators" which add semantic annotations to dense region proposals generated by state-of-the-art object detectors (Kirillov et al., 2023; Fang et al., 2023a; Li et al., 2022b; Wang et al., 2023c). Subsequently, human annotators verify the generated pseudo labels and provide feedback with high high-quality data, which is then used to fine-tune the models to improve their performance. The enhanced models are then used to re- c--anootate the data, starting another itcration of the loop. As shown in Fig. 2, AS-1B contains wide range of opon-world concepts, including over 3.5 million different semantic tags ranging from common categories (e.g., . human, backpack) to fine-grained or rare categories with attributes (e,g., old metal latches). AS-IB also encompasses the 3.3 billion visual question-answertg pairs and 1.2 billion region captions for 1.2 billion regions.
In terms of the model perspective, we propose the All-Seeing model (ASM), unified location- aware image-text foundation model. The model consists of two key components: a location-aware image tokenizer and an LLM-based decoder. The location-awart aware image tokenizer uses location information such as bounding box as conditions (see Fig. Ic) to extract image features, which contribute to the location capability of ASM. The LLLM-based decoder inherits the world knowledge and reasoning capability from LLMs, providing a strong foundation for visual recognition and understanding. In addition, to unify image-text aligning and generation tasks, we introduce a new decoding approach, where the aligning tasks are reformulated as a "special" generation task, enabling our model to generalize to various Dision-languee methods (Radford et al., 2021; Li et al., 2023a; Liu et al., 2023a: Zhu et al., 2023b), our work offers several advantages as follows: (1) Our model not only excels in image-level understanding but also demonstrates exceptional capability in recognizing and comprehending objects at the instance
tasks with shared weights. Compared to previous
2

---

Published as a conference paper at ICLR 2024
Frequency of Semanntc Tags (1g)
batting
6.0
Stone Sph
Skurmay cludds
4.0
3.0
metal latches
Blue and ack eyes
1.0
Sematic Tags
3.5M
Semantic Tag
Semantic Tag
Stone Sphinx
Pyramid
Detailed Annotation
Deteiled Annotation
What materal of sphine?
Q1:
The
What the rame of this pyramid?
aramndd Great
materill of sphinx's is sardition.
Pyramid
Q2: What is the height of this pyramid?
A2: The oolor of the sphinne brown.
Q2: What is the color of this sphins??
aoprosimately 144 meters (479 feet). bullt pyramid?
A2: The height of the pyramic
height
A3: The height of the Sphins of Giza is spprrteimetell 65 feet (1.55 meters) tell.
A3: The Great Pyromid of Git, also
known as the Pyramid of Khul,, was built by the Fgyptians mee 4,500 years. ago. Detailed Caption: The Great Pyramid Giz,, known the Pyramid Khufa the Eayptians...
Detailed Caption: Sphinu Glia sanditone shetue with a brown color holght of spproximatoly 65 feet (18.5 neters).
Figure 2: Semantic concepts and annotations in the AS-1B dataset. The semantic tags in AS-IB dataset encompass a wide range of concepts. Beyond brief semantic tags, detailed annotations, including visual-question-ansveiii pairs and region captions are also provided.
level, closely aligning with human cognitive processes. (2) Our model is a unified framework that supports a wide range of image-text tasks, including discriminative tasks and generative tasks such as visual captioning and question-answering answering. (3) Our model comes with AS-IB the largest dataset with open-world panoptic semantics. Data and models feed each other the data engine, iteratively improving the model performance, data scale and diversity.
In summary, our contributions are three folds:
(1) We propose a new large-scale dataset (AS-1B) for open- world panoptic visual recognition and understanding, using an economical semi-automatic data engine that combines the power of off tfe-the- shelf vision/language models and human feedback. As reported in Table 1, we have 159 times more semantic tags and 33 times more regions compared with its counterparts.
(2) Based on the dataset, we develop unified vision-language foundation model (ASM) for open- world panoptic visual recognition and understanding. Aligning with LLMs, our ASM supports versatile image-text retrieval and generation tasks, demonstrating impressive zero-shot capability.
(3) We evaluate our model on a representative vision and vision-language tasks. Our ASM outperforms CLIP (Radford et al., 2021) by 10.7 and 13.4 (mAP) on COCO (Lin et al., 2014) and LVIS (Gupta ct al., 2019) in zero-shot region recognition tasks. When trained with AS-1B (region-level data) and LaionCOCO (Schuhman et al., 2022) (image-level data), our model achieves superior zero-shot and fine-tuned performance compared to recent image-level (Dai et al., 2023; Wang et al., 2023b; Huang et al., 2023) and region-level (Yu et al., 2017; Wu et al., 2022; Peng al., 2023) VLLMs.
2 RELATED WORK
Datasets for Visual Recognition and Understanding, As one of the three pillars of deep learning. datasets play a critical role in the advancement of deep learning models, especially in the field of visual recognition and comprehension. Prior to the era large-scale models, datasets (Deng et al., 2009; Lin et al., 2014; Goyal et al., 2017) primarily closed-world have limited data scale. Additionally, datasets like Visual Genome (Krishna et al., 2017) and Visual7W (Zhu et al., 2016) integrate visual location and understanding, offering more comprehensive tasks to describe the visual
3

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with popular vision and vision-language datasets. "#" denotes the number of something. "Open" and "Closed" means Open-World and Closed-Set respectively. We see that the proposed AS-IB dataset has a significantly larger data scale and diversity than its counterparts.
Type/
Dataset
#lmages #Regions #Concepts #Tokens Location Semantic
Image- Leve-LLvell
ImageNet-22K (Deng et al., 2009)
15M
22,000
Closed
COCO Caption (Chen et al., 2015)
0.1M
8.4M
Closed
CC12M (Changpinyo ct 2021)
12.4M
250.9M
Open
YFCC15M (Kalkowski et al., 2015)
15M
1.0B
Open
COY0700M (Byeon et al., 2022)
700M
15.0B
Open
LB (Schuhmann et al., 2022)
5B
135.0B
Open
SA-1B (Kirillov et al., 2023)
11M
1.1B
Open
Region-Leve Level
COCO (Lin al., 2014)
0.IM
0.9M
80
Closed
Closed
LVIS (Gupta al., 2019)
0.IM
1.5M
1,203
Closed Closed
Open Images (Kuznetsova et al., 2020)
L.SM
14.8M
600
Closed Closed
BigDetection (Cai et al., 2022)
3.5M
36.0M
600
Closed Closed
V3Det (Wang et al., 2023a)
0.2M
1.5M
13,029
Closed Closed
Visual Genome (Krishna al., 2017)
0.IM
0.3M
18,136
51,2M
Open Open
AS-IB (ours)
11M
1.2B
3.5M
132.2B
Open
Open
world. However, these datasets have limited semantics and fail to encompass diverse scenarios in the open world, which hinders the generalization ability of models. To achieve open-world capability, CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) propose training models using web-scale image-text pairs collected from the internet. Subsequent works, such Laion (Schuhmann et al., 2021; 2022), COYO-700M (Byeon et al., 2022) and DataComp (Gadre et al., 2023), have also been introduced for open-source research. However, these approaches only include descriptions or question-anssering pairs corresponding to the entire image, resulting in models struggling to accurately recogmize and understand specific objects the instance level. Recently, Kirillov al. introduced SA- AB (Kirillov et al., 2023), which provides open-w open-world location information such as boxes and masks but still lacks semantic details. So existing datasets cannot meet the requirements of data scale, open-world location, and semantics necessary for achieving visual AGI models, thus posing challenges in supporting human-like panoptic visual recognition and understanding.
Models for Visual Recognition and Understanding. Significant advancements have been made in the field of visual recognition and understanding in recent years. Previous methods (He et al., 2017; Chen et al., 2023b; 2022b; Kirillov et al., 2019) mainly concentrate the close-set recognition while recent works begin to focus on the open world understanding. Subsequent works (Radford et al., 2021; Li et al., 2021; Yu et al., 2022) can recognize and understand the open world semantics while failing to capitalize on the powerful perception capabilities of existing powerful pre-trained models, increasing the cost of developing
models. In recent years, Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023) have demonstrated excellent performance across various tasks, showcasing their potential for semantic understanding, dialogue generation, programming, mathematical problem-solving etc, which leads to the emergency of many LLM-based multimodal models (Li et al., 2023a; Zhu et al., 2023b; Liu et al., 2023a; Wang et al., 2023b; Liu al., 2023b; Li et al., 2023b; Zhai et al., 2022: Tschannen et al., 2023). However, these works are only capable of recognizing the entire image, lacking the ability to comprehend specific regions within the image. Some concurrent methods (Chen et al., 2023a; Peng et al., 2023; Zhang et al., 2023) begin to focus on location- aware understanding. However, without the support of large-scale instance-level visual understanding data, the generalization ability of these models is still limited.
3 THE ALL-SEEING DATASET (AS-IB)
In this section, we introduce the All-Seeing-1B (AS-1B) dataset for open-world panoptic visual recognition and understanding. The dataset consists of 1.2 billion regions 11 million images?. Each region is annotated with comprehensive information, including semantic tags, question-answer pairs, and captions. Compared with the previous visual recognition datasets like ImageNet (Deng et al., 2009) and COCO (Lin et al., 2014), visual understanding datasets like Visual Genome (Krishna et al., 2017) and Laion-5B (Schuhmann et al., 2022), the proposed AS-IB dataset stands out due to its rich and diverse instance-level annotation and corresponding detailed object concepts and descriptions.
Images source from SA-1B (Kirillov et al., 2023)
4

---

Published as a conference paper at ICLR 2024
6 Osge World Locatin Gercraaion
1d Semanik Loaatio Magching
(f Deta lod Descrtstin Genevatinn
Senvantic
Oscitioner
Tags
buttion
Goscll Sct
CUR
Reglan
Datt led Descripton
Openllembuarry
Rag Proposels
Bi] Opee- Wterts Semonnic lags Gineration
Statue
Fmnmd
Sphira 2 Brebs Statue
Imagihator
Bricks
Dae libe
Splttes
West
Human Footbick
Autoraatic Generation
Magsfier
Semrnntcc TTgs
A Algned Region Tags
Figure 3: pipeline (i.e., (a), (b), (c), (d)) and a human verification stage (i.e., (e)). We combine various powerful models to produce annotations for different regions, which are sampled and verified by human experts. Automated annotations are used together with human validation results to train region-aware models, which are then used in the automated annotation pipeline to improve data quality.
Data engine for AS-1B dataset. Our data engine consists of an automatic annotation
3.1 DATA ANNOTATION ENGINE
We develop a semi-automatic data engine that efficiently uses wide range of state-of-the-an foundation models as annotators, reducing the enormous labeling cost to an acceptable level. As depicted in Fig. 3, the process of the data engine begins by generating noisy pseudo data using well-trained off-the-shelf foundation models (Kirillov et al., 2023; Wang et al., 2023c; Fang et al., 2023a; Li et al., 2022b; Radford et al., 2021; Lüddecke & Ecker, 2022; Chiang et al., 2023; Liu et al., 2023b) from diverse fields. Subsequently, subset of these pseudo data are sampled to be verified and corrected by human annotators. After that, we pre-train our All-Seeing- Seeing-MMoee (ASM) with the noisy pseudo data and finetune it with the human human-verified data. Then we re-annotate the data with the aid of ASM. The process of annotation, verification, and fine-tuning are repeated to iteratively refine the annotation quality. By employing this mada--human-mode!" cycle, we generate large number of region-level annotations with exceptional quality. See Appendix E.3 for more details about the cycle.
As the core component of the data engine, the data generation pipeline consists of five steps as follows: (1) Creating open-world location (e.g., bounding box, mask, point set) with an ensemble of sfte-of-theearr class-agnostic, visual grounding, and closed-set set perception models (Kirillov et al., 2023; Li et al., 2022b; Wang et al., 2023c; Fang et al., 2023a); (2) Generating open- open-world semantic tags using the combination of image captioning models (Li al., 2022a; Zhu et al., 2023b) and LLMS (Chiang et al., 2023); (3) Matching the semantic tags to proper regions with image-text aligning models (Radford al., 2021; Lüddecke& Ecker, 2022); (4) Using LLM (Chiang al., 2023) and VQA models (Liu al., 2023b) to generate the attributions of each region based on the matched semantic tags; (5) Generating detailed captions based on the semantics and attributions each region.
3.2 WORLD LOCALIZATION
To obtain comprehensive locations of all instances in image, combine the results of state- of-the-art perception models from different fields, including (1) class-agnostic model: we adopt the SAM (Kirillov al., 2023) to provide initial proposals of most objects in view. (2) closed-set detection model: we use InternlmageHH (Wang et al., 2023c) and EVA-02 (Fang et al., 2023a) trained on BigDetection (Cai et al., 2022) and LVIS (Gupta al., 2019), respectively, to detect the common-seau objects. (3) grounding model: we use GLIP (Li et al., 2022b) to ground open- world semanties generated by L.LMs (see Sec. 3.3). All the bounding boxes are gathered together to ensure that all possible objects in view are covered. See Appendix B.1 for details of the gathering strategy.
3.3 OPEN-WOO WORLD SEMANTIC
Manually labeling billions of regions for an open-world semantic description is impractical due to the enormous cost and time required. To remedy these challenges, we draw inspiration from the recent
5

---

Published as a conference paper at ICLR 2024
advancements in Large Language Models (LLMs) and Visual Large Language Models (VLLMs). We leverage a series of LLMs (Chiang et al., 2023) and VLLMs (OpenAI, 2023; Liu et al., 2023a; Li et al., 2022a; Liu et al., 2023b; Wang et al.. 2023b; Zhu et al., 2023b) as "semantic generators" and tap into their vast world knowledge and reasoning abilities for open-world semantic generation. These "semantic generators" can be specialized for producing short semantic tags detailed descriptions, including attributes, iresstoon-answerigg pairs, and captions, based on specially-designe prompts.
Semantic Tags.
To generate as many semantic tags as possible for a view, different instructions are employed to harness the diverse capabilities of LLMs and VLLMs, turning them into annotators with different focuses and skills. Specifically, we have (1) spotter, which identifies major instances and provides an overview of the scenes, (2) imaginator that leverages world knowledge to imagine plausible objects, (3) splitter that divides complicated objects into parts, as well as (4) a magnifier which zooms on each region to produce region-specific candidates. These models complement each other to create a powerful system that can generate comprehensive open-world semantic tags for each region and the entire image. See Appendix B.2.1 for more details.
Detailed Descriptions. To provide detailed descriptions that include attributes and statuses of each region, we develop pipeline that expands the region description using the open-world location and its matched semantic tags (see Sec. 3.4 for rocation-semannee matching). We utilize a series of skilled LLMs, including (I) questioner that asks specific questions about the attributes status of a given semantic tag; (2) a responder that provides the accurate answers for these questions based on the region's 's content; and (3) writer responsible for composing detailed caption for each region, according to the generated semantic tags, attributes, and status. See Appendix B.2.2 for more details.
3.4 MATCHING LOCATION AND SEMANTIC
In the matching process, we employ a region-text aligning model to measure the similarity between a certain region and its semantic tag list. For each region, the semantic tag list is constructed by LLMs (I.e. "spotter", "imaginator", "divider" and "magnifier") and closed-set object detectors. Initially, in the first iteration of the data engine, we use a CLIP model (Radford et al., 2021) for the region-text alignment, where the input is the cropped region. To make sure the semantic tag is matched with the major object in the bounding boxes. CLIPSeg (Lüddecke Ecker, 2022) is also utilized generate the mask for each candidate. The original CLIP confidence is then modulated with the corresponding mask area. the subsequent iterations, we upgrade the CLIP model our All-Seeing Model.
3.5 HUMAN VERIFICATION
Albeit efficient, annotations from the automated pipeline still contains some noise due the cropping process, which might discard essential context information. Therefore, to enhance the data quality, we find it crucial to include human verification and implement a "data-human-moode continuously improve the data quality. Specifically, the human annotators are asked to verify the annotations of semantic tags and question-answer pairs. For semantic tags, we simply the task for annotators by focusing question-answer pairs, a two-stage verification procedure employed. In the first stage, human annotators are asked to annotate the pairs with one of four choices: correct answer, wrong answer, unanswerable question, or wrong semantic tag. Samples annotated as the latter two options are annotated with rejection answer, while those annotated as "wrong answer" will be further corrected in the second stage. With such an annotation strategy, human annotators only need to verify the outputs of the model instead of writing any long sentence, which greatly reduces the annotation cost. See Appendix B.3 and B.4 for more details.
loop to
picking the incorrect ones from the top-5 candidates in each region. For
4 THE ALL.-SEFING MODEL (ASM)
4.1 OVERAL ARCHITECTURE
Our objective is to create a unified framework that supports contrastive and generative image-text tasks at both the image level and region levels. By leveraging pre-trained LLMs and powerful vision foundation models (VFMs), this model demonstrates promising performance in discriminative tasks like region recognition, well generative tasks such as image captioning, region captioning, etc.

---

Published as a conference paper at ICLR 2024
Random Queries
Location-Aware Image Tokenizer
Region Queries
Desired Output
Backbone
5
Transformer Decoder
LLM-based Decoder
R
Task Instruction, Question
Image w/ Location
For generative tasks:
discriminative tasks:
This is the Sphinx Generative
Discriminative Loss Eiis
Loss Aignn
LLM-based Decoder
LLM-based Decoder
LLM-based Decoder
(2)) (bos) Human: is
[Pa) (bos) Human: What is this? (altgn)
(P&) (boz) Assistant: of the Sphinx. (align)
this? Assistant:
Figure 4: Overview of the All-Seeing Model (ASM). ASM incorporates location-aware image tokenizer to extract image-level and region-level features and specific prompt design handle both generative tasks and discriminative tasks using a unified architecture with shared parameters.
As illustrated in Fig. 4, our All-Secing Model (ASM) comprises three key designs: (I) location- aware image tokenizer extracting features from both the image and region levels based on the input image and bounding box, respectively. (2) trainable task prompt that is incorporated the beginning of the vision and text tokens to guide the model in distinguishing between discriminative and generative tasks. In the case of the discriminative task, trainable align token is appended to the input sequence to gather the overall representation, and its embedding is then used in the matching process. (3) LLM-based decoder that is utilized to extract vision and text features for discriminative tasks, as well as to auto rugr-segrysyyvely generate response tokens in generative tasks.
The training objective of ASM Listal Cgen + Ldis where the generation loss Egen is the same as the loss of GPT series (Radford et al., 2019) and the discriminative loss follows the contrastive loss of CLIP (Radford al., 2021), where each region treated as an image when calculating the loss.
4.2 LOCATION-AWARE IMAGE TOKENIZER
Here, we propose extension of the Query Transformer (Li al., 2023a) for location-aware image tokenizer that conditions its queries on location information, such as bounding boxes. As depicted in Fig. 4, we first apply the ViT-g/14 (Fang al., 2023b) to encode the input image and utilize RoIAlign (He et al., 2017) to extract the region features R from the image features F based on the given bounding box. Then, the flattened region features R are projected to the same shape as the randomly initialized query tokens Q'. They are concatenated to form the location-aware query tokens Q. Subsequently, these location-aware query tokens Q are passed through the Q-Former extract output features from F. Finally, the output features projected to match the feature dimension of the LLM and are used as the soft prompt for subsequent decoding processes. Particularly, when no location information is provided, the bounding box is assumed to cover the entire image. This method guarantees a consistent approach for both local region and whole image tokenization.
4.3 LLM-BASED DECODER
To develop unified L.LM framework that can effectively handle both generation tasks and discriminative tasks, utilize Husky-7B (Liu et al., 2023b) our foundation language model.
For generative tasks, the input sequence comprises three types of tokens, including (1) learnable generative task prompts P,, RM which informs the model that it should perform generative task. (2) location-awe awwre image tokens that contain the extracted image-level and region-level information from the input image and (3) user prompt that expresses his/her requirements. Given such an input sequence, the LLM generates text tokens sequentially in an autoregressive manner until an end token (cos) is reached, An example prompt is shown in Appendix D.
7

---

Published as a conference paper at ICLR 2024
Table 2: Performance on the region-level captioning task. "FS: refers to the few-shot result with 4 samples. "-FT" denotes ASM with fine-tuning.
Visual Genome Meteor
RefCOCOg
Model
Zero-Shot
CIDEr
Meteor
CIDEr
GRiT (Wu et al., 2022)
X
17.1
142.0
15.2
71.6
*K
SLR (Yu et al., 2017)
15.4
59.2
SLR+Rerank (Yu al., 2017)
15.9
66.2
Kosmos-2 (FS: 4) (Peng al., 2023)
14.1
62.3
Kosmos-2 (Peng ct al., 2023)
12.2
60.3
ASM (ours)
13.0
46.8
15.0
48.8
ASM-FT (ours)
X
18.3
148.7
21.8
107.8
For discriminative tasks, the input sequence of input image consists of soft prompt tokens that indicate task information, as well as vision tokens. Similarly, the input sequence of input text consists of the same soft prompt tokens as those of input image and text tokens that represent the corresponding region caption or object class name. In addition, we append a trainable align token (align) to each of the input sequences to extract the holistic representation of the current input sequence. During the process of region-text matching, we achieve image-text retrieval by simply computing the similarity of the embedding of the (align) token. The example prompts provided in Appendix D.I.
EXPERIMENTS
We compare our ASM with CLIP-based and other VLLMs on representative vision tasks such as zero-shot region recognition, image caption, and region caption. In addition to standard metrics, also use human evaluation for LLM-based models (see Appendix E) For the AS-1B dataset, a brief analysis has been provided in Fig. 2 and Table 1. The detailed analysis is present Appendix
5.1 TEXT GENERATION
Settings. We evaluate the image-level caption ability of our model on Flickr30K (Young al., 2014) and NoCaps (Agrawal et al., 2019) dataset. Following the common practice (Huang et al., 2023; Li et al., 2023a), we report the CIDEr (Vedantam al., 2015) and SPICE (Anderson et al., 2016) metrics on these benchmarks. To assess the region-level caption ability, also evaluate ASM on the Visual Genome (Krishna et al., 2017) and RefCOCOg (Mao et al., 2016). On the region caption task, adopt both the Meteor (Banerjee & Lavie, 2005) and CIDEr metrics following Kosmos-2 (Peng et al., 2023). All of the metrics are computed by COCOEvalCap.
evaluation metrics,
During training, a two-stage training process is employed. The first stage utilizes AS-1B (region-leve) and LaionCOCO (image-level) for pre-training, while the second stage utilizes a subset of AS- 1B that has been verified by human annotators, along with other high-quality data, for supervised fine-tuning. The fine-tuned ASM is denoted as ASM-FT. See Appendix D for more training details.
Results. For region-level captioning, as shown in Table our ASM model surpasses the concurrent region-aware VLLMs, Kosmos-2 (Peng et al., 2023), by 2.8 points on the RefCOCOg dataset, under the zero-shot setting. After the second-stage fine-tuning, our ASM model has achieved new record for referring expression generation on RefCOCOg. Besides, on the Visual Genome (VG) dataset, although the Meteor score of zero-shot ASM is inferior GRiT (Wu et al., 2022), ASM-FT achieves significantly better results than GRiT given relevant data.
In addition, our model also excels image-level captioning, as presented in Table 3, our ASM model demonstrates promising zero-shot performance on Flickr30K (Young et al., 2014) and No- Caps (Agrawal et al., 2019) dataset. Specifically, under the zero-shot setting, our model achieves a CIDEr score of 79.5 without the second-stage fine-tuning and 88.0 after the second-stage fine-tuning, which outperforms all the concurrent VLLMs, such as InstructBLIP (Dai et al., 2023), Shikra- the NoCaps dataset,
13B (Chen et al., 2023a) and Kosmos-2 (Peng al., 2023). Furthermore, ASM also achieves comparable performance compared to the baselines. These results indicate that ASM model retains strong image-level comprehension ability while also being region-aware.
8

---

Published as a conference paper at ICLR 2024
Table 3: Zero-shot performance on the image-level captionnng tasks.
Flickr30K
NoCaps
Model
Zero-shot
CIDEr
SPICE
CIDEr
SPICE
Flamingo-9P (Alayrac et al., 2022)
61.5
SimVLM (Wang al., 2022)
110.3
14.5
BLIP (Li et al., 2022a)
113.2
14.7
BLIP-2 (Li et al., 2023a)
121.6
15.8
InstructBLIP (Dai et al., 2023)
82.8
123.1
Shikra- 13B (Chen et al., 2023a)
73.9
Kosmos-1 (Huang et al., 2023)
67.1
14.5
Kosmos-2 (Peng et al., 2023)
66.7
ASM (ours)
79.5
17.6
107.7
14.6
ASM-FT (ours)
88.0
18.8
116.9
15.6
Table Zero-Shot object recognition performance. We report the zero-shot recognition perfor- mance on COCO and LVIS dataset. The ground-truth boxes are used for inference.
COCO
LVIS
Model
mAP
APs
APM
AP,
mAP
APs
APM
AP₁. I.
CLIP (Radford al., 2021)
58.9
50.7
70.4
58.3
47.1
40.3
59.2
$7.4
OpenCIIIP (Ilharco et al., 2021)
63.3
47.8
75.6
60.9
49.1
37.4
62.8
66.5
R-CLIP (our baseline)
68.6
61.4
75.4
79.3
54.8
49.3
60.6
66.6
ASM (ours)
69.6
63.7
77.3
72.2
60.5
55.8
67.3
69.3
5.2 ZERO-SHOP REGION RECOGNITION
Settings. We use zero-shot region recognition evaluate the region- text alignment ability of our model. We use COCO (Lin et al., 2014) and LVIS (Gupta et al., 2019) detection dataset for evaluation. Since our current focus to predict the categories given the corresponding texts following RegionCLIP (Zhong et al., 2022). report the mean Average Precision (mAP) metrics for this evaluation. The model shared with text generation tasks, which is trained with two-stage schedule.
on object localization, we use the ground-trutt boxes and use the model
Results. As shown Table 4, both our baseline model R-CLIP (see Appendix D.3) and the proposed ASM achieve promising zero-shot region recognition performance. On the COCO dataset, R-CLIP outperforms the original CLIP by 9.7 mAP, and ASM further increases the performance by 10.7 mAP. On the more challenging LVIS dataset with 1 1,203 categories, R-CLIP outperforms CLIP by 7.7 mAP, and ASM achieves a more significant improvement 13.4 mAP over CLIP. These results demonstrate the effectiveness of region-text data in AS-1B dataset and the proposed ASM in region-text alignment tasks. Notably, our ASM simultaneously performs caption and region recognition tasks with the same weight, showcasing its versatility and efficiency.
CONCLUSION
In this paper, present the All-Seeing (AS) Project, which develops comprehensive system for panoptic visual recognition and understanding in the open world, from both dataset and model perspectives. In terms of data, elaborate comi-automatic data engine consisting of automatic annotation pipeline and human verification step. Using this data engine, we annotated the AS- AS-IB dataset comprising over billion region-level comprehensive annotations, with controllable costs. From the model aspect, we propose region-aware multi-modal large language model called the All-Sceing Model (ASM). The ASM utilizes unified LLM decoder to model both region-text alignment and generation tasks. Leveraging the AB-IB dataset and other high-quality data, ASM achieves promising results on image and region- level tasks. We believe that the data engine, AS- AB dataset, and the ASM model proposed in the All-Secing Project will inspire further research and development towards empowering artificial intelligence systems with an "all-seeing eye", enabling them to achieve deeper understanding of the world.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The work supported by the National Key R&D Program of China (NO. 20222D0161300), and the National Natural Science Foundation of China (Grant No. 62376134).
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Int. Conf. Comput. Vis., 2019. 8
Jean-Baptie Banttisee Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Adv. Neural Inform. Process. Syst. 2022. 9, 32, 33
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Eur. Conf. Comput. Vis., 2016.
Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin- Yew Lin, and Clare R. Voss (eds.), Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@anr ACL 2005, Ann Arbor, Michigan, USA. June 29, 2005, pp. 65-72. Association for Computational Linguistics, 2005. 8,31
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Ady. Neural Inform. Process. Syst. 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://giit wtt.c/co///kk/eeee kakaobraik coyo-datased 2022. 4, 15
com/
Likun Cai, Zhi Zhang, Yi Zhu, Li Zhang, Mu Li, and Xiangyang Xue. Bigdetection: large-scale benchmark for improved object detector pre-training. In IEEE Conf. Comput. Vis. Pattern Recog., 2022. 4, 5, 25
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In IEEE Conf. Comput. Vis. Pattern Recog., 2021. 4
Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually impaired people. In JEEE Conf. Comput. Vis. Pattern Recog., 2022a. 30
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306.. 15195, 2023a. 4, 8, 9, 33
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection, arXiv preprint 2xXiv2221..09788 2022b.
Xinlei Chen, Hao Fang, Tsung- Yi Lin. Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and CLawrence Zitiick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.003335 2015. 4, 30
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In Int. Conf. Learn. Represent. 2023b. 4
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http:://vicuna lmsys. org (accessed 14 April 2023), 2023. 1, 2, 5, 6, 17, 22, 23, 25
EasyOCR contributors. Easyoer. https: http://ggtuuuucco/////aaa///////
2023. 22
10

---

Published as a conference paper at ICLR 2024
Wenliang Dai, Jumnan Li, Dongxu Li, MethonyyMeng Huat, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv preprint (.XXx:23350060000 2023. 3, 8, 9, 33
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Conf. Comput. Vis. Pattern Recog, 2009. 3, 4, 25, 26
Yuxin Fang. Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 11331, 2023a. 2, 5, 17, 24
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In IEEE Conf. Comput. Vis. Pattern Recog, 2023b. 7, 30
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023. 15
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In JEEE Conf. Comput. Vis. Pattern Recog., 2017. 3, 30, 31
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 3, 4, 5,
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Int. Conf. Comput. Vis., 2017, 4, 31
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Int. Conf. Learn. Represent. 2022. 30,
Shaohan Huang. Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. /4045, 2023. 3, 8, 32
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 30, 31
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. doi: 10.528//2ecodo.5544333.. URL 9
https: doi.. org/logg//oooooooooo 10.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and -sionnlanguage representation learning with noisy text supervision. In International Conference on Machine Learning., 2021. 4
Sebastian Kalkowski, Christian Schulze, Andreas Dengel, and Damian Borth. Real-time analysis and visualization of the 10cc 100m dataset. In Proceedings of the 2015 workshop on community- organized multimodal mining: opportunities for novel solutions, pp. 25-30, 2015. 4
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen-
tation. In IEEE Conf. Comput. Vis. Puttern Recog., 2019. 2,
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprini SXXi:2304400643: 2023. 4. 5, 15, 16, 17, 24
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123:32-73, 2017. 1, 3, 4, 8, 15, 25, 30
11

---

Published as a conference paper at ICLR 2024
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Int. J. Comput. Vis., 128(7):156--198) 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Adv. Neural Inform. Process. Syst., 2021. 4
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping
pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning.. 2022a. 2, 5, 6, 9, 17, 22, 25,
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: training with frozen image encoders and large language models. arXiv preprini arXiv:2301.12597, 2023a. 1, 4, 7, 8, 9, 30, 31, 32, 33
Bootstrapping language-image pre-
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355.
2023b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In IEEE Conf. Comput. Vis. Pattern Recog.. 2022b. 2. 5, 17, 24
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., 2014. 3, 9,
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 08485, 2023a. 1, 2, 4, 6, 30, 31
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang. Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centria tasks by interacting with chatbots beyond language. arXiv preprint 2305.:2305.0566 2023b. 2, 4, 5, 6. 7, 17, 22, 30
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Int. Conf. Leurn. Represent., 2019. 30
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In IEEE Conf. Comput. Vis. Pattern Recog., 2022. 5, 6, 17,32
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. JEEE Conf. Comput. Vis. Pattern Recog. 2016. 8, 30
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In IEEE Conf. Comput. Vis. Pattern Recog., 2019. 31
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: 2023. 4,
TB OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAI, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint
arXiv:2306. 14824, 2023. 3, 4, 8, 9, 32,
Shi Qiu. Xiaogang Wang, and Xiaoou Tang. Visual semantic complex network for web images, In Int. Conf. Comput. Vis., 2013. 22
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl, 2019. 7
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning.. 2021. 2, 3, 5, 6, 7,9, 9, 17.31 31
Christoph Schuhman, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m synthetic captions from laion2b-en. 2022. 3, 3,
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint 1 14, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Adv. Neural Inform. Process. Syst., 2022.
15, 25
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: a dataset for image captioning with reading comprehension. Eur. Conf. Comput. Vis., 2020. 30
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality, arXiv preprint ar arXiv:2307.0522 2023. 33
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023. 1,4
Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychology, 12(1))97-136, 1980.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. arXiv preprint NXXi::2306.09//15 2023.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In IEEE Conf. Comput. Vis. Pattern Recog 2015. 31
Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. arXiv preprint arXiv:2304.03752 2023a. 4, 26
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305. 1115, 2023b. 3, 4, 6
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Int. Conf. Comput. Vis., 2023c. 2, 5, 17, 24
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. Int. Conf. Learn. Represent., 2022. 9
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.002280, 2022. 3, 33
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014. 8
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res., 2022.
13

---

Published as a conference paper at ICLR 2024
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speaker- bieekerrlittener-reifooroee model for referring expressions. IEEE Conf. Comput. Vis. Pattem Recog., 2017. 3,
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18123-18133 2022.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interee arXiv preprint arXiv:2307.03601, 2023. 4
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region- language-image pretraining. In IEEE Conf. Comput. Vis. Pattern Recog., 2022. 9, 32
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, b answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594 2023a. 31
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arX5v:2304.10592, 2023b. 1, 2, 4, 5, 6, 17, 22, 25, 30, 31
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In IEEE Conf. Comput. Vis. Pottern Recog., 2016. 15
14

---

Published as a conference paper at ICLR 2024
APPENDIX OF "THE ALL-SEEING PROJECT: TOWARDS PANOPTIC VISUAL RECOGNITION AND UNDERSTANDING OF THE OPEN WORLD"
A DATASHEET FOR AS- 1B DATASET
A.1 MOTIVATION
QI For what purpase was the dataset created? Was there specific task in mind? Was there a specific gap that needed to be filled? Please provide description.
AS-IB was created the first large-scale dataset with comprehensive and detailed instance-level annotations. This dataset contains a wide range of open-world concepts, including over 3.5 million different semantic tags ranging from common categories (e.g.. human, backpack) to fine-ge teee-graied or rare categories with attributes (e.g., old metal latches), which serves the data foundation to train powerful model for open-world panoptic visual recognition and understanding. Before the curation of AS-1B, existing large-scale image-text pair datasets (Schuhmann al., 2022; Byeon et al., 2022; Gadre et al., 2023) only include descriptions or guesttio-ansserrin the entire image. On the other hand, datasets like Visual Genome (Krishna al., 2017) and Visual7W (Zhu et al., 201 6) integrate visual location and understanding, offering more comprehensive tasks describe the visual world. However, these datasets have limited data scale and fail to encompass diverse scenarios in the open world.
pairs corresponding to
Q2 Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
This dataset is presented by OpenGVLab of Shanghai AI Laboratory.
Q3 Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
This work was sponsored by Shanghai AI Laboratory.
Q4 Any other comments?
No.
A.2 COMPOSITION
Q5 What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide description.
Each instance in AS-IB represents a region in a certain image.
Q6 How many instances there in total (of each type, if appropriate)?
AS-1B IB consists of over 11 million images, 1 billion region-text pairs, 3.5 million open-world concepts, and 100 billion tokens of region-related question-answering and captions. A further overview of the statistics may be seen in the Table 1.
Q7 Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from larger set? If the dataset is sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verifed. if is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
AS-1B is created based on the SA-1B dataset (Kirillov et al., 2023). From this collection of images, we extend the original annotations in SA-IB with scalable semi-automatic data engine. Please see Section 3 for more details about the data engine.
Q8 What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
15

---

Published as a conference paper at ICLR 2024
We provide 1.2 billion instance-level annotations. Each annotation consists of the following: an image file url; box coordinates in the format of (01. 1/1 22, V2); semantic tag; three question-anse -nnwerr pairs; detailed region caption.
Q9 Is there label or target associated with each instance? so, please provide description.
No, we do not define any label or target for the instances. Targets are task-dependent. AS-IB can be used for variety of tasks such region recognition (inputs regions, targets = semantic tags), region captioning (inputs = regions, targets = captions), or region-level visual question answering (inputs images and questions, targets answers).
Q10 Is any information missing from individual instances? If so, please provide description, explaining why this information missing (e.g.. because it was unavailable), This does not include intentionally removed information, but might include, e.g., redacted text.
No.
Q11 Are relationships between individual instances made explicit (e.g,, users' movie ratings, social network links)? If please describe how these relationships are made explicit.
No.
Q12 2 Are there recommended data splits (e.g,, training, developmentUvaliiattion, testing)? If so, please provide description of these splits, explaining the rationale behind them.
No.
Q13 Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
AS-IB is noisy by design since the annotations are produced by semi-automatic data annotation pipeline. However, this pipeline operated in a "data-human munta--uma-model"" loop, iteratively refining data quality. Besides, we also provide a fully human- verified subset of IB with over 800k instance-level annotations, which is much more clean.
Q14 Is the dataset celf-contained, does it link to otherwise rely on external resources (e.g,, wehsites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links other access points, as appropriate.
This dataset is established based on SA-IB dataset (Kirillov al., 2023). In response sub-questions: (a) These image servers ensure stable access unless the SA-1B authors delete their images; (b) Yes, AS- IB archives all the annotations. For images, AS-IB only archives the URL and not the media content: (c) All restrictions follow the SA- 1B dataset. We do not introduce any additional restrictions.
Q15 Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a description.
No.
Q16 Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
The annotations in this dataset are produced by semi-automatic data engine. For the image component of AS- IB, we utilize high-resolution images from SA-1B. These images have undergone rigorous selection and privacy protection by Meta AI to ensure their suitability and compliance with privacy standards. For the text component of AS-1B, have constrained the model not to generate offensive content and have removed annotations containing potentially offensive words or phrases?.
Ohscheee--nccccccn-aeaaa Ottheeewiee---e-e-eeeee Bad- Bad-Words
16

---

Published as a conference paper at ICLR 2024
Q17 Does the dataset relate to people? If not, you may skip the remaining questions in this section.
People might be found in the images or textual descriptions, but they are not the primary emphasis of the dataset.
Q18 Does the dataset identify any subpopulations (e.g., by age, gender)?
We don't include any indicators of subpopulations as attributes for the region-text pairs, although might be inferred in certain annotations.
Q19 Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e,, in combination with other data) from the dataset? If so. please describe how.
No. The images in AS- are sourced from the SA-1B, where faces and vehicle license plates have been blurred in the released images.
Q20 Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious heliefs, political opinions or union memberships, genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
locations; financial or health data; biometric or
No. The annotations in this dataset are generated by autimautomatic data engine. Our model is not able to generate any sensitive content.
Q21 Any other comments?
No.
A.3 COLLECTION PROCESS
Q22 How was the data associated with each instance acquired? Was the data directly observable (e.g.. raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g. part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
The annotations of AS-1B are generated by semi-automatic data engine. As the core component of the data engine, the data generation pipeline consists of five steps as follows: (1) Creating ppendworld location (e.g., bounding box, mask, point set) with an ensemble of sfate-of-the-art class-agnostic, visual grounding, and dosed-set perception models (Kirillov et al., 2023; Li al., 2022b; Wang et al., 2023c; Fang et al., 2023a);
(2) Generating open-world world semantic tags using the combination of image captioning models (Li et al., 2022a; Zhu et al., 2023b) and LLMS (Chiang et al., 2023); (3) Matching the semantic tags to proper regions with image-text aligning models (Radford et al., 2021; Lüddecke & Ecker, 2022); (4) Using LLM (Chiang et al., 2023) and VQA models (Liu et al., 2023b) to generate the attributions of each region based on the matched semantic tags; (5) Generating detailed captions based on the semantics and attributions of each region. See Section 3 for more details about the data engine.
To verify the quality of the generated annotations, we asked 10 experts to annotate 1000 randomly sampled data from AS-1B with correct or wrong after each human-in-loop iteration separately. These annotations achieve an accuracy rate of 83.5 33.5%. In addition to the large-scale automatic annotations for over billion regions, we will also release a clean version dataset, containing over 800k annotations that have been fully verified by human annotators. The accuracy of annotations from such version is 95.7%.
Q23 What mechanisms procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?
We the data engine in Python, over 32 8-A100(8000 GPU machine. We validate our implementation by manually checking a few results of the data engine.
Q24 If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)
17

---

Published as a conference paper at ICLR 2024
AS-1B is created based on the SA- dataset. We annotate all images in the SA-1B.
Q25 Who was involved in the data collection process (e.g., students, crowdworkers, contrac- tors) and how were they compensated (e.g., how much were crowdworkers paid)?
Our data engine requires crowdworkers to verify subset of the generated annotations. Specifically, human annotators are tasked with verifying the annotations of semantic tags and question-an -answer pairs. For semantic tag verification, pay 0.02 dollars for each region. Regarding the verification of question-answer pairs, we pay 0.02 dollars for the first stage and 0.07 dollars for the second stage.
Q26 Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
The licensed photos vary in their date taken over wide range of years up to 2022
Q27
Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
We did not conduct a formal ethical review process via institutional review boards. However, as described in Q16, we employed several filtering mechanisms to try and remove instances that could be problematic.
Q28 Does the dataset relate to people? If not, you may skip the remaining questions in this section.
People might be present in the images and descriptions, although they are not the sole emphasis of the dataset.
Q29 Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
We collect the data by annotating the images from the SA-1B dataset with our proposed semi-automatic data engine.
Q30 Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots other information) how notice was provided, and provide link or other access point to, or otherwise reproduce, the exact language of the notification itself.
For the image component of AS-IB, we utilize high-resolution images from SA- IB, which are licensed from a third party who provided appropriate representations regarding the collection of any notices and consents as required from individuals. For the text component of AS-IB, annotations are generated by semi-automatic data engine, and the individuals in the image were not notified since their personal information has been hidden via facial blurring.
Q31 Did the individuals in question consent to the collection and use of their data? so, please describe (or show with screenshots other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
No. See Q30 for more details.
Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide description, as well as a link or other access point to the mechanism (if appropriate).
Users can contact the research team of the SA- IB dataset for image(s) removal. Besides, users can contact us to remove any annotation in our proposed AS-IB.
Q33 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis. including the outcomes, as well as a link or other access point to any supporting documentation.
18

---

Published as a conference paper at ICLR 2024
To eliminate any potential impact on people whose photos included in the dataset, identifiable information (faces, license plates) has been blurred by the research team of the SA- IB dataset.
Q34 Any other comments? No.
A.4
PREPROCESSING, CLEANING, AND/OR LABELING
Q35 Was any seeproeesinggcceannn//aaae ing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions this section.
of the data done (e.g.. discretization or bucket-
The data engine operates in a sas--human-oodeee loop, iteratively refining data quality. Besides, we will also release subset of AS-IB with over 800k annotations, which has been fully verified by human annotators.
Q36 Was the "raw" data saved in addition to the preproceseddccenne///aae support unanticipated future uses)? If so, please provide link other access point to the "raw". data
data (e.g.. to
No.
Q37 Is the software used to preprocess/clean//aee the instances available? If so, please provide a link or other access point.
Yes, the data collection code will be open-sourced and accessible from the dataset website.
Q38 Any other comments?
No.
A.5
USES
Q39 Has the dataset been used for any tasks already? If so, please provide a description.
The AS-1B has been employed to train the All-Seeing Modell (ASM). As discussed in Section 5, ASM exhibits powerful performance in both region recognition and region captioning tasks, which strongly demonstrates the effectiveness of AS-IB.
Q40 Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
No.
Q41 What (other) tasks could the dataset be used for?
The dataset could used for variety of region-level iision-and-languaa (V&L) tasks, such region recognition, region captioning, and region-level visual question answering.
Q42 Is there anything about the composition the dataset or the way it was collected and prepooessealcleaneelabeeee that might impact future uses? For example, is there anything that future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g. fiaaccial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?
No.
Q43 Are there tasks for which the dataset should not be used? If so, please provide description.
Our dataset should only be used for non-commercial academic research.
Q44 Any other comments?
No.
19

---

Published as a conference paper at ICLR 2024
A.6 DISTRIBUTION
Q45 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
Yes, the dataset will be open-source.
Q46 How will the dataset be distributed (e.g,, tarball on website, API, GitHub)? Does the dataset have digital object identifier (DOI)?
The data will be available through GitHub.
Q47
When will the dataset be distributed?
3 1/03/2024 and onward.
Q48 Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
Apache 2.0 license
Q49 Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point otherwise reproduce, uny relevant licensing terms, well as any fees associated with these restrictions.
AS- IB owns the metadata and release as Apache 2.0 license.
We do not own the copyright of the images.
Q50 Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? so, please describe these restrictions, and provide link other access point to, or otherwise reproduce, any supporting documentation.
No.
Q51 Any other comments?
No.
A.7
MAINTENANCE
Q52 Who will be nunporriingnosiinn/mmmanninnnnnn the dataset?
Huggingface will support hosting of the metadata.
OpenGVLab of Shanghai AI Laboratory will maintain the samples distributed.
Q53 How can the owner/cuurator/maaage of the dataset be contacted (e.g., email address)?
https: https:///ithus com/ Opwn//Lab//eaaeeeeeeeeeeaaaaa all sll-sseeing
Q54 Is there an erratum? If so, please provide a link or other access point.
No.
Q55 Will the dataset be updated (e.g., correct labeling errors, add new instances, delete in- stances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
No. However, specific samples can be removed on request.
Q56 If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period time and then deleted)? If so, please describe these limits and explain how they will be enforced.
People may contact us add specific samples to blacklist.
Q57 Will older versions of the dataset continue to be supported/hostedlainnte please describe how. If not, please describe how its obsolescence will be communicated to
so,
20

---

Published as a conference paper at ICLR 2024
We will only support and maintain the latest version at all times and new version release of AS- IB will automatically deprecate its previous version.
Q58 If others want to extend/augment/huil on/contribute to the dataset, is there mech- anism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for goamuunicaing//dstrbbtttii these contributions to other users? If so, please provide description.
We welcome any contributions to AS-1B and we will announce updates regarding dataset extensions on GitHub. However, contributors must demonstrate the quality and harmlessness of the extended data annotations; otherwise, we will not accept these extensions.
Q59 Any other comments?
No.
B DETAILS OF DATA ANNOTATION ENGINE
B. OPEN-WORLD LOCALIZATION
Due to the incomparable score ranges of different models, directly using non non-maximum suppression (NMS) eliminate duplicated proposals from multiple resources is infeasible. Therefore, we develop an effective strategy that keeps all the semantics while removing highly overlapped regions. As shown in Alg. the merging strategy works follows: (1) We start by initializing the result region proposal set R with the class-agnostic bounding boxes generated by SAM. (2) When a set of region proposals R' from new source (e.g., closed-set/groundding detector) comes in, we calculate the Intersection over Union (IoU) between the regions in R' and R. (3) If the IoU between new region r' E R' and an existing region r E greater than a threshold TioU, the region r' is removed, and its (losed-set/grounddil remaining low-loU regions R' along with their tags are added to R. By employing this strategy, we sequentially combine the results of SAM, Internlmage. EVA-02 and GLIP to obtain comprehensive location information for an image.
tags are appended to the tag list of the matched region (3) Finally, the
Algorithm Region Proposal Merging Input:
Existing region proposals R
New region proposals R'
IoU threshold Ti.U
Output:
Merged region proposals R
1: for region r' E R' do
2: Calculate IoU between r' and proposals in R
3: if maximum IoU Tiou then
4: Merge semantic tags from r' into the semantic tag of corresponding regions in R
5: Delete r'
6: else
7: Add r' into R
8: end if
9: end for
B.2 WORLD SEMANTIC
Expanding on our brief description in Section 3.3, this section provides an illustration of the modules utilized to generate the semantic tags and detailed descriptions.
21

---

Published as a conference paper at ICLR 2024
B.2.1 SEMANTIC TAGS
Spotter. This module aims to list the prominent and major objects present in the given image. To achieve this, we use MiniGPT4 (Zhu et al., 2023b) to provide an overall caption of the input image. From the generated captions, we extract noun phrases to serve as the semantic tags shared by all the regions in the input image. In addition, we also add OCR detector (contributors, 2023) detect the texts as semantic tags in the scenes. Note that the generated caption will also be passed to other annotators, which gives visual signal for the LLMs, serving as their eyes.
Imaginator. Although the "spotter" can find out the major objects in the scenes, it fails to identify many insignificant objects. To address this limitation, we develop "imaginator" further expand the semantic tag list with plausible imagination. The "imaginator" emulates human-lite like thinking. When provided with descriptions of particular scene, humans can effortlessly imagine the potential objects present. For instance, if informed that an image depicts a group of children standing in classroom, one may envision objects like "teacher", "blackboard", and "stationery"". In our data engine, we utilize Vicuna (Chiang et al., 2023) to imagine possible objects in scenes based on the captions generated by the "spotter", and then extend the set using web search engines (Qiu et al., 2013). The "imaginator" excels at supplementing scene-specific object candidates, such as suggesting "airport stuff" instead of simply "person". This significantly enhances the concept diversity within this project.
Splitter. This model is proposed to divide the generated concepts into more fine-grained parts. We find that some region proposals only cover a part of the objects, such the wing of plane the windshield of a car. However, most of the existing perception or caption models are not capable of detecting parts. To this end, we further instruct the Vieuna (Chiang al., 2023) to divide the semantic tag into parts. For example, "building" will be decomposed into "roof", "door", "windows" and "walls". We tailor the prompt for LLM that the model only divides the semantic tag that represents concrete object into parts. LLM is instructed to ignore the semantic candidate that is non- con-physical or cannot be further divided, such as "water", "sky", etc.
Magnifier. Although hundreds of open-world semantic tags can be generated by the aforementioned annotators for each image, there still exists some regions whose semanties are absent from the generated tag lists. So we introduce a "magnifier" to zoom in on each region and add semantic tags for them. We simply crop the region and use a caption model to describe the cropped image, and then extract the noun phrases, which are used the semantic candidates exclusive for the corresponding regions. In this model, we use BLIP (Li et al., 2022a) for efficiency.
B.2.2 DETAILED DESCRIPTIONS
Questioner. Given semantic tag, to determine its commonly-used attributes, we use Vicuna (Chiang et al., 2023) as questioner to generate three questions about the attributes or statuses. The prompt is shown below. In this way, we leverage the world knowledge and reasoning capabilities of LLMs to identify the most relevant attribute of object.
Instruction: will give you some objects. Please list questions about the given objects. These questions must be answerable based on photograph of the object and cannot rely on any outside knowledge. Some examples are listed as follows:
Human: Person
Assistant: Q1: What is the of this person? Q2: What is the hairstyle of this person? Q3: What is this person doing?
Human:
(Semantic Tag)
Assistant:
Responder. After obtaining the questions related to semantic tag, we employ Husky (Liu et al., 2023b), an LLM-based VQA model, to generate the responses to each question. The responses are generated in several sentences, taking into account the content of the region. An example prompt is shown below. This approach enables us to gather additional information about a region while preventing the inclusion of irrelevant content.
22

---

Published as a conference paper at ICLR 2024
Instruction: <ing><lmgeeooneen<</mngg Human: (Question for the image] Assistant:
Writer. Based on the question-answert pairs, we proceeded to use Vicuna (Chiang et al., 2023) to rephrase them into a single sentence, resulting in a detailed description of the region. The prompt used during annotation is shown below. It is notable that both the question steps and the region captions from this step are valuable for visual recognition and understanding models.
pairs from previous
Please paraphrase the following sentences into one sentence. (answer for question } (answer for question 2} [answer for question 3}
Human:
Assistant:
B.3 HUMAN ANNOTATION
Here, we introduce the details of human annotation process.
Semantic tags. We design a data sampling strategy and simplify the task for annotators by focusing on picking the incorrect ones from the top-5 candidates in each region. In the real world, concepts exhibit long-tail tail distribution as shown in Fig. 2. Therefore, many the region is randomly sampled for validation. To address this issue, we implement a concept-wise sampling strategy. Specifically, we collect list of concepts in the first IM images in the AS-1B dataset. From this list, we select most concepts for verification. We randomly sample regions from the least frequent concepts and 90 regions from the concepts with the highest number of regions. During the human verification process, the semantic tag list for the sampled regions is provided to the annotators, who then tasked with filtering out any incorrect tags.
concepts will be missed if
Visual Question-Answering Pairs. Although using LLMs/VL. LLLM/ greatly reduces the annotation cost of generating visual question-ansse answer pairs, there are still some issues that may introduce noise into the data. (1) The answer to the question is wrong since the VLLM is not perfect. (2) The generated question for the semantic tag may be unanswerable according the given image content. (3) The semantic tag assigned to region may be incorrect, leading to meaningless generated questions. For example, if a region containing a dog is wrongly labeled as a cat, asking about the color of the cat would be nonsensical.
To address these issues, we perform a two-stage verification procedure. the first stage, human annotators are provided with the image, location (hounding box), and corresponding question-answer pairs. They are then asked to annotate the visual question question-answer pair with one four choices: correct answer, wrong answer, unanswerable question, wrong semantic tag. Samples annotated as "correct answer" are retained, while those annotated as "wrong answer" are re-annotated with a correct answer generated by human annotators in the second stage. Samples annotated "unanswerable question" "wrong semantic tag" are annotated with a rejection answer, such as "This question is unanswerable according to the image" "The object in this region is incorrectly labeled", respectively.
We engaged 50 human annotators to perform verification on the annotations generated by our model. To guarantee the quality of this verification process, we additionally request 10 experts to review the verified annotations. These experts are selected based on their domain knowledge and experience in annotation tasks. To streamline the process, we organize the regions requiring review into groups of 100. Each group is assigned to one expert, who checks the accuracy and consistency of the annotations within the group. Any package with an accuracy rate below 95% % will be sent back for re-verification by another annotator. This review process double-checks the annotations, further ensuring their reliability and validity for our models.
Verification Review.
B.4 DATA ENGINE ITERATION
To continuously improve the data quality, we implement "data-human-moder" loop that maximizes the utilization of both human-verified data and models. As depicted in Alg. 2, the data engine iteration comprises three steps follows: (1) The images are processed with the annotation pipeline which produces automatic annotations. (2) The ASM model is then trained using these coarse annotations, enabling it to perform both discriminative and generative tasks such region-text matching and
23

---

Published as a conference paper at ICLR 2024
Algorithm 2 Data Engine
Input:
Iteration Number
Images I
Models M
Annotation Pipeline P)M,I)
Output:
Annotations: A
Improved Models M
1: Generate initial annotation An by off-the-shelf models;
2: Train ASM with Ao. yield Mu;
3: <-0
4: while <<n n do
5: Perform Human verification on A:, yield A/;
6: Fine-tune M, with A', obtain M4+1;
7: Obtain Annotation A++1 by P(M++I,I);
8: i i i+1
9: end while
region captioning. (3) The automatic annotations sampled, reviewed, and corrected by human annotators, yielding high-q qugliiy human annotations. This verified data is then used to fine-tune the ASM model, thereby enhancing its performance. (4) The fine-tuned model is utilized re-rank the semantic tags and generate more accurate region captions and answers. Repeat the third and fourth steps until the data quality meets the requirements. By following this data iteration process, ensure continuous optimization of data quality, ultimately leading to superior results. In Appendix E.3, we demonstrate the effectiveness of this iteration process with quantitative experiments.
C DATA ANALYSIS
We conduct an in-depth analysis of our AS-1B dataset. We begin by showcasing the abundance of data in terms of quantity. Next, we explore the data diversity and open-world semantics captured in AS-IB. Finally, we thoroughly analyze the data quality of the initial automatic annotation pipeline and explain how we have improved it through data engineering and human feedback.
C.I DATA SCALE
Statistics. The AS-IB dataset consists of a vast collection of 1.2 billion region-text pairs extracted from 11 million images, encompassing 3.5 million distinct semantic tags. Regions in the dataset categorized into five different resolution scales: tiny, small, medium, large, xlarge, and huge. As indicated in Table 5, the distribution of region resolutions follows roughly normal distribution. Over half of the regions are on the medium or large scale. In Sec. 3.2, we utilize several region proposal generators, including SAM (Kirillov al., 2023), Internlmage (Wang al., 2023c), EVA-02 (Fang et al., 2023a), and GLIP (Li et al., 2022b), to generate region proposals for the AS- 1B dataset. Table 6 presents the proportion of regions provided by each model in the 1.2 billion regions. SAM generates 36.4% of the regions, while the other three models contribute to 63.6% of the regions. Therefore. although our dataset shares images with SA-1B (Kirillov et al., 2023) and has similar number of regions, the actual regions are different due to the use of diverse region proposal generators.
Each region also annotated with detailed question-answe pairs and a caption, which yields a total of 3.3 billion visual question-answering pairs and 1.2 billion detailed region captions. As seen in Table 7, the average token number of the answers is 16.91, while the average token number of the composited caption is 34.84. The total number of tokens in detailed region captions amounts to approximately 42.2 billion. This extensive collection of detailed captions provides valuable textual descriptions of regions within the images.
24

---

Published as a conference paper at ICLR 2024
Table 5: Region statistics and semantic sources. The percentage of semantic tags generated by different models at each resolution are reported. LLM/VLLMs (Chiang et al., 2023; Zhu et al., 2023b; Li al., 2022a) contribute significantly the semantic diversity of our dataset.
Region Type
Area Range
Proportion
(V)LLMs
BLIP
InternImage
EVA-02
GLIP
Tiny
< 20 20 ~ 40
4.2%
33.8%
16.5%
24.6%
25.1%
0.0%
Small
8.7%
34.5%
14.3%
24.6%
25.9%
0.7%
Medium
402 1002
35.8%
55.6%
22.9%
8.3%
11.6%
1.7%
Large
100 200
23.7%
58.5%
26.2%
5.0%
7.9%
2.3%
Xlarge
200 500
18.3%
62.6%
27.1%
3.0%
4.3%
3.0%
Huge
500
9.5%
69.7%
24.9%
1.6%
1.2%
2.7%
All
100%
55.4%
24.0%
8.2%
10.4%
2.1%
Somanti: Tag accuracy (4)
BUP
Intermmagge
ENW
GUP
80
274
765
60
70
52.3
15.0
52.6
02.5663
50.
15. 96
51.6
50
40.11
73.
14.
Tiyy
Lange
Xlarge
Huge
Figure 5: The accuracy of semantic tags from different sources. LLM/VLLMs (Chiang et al., 2023: Zhu al., 2023b; Li et al., 2022a) show lower accuracy than other models, especially on low resolution regions.
Comparisons. When comparing the AS-IB dataset with popular datasets containing region-level annotations, AS-1B stands out with significantly larger number of regions. It has about 33 times more regions than the current largest detection dataset, BigDetection (Cai al., 2022). While AS- IB has fewer images compared to close-set classification datasets (Deng et al., 2009) or vision-language datasets (Schuhmann al., 2022), compensates with valuable region annotations. Additionally, AS-1B offers abundant collection of detailed region annotations. Compared to the largest region- level dataset, Visual Genome (Krishna et al., 201 7), AS-1B's detailed region annotation about 1941 times larger than Visual Genome's 's 1.7 million pairs of VQA annotations and 222 times larger than its 5.4 million region captions.
C.2 DATA DIVERSITY
Statistics. distinctive feature of AS-IB its vast inclusion of open-world concepts, demonstrated through two key aspects: 1) large number of semantic tags and 2) long and informative detailed descriptions. Fig. visually demonstrates the wide range of open-world concepts present in AS-1B The dataset covers diverse categories, including fine-grained categories like "lynx", proper such as "The Sphinxs", object parts like "charging cords", and attributes like "pink and white baby cribs". In Fig. 2, we display the frequency distribution of semantic tags, revealing clear long-tai tail pattern. The most frequent semantic tags predominantly represent broad category names, while less frequent tags correspond fine- ane-grained category names or instances with specific attributes.
In Table 5, we analyze the sources of each semantic tag to understand how open-world concepts are enriched. We report the proportion of sources for the top-1 semanties in the semantic tags at different scales. The results reveal that 55% of the top-1 semantic candidates are from the LLM,, while 24% originate from the BLIP (the "magnifier" Sec. 3.3). Interestingly, only 19% of the top-1 candidates are generated from the closed-set detectors, InternImage. and EVA-02. This highlights that the majority of concepts in the AS-IB dataset are obtained from open-world sources, especially the LLMs and VLLMs.
However, although semantic tags generated by LLM/VLLMs can introduce comprehensive open- world semantic tag candidates. they may also introduce the image, thus giving rise to hallucinations. So we further analyze the accuracy of these retained semantic tags generated by LLLM/VLLMs through human evaluation. Through human evaluation,
semantic tags that do not exist in
25

---

Published as a conference paper at ICLR 2024
Table 6: The proportion of region proposals generated by different models. Only 40% re- gions are generated from SAM.
Table The statisties of detailed description in AS-1B dataset. The overall number of tokens reaches 132.2 billion.
Model
SAM Internlmage EVA-02 GLIP
Type
Nuumber #Tokens Average Tokens
Proportion
36.4%
20.5%
22.5%
20.6%
Question
3.3B
34.6B
10.50
Answer
3.3B
55.4B
16,91
Caption
1.2B
42.2B
34.84
Table 8: The statisties of annotation accuracy. AS-LoopX denotes the accuracy after the X-th human-in-loop iteration. AS-Human represents the accuracy of the subset of AS-IB which has been fully verified by human annotators.
Stage
AS- AS-LoopO
AS-Loop1
AS-Loop2
AS-Loop3
AS-Huumn
Acc for semantic tags
54.8%
70.2%
76.7%
80.6%
95.3%
Acc for question-answer pairs
54.8%
75.0%
80.3%
83.5%
95.7%
the accuracy for the automatic data annotations is approximately 80.6%. For the data annotations that have undergone complete manual verification, the accuracy is 94.6%. Furthermore, the retained proportion of semantic tags ultimately produced from LLM/VLLMs is approximately 26.0%. These data demonstrate that, despite the fact that most semantic tags generated by LLM/VLLMs may not actually be present in the image, they do not introduce excessive noise after the semantic-location matching stage.
As for the detailed region caption, the VQA-based generation approach in AS- has proven advantageous, resulting in longer and more informative region descriptions. more straight-forward way is to directly ask the VLLM to generate region captions. However, without guidance from semantic tags and questions, the model tends to output inaccurate information or hallucinations.
Comparisons. Instead of using fixed labels from pre-defined set, the AS-IB dataset employs flexible and open-world semantic tags to label each region. Table highlights that AS- contains a significantly larger number of semantic tags and concepts compared close-set classification datasets or object detection datasets. For example, the number of semantic tags in AS-1B is approximately 159 times greater than the widely-used classification dataset ImageNet-22K (Deng et al., 2009), and it is 268 times larger than the category number in V3Det (Wang et al., 2023a).
C.3 DATA QUALITY
The Accuracy of Automatic Annotations. We asked 10 experts annotate 1000 randomly sampled data from AS- IB with correct or wrong after each human-in-loop itcration separately and report the accuracy rates of our annotations for semantic tags and question-answer pairs after each human-in-loop iteration separately. As shown in Table 8, the introduction of human verification significantly improves the data quality, from 54.8% before human involvement 75.0% after the first human-in-loop iterations. Moreover, as the number of loop iterations increases, the data annotation quality gradually improves. After the third loop, are able achieve an accuracy rate of 83.5%. In addition to the large-scale automatic annotations for over billion regions, we will also release clean version dataset, containing over 800k annotations that have been fully verified by human annotators. The accuracy of annotations from such a version is 95.3% for semantic tags and 95.7% for question-answer pairs.
As shown in Figure 5, we find that different models in the annotation pipeline exhibit complementary behavior. The LLM and BLIP models show lower accuracy for small regions as they are not robust for cropped low-resolution images. In contrast, close-set detectors perform better on these small regions, providing more accurate semantic candidates. For larger regions, LLMs and VLLMS become more accurate. Hence, the inclusion of close-set detectors provide trade-off off between data quality and open-world semantics. This interplay of models contributes to the overall improvement of data quality in AS-1B.
26

---

Published as a conference paper at ICLR 2024
Consumption Analysis. Here focus the consumption and efficiency of human verification in the context of the semi-automatic data engine we constructed. This approach significantly reduces the human labor required for data refinement compared with annotating all the data by humans. For verifying semantic tags, it takes approximately 10 seconds for one annotator to complete one region. Verifying every million regions would take about 2,750 working hours. Considering group of 50 annotators in our case, the entire verification process takes approximately 15 days. If we were to annotate all regions, the annotation consumption would become 1,000 times larger, approximately 42 years. Such a large-scale human annotation effort would be unaffordable.
Moreover, for detailed captions with longer texts, the verification process would take even longer. e.g. 15 seconds for each VQA annotation. Therefore, for large-scale annotation involving billions of regions in our case, utilizing models to annotate data scale and correcting the models' bias with limited human annotation proves to be both feasible and efficient.
27

---

Published as a conference paper at ICLR 2024
Burj khalifa
Vondtstaring
air traffic
control tower
and baby cribs
AIRSPOT
3
The Sphinxs
boat lift
Charging cords
candle drippings
Trump
lynx
Eagle beak
ON
TRemppiilee NEd LINE MAEH Exiit
RUMP:
JERL
Figure 6: Examples of the semantic tags. Benefitting from the world knowledge of LLMs/VLLMM. the AS-IB dataset covers diversity semantic tags in the real world.
28

---

Published as a conference paper at ICLR 2024
Somantic Tag
Semantic Tag
stotue
small windows
Detailed Annotation
Detailed Annotation
saade
Yo. small windows operable
Qu: What the liigtt of this statec?
hought catar nor provided in
parpace
image
windows?
G+: Wa is oxtject of this siiter?
Al: The pagous of tane scall wirdlors is porode rutural light and ventianee to the buiMing alowing fook dircalate hrigt kis specn
Alt: The subject of the statac is seccan.
provitly pooklc
fianc, shanling
qJ: Are thone emall whsdowe locared the eateriee isterite of building? Al: small windens are located
Detailed Caption::
The satue, amde Iesnue,
depece wortn, posebly a aoddave ora Iemenoil fypnn eteeing in of hualding, heigd krage
Detuiled Caption:: The parpos dae srtall srndow tmugs, which located or
light spenabll,
herlding allwwtng trus cirabar beigltor
Somantic Tag
Semantic Tag
banner
yellow shirt
Detailed Annotation
Detailed
Whst buuner?
Qt: shirt this person
ycllewe
q2: What materbal make this
0t: ahirt sachedd
@t: Whar tspe manerbal is the siire made off
continstion fibrrl and
Al; The abint is raade of lightociali matiotaa.
purpone
seca outles lawas aildiaatooo aay brartshibey.
perpour kistty
public.
Caption:
The perems weaningu
the in the image in made of ceribruaton tiick plastic
Detated Caption:
bevergg
shiich slowe movnmare tmortaiiiiey.
specified
Semantic Tag
Semmantic Tag
building
barge
Detailed Annotation
Detailed Annotation
Qhi What the nomo belldin??
Qt:
bacas?
Al:
the building pooiided
liigh hangy
Q2: What arehttictural style briiking??
O2: bast the color this harge?
A2: The oxlor of hange is not.
A2: The anbioctaaal strle the buiidiin is image medern and dedic fanerrng misiccals deiig.
QA: What is the parpese of thi barge??
Al: parpose the harge ranaquer
gray woken
people goods
go: What
of this tnatading??
Detalled Caption: the leauts
ionot
ARi The pargost of building is serve ros dertal commercial qucce for people ixssos.
specifie in the image, bur is rot and ased morqprr people and goods suroes the deer.
Detailed sleck buildnng wilu wlike and jasy ools sclicmee and mentroist desig,, oricc serves recdestal conneercid space So prople heassnss.
The
Figure 7: Examples of the detailed region annotations. Visual guestion-answenr pairs and captions are provided based on the semantic tags. Failure cases marked in red.
29

---

Published as a conference paper at ICLR 2024
D INPLEMENTATION DETAILS
D.1 MODEL DETAILS
The example prompts used by ASM are shown below. Prompt #1 is an example prompt for generative lasks, while Prompt #2 and #3 are example prompts for discriminative tasks.
Prompt #1: "(Pg} (bos) Human: {V} What is this? Assistant:"
where the token number of task prompt M is set to 5. (bos) represents the beginning of the sentence.
Prompt #2: "(Pa) (bos) Human: (V} What this? (align)"
where Pa E RMXD: represents the learnable task prompt used for discriminative tasks.
Prompt #3: "(Pa} (bos) Assistant: photo of the Sphinx. (align)"
It notable that the learnable task prompt tokens and align tokens used in Prompt #2 and #3 are shared, while the task prompt tokens differ between generative tasks (Prompt #1) and discriminative tasks (Prompt #2 and #3).
D.2 TRAINING DETAILS
The pre-training of the All-Seeing Model (ASM) involves three types of labels obtained from the AS-IB dataset, including region-level semantic tags, question-ansser pairs, and detailed captions. The semantic tags used for aligning regions with corresponding text, while the other annotations are used to train the text generation task. addition, we also include LaionCOCO (Schuhman et al., 2022) in our pre-training process, since the image-level caption data from LaionCOCO is beneficial for ASM's ability to comprehend the whole images. For the supervised fine-tuning process, in addition to human-verified region annotations,
also utilize MiniGPT-4 (Zhu al., 2023b), LLaVA- 150k (Liu et al., 2023a), TextCaps (Sidorov et al., 2020), and COCO caption dataset (Chen et al., 2015) as image-level text generation data, along with VG (Krishna et al., 2017) and RefCOCOg (Mao et al., 2016) datasets region-level text data. VQA datasets (Goyal et al., 2017; Hudson & Manning, 2019; Krishna et al., 2017: Chen et al., 2022a) are also utilized enhance the vqa ability of ASM. In both of the stages, we optimize the Q-Foree, soft prompt tokens and align tokens. In addition, LoRA (Hu et al., 2022) is employed to finetune the Vision Foundation Model (VFM) (Fang et al., 2023b) and Large Language Model (LLM) (Liu et al., 2023b).
We adopt a multi-task training approach that combines text generation and region-text alignment tasks to train our ASM. The batch size for text generation is set to 256, while for region text alignment is set to 32,768. We employ the AdamW optimizer (Loshchilov & Hutter, 2019) with the B1 of 0.9, the 83 of 0.999, and the weight decay of O. During training, the learning rate is initialized as 5 x 10-4 and includes a linear warmup that lasts until the first 10% training steps. The warmup is followed by cosine decay strategy with minimum learning rate of O. The image resolution of ASM is set to 224 x 224 for pre-training and 364 x 364 for fine-tuning. We initialize the model parameters using Husky (Liu et al., 2023b) and train the model for one epoch. In addition, also provide a second-stage fine-tuning setting to further improve the effectiveness of ASM. Specifically, we utilize high-quality multi-moda. data MiniGPT-4 (Zhu et al., 2023b), LLaVA- (Liu et al., 2023a), and COCO caption dataset (Chen et al., 2015) as image-level text generation data, along with VG (Krishna al., 2017) and RefCOCOg (Mao et al., 2016) datasets as region- text data. Human-verified region annotations are also included. During fine-tuning, we set the learning rate to x 10-5 and apply a weight decay of The other settings remain the same as during pre-training.
D.3 BASELINE MODELS
To make comparison with recent popular multi-modality large language models (VLLMs) (Zhu et al., 2023b; Liu et al., 2023a; Li et al., 2023a) that only focus on processing the entire image, crop region from the image and input it to these models for region-level visual recognition
30

---

Published as a conference paper at ICLR 2024
Table 9: Human evaluation results on caption tasks. We ask the users to select the caption that contains the most information regarding the image/region while does not producing any factual errors.
Model
Visual Genome Rate
RefCOCOg Length
Flickr30K
NoCaps
Length
Rate
Rate
Length
Rate
Length
Human
47.8
13.6
10.3
6.3
30.0
16.0
27.3
15.1
LLaVA (Liu et al., 2023a)
4.3
110.8
15.4
100.6
17.5
114.0
9.1
108.4
MiniGPT4 (Zhu et al., 2023b)
8.7
110.9
15.4
113.5
14.2
114.6
13.6
101.0
ASM (ours)
39.2
34.5
46.1
110.8
38.3
121.4
50.0
115.9
and understanding. However, this cropping may result in the loss of some contextual information from the entire image. For better comparison, we implement a simple region-text contrastive model based on CLIP (Radford et al., 2021) as baseline. The baseline model, named Region-Aware CLIP (R-CLIP), is equipped with a RolAlign layer (He et al., 2017) on the feature maps obtained from the vision encoder in the CLIP model. To initialize the model weights, et al., 2021) (ViT-L/14) and then train the CLIP model on our AS- dataset. The model trained for 10, 000 steps with batch size of 32,768. Other training settings is the same those of ASM. Unless otherwise specified, the image resolution of R-CLIP is set to 224 x 224.
leverage CLIP (Radford
E SUPPLEMENTARY EXPERIMENTS
E. VISUAL CAPTIONING
As discussed in ChatCaptioner (Zhu et al., 2023a), using conventional image caption metrics such as Meteor (Banerjee & Lavie, 2005) and CIDEr (Vedantam et al., 2015) may not reliably evaluate relatively lengthy texts generated from LLM-based models. To better assess the text generation ability from a human perspective, we conducted a user study.
Evaluation Setting, In user study, involve total of 5 participants to evaluate the performance of the All-Seeing Model (ASM) along with two other powerful VLLMs: MiniGPT4 (Zhu et al., 2023b), and VA (Liu et al., 2023a). We evaluate image and region-level captioning. For the evaluation. we randomly select 20 samples from each of the Visual Genome, RefCocOg, COCO, and Flickr30K datasets. Participants are asked to choose the most informative captions without any factual errors or hallucinations. Aside from model outputs, we also add the ground truth captions as options, which can be viewed as human outputs.
The human evaluation results in Table 9 indicate that captions generated by our ASM are preferred over those from MiniGPT4 and LLaVA. While LLaVA and MiniGPT4 may produce longer captions for region-level tasks (VG and RefCOCOg), they often introduce over-association, hallucinations, and factual errors. In contrast, ASM generates captions with moderate length and more accurate information. On ReCCOCOg, Flickr30K, and NoCaps datasets, ASM even outperforms human annotations with longer and more detailed captions. This is because human annotators tend to write short captions, while users prefer longer, detailed captions generated by ASM, which also contain fewer factual errors. For image-level generation tasks, ASM produces captions with similar lengths to those from MiniGPT4 and LLaVA but is more frequently favored by users.
Results.
The results clearly demonstrate the effectiveness of ASM and the AS-2B dataset. The VQA-based annotation pipeline provides region-specific information with less irrelevant content, reducing the occurrence of hallucinations. Moreover, human verification further enhances the data quality, leading to significantly better performance on region-level tasks.
E.2 VISUAL QUESTION ANSWERING
Setting. We evaluate our model on the general visual question answering (VQA) benchmarks, including VQAv2 (Goyal et al., 2017), OKVQA (Marino et al., 2019), and GQA (Hudson & Manning, 2019).. Following BLIP2 (Li et al., 2023a), we report VQAScore (Goyal et al., 2017) metric on GQA. Besides, considering the lack of widely recognized
VQAv2 and OK OKVQA and Accuracy region-level VQA benchmarks, we perform the human evaluation to evaluate the region-level VQA ability of our ASM. We randomly select 100 question-anser pairs from the proposed AS-1B. Note
31

---

Published as a conference paper at ICLR 2024
that these question answer pairs annotated by our proposed ASM and have not been manually verified. Then, 5 participants are involved, and each of them is asked to annotate 20 question-ane answer pairs with one of four choices: correct answer, wrong answer, unanswerable question, wrong semantic tag gee question has nothing to do with the object in the region).
Results. For image-level visual ques- tion answering. as shown in table 10, our ASM model achieves comparable perfor- mance with recent Visual Large Language Models (VLLMs) (Dai et al., 2023; Peng et al., 2023). Specifically, on VQAv2 and OK OKVQA benchmarks, ASM achieves VQAScore of 73.1 and 48.6, respectively. On GQA benchmark, ASM achieves 59.0 accuracy. It is notable that the ground truth answers to the questions in these VQA benchmarks are quite brief and consist of only several words, while our ASM model is proposed to annotate the region with detailed descriptions and tends to generate more detailed and complete sentences. So these traditional VQAScore are not proper to evaluate the ability of LLM-bbsee VQA systems like ASM. For region-level visual question answering, as there are no standard benchmarks, we score our model via human verification. As presented in Table 8, our ASM achieves remarkable performance. which demonstrates the ability of ASM to generate detailed and accurate answers to region-level visual questions.
Table 10: Quantitative results on the VQA task.
Model
VQAv2 OKVQA GQA
Flamingo-99 (Alayrac et al., 2022)
51.8
44.7
Flamingo- 80B (Alayrac ct al., 2022)
56.3
50.6
Kosmos-1 (Huang et al., 2023)
51.0
Kosmos-2 (Peng et al., 2023) BLIP-2 (Li et al., 2023a)
51.1
65.0
45.9
32.3
InstructBLIP (Dai et al., 2023)
49.5
ASM-FT (ours)
73.1
48.6
59.0
E.3 DATA ENGINEERING
Here, we use quantitative results to show the impact of data quantity and data engineering, which shown in Alg. 2. Considering the cost the experiment, we use our baseline model R-CLIP. We use the Zero-shot object recognition metrics as in Sec. 5.2 to inspect the impact of data engi- neering, i.e., we use the ground-truth boxes and use R-CLIP to determine the categories following RegionCLIP (Zhong et al., 2022).
Data Scaling-Up. We find that scaling up the semantic tags can be helpful for zero-shot region recognition. To verify this, we train our baseline R-CLIP with different amounts of semantic tags. As shown in Table 11, with more training data (from 1M to 5M images), the R-CLIP model attains better zero object recognition performance.
Data Cleaning. Data cleaning and post-processing are important. In practice, the original data annotation pipeline outputs total of 2. 14 billion regions. We devise simple data cleaning strategy: (1) we sample the top 100 regions with the highest CLIP score at different scales from each image in the AS-IB dataset and (2) we further re-rank the semantic candidates with CLIPSeg (Lüddecke & Ecker, 2022), as discussed in Sec. 3.4. This data cleaning process will compress the original 2. 14B regions into 1.2B regions. As shown in Table 12, adding data cleaning can significantly improve the mAp by 6.0% and 7.5% on COCO and LVIS datasets.
How human verification improves the model? An important part of our data engine to improve the model with human feedback. In this way, the improved model can be used to refine the initial data which is automatically generated. In this section, we investigate the effectiveness of the human verification process. We fine-tune the trained R-CLIP model with human-verified region annotations and find that a small number of human labels can significantly boost the model performance.
Specifically, to make the most of human labels, we utilized both the positive and negative candidates marked by the human annotators. When calculating the contrastive loss, for each region, we randomly selected one positive candidate and use all the unselected candidates negative samples. Compared with the image-to-text part in the original CLIP-style contrastive loss, each region will be compared with more negative text samples. The unselected candidates can be viewed valuable hard samples, indicating when the model will make mistakes.
As shown in Table 13, fine-tuning the model with human data can yield significant performance gain. This demonstrates that a small amount of human data can correct the model's 's bias and hard cases, thus improving performance. The effectiveness of human verification lays the foundation for data
32

---

Published as a conference paper at ICLR 2024
Table 11: Zero-shot object recognition performance (mAP) with different train- ing data scale.
Table 12: Zero-shot object recognition performance (mAP) with and without data cleaning.
Table 13: Zero-shot object recogni- tion performance (mAP) with and without fine tuninu on human-verified annotations.
Data Scale
COCO
LVIS
Data Cleaning COCO LVIS
Human Data COCO LVIS
IM
67.8
54.0
61.8
46.5
67.8
54.8
2M
67.5
55.0
67.8
54.0
70.2
55.0
5M
68.6
54.8
CUP
generate
Label 1
train
R-CUIP ASM
generate
Label 2
Human Verification
train
ASM R-CUP
generate
Label
Label 1: car
Label white
Label filpers Label Vassss Label 3: Maves
buay Label tires
spoons
front grilla Label front grille Label handle
Label 2: reflector
Label 3: reflector
Figure 8: Visualization of the data iteration process. The iteration process improves the label accuracy. We visualize three types of models: (1) Label 1: labels produced the original CLIP; (2) Label 2: labels produced by R-CLIP ASM, trained with Label 1 as input data; (3) Label 3: labels produced by R-CLIP or ASM which is further tuned with human verification data.
quality improvement in the data engine iterations. To intuitively show the data quality improvements, we show the labeling results for CLIP as well as the outputs of R-CLIP before and after the human data fine-tuning in Fig. 8. The original CLIP is unreliable for small objects. Thanks the data cleaning strategy, R-CLIP pre-trained on AS-1B data performs better in these small objects. However, may fail recognize some objects due noisy labels, e.g., labeling the tires hung by the boat "life buoy". The human data fine-tuning process can correct the pre-trained R-CLIP.
E.4 ABLATION
Table 14: Ablations on captioning tasks.
Model
RefCOCOg
COCO Caption
Flickr30K
NoCaps
zerv-shot setting
Flamingo-9B (Alayrac al., 2022)
73.9
61.5
Kosmos-2 (Peng al., 2023)
60.3
67.1
Shikra (Chen et al., 2023a)
117.5
73.9
Emu (Sun et al., 2023)
117.7
ASM
46.8
119.8
79.5
107.7
ASM w/o LoRA
42.6
45.3
123.0
81.7
111.0
setting
GRiT (Wu al., 2022)
142.0
71.6
BLIP (Li et al., 2022a)
133.3
113.2
BLIP2 (Li et al., 2023a)
145.8
121.6
InstructBLIP (Dai et al., 2023)
82.8
123.1
ASM-SFT
148.7
107.8
140.2
88.0
116.9
ASM-SFT w/o LoRA
144.9
102.4
142.3
84.9
118.7
Considering the fine-tuning strategy utilizing LoRA (Hu al., 2022) different from other meth- ods (Li et al., 2022a; 2023a; Dai et al., 2023), we supplement our results in Table 14, which includes outcomes without LoRA. Here, We report CIDEr. Note that the results under the SFT setting are fine-tuned with the COCO Caption dataset. We can observe that our model achieves powerful perfor- mance across the board in both zero-shot and finetuning settings. Besides, the results also reveal that ASM without the LoRA fine-tuning strategy also demonstrates comparable performance with the counterpart tuned by LoRA. Since the ASM tune with LoRA achieves slightly better performance on region-level captioning tasks, we this training strategy for human-in-loop iterations.
33


Reviews:
[{"review_id": "HNMZEvqVx5", "reviewer": "Reviewer_Wviv", "summary": "In this paper, the authors present a large dataset and model for panoptic visual understanding, collectively named the \"All-Seeing Project\".\nThe dataset (AS-1B) contains more than one billion region-text pairs, where the text comprises semantic tags, question-answer pairs, and captions.\nText in AS-1B entails a rich vocabulary of visual concepts, e.g. the authors state the presence of 3.5 million unique semantic tags.\nThe authors design a scalable, semi-automatic data collection engine to collect AS-1B --\ntheir pipeline is composed of several large vision models that generate region-text annotations,\nand human annotator to verify the correctness of the generated annotations.\nThe authors train All-Seeing Model (ASM) using their dataset and show strong empirical performance on several downstream vision-language tasks.", "questions": "Please see weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. AS-1B is a large dataset of region-text pairs, perhaps currently the largest of its kind.\n2. The design choice of using the same images as SA-1B is excellent to mitigate ethical risks regarding copyright and privacy of users,\n   as these images are meticulously verified by Meta AI, and released with a permissible license for research.\n3. The proposed model (ASM) achieves strong empirical performance on several region-level visual understanding tasks.\n4. The design of ASM allows it to be \"composable\" in a larger system that may include localization models like SAM.", "weaknesses": "**Note:**\nI wrote my review a few days ago, but it comes late due to my oversight -- apologies for the delay.\nIn the interest of time, I have updated my initial asessment to incorporate the authors' rebuttal.\nMy outstanding concerns do not require the authors to run additional experiments.\n\nI concur with the authors that the main contribution of this paper is a large-scale dataset.\nAS-1B is currently the largest dataset of its kind (to the best of my knowledge)\nand its availability will open new avenues for empirical study in the research community.\nI view the model (ASM) as an important, however secondary contribution --\nit serves as a baseline to provide a strong guarantee of data quality,\nthat training with AS-1B can yield strong empirical improvements across several task benchmarks.\n\nMy remaining concerns listed below are all geared towards ensuring a stronger guarantee of data quality\nand repsonsible release of the dataset.\n\n1. **Collection engine prone to hallucinations:**\nThe authors use large language models (LLMs) in the \"imaginator\" and \"splitter\" to produce semantic tags that are NOT conditioned on the visual content.\nThe imaginator produces plausible semantic tags that are _likely_ to occur, but not guaranteed to occur in images.\nIs there is a way to quantify the amount of hallucination by checking the response of human annotators?\nI suggest the authors to provide ample of qualitative examples in the paper showing the initial pool of semantic tags *before* they are assigned to the region proposals.\n\n2. **Redundant caption annotations:**\nThe detailed caption of a region is produced by paraphrasing three question-answer pairs.\nBased on the limited examples in the paper, the captions sound like a \"dry\" paraphrasing of the question-answer pairs (understandably so).\nI wonder if having such redundancy contributes to the uniqueness of AS-1B, or simply adds redundancy and increases the size of dataset.\n\n3. **Consider adding a datasheet:**\nThe authors should consider adding a datasheet () or a similar supplemental material outlining the characteristics of AS-1B.\nFor example, the Segment Anything paper includes a datasheet for SA-1B.\nDatasheets serve as as a medium of communication between the authors (creators of the dataset) and future works (users of the dataset).\nMany papers published in NeurIPS datasets track have datasheet templates which can be suitable in the ICLR format,\ne.g. some image-text datasets like [LAION-5B](https://arxiv.org/abs/2210.08402) and [RedCaps](https://arxiv.org/abs/2111.11431).\n\n4. **Needs a train/val split:**\nI agree with the other reviewers' assessment that the authors should consider defining a train/val split for AS-1B.\nIf the authors do not define a split, different future works will regardlessly split AS-1B in ad-hoc ways and lead to inconsistencies.\nI suggest the authors to \"hold out\" ~1% data as a validation set for the sake of consistency.\n\n6. **Missing References:**\nThe proposed All-Seeing Model is trained with both, a generative and contrastive loss, to facilitate its use for generative (e.g. captioning) and discriminative (e.g. object recognition) tasks.\nDue to the similarity in its architectural design, I believe the authors should cite a few prior works in their discussion to provide a broader context for the reader:\n\n    - [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://arxiv.org/abs/2205.01917) - trains with both objects as ASM.\n    - [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://arxiv.org/abs/2111.07991) - repurposes *any* image backbone to a contrastive image-text model.\n    - [Image Captioners Are Scalable Vision Learners Too](https://arxiv.org/abs/2306.07915) - trains with generative objective first, then uses LiT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jc2nId5M3e", "reviewer": "Reviewer_WmLf", "summary": "The papers presents a large-scale dataset collected using a semi-automatic data engine for open-world panoptic visual understanding. The dataset consists of 1 Billion + region annotations spanning semantic tags (3.5 Million +), question-answer pairs (3.3 billion) as well as detailed captions (1.2 billion). The paper proposes a VLLM called All-Seeing Model (ASM) trained on this dataset consisting of a location aware image tokeniser and a LLM based decoder. ASM achieves promising results on image and region-level captioning and recognition tasks.", "questions": "See weaknesses section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper does a great job explaining the details related to the dataset. The appendix contains several details that help understand the semi-automatic approach mentioned in the paper better (percentage of annotations from LLMs/VLLMs, accuracy of automatic annotations.\n- The paper presents a fairly exhaustive benchmark (VQA, OK-VQA in supplementary). The paper also attempts to evaluate the model’s performance on region-based tasks like region-based visual question answering by conducting human studies.\n- The paper also presents a good summary of many factors that are responsible for improving the performance of the model such as the role of data-engineering (D.3)", "weaknesses": "- Phrase Grounding Evaluation: The proposed method also missed an opportunity to leverage the dataset to learn the ability to ground language into the image by generating the bounding boxes corresponding to the text. I would have liked to see the models performance on the phrase grounding task on Flickr30K Entities.\n- I think the paper misrepresents the state of the art in the community. For instance, the claim that current systems “are primarily focused on understanding images as a whole, rather than recognizing and comprehending individual instances within the scene” seems ungrounded, and several state of the art systems (e.g, Unified IO, including more recent ones like KOSMOS-2) show a fairly good understanding of the image on benchmarks that test this visual grounding like referring expressions, and phrase groundings. Since the authors compare and cite KOSMOS-2, for completeness the authors should also put the proposed dataset (AS-1B) in perspective of other comparable datasets such as GRiT which consists of region annotations for 90 million images.\n- The paper uses LORA to fine-tune the LLM on various tasks COCO, VQA, etc which is different from other methods (BLIP, InstructBLIP, etc) that the method have compared against. This makes the evaluation unfair because these evaluations heavily penalise the peculiarities of the the evaluation protocol (one-word answers as opposed to natural language generation). Since methods like BLIP use a frozen LLM, it’s much harder for them to conform to the expected style of answers as opposed to the ASM which adapts the LLM using LORA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FphlXDXcsJ", "reviewer": "Reviewer_5eEK", "summary": "This paper proposes a large dataset and model for detailed VQA and captions about image regions. The data engine involves the use of a combination of localization models, contrastive vision language models, and other LLMs/VLLMs, as well as humans in the loop to verify the outputs. The resulting dataset has 1.2B regions covering a wide range of 3.5M concepts. The authors also propose a model to ingest this data and handle both discriminative/generative tasks.", "questions": "See weaknesses. My main concerns are with the data quality and evaluation benchmarks based on this dataset/model.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important space of problems that existing vision and language foundational models are focused on image-level understanding, and there's a clear need to build region-level vision and language foundational model. The proposed dataset is based on the recent SA-1B dataset and extend it with semantic tags/QA pairs, and detailed caption, all of which can be useful to the community.", "weaknesses": "1. It'd be helpful to get some analysis on the quality of the final data after human verification. Appendix B.3 shows the accuracy of automatic annotation is around 50-60%. How much of the error is fixed by human verification, and how much is still there? \n\n2. I'm wondering if it'd be better to set apart a high-quality split for region-level validation/testing of captioning. Existing dataset don't seem to serve this purpose very well e.g. RefCOCOg is not intended for region-level captioning. Visual genome is not a common benchmark for captioning evaluation either.\n\n3. Image-level captioning in Table 3 is helpful, but not the focus of this work in my view. To make this more complete, it might be good to add COCO captions too.\n\n4. To make a strong claim on region level understanding, I feel that the model should be able to predict regions from images rather than accepting regions as input. For example, in Table 4, it'd be more useful to have a simple ASM model that can predict regions without groundtruth box inputs.\n\n5. It'd be great to have a region-level VQA benchmark as well since the dataset includes VQA. I see the image-level VQA results in Table 10, but that does not seem to capture the uniqueness of this data.\n\n6. It'd be helpful to have some analysis on bias/fairness considerations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)