Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation
Authors: Jiangyong Huang
William Yicheng Zhu
Baoxiong Jia
Zan Wang
Xiaojian Ma
Qing Li
Siyuan Huang
Venue: Submitted to ICLR 2023 (2023)
Keywords: general-purpose vision
benchmark
visual representation

Abstract: Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts at general vision models are limited to a narrow range of tasks and offer no overarching framework to perform visual tasks holistically. We present a new comprehensive benchmark, General-purpose Visual Understanding Evaluation (G-VUE), covering the full spectrum of visual cognitive abilities with four disjoint functional domains — Perceive, Ground, Reason, and Act. The four domains are embodied in 11 carefully curated tasks, from 3D reconstruction to visual reasoning and manipulation. Along with the benchmark, we provide a general encoder-decoder framework for the tasks in G-VUE, to accommodate arbitrary visual representations on all 11 tasks. With our benchmark and framework, we evaluate 7 typical visual representations and observe that (1) transformer and more data empirically lead to more general-purpose, (2) language plays a significant role in learning versatile visual representation, and (3) correlations indicate a subtle constituent among tasks despite the distinctions, which could be evidence of general-purpose. We argue that instead of pursuing general-purpose vision models by end-to-end multi-task training, it is more reasonable to evaluate and investigate representations, which helps digest emerging pre-trained vision models and hopefully shed light on general intelligence.

Under review as a conference paper at ICLR 2023
Perceive, Ground, Reason, and Act: A BENCHMARK FOR GENERAL-PURPOSE VISUAL REPRESENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts create a general vision model are limited in the scope of assessed tasks and offer no overarching framework to perform them holistically. We present a new comprehensive bench- mark, -ureose Visual Understanding Evaluation (G-VUE), covering the full spectrum of visual cognitive abilities with four functional domains - Perceive, Ground, Reason, and Act. The four domains embodied in 11 carefully curated tasks, from 3D reconstruction to visual reasoning and visual manipulation. Along with the benchmark, we provide general encoder-decoder framework to allow for the evaluation of arbitrary visual representation on all 11 tasks. We evaluate various pre-trained visual representations with our framework and observe that visual backbone generally outperforms CNN-based back- bone on G-VUE, (2) visual representations from vision- superior to these with vision-only pre-training across visual tasks. With G-VUE, we provide holistic evaluation standard to motivate research toward building general-purpose visual systems via obtaining better, more general general-purpose purpose visual representation.
(I) Transformer-basee
pre-training are
INTRODUCTION
The long-term goal of machine vision (Mar,, 1982; Barrow and Tenenbaum, 1981; Ikeuchi and Hebert, 1996; Marr, 2010) is to build general-purpae vision system that can perceive, understand, and react to visual inputs from unconstrained environments. The term generul-purpose can be best understood through observing our own visual systems, which support various complex higher-order visual tasks, from edge detection and object recognition to visual navigation and manipulation, all while rooted in common brain region that produces core visual representations.
On the other hand, even though the field of computer vision has been thriving with models that solve complex tasks increasingly well over the last decade (He et al., 2016; 2017; Ronneberger et al., 2015; Dosovitskiy et al., 2020), there is still considerable gap between vision models and the aforementioned human visual systems. In particular, current vision models are mostly task-specific and often contain specialized components or architectures designed for a specific setting. They are also hindered by diverse input-output formats that vary from task to task. Recent works in self-supervised learning (Oord et al., 2018; He et al., 2020; Chen et al., 2020; Grill et al., 2020; He al., 2021; Huang et al., 2021; Ma al., 2022) explore more general visual representations by learning from enormously-sizen visual domains. Advances in vision-languaa 2020; Hu and Singh. 2021; Cho al., 2021; Gupta et al., 2021; Kamath et al., 2021; Wang et al., 2022) also seek to unify the vision tasks with task-agnostic model architecture. However, these models are evaluated solely on traditional tasks like classification, detection, or vision-language understanding. It is still unclear whether current computer vision models are general-purpon like the human visual systems, as there is no overarching benchmark that covers all visual tasks holistically, from lowelevel perception to high-level reasoning and acting.
modeling (Lu et al.,
To facilitate the research toward the general- purpose vision, we present General- purpose Visual Understanding Evaluation (G-VUE) benchmark. We carefully curate 11 tasks from four functional domains that visual systems should support PPecceive Ground, Reason, and Acr- ordered by their cognitive complexity. These four domains cover the full spectrum of human visual tasks and

---

Under review as a conference paper at ICLR 2023
extend far beyond the standard set of visual tasks like detection or V-L grounding discussed in previous models or benchmarks. Specifically, Perceive tests a model's geometry understanding. Ground examines a model's acquisition of visual semantics. Reason probes model's capacity for logical deduction and common sense reasoning. Act investigates a model's ability for planning and decision-makinn by learning visual policies. Fig. illustrates benchmark and Tab. shows the task details. Principles and details about task and dataset selection are presented in Sec. 3 and Appendix B. G-VUE provides test-bed for a general-purposs vision system and allows for fair comparisons between different visual representations over full spectrum of visual tasks.
The diverse visual tasks in G-VUE require the model to navigate in a wide range of input and output formats, including image, text, bounding box, dense map, and policy. There is no exist- ing model that can accomplish all these tasks. It is quite different from the Natural Language Processing (NLP) field, where most tasks (Wang al., 2018; 2019) can be accomplished using the same sequence-to-se -ee-nnce-----suunneeee model based on RNN (Mikolov et al., 2010), LSTM (Hochreiter and Schmidhuber, 1997), or Transformer (Vaswani al., 2017). Therefore, we carefully equip the visual tasks in G-VUE with a general encoder- decoder framework mitigate the gaps among various visual tasks and accommodate arbitrary visual encoder, which makes it possible uate any visual representation Sec. further describes the principles and details of framework design. Notably, although such framework facilitates handling tasks across various domains, it is not designed for massive multi- malti-task training but for a flexible and fair evaluation of various existing visual representations G-VUE.
RASKON
R
Percewn
Visual Representation
A
Hocipttor
Cane's Nea
V.d Bestiall Samardi Saginar
Figure An overview of G-VUE, key idea of G- VUE is to evaluate visual representation in a general- purpose standard, where P (Perceive), G (Ground), R (Reason), and A (Act) represent four functional domains for general vision system.
eval- these 11 tasks.
To understand the challenges posed by G-VUE and the limitations current models, we evaluate the most representative visual representations obtained with various architectures (e.g.. ResNet vs. ViT) and pre-training mechanisms (e.g.. supervised vs. self-supervised). The experimental results show that (1) Transformer- architectures generally beat CNN-based architectures, and (2) visual representations pre-trained on massive image-text data (e.g., g., CLIP) significantly outperform traditional ImageNet pre-training across visual tasks.
We make the following contributions: (I) we present a general-purpose vision benchmark G-V that consists of 11 visual tasks covering diverse visual skills: (2) we introduce an encoder-decoder framework for G-V that allows for the evaluation of arbitrary visual representations: (3) we examine the performance of state-of-the-as visual representations on G- G-V and present detailed analyses. Additionally, we construct an evaluation platform and leaderboard for G-VUE that will be made publicly available upon acceptance.
2 RELATED WORK
2.1 VISUAL PRE-TRAINING
Large-scall visual pre-training has been viewed as potential way to obtain general visual represen- tation that adapts well to any downstream tasks. There has been successful attempts at learning robust visual features from discriminative (Oord et al., 2018: He al., 2020; Chen et al., 2020; Grill et al., 2020; Chen et al., 2021) and generative (He et al., )021) self-supervised learning and vision- vision-language alignment learning (Radford et al., 2021). These models use various visual backbones to learn from diverse visual domains, including images (Deng al., 2009), egocentric videos (Grauman et al., 2022), and image-text pairs (Radford et al., 2021). Their differences in learning mechanism, visual backbone, and source learning domain affect their transferability to downstream tasks. Although these models have been extensively evaluated in existing tasks such as image classification and object detection, remains unclear how they be used to achieve broader domain of visual tasks toward
2

---

Under review as a conference paper at ICLR 2023
general-purpose purpose vision. One essential goal of G-VUE is to evaluate these models and provide tools for seeking more generalizable pre-trained models.
general
2.2 GENERAL-PURPOST VISION
Models The pursuit of general- purpose vision emerges from the early stage of computer vision. In the last 70s, Marr (2010) introduced a fixed layered representation and unified visual processing for general visual tasks. Despite suffering from its inflexibility to adapt to visual tasks, it provides a paradigm for general visual perception. Recently, progresses in deep learning drives the vision model towards unified architectures (Lu et al., 2020; Hu and Singh, 2021; Cho et al., 2021; Gupta et al., 2021; Kamath et al., 2021; Wang et al., 2022; Lu et al., 2022) that are agnostic to the input and output formats. Among them, VL-T5 (Cho al., 2021) and GPV-1 (Gupta al., 2021) unify the visual tasks through dision-languse tokenized representation to model various vision-language tasks jointly. Although these methods tackle a larger set of visual tasks, they are still limited to the conventional vision-language domain, lacking the general capabilities for low-level geometric understanding and high-level reasoning and acting.
models, OFA (Wang et al., 2022) and Unified-IO IO (Lu et al., 2022) utilize
The general understanding benchmarks have become standard in NLP (Wang et al., 2018; 2019) for testing pre-trained language models. However, few benchmarks are designed for general vision understanding, mainly due to the enormous complexity of diverse visual tasks. Among these benchmarks, Zhai et al. (2019) and Shao et al. (2021) focus on the transferability of visual representation, which perform domain adaptation by exposing pre-trained visual models to a small fraction of tuning data. GRIT (Gupta et al., 2022) emphasizes both generality and robustness of visual representation in visual perception and grounding tasks. Parisi et al. (2022) focuses on the evaluation of pre-trained visual representation on policy learning and explores how ingredients of pre- training affect the learning of control policy. While each of the previous benchmark only addresses narrow aspect of human visual function, G-VUE provides holistic coverage of visual tasks spanning the functional domains of Perceive, Ground, Reason, and Act. With the equipped general encoder-decoder framework, G-VUE serves as a platform for evaluating any arbitrary visual encoder and representation in the context of general-purpose vision.
Benchmarks
THE G- G-E BENCHMARK
Mirroring human cognitive abilities, our benchmark comprised of tasks from four functional domains that a general vision system should support: Perceive, Ground, Reason, and Act. Perceive characterizes the basic ability of understanding geometry from raw visual input. Ground connects geometry with concepts and semantics. Reason operates on grounded objects and performs deductions. And finally, Acf makes decisions and plans actions out of extensive reasoning. For each domain, we select visual tasks and datasets that representative, challenging, from the real-world domain, and available to the public. The full list of the tasks is shown in Tab. 1. We provide an in-depth description in this section and Appendix B for task selection, dataset selection, and rationales.
3.1 PERCEIVE
Visual perception underlies human's 's subconseious mental approximation of geometric relationships in the physical world (Clements, 2004). In the domain of Perceive, we start with the evaluation of the sense distance and space through Depth estimation. We move onto egocentric pose understanding with Camera pose estimation. Lastly, we examine the ability for detailed, shape-based geometric understanding with Single-view 3D reconstruction.
Depth estimation. Monocular depth estimation is the task of estimating the egocentric distance to objects on a single 2D image. We adopt NYUv2 (Silberman et al., 2012), a common practice for evaluating monocular depth estimation. We make use of the updated version proposed in Lee et al. (2019) as the training set to provide a sufficient amount of data samples. The model's 's performance is evaluated by relative and absolute errors.
Camera pose estimation. Learning- based absolute camera pose estimation is an instance-level task that requires the model infer parameters about the point of view from scene's geometry.
3

---

Under review as a conference paper at ICLR 2023
Table 1: The full list of tasks in G- GUE We categorize tasks according their functional domains and provide details on the selected datasets, train/val/test splits, input/output modalities, and evaluation metrics.
Task
Dataset
/Taiin/ / [Vall /esst
Text
Output
Metries
Perceive
Depth Estimation
NYUv2
24k / 0.6k
N
Depth Map
d<1.25, AbsRel, RMSE
Camera Pose Estimation
CL & 7 Scenes
(3.8k, 26k) / (L.1K, 17k)
N
Camera Pose
Mean Trans, & Orient. Ertor 3D IoU
3D Reconstruction
StupeNeiCore
30k 7
N
Volumetric SDF
Ground
Image-Texx Retrieval
Flickr30k
291 1.0k/ 1.0K
Matching Score
Recall 1,5,10
Phrase Grounding
RefCOCO
42% / 3.8k (2.0k, 1.8k)
Y
Bhox
Acc@00.5
Semantic Segmentation
ADE20K
20k / 2.0k/
N
Segmentation Map
mloU
Reason
Question Answering
GQA
943k / 132K 12.5k
Y
Choice
Accurncy
Commonsense Reasoning
213k 26.5k/
Accuracy
Abstract Reasoning
Bongard-HO1
23k /7k/ 14k
Binary Label
Accuracy
Act
Navigation
R2R
14k / (1.06 2.3k)
Y
Next Move
SPL
Manipulation
Ravens
0.8k / 0.08k / 0.8k
Y Pick & Place
Success Score
Following the general setting in previous works (Kendall et al., 2015), use both outdoor urban localization dataset, Cambridge Landmarks (Kendall et al., 2015) and an indoor scenes dataset, 7 Scenes (Shotton et al., 2013) as our data source. We use the training and testing splits provided by Kendall et al. (2015), and report the mean errors of position (meters) and orientation (degrees) respectively over the scenes as our metric.
Single-view 3D reconstruction. The task of single-view 3D reconstruction requires the model to predict a 3D voxel shape from a single 2D image. We evaluate on the synthetic images (Choy et al., 2016) of 3D objects from ShapeNetCore (Chang et al., 2015) and use the train/test split provided by Xu et al. (2019). We use 3D loU in 643 resolution as the metric to evaluate the performance.
3.2 GROUND
Visual grounding is defined as the process of connecting and aligning perceptual visual inputs with natural language. This alignment comes at different granularities, which form the basis of task progression in the Ground domain. We start with image-level content matching in Image-text retrieval, and proceed onto instance-level content matching in Phrase grounding. We further consider the composition of both granularities, i.e., scene-level instance property grounding, in Semantic segmentation.
Image-text retrieval. Image-text retrieval requires the model match the image-text pairs given a sample from one modality. We select Flickr30k (Young et al., 2014) for this task. We follow the dataset split in Karpathy and Li (2015), with 29k/Ik/Ik images for training, validation, and test sets respectively. We use the recall of finding the correct caption in top-1 (R I), top-5 (R@ (),5, and top-10 (R @ 10) predictions as the metric for retrieval.
Phrase grounding. In phrase grounding, model is asked to predict a bounding box that crops out an object referred to by a natural language phrase. We select RefCOCO (Yu al., 2016) as our source. It is dataset built upon MSCOCO (Lin et al., 2014) with objects and bounding boxes labeled for each image. We evaluate on the benchmarking splits of person reference (testA) and non-person reference (testB) in RefCOCO, in addition to the validation set.
Semantic segmentation. Semantic segmentation requires the model to ground semantic labels to pixels in each image and make dense predictions. For this task, we select the curated version of ADE20K (Zhou et al., 2017) and the corresponding train/test split from the MIT Scene Segmentation Benchmark (Zhou al., 2017). It contains 150 categories of stuff (e.g.. sky) and discrete objects (e.g., car) with their corresponding semantic masks on each image. We use the mean Intersection over Union (mloU) between predicted and ground truth segmentation mask as the evaluation metric.
4

---

Under review as a conference paper at ICLR 2023
3.3 REASON
Visual reasoning characterizes the process of collecting grounded facts from images and making a complex deduction, e.g., answering questions. In Reason tasks, we first visit Visual question answering, a seminal task that comprehensively examines whether a learner can jointly reason with images and text. We then advance to visual Commonsense reasoning, where model also needs to acquire physical and societal commonsense through learning. Our final task examines a not often discussed niche in visual tasks: Abstract and few-shot reasoning.
Visual question answering. In a VQA problem, the model needs to generate the answer to a question (a single word compound word) after viewing an image. We select GQA (Hudson and Manning, 2019) for this task as it requires challenging relational and multi-step reasoning. By default, GQA provides image region features extracted with Faster-RCNN and ResNet-50. We instead use grid features output by the backbone directly without object detection. GQA also offers rich metrics for evaluation, but only use the overall accuracy in our benchmark to ensure consistency with other tasks. Accuracy on the test-dev set is reported.
Commonsense reasoning. This task aims at answering multiple-choice questions out of scenarios that require commonsense knowledge. To crack these puzzles, a learner needs to understand human states, activities, social norms, and other commonsense knowledge. We select VCR (Zellers et al., 2019) dataset to evaluate commonsense reasoning with images. In this dataset, we follow the data preprocessing pipeline in Zellers et al. (2021), referring to entities by highlighting regions with pre-defined colors. For simplicity, only incorporate the question answering (Q + A) subtask and report the overall accuracy on the validation set.
Abstract and few-shot reasoning. We employ Bongard- -OoI (Jiang al., 2022) proxy to evaluate the ability of abstract and few-shot reasoning. The task can be briefly described as follows: given small of positive and negative images that either depict don't 't depict human-object interaction (HOI) concept, the objective to determine whether the additionally provided query image depicts that HOI concept. Bongard-HOF offers four val and test sets, respectively, which comprise seen/unseen objects and seen/unseen actions, allowing systematic evaluation of the model's generalization capability. We follow the evaluation protocol introduced in Jiang et al. (2022) and evaluate the overall accuracy on the four test sets.
3.4
ACT
Act is the most complex function vision systems could support. We consider two aspects of acting: navigating in an environment and manipulating objects. Both tasks require extensive reasoning about the environment and planning accordingly. Specifically, Navigation proposes the challenge of identifying one's location, understanding their possible movements, and directing to the target location from an egocentric perspective. Manipulation, the other hand, focuses learning from allocentric-viet observations for planning with granular visual-mon motor skills.
Navigation. We adopt the Matterport3D Simulator (MatterSim) (Anderson et al., 2018a) and its associated Room to Room (R2R) dataset as the setting for vision-language navigation (VLN). This task requires the agent to follow natural language instructions and traverse virtual house on mesh formed by viewpoint locations, each with panoramic view of the surroundings. Following previous works, we evaluate the performance on validation subsets and use SPL (Anderson et al., 2018b) in the unseen environment as the primary metric.
Manipulation. We follow CLIPort (Shridhar et al., 2022) and adopt their environment for the manipulation task. This task requires the agent to perform compositional actions (e.g., pick and place) based on visual observations and language instructions. As in Shridhar et al. (2022), visual modules make affordance predictions to guide the subsequent action module where and how to manipulate. subtasks
We collect 8 subtasks with seen/unseen splits out of the 10 initial language- sanguage-goal-con proposed by Shridhar et al. (2022) and perform affordance learning seen split of subtasks jointly, with 100 demonstrations for each task. We adopt the success scores used in Zeng et al. (2020) and report average scores on the unseen subsets of subtasks for evaluation.
5

---

Under review as a conference paper at ICLR 2023
4 FRAMEWORK
Due to the diverse nature of tasks in G-VUE, evaluating visual representations could be challenging. To this end, we propose an encoder-decoder framework for the visual tasks, making possible to adapt arbitrary visual representations to all 11 tasks.
An overview of our framework can be found Fig. 3. Firstly, the visual representation is viewed as the output of an image encoder, e.g., ResNet-50. For tasks that also incorporate textual input, a text encoder is used to produce appropriate language embedding. Specifically, we employ RoBERTa (Liu et al., 2019) for language-involved tasks. Then the visual representation (and possibly the language embedding) will be sent to a task decoder. Since the tasks G-VUE require various formats of output, we design different decoders for each class of output format. We briefly summarize the variants of our decoder as follows. More details are deferred to the Appendix C.
Label decoder. This decoder composed of several Transformer blocks and consumes tokenized visual representation (possibly concatenated with text embedding). We insert an extra [CLS] token for prediction. works with most of the classification and regression tasks in G-VUE.
Dense decoder. This decoder adopts the Segformer head (Xie et al., 2021), a light-weight module that maps tokenized visual representation to dense output. We use it in dense prediction tasks. 3D decoder. This decoder resembles the pipeline in Mittal et al. (2022), where Transformer- based decoder maps tokenized visual representation to a joint distribution over shapes and finally produces volumetric SDF via a pre-trained VQ-VAE. It only invoked in the 3D reconstruction task. Navigation decoder. This decoder follows Hong et al. (2021), which maps visual representations and instruction embeddings to movement actions in a recurrent manner.
Manipulation decoder. We employ the design in Shridhar et al. (2022) and map the visual representation and instruction embeddings to an affordance map, which then guides a model-based planner to draw the next manipulation action.
Albeit our efforts to make the encoder-decoder framework more generic and powerful, we do ac- knowledge that many task-specific decoders could generate better results given the same visual representation. However, we would like to highlight that our design principle is to balance per- formance and the amount of inductive bias, as the purpose of G-VUE is provide generic and accessible platform for evaluating penrroe- visual representation. Architectures with too many complex and task-specific designs are not consistent with the design principle and long-term goal of G-VUE. More discussions can be found in the Appendix.
5 EXPERIMENT
5.1 VISUAL REPRESENTATIONS FOR EVALUATION
We analyze three factors that may affect the quality of visual representation: architecture, pre-training mechanism, and source data. In our experiments, we adopt two architectures commonly used in large-scale pretraining: ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020). On top of these backbones, we consider variations of pre-training mechanisms including the extent of supervision (e.g., supervised vs. self-super) and learning objectives (e.g., discriminative (He al., 2020) vs. generative (He et al., 2021)). For source data, we consider ImageNet (Deng et al., 2009), large-scale image-caption pairs (Radford et al., 2021), 1), and Ego4D videos (Grauman et al., 2022). To fully explore the effect of these factors, select 7 representative visual representations shown in Tab. 2.
During training, we keep these representations fixed and tune task-specific decoders for each task. Notably, we mainly chose grid features in our evaluated visual representations, it is more generic, flexible for adaptation, and efficient compared with object-centric features (Jiang et al., 2020; Kim et al., 2021). Recent studies (Carion al., 2020; Caron et al., 2021; Ma et al., 2022) further shows emerging object-centric features in grid feature learning, which reveals their potential in tasks (e.g. VQA) where object-centric features commonly adopted.

---

Under review as a conference paper at ICLR 2023
Table 2: An overview of evaluated visual representations.
Representation
Architecture
Pre-truining mechanism
Data
RN-IN
ResNet-50
Supervised classification
ImageNet
RN-MoCo
ResNet-50
Self-supervised Contrastive Learning
ImageNet
RN-CLIP
ResNet-50
Vision-language Contrastive Learning
Weblmage
RN-Ego
ResNet-50
Vision- Vision-language Contrastive Learning
Ego4T
ViT-32-CLIP
ViT-B/32
Vision-language Contrastive Learning
WeblmageToxxt
ViT-16-CLIP
ViT-B/16
Vision-language Contrastive Learning
WebImage
ViT-16-MAE
VIT-B/16
Sulf-supervised Masked Image Modeling
ImageNet
Table Quantitative results visual representations on G-VUE, For space considerations, we use the abbrevia- tion of words for identifying tasks (e.g. "Cam. Pose." for camera pose estimation, "I-T Retr." for image-to-text retrieval, "Phr. Grad." for phrase grounding. "Sem. Seg." for semantic segmentation, "Com. Res" for common reasoning, "Abs. Res." for abstract reasoning, "Nav." for navigation and "Manip." for manipulation). Summary scores are attached at bottom, which only consider test-dev subset score on VQA task, subset for score on abstract reasoning task, and unseen SPL for score on navigation task. We highlight the best results on each task both in bold and with underline.
Representation
Task
RN-IN RN-MoCo
RN-CLIP
RN-Ego
VIT-32-CLIP VIT-I6-LLIP VAT-I6-MEE
Perceive
test d<c1.2
0.6394
0.6400
0.6523
0.4548
0,7363
0.7608
0.7699
Depth Est.
test AbsRel
0.2190
0,2184
0.2097
0.3409
0,1699
0.1606
0.1554
RMSE J.
0.7292
0.7284
0.6987
1.0302
0.5888
0.5535
0.5486
Trans.(CL)
2.909
1,972
2.446
3.045
2.160
2.196
2.177
Orient
7.298
6.814
7,000
5,974
6.020
4,911
Cam. Pose
Trans.(78)
0.281
0.241
0.279
0.266
0.290
0.263
0.271
Orient.(78) |.
10.299
8.797
9,502
9.058
9.682
9.752
6.699
3D Recon.
test mloU
39.70
41.98
42,00
37.66
40.63
39.43
43.95
Ground
test R
10.4
7.1
18.5
1.5
15.3
23.0
5.8
I-T Retr.
test R R@8 1
27.2
22.5
42.2
4,7
38.6
50,9
20.5
test R
38.3
33.8
55.2
8.4
51.2
64.2
30.9
Aco@0.5
48.57
54.40
63.45
36.5
67.99
64.55
65.18
Grnd.
testA Acc 00,5 f
54.48
70.63
41.92
75.70
70.73
67.59
testB Ac.@0.5
43.87
50.88
55.86
31.38
61.16
57.24
62.10
Sem, Seg.
val mloti
18.83
18.05
22,32
5,00
29.14
35.86
26.25
Reason
val Acc. ?
52.79
50.20
56.09
45.56
$9.41
59.18
54.82
VQA
dest-dev Acc. f
47.54
45.02
50.01
41,20
$1.88
51.65
48.50
Com, Res.
val Acc. 1
52.61
50.98
57.30
52.79
61.94
64.11
60.76
val Acc.
59.85
62.38
64.54
55.72
63.24
65.65
62.12
Abs. Res.
62.14
64.63
67.32
56.09
65.55
69.43
62.85
Act
Unseen Succ. t
48.62
48.49
49.00
43.89
49.21
47.13
49,55
Unseen SPL
42.61
43.49
43.49
37.88
43,99
43.60
43.72
Manip.
Unseen Score
32.19
36.94
37.47
27.78
28.80
40.80
37.98
Summary Score
40.23
41.24
44.96
32.86
45.44
48.61
44.62
5.2 RESULTS AND ANALYSES
We present quantitative experiment results in Tab. 3. In addition to metrics each task, we provide summary score (the bottom line of Tab. 3) as an overall measure across all tasks by converting all metrics into percentages and taking average. We summarize major findings follows.
Architecture. general, ViT outperforms ResNet on most visual tasks. This confirms similar observations made by few recent works (Liu et al., 2021; Wang et al., 2021; Shen al., 2021; Ma
7

---

Under review as a conference paper at ICLR 2023
et al., 2022). Specifically, we find ViT-32-CLID and ViT-16-CLI significantly outperform RN-CLIP on dense prediction tasks, i.e, depth estimation and semantic segmentation. This may indicate Transformer architecture is endowed with better compatibility on fine-grained discriminative tasks.
Pre-training mechanism. We compare three pre-training mechanisms: supervised learning, self- supervised learning, and vision-language contrastive learning. Supervised learning on ImageNet was once the most common practice for producing general visual representations, but it is recently challenged by the simple yet effective self-supervised learning (He et al., 2020; Chen et al., 2020). We observe similar results on G-VUE Representation from supervised pre-training (RN-IN) is only better than self-supereised one (RN-MoCo) in handful of tasks that require semantics (e.g.. image-text retrieval) while significantly inferior to self- sepe-vuue one in tasks highly dependent on geometric pereeption (e.g., camera pose estimation).
Vision-language contrastive learning is another competitive pre-training scheme equipped with semantics alignment. Due to the contribution of language, visual representation obtained in this way shows considerably stronger generalizability regardless of the lack of direct supervision (Radford et al., 2021). This is demonstrated by the observation that CLIP visual representation dominates most V-L tasks, and achieves the highest summary score.
Source training data. The experimental results also indicate that, on visual tasks that require semantics, especially language alignment, the gap between representations trained from supervised learning and self-supervised learning on ImageNet could be less significant than the gap with CLIP visual representations. We hypothesize that the cause lies in the source training data. It appears that ImageNet is already insufficient to support the learning of a ceneral-purposs visual representation that can accommodate the broad spectrum of tasks in G-VUE. On the other hand, the success of CLIP visual represeniations in G-VUE supports the recent paradigm shift to multi-modal pre-training, which leverages massive source data (e.g.. WebImageTText and facilitates better representation with grounded concept learning. Such large-scale pre- pre-training provides a promising path towards general-purposs visual representation and could potentially help to solve G-VUE As side note, we also notice that visual representations learned from egocentric video clips (RN-Fgo) fail to achieve comparable performances on G- G-VUE, The drastic domain shift from the hand-centered Ego4D data to more general natural images in G-VUE might be the primary cause for such performance drop.
Language representation. To make the evaluation of visual representations fair, we provide them with an identical language representation, i.e., RoBERTa as language encoder. Nevertheless, using language representation pre-trained on the text-only corpus to collaborate with visual representations may cause sub-optimal performances, see Appendix D. This is similar to the shortcoming of vision- only pre-training. Check Appendix D for detailed ablation studies on the language representation.
In addition, we explore some other factors that could improve the performances of visual representa- tions on G- VUE, including fine-tuning and resolution. We summarize these studies to Appendix D.
5.3 TASK CORRELATION AND GENERALIZATION
It is believed difficult for visual representations learned from specific tasks to adapt to other do- mains. However, as our results suggest, this is not necessary the case (e.g. MAE models perform sur- prisingly well on Ground and Reason tasks with reconstruction objectives). Such findings further motivates the question: are there shared require- ments on representations over the seemingly di- vergent tasks? As there is no clear relationship between tasks, we empirically estimate task corre- lations with our experimental results. Specifically, we construct a task correlation matrix by treating the representation's performance on each task a variable. As shown in Fig. 2, we observe that most tasks show high correlations. Compared with other tasks, camera pose estimation and 3D reconstruction show fewer connections to other tasks. Nonetheless, camera pose estimation still correlates with manipulation since they both require 3D)
1.0
cam, poo_peee
54 recon
0.8
phr_omd.
6:29
0.5
52g
6:22
0.4
vor-
5.000
0U77
bongand
0.2
6.55
634
0.2
150
344
625
dapth
poase
Figure 2: Correlations of 11 tasks in G-V
8

---

Under review as a conference paper at ICLR 2023
Table 4: Evaluation of general purpose vision- vision-language models G-VUE. For simplicity, omit the tasks they cannot solve, e.g., reconstruction, navigation, and manipulation. *Note that performances on commonsense reasoning task are only for reference since models have no awareness of the referred objects from text description (which is adopted in Zellers ct al. (2021) and G- G-VUE).
Depth/ /test
I-T retrieval
Phrase grounding / Acc@0.5
VQA
Commonscnse
Model
dc1.25
AbaRel
RMSE
R@1
R@5
R@
testA
testB
test-dov Acc.
val Ace.
OFA-Ham Huge
26.1
46.3
54,9
92.04
94.03
88.44
59.92
28.19
UIO-XL
0.9439
0,0860
0.3505
33.0
53,9
62.8
86.17
86.28
85.30
36,91
32,09
geometric understanding. We also observe significant correlations in Ground and Reason domain. This result is intuitive since these tasks share similar objectives. We hope these findings could foster new insights in multi-task learning.
On the other hand, we find that visual representations may not be able to generalize across large modality gaps. RN-Ego is trained with first-person videos that have drastically different viewpoints compared to other data sources. As a result, we observe inferior performance of RN-Ego on G-VUE compared with other image-based representations. Additionally, we recognize that visual represen- tations learned with strong biases not suitable for generalizing across tasks, e.g., representation fine-tuned on depth prediction shows significant performance drops when transferring to other tasks despite the high task correlation.
5.4 EVALUATION OF GENERAL VISION-LANGUAGE MODEL
Recently there emerges several unified vision-languag model (Wang et al., 2022; Lu et al., 2022) that can perform multiple visual tasks by treating the task specification as language prompt and performing large-scale pre-training. To examine how general-purpose they are, evaluate the OFA (Wang et al., 2022) and Unified-IO (Lu et al., 2022) on G-VUE Specifically, we select their largest model variant and directly perform inference tasks they solve. Tab. 4 shows the results.
The evaluation results demonstrate the remarkable performance of recent unified models, particularly on tasks that they have been pre-trained. For example, Unified-IO achieves superior performances on depth estimation and phrase grounding, and both of OFA and Unified-IO reach the state-of-the-ar several VL tasks. We conclude that the large-scale multi-task pre-training significantly helps to elevate the performances on tasks that they have been pre-trained and enables certain level of generalization to various task prompt.
However, these models are still in lack of general capability to deal with diverse tasks in G-VUE, e.g., camera pose estimation, 3D reconstruction, abstract and few-shot learning, navigation, and manipulation. These visual tasks are substantial and form the core of human visual system, hence we advocate that the general vision model should expand broader domain, but not limited to vision-language tasks.
6 CONCLUSION
In this work, we present novel general-purpose vision benchmark G-VUE, consisting of 11 meticu- lously chosen tasks. G-VUE covers the full spectrum of visual skills over four domains: Perceive, Ground. Reason and Act. We further introduce an encoder-decoder framework that supports the evalu- ation of arbitrary visual representations on all 11 tasks. With G- GUE, we evaluate the performance of state-of-the-art visual representations pre-trained with different learning paradigms and data sources. In particular, we find that mranfooreer-bssee architectures beat CNN-based models in most visual tasks and that pre-training massive image-text pairs such CLIP achieves significantly better results than ImageNet pre-training. Such findings shed light on the path to building a unified model for ceneral--uupoon vision.
We will also spen-source an evaluation platform with a public leaderboard that can fairly and holistically evaluate different models G-VUE We hope this effort will promote the study of general- purpose representation learning and encourage the computer vision community to pursue universal, easily adaptable, and general-purpor visual models.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
David Marr. Vision: A computational investigation into the human representation and processing of visual information. 1982.
Harry G Barrow and Jay M Tenenbaum. Computational vision. Proceedings of the IEEE, 1981. 1
Katsushi Ikeuchi and Martial Hebert. Task-oriented vision. In Exploratory Vision, pages 257-277. Springer, 1996.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. 1,3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Conference on Computer Vision and Puttern Recognition (CVPR), 2016. 6
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015. 1
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929. 2020. 1,6
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03788 2018. 1,2
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020. 1, 2, 8
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057099 2020. 1, 1,2,8
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 2020. 1,2
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021. 1.
Siyuan Huang, Yichen Xie, Song-Chin Chun Zhu, and Yixin Zhu. Spatio-temporal sule-supervised representation learning for 3d point clouds. In International Conference on Computer Vision (ICCV), 2021.
Xiaojian Ma, Weili Nie, Zhiding Yu, Huaizu Jiang, Chaowei Xiao, Yuke Zhu, Song-Chun Long-Chun Zhu, and Anima Anandkumar. Relvit: Concept-guided vision transformer for visual relational reasoning. In International Conference Learning Representations (ICIR), 2022. 1, 6,
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 3
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1,
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying iision-and-languaaaa tasks via text generation. In International Conference Machine Learning (ICML., 2021. 1,3
10

---

Under review as a conference paper at ICLR 2023
Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. arXiv preprint arXiv:2104.00743, 2021. 1,3
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Modula-md detection for end-to-end multi-modal understanding. International Conference on Computer Vision (ICCV), pages 1780-1790, 2021. 1,3
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequenee learning framework. International Conference on Machine Learning (ICML), 2022. 3,9
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. 2018. 2,
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general purrpol- language understanding systems. Advances in Neural Information Processing Systems (NeurIPS), 2019. 2,
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech. Makuhari, 2010. 2
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017. 2
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. 6, 8, 19
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference Computer Vision and Pattern Recognition (CVPR), 2009. 2, 6
Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric vidco. Conference Computer Vision and Pattern Recognition (CVPR), 2022. 2, 6
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08911 2022. 3,
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019. 3
Jing Shao, Siyu Chen, Yangguang Li, Kun Wang, Zhenfei Yin, Yinan He, Jianing Teng, Qinghong Sun, Mengya Gao, Jihao Liu, et al. Intern: A new learning paradigm towards general vision. arXiv preprint Xiv: 1.00687, 2021. 3
Tanmay Gupta, Ryan Marten, Aniruddha Kembhavi, and Derek Hoiem. Grit: General robust image task benchmark. arXiv preprint arXiv:2204. /3653, 2022.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.035.0 2022. 3
11

---

Under review as a conference paper at ICLR 2023
Douglas H Clements. Geometric and spatial thinking in early childhood education. 2004. 3
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. European Conference on Computer Vision (ECCV), 2012. 3
Jin Han Lee, Myung-Ko Kyu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv: Nekiv:/9077 /907. 1a0.. 10326, 2019. 3
Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In International Conference on Computer Vision (ICCV), 2015.
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization rgh-d images. In Conference on Computer Vision and Puttern Recognition (CVPR), 2013.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European Conference on Computer Vision (ECCV), 2016. 4
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information- 3d model repository. arXiv preprint arXiv: 15t.:::5/2.00110 2015. 4
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Dism: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 2019. 4
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Anmual Meeting of the Association for Computational Linguistics (ACL), 2014. 4
Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 4
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context in referring expressions. In European Conference on Computer Vision (ECCV), 2016. 4, 16
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014. 4, 16
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 5
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems (NeurIPS), 2021. 5,
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard- hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 5
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and- Iision-and-Languae Language Navigation: Interpreting visually- grounded navigation instructions in real environments. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018a. 5
12

---

Under review as a conference paper at ICLR 2023
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv: 2018b. 5
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning. 2022. 5, 6, 17, 18
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010.14406, 2020. 5, 17
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019. 6
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems (NeurIPS), 2021. 6, 17
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. arXiv preprint arXiv:2203.09516. 2022. 6, 17, 18
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. VIn bert: A recurrent laing-a bert for navigation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 6, 17, 18,
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen. In defense of grid features for visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 6
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vidion-and-language transformer without convolu- tion or region supervision. In International Conference on Machine Learning (ICML), 2021. 19
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference Computer Vision (ICCV), 2021. 7
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In International Conference on Computer Vision (ICCV), 2021.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit lision-and-langee tasks? arXiv preprint arXiv:2107.06388 2021. 7
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 15
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
16
13

---

Under review as a conference paper at ICLR 2023
Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV), 2010. 16
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
16
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: platform for embodied ai research. 2019. 17
Ross Wightman. Pytorch image models.
pytorch- iew-i-maaaeeooe modelo ss, 2019. 17
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic
agent for vision-and-lane -asinnaand-aanguage navigation via pre-training. Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 18
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre- training for vision-language tasks. European Conference Computer Vision (ECCV), 2020. 18
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
arXiv: 2017. 18
Hao Tan and Mohit Bansal, Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019. 20
14

---

Under review as a conference paper at ICLR 2023
A OVERVIEW
To present intuitive visualization of our benchmark, we gather all the 11 tasks, as well as their inputs and outputs, and exhibit how the pipeline works in holistic layout. See Fig. 3.
Task
Input
Encoders
Decoder
Output
3D Reconstruction
3D
Depth Extimation
Dense
Segmentation
Camera Pose Extimationn
(063) J.S 953--33666
Visual Representation
Phrase Grounding
Rrimnig
Answering
Sundinn
Vinil hetee
Label
Text Encoder
Abstract Reasoning
The
Commonsense Reasoning
Anaser (ol)
Image
arallcyy Sare
Navigation
Nastgation
Manipulation
Manipulation
Figure 3: An illustration of our encoder-decoder framework. The visual representation is viewed as the output of image encoder, e.g. ResNet-50. For tasks that also incorporate textual input, text encoder is used to produce appropriate embeddings. The visual representation (and possibly the text embedding) will sent task decoders to accomplish full spectrum visual tasks.
B DESIGN PRINCIPLE AND RATIONALITY
To present our detailed task and dataset selection process, we separate four domains and organize each domain by first delivering the motivations behind task selections and then explaining dataset choices.
B. PERCEIVE
Task. Depth perception is the foundation of transition from 2D image to 3D space, which makes it essential for geometric perception. As such, this constitutes the first task the Perceive domain. Specifically, we select monocular depth estimation, as it is a common practice. Apart from depth, there are many other tasks reflecting the capability of perception, most of which are covered Zamir et al. 12) Surface normal estimation is also a critical factor about scene geometry, but it has a large overlap with depth, e.g., both are making a dense prediction. Line segment recognition another one, but is ruled out for being too low-level. We would like to augment this domain with task from a different perspective that can better incorporate scene-level properties, and thus camera pose estimation becomes choice. Inferring the camera pose from single image is erucial for egocentric downstream applications, e.g., robotics.
With depth and camera pose, able recover the scene layout from macroscopic perspective. Next, we examine detailed perception on objects. One task candidate is 6D pose estimation, which requires the model to estimate both location and orientation of objects. However, it ignores some key
15

---

Under review as a conference paper at ICLR 2023
components in perception, e.g., the appearance and shape of objects. The ultimate goal of geometric understanding is to reconstruct objects, and this can further support 3D holistic scene reconstruction. Therefore, choose single-view view 3D reconstruction as comprchensive test Perceive domain.
Dataset. We choose NYUv2 dataset for evaluation on depth estimation task. Compared with KITTI (Geiger et al., 201 2), another dataset often used for autonomous driving, NYUv2 covers more curated indoor scenes, which is consistent with tasks in Act domain. We choose the 7Scenes dataset and the Cambridge Landmarks dataset for evaluation on camera pose estimation. The former contains seven small-scale indoor scenes with only a few squared meters and the latter includes six large-scale outdoor scenes whose space ranges from 35x25m (Shop Façade) to 500x100m (Street). Regarding this choice. we consider two aspects: (1) they are widely adopted by previous works, (2) they encompass typical scenarios and challenges associated with estimating camera pose, e.g., scales, indoor scenes vs. outdoor scenes, and varying viewpoints. We select Shape dataset for evaluation on 3D) reconstruction task, because excellent materials and tools provided by the community can be utilized, including rendered images with diverse view directions, voxel grids varying resolutions etc.
the most popular choice for single-view 3D reconstruction, and
B.2 GROUND
Task. In general, we develop this domain in an incremental manner. The most accessible multi- modal data are image-text pairs, and the identification of these pairs reflects the basic capability of grounding. Naturally, we select the image-text retrieval task to evaluate this capability.
The capability to match image-text pairs cannot guarantee correct spatial grounding. A complete vision model needs to tell not only what, but also where. Therefore we must consider the evaluation on correctly located objects. The reason for selecting phrase grounding instead of a conventional detection task is that we care more about precise recognition aligning with language. And since this task is similar to text-to-image manner, implement the previous image-text retrieval task in image-to-text setting. To this end, we have instance-level grounding, expanding to object-phrase pairs.
But rectangle boundary prior in phrase grounding task limits the flexibility of grounding. Hence, We seek finer-grained grounding task, which can be achieved by tagging each pixel. We select semantic segmentation rather than referring expression segmentation, since the ability to precisely ground single object has been included in phrase grounding task and here hope to evaluate holistic scene parsing. Such a skill is complementary to object-centered grounding. With these three tasks, we cover rich diversity from several aspects, e.g., granularity, aligning with language vs. tagging, object-centered vs. scene-centered. Image captioning and human-object interaction (HOI) are also in our consideration, but the former has overlap with image-text retrieval and the latter is incorporated in Bongard-HOO.
Dataset. For I-T retrieval task, We choose Flickr30k, rather than MSCOCO (Lin al., 2014) for the purpose of reducing data overlap with phrase grounding, which adopts RefCOCO dataset. RefCOCO particularly proposed for referring expression comprehension task (also referred to as phrase grounding), incorporating cases where the model should be able to discriminate between similar instances according to subtle hints in language descriptions. RefCOCO+ (Yu et al., 2016) and RefCOCOg (Yu et al., 2016) are very similar to RefCOCO, so picking We choose ADE20K for semantic segmentation because it is more challenging compared with other counterparts e.g. PASCAL (Everingham et al., 2010), Cityscapes (Cordts et al., 2016). And ADE20K also includes rich contents, egg., stuff annotations like wall, sky elc.
of them is enough.
B.3 REASON
Task. For this domain, we consider vision tasks that require reasoning. As the evaluation of reasoning capability commonly comes in the form of question answering, we select visual question answering as the basis for task selection in this domain. Specifically, we consider question answering that requires different capabilities, starting from the simplest object grounding and relation inference, to commonsense reasoning, and finally to analogical reasoning between sets of images. We select this task based their difficulties, where relational inference requires grounding capabilities within
16

---

Under review as a conference paper at ICLR 2023
the image, common reasoning requires common knowledge outside of the image, and finally, analogical reasoning requires finding similarities between different sets of images.
Dataset. To address the aforementioned critical dimensions of visual reasoning in the real-image domain, we select GQA, VCR, and Bongard- HOOI as our benchmarking datasets. Specifically, we prefer the GQA dataset over the other commonly adopted dataset VQAv2 it incorporates object grounding and more complex compositional reasoning over relationships of objects. VCR contains question answering problems on common sense knowledge. Bongard-HOI dataset is proposed for solving the Bongard-problent and next rule out the odd one from the other that does not follow the concept. Therefore, we adopt it as our benchmark for concept abstraction and analogical reasoning.
where one needs to infer the common concept from a set of images
B.4 ACT
Task. For this last domain, we consider visual tasks in an embodied setting. Navigation asks an agent to understand its positioning in an environment using visual information and planning movements to desirably move to different place. Manipulation, on the other hand, requires the agent to achieve the desired configuration of the environment through planning the movement of environmental elements. We assume these two tasks can cover most embodied visual tasks.
Dataset. For navigation, we adopt the Matterport3D dataset and simulator to create virtual envi- romment to train and evaluate virtual agents. We choose this dataset because it covers real-world image domain with much more visual complexity, as well as a diversity clean indoor environments, and is also frequently used for vision-language navigation compared to other navigation datasets like Habitat (Savva al., 2019). For manipulation task, we follow CIIPort, leveraging the advan- tages of integrated methodology. The Ravens benchmark (Zeng et al., 2020) is proposed with rich ingredients to facilitate systematic evaluation. In addition, another important reason for our choices is that selected datasets and environments of the two tasks both provide seen and unseen splits for generalization testing.
C IMPLEMENTATION DETAILS
Visual backbones. We use Timm (Wightman, 2019) to build most of the visual backbones. Feature extractors are manually implemented if not supported. To obtain CNN-like feature pyramids from ViT backbones, we simply apply bilinear interpolation on intermediate layers as it shows little difference compared with transposed convolution. All ResNets are ResNet-50, and all ViT's are ViT-Base.
Decoders. We design three types of decoders for tasks other than Act, and their architectures are visualized in Fig. Tasks in Act, i.e. navigation and manipulation, are performed by integrated agents in simulation environments, whose structures can be found in Hong et al. (2021) and Shridhar et al. (2022). The details of the decoders are as follows:
stack of Transformer blocks. The decoder flattens the last feature map from the encoder and then regards the feature as a sequence of tokens, to which fixed 2D positional embedding is applied. We concatenate the text token sequence with the visual feature sequence if needed. We insert an extra label token as the proxy to regress desired outputs. We pass this token through MLP after Transformer layers for the final predictions. Note that we do not adopt this decoder in V-L retrieval task, where we instead model cosine similarity after processing embeddings of the two modalities through simple MLP.
Label Decoder. We implement the label decoder
Dense Decoder. We adopt the lightweight Segformer head (Xie et al., 2021) as our dense decoder handle dense prediction tasks. performs efficient spatial and channel-wise feature aggregation on the visual feature pyramid representation.
3D Decoder. Implemented for the single-view 3D reconstruction task, the 3D decoder consists of three sub-decoders. The first up-convolution decoder maps 2D visual representation into conditional distribution over the latent variables of 3D shapes. We then combine the predicted conditional distribution with autoregressive priors produced by a pre-trained Transformer- decoder (Mittal et al., 2022) to obtain a joint distribution over 3D shapes. We finally obtain the volumetric SDF by forwarding the joint distribution to the third decoder, the decoder of a pre- tre-trained Patch-wise Encoding VQ- VAE (Mittal et al., 2022). We freeze the weights except for the first sub-decoder.
17

---

Under review as a conference paper at ICLR 2023
Ourput N
Label
MLP
Visual
Feature Pyramid
Upsaapie
Visual Fealure
Transfonmer bases Label Decoder
MILP UpSanpple
Feature
Uptarmple
(a) Label decoder.
(b) Dense decoder.
Visual
Linear Projection
Decoder
P-VOVAE Decoder
Volumetric SDF
Grid Feature
Cube Feature
Quantized Tokens
Shape Prior
(c) 3D decoder.
Figure 4: An illustration of the decoders. Let N denote the output dimension, e.g. 4 for bounding box. (a) The output Label Decoder is single each element being an N-length vector. This dense grid is then upsampled back to the original image dimension through a bilinear interpolation. (c) The output of 3D Decoder a volumetric SDE The Autoregressive(AR) decoder is fixed module with pre-trained weights from Mittal et al. (2022).
of length N. The output of the Dense Decoder is dense grid with
Navigation. Following Hong et al. (2021), we use visual backbones to pre-compute visual representations (using the last feature map) at every possible viewpoint in a given room setting to accelerate RL training. We then adopt Recurrent VLN Bert as the task decoder, specifically the PREVALENT-like (Hao et al., 2020) variant due its superior performance over the original OSCAR-like (Li et al., 2020) model. And in that paper, initialize the decoder with pre-trained PREVALENT weights and use combination RL loss and imitation learning (IL. loss as training objective.
Manipulation. This decoder inherited from CLIPort (Shridhar et al., 2022). Similar to the dense decoder, this decoder depends the feature pyramid from visual representation and performs up-sampling at each level. Differently, language instructions are required to finish manipulation task. To process language input, the decoder tiles features and aggregates them with visual features. The output of the decoder is a dense affordance map to guide the agent picking and placing. It is important to bring up that we remove the extra spatial stream prevent the intervention of other visual representations, while the original CLIPort (Shridhar et al., 2022) includes that stream attain improved performance.
Table 5: Settings for Perceive, Ground, and Reason tasks. Act tasks are omitted here since they belong to separate training schemes. Abbreviations: "Cam. Pose." for camera pose estimation, "I-T Retr." for image-to-text retrieval, "Phr. Grnd." for phrase grounding, "Sem. Seg." for semantic segmentation, "Com. Res" for common reasoning, "Abs. Res." for abstract reasoning.
Setting
Depth Est.
Cam. Pose.
3D Recon.
Retr.
Phr. Grnd.
Seg.
VQA
Com. Res
Abs. Res.
Decoder
Dense
Label
3D
MLP
Label
Dense
Label
Label
Lahel
Out Dim
7
512
151
1843
2
Batch Size
128
32
32
256
128
128
128
128
32
Epoch
50
100
100
100
50
50
30
20
30
Most of the training settings are listed in Tab. 5. We use AdamW (Loshchilov and Hutter, 2017) optimizer, with 3n 0.9, B2 0.99 and weight decay 10-2, During training, warm up
Training.
18

---

Under review as a conference paper at ICLR 2023
the learning rate linearly in the first 2% steps to reach 1 x 10 and adopt linear decay schedule gradually decreasing the learning rate to zero during the rest of steps. The gradient norm is clipped to I. Navigation and manipulation tasks trained in simulation environments and stopped when convergence is observed. For simplicity, we resize input images to 224 to align with the resolution during pre-training phase of visual representations, which is necessary since these representations stay fixed during evaluation phase. All experiments are affordable on a single A100 GPU.
Summary score. To summarize the overall performance of visual representation on various tasks it is, we propose a comprehensive summary score. However,
and conclude how performances on different tasks are in divergent metrics. In general, there are two types of metrics in our benchmark: percentage and error. We create a normalized mapping to convert all error metrics to percentage metrics: given metric in error form F., the corresponding metric in percentage form P is formulated as P(F) = e -.336E where the constant - 1 1.386 is determined by two conditions: P(O) and P(0.5) = 0.5. With this normalization, we can calculate score for each task by averaging all aligned metrics within the task, and then calculate the final summary score for visual representation by averaging scores on all tasks. To make domains more balanced, we augment each task in Act with weight at 1.5.
D ABLATION
D.1 FINETUNING
In the experiment section, we mainly report results of fixed visual representations, since the evaluation of coneral-purpose requires shared visual representation for various tasks. To further study the effects finetuning the visual backhones, we provide experiments on two visual representations learned in self-supervieed fashion: RN-MoCo and ViT-I6-MAE. During finetuning, we set the base learning rate visual backbones to 10 and keep the same warm-up and linear decay schedule. As shown in Tab. 6, dense prediction tasks, e.g. depth estimation, benefit most from finetuning visual backbones. Other tasks in general can benefit marginally from finetuning. While a few cases turn out to be harmful finetuning. We hypothesize that learning decoder with dynamic encoder is hard and prone to overfitting, especially on relatively small amount of training data.
D.2 RESOLUTION
Another factor that may cause inferior performance the resolution of the input image. We demon- strate this by showing performance gain from scaling up input resolution, Tab. 7. The results reveal the headroom of performance when scaling up resolution.
D.3 LANGUAGE ENCODER
Image-text retrieval. From the experimental results in Tab. 3, we find large variance on image-text retrieval task. As mentioned before, we hypothesize the reason for unsatisfactory performances this task lies in the language encoder. With limited cross-modal processing in the implementation, the cross-modal matching highly depends on the quality of pre-trained representation. Since the default language encoder RoBERTa is pre-trained on text-only domain, the representation it produces may not align with vision branch well. We substitute with the CLIP language encoder (Radford et al., 2021) pre-trained in vision- language contrastive learning manner. As we can see in Tab. 8, switching to the CLIP language encoder leads significant improvements. This demonstrates the effectiveness of coupled representations, and also indicates the performance drop due to mismatching representations, though GPT2-CLIP and RoBERTa are both commonly recognized language encoders.
Actually, the brilliant performance current state-of-the-art models on image-text retrieval task rely heavily coupled representations obtained by large-scale vision- pre-training, Besides, fixed language encoder initialing from text-only pre-training proves sub-optimal results on V-L matching task, as reported in Kim et al. (2021). To pursue general-purpose visual representation, how to overcome the gap between diverse disentangled representations of different modalities and provide fair yet efficient evaluation methods is still challenging problem that remains unsolved.
19

---

Under review as a conference paper at ICLR 2023
Table 6: Influence of finetuning visual backbones. For space considerations, for identifying tasks (e.g. "Cam. Pose." for camera pose estimation, "I-T Retr." image-to-tex retrieval, "Phr. Grnd." for phrase grounding, "Sem. Seg." for semantic segmentation. "Com. Res" for common sense reasoning. "Abs. Res." for abstract reasoning). Finetuning on Act tasks is omitted here. We also remark the gains from finetuning in separate column. Green indicates positive gain, while red indicates drop.
the abbreviation of words
RN-MoCo
ViT-16-MAE
Task
Fixed
Finetune
Gain
Fixed
Finetune
Gain
Perceive
lest d<1.25
0.6400
0.7302
+0.0902
0.7699
0.8605
+0.0906
Depth Est.
test AbsRel J
0.2184
0.1725
-0,0459
0.1554
0.1203
-0.0351
test RMSE
0.7284
0.6042
-0.1242
0.5486
0.4444
-0.1042
TCans.(CL)
1.972
1.943
-0.029
2.177
1.926
-0.251
Orient((CL) |
5.427
5.671
+0.244
4.911
3.837
-1.074
Cam. Pose
Trans.(7S) |
0.241
0.255
+0.014
0.271
0,250
-0.021
Orient.(7S)
8,797
8,452
-0.345
6,699
5,823
-08776
3D Recon.
test mloU f
41.98
42.87
+0.89
43.95
48.25
+4.30
Ground
test R@ 1
7.1
5.9
-1.2
5.8
6.8
+1.0
1-T Retr.
test R@5
22.5
17.3
-5.2
20.5
24.1
+3.6
test R@ 10 f
33.8
26.6
-7.2
30.9
35.9
+5.0
val A.c@0.5
54.40
56.73
+2.33
65.18
64.55
-0.63
Phr. Grnd.
testA A.c@0.5 f
59.14
61.37
+2.23
67.59
66.13
-1.46
testB Acc@0.5
50.88
52.10
+1,22
62.10
62.38
+0.28
Sem. Seg.
val mIoU 1
18.05
27.40
+9.35
26.25
35.21
+8.96
Reason
val Acc. f
50.20
51.49
+1.29
54.82
54.49
-0.33
VQA
test Acc.
45.02
45.44
+0.42
48.50
47.89
-0.61
Com. Res.
val Acc. f
50.98
55.22
+4.24
60.76
55.68
-5.08
val Acc. f
62.38
62.58
+0.20
62.12
64.38
+2.26
Abs. Res.
Acc. 64.63
62.36
-2,27
62.85
65.14
+2.29
Table 7: Performance progressively gains from fine-tuning and higher resolution. Take RN-IN on phrase grounding task for example. The results are reported in A.c@0.5 metrics.
Setting
val
testA
testB
Fixed 224
48.57
54.48
43.87
Tuned on 224
58.65
64.15
53.26
Tuned on 512
65.55
71.95
$9.83
Table 8: Performance of different vision-language encoder pairs on Flickr30k test set.
Eneoder Pair (V+L.)
Recall@
Recall@
Recall @ 10
I6T-16-CLIP RoBERTa
23.0
50.9
64.2
ViT-16-CLIP GPT2-CLIP
45.5
74.2
83.0
Navigation. Similar to Image-text retrieval, we also conduct ablation of language encoder navigation task. It is customary in languaee navigation task to use pre-trained language encoder that produces visually-aligee interested to what extent such a gap can affect the performance on navigation. We re-evaluate the visual representations with the more VLN-fitting LXMERT (Tan and Bansal, 2019) language encoder
representations to better aid navigation. However,
also
20

---

Under review as a conference paper at ICLR 2023
from Hong al. (2021) apart from the results using RoBERTa. We show full comparison between LXMERT and RoBERTa as the language encoder in Tab. 9.
Table 9:
Exhaustive ablation on language encoder (RoBERTa vs. LXMERT) on navigation task. ResNet shows stronger fitting on seen scenarios, while ViT generalizes better on unseen scenarios.
Encoder Pair (V+L.
Seen Suce.
Seen SPI.
Unscen Suec.
Unseen SPI.
RN-IN + RoBERTa
54.75
$1.02
48.62
42.61
+ LXMERRT
58.18 (+3.43)
54,28 (+3.26)
55.17 (+6.55)
50.40 (+7.79)
RN-MoCo RoBERTs
52.50
48.92
48,49
43.49
LXMERT
57.49 (+4.99)
54.20 (+5.28)
55.21 (+6.72)
50.73 (+7.24)
RN-CLIP + RoBERTa
52.40
48,39
49.00
43,49
+ LXMERT
58.86 (+6.46)
55.03 (#6.64)
55.98 (+6.98)
50.73 (+7.24)
RN-Ego RoBERTa
44.07
39.41
43.89
37.88
LXMERT
45.45 (+1.38)
41.89 (+2.48)
48.45 (+4.56)
44.00 (+6.12)
VIT-32-CLIP RoBERTa
51.03
46,98
49.21
43,99
LXMERT
54.95 (43.92)
50,53 (+3.55)
56.88 (47.67)
51.70 (+7.71)
ViT-16-CLIP + RoBERTa
46.23
44.15
47.13
43.60
LXMERT
61.90 (+15.67)
55.22 (+11.07)
59.51 (+12.38)
53.15 (+9.55)
ViT-16-MAE RoBERTa
48.87
43.91
49.55
43.72
LXMERT 48.29 (-0.58)
43.16 (-0.75)
52.41
(+2.86)
45.99 (2.27)
We observe that performances improve nearly all visual representations when LXMERT instead of the text-only praitrained RoBERTa. This demonstrates that esuuall--aligned language representations do massively improve downstream performances. And further eliminate the influence of subsequent navigation decoder which is pre-trained along with the LXMERT language encoder, we conduct an extra ablation to test whether from-scratch navigation decoder with randomly initialized weights, which loses the encoder-decoder pre-training alignment, can still achieve similar result. The results are shown Tab. 10.
Table 10: Performance of LXMERT language encoder paired with pre-trained decoder vs. from-scratch decoder.
Eneoder Pair (V+L)
Decoder
Seen Suce.
Seen SPI.
Unseen Suce.
Unseen SPI.
VIT-CIJP--6 + LXMERT
pre- traitrained
61.90
55.22
59.51
53.15
VIT-CLIP-16 + LXMERT from-scratch
60.92 (-0.98)
56.54 (+1.32)
57.13
(-2.38)
51.40 (-L.75)
We observe that the model with a from-scratch navigation decoder still achieves comparable perfor- mance despite losing the encoder-decoder pre-training alignment. This shows that the improvements shown in Tab. 9 is not due to the encoder-decoder coupling, but rather due to the visually-aligned semantics of the LXMERT language encoder.
E SOCIAL IMPACT
With the emergence of foundation models, tremendous efforts have been put on improving multi- tasking capabilities of large-scale pre-trained models. Instead of improvements from the model's side, we focus on the problem of designing general, sophisticated and fair evaluation metric that reveals the models' capabilities on the full spectrum of vision tasks. We believe our G-VUE benchmark will make an excellent complement existing works as they can be easily adapted to be evaluated in G-VUE, There no known negative societal impact at this point.
F LICENSE
We list the licenses of all dataset involved in G-VUE follows:
21

---

Under review as a conference paper at ICLR 2023
NYUv2:
Unknown
ShapeNet: Custom
CambridgeLandmarks: BY-NC-S 2.0 UK
Scenes: Custom
Flickr30k: Custom
RefCOCO: Apache License 2.0
ADE20K: 3-Clause BSD License
GQA: CC B. 4.0
VCR: Custom
Bongard-HOI: Custom
R2R: Unknown
Ravens: Apache License 2.0
22


Reviews:
[{"review_id": "KOoVSQZYOnV", "reviewer": "Reviewer_7KYb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a General-purpose Visual Understanding Evaluation (G-VUE ) benchmark that covers 11 tasks in four functional\ndomains (Perceive, Ground, Reason, and Act). The benchmark is accompanied by several decoders that allow the evaluation of arbitrary visual representation models on these tasks. ", "summary_of_the_review": "This paper aims to create a benchmark that can holistically evaluate the cognitive abilities of visual systems. This is work in the right direction, with functional domains properly articulated and motivated. However, the current implementation is a compilation of existing datasets, each with its own evaluation metrics. Improving the overall evaluation, for example by considering task difficulty, or a curriculum based on these tasks, or a unified evaluation metric and dataset, would make the realized work more aligned with the overarching goal of human-like general-purpose vision systems.", "strength_and_weaknesses": "Strengths:\n- The benchmark will be made easily accessible through an evaluation platform and a leaderboard.\n- The task correlation analysis and the decoders that can be used for evaluating any arbitrary model are nice pluses.\n\nWeaknesses:\n- It seems that the tasks are a composition of existing datasets with existing evaluation metrics. As such, the contribution seems limited. \n- It is unclear whether the tasks in each functional domain are necessary and sufficient for evaluating general-purpose human-like vision systems, let alone if evaluating each functional domain separately is the best way to test human-like visual cognitive abilities. \n- In Table 3, it is really difficult to extract from the summary score that ViT-16-CLIP is the best general-purpose model because, for some tasks, models have large deviations (such as I-T Retr.) while others do not. Would the underlying assumption of a summary average be sufficient, or should an overall evaluation metric consider task difficulty? Please also clarify the sentence \"To make domains more balanced, we augment each task in Act with weight at 1.5\".\n- It is unclear how extensible the benchmark is, e.g., how easy it is in the future to include more tasks as newer datasets are constantly being constructed.\n- Most evaluation does not consider multi-task models, but rather self-supervised or pretrained visual representations. It is a bit of a stretch to refer to this as a general-purpose vision benchmark in the conclusion and introduction (list of contributions). In addition, it would be great to have a direct comparison of a multitask model with the ones shown in Table 3. Adding this would demonstrate how far away self-supervised visual representation models are from finetuned multitask models.\n\nA few additional questions:\n1. In terms of the claim, that unified architectures are \"still limited to the conventional vision-language domain, lacking the general capabilities for low-level geometric understanding and high-level reasoning and acting\", what about GATO [1]?\n2. Also, why are the unified architectures on Table 4 only evaluated on tasks they can solve, and not the full pool of benchmark tasks? Aren't the decoders described in this work specifically proposed for facilitating general evaluation purposes? \n3. If the language encoder can be substituted with a CLIP language encoder, then what is the reason for not using this  CLIP language encoder for all models, and evaluating these models with improved cross-modal understanding capabilities?\n4. How do the custom licenses for some of the datasets affect benchmark license and usage?\n\n[1] Reed, Scott, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez et. al. \"A generalist agent.\" arXiv preprint arXiv:2205.06175 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "88a8Ril_6S", "reviewer": "Reviewer_xbAo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines 11 tasks and benchmarks into a new benchmark for general-purpose visual representation learning. Four kinds of visual tasks are considered, i.e., perception, grounding, reasoning, and action. Then, by choosing different pre-trained backbones, a unified experiment is conducted to compare the learned representations from these backbones. Discussion and analysis are provided according to the results of the above investigation.", "summary_of_the_review": "Overall, the experiments are extensive and implementations are detailed. However, besides the composition of previous datasets and tools cannot result in a \"new benchmark\", at least not enough. New insights into the benchmark design, the relation between previous datasets and tasks, and how to design a new general model are all lacking. I appreciate the efforts of this work. However, deeper discussions are essential. Moreover, the contribution claim about the model is also not suitable. I encourage the authors to make a major revision to make the analysis and discussion deeper and more inspiring. Thus, I think this paper is not ready yet currently for ICLR.", "strength_and_weaknesses": "Pros:\n+ Pursuing the general purpose visual representation learning is important for our community.\n\n+ Some results and discussions are interesting and verify some conclusions of previous works, e.g. the comparison between CNN and Transformer, and the superiority of CLIP.\n\nCons:\n- There is no deep analysis of the relationship between the selected 11 tasks, only a performance correlation matrix is given. This is weird as the analysis of tasks is the most important thing to build a real unified benchmark, instead of simply compositing all existing sole datasets with domain gaps, different raw data, definition gaps, and annotation quality.\n\n- Few new insights and discussions are provided, except for sec. 5.3. Moreover, \"camera pose estimation and 3D reconstruction show fewer connections to other tasks\", this seems weird, camera extrinsic parameter estimation matters most in 3D reconstruction in tasks like structure from motion. Please give a discussion.\n\n- The claim about the proposed model is confusing. We these pre-trained encoders and fine-tuned decoders for these tasks in evaluation. But if we want to propose a new general model, what is the point of this model's existence? I think an experiment is suitable for this contribution instead of a model structure.\n\n- Some discussion about Pathway from Google?\n\n- ViT-B/32 MAE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0FdhDE2agd", "reviewer": "Reviewer_9fAp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a general purposed visual understanding benchmark (G-VUE), which covers four different abilities -- Perceive, Ground, Reason, and Act. The authors chose 11 existing tasks in this benchmark. The authors also provide a general encoder-decoder framework with pre-trained representations and tests on the benchmark. ", "summary_of_the_review": "The paper proposed an interesting benchmark to evaluate more unified models. However, there are multiple issues related to the connection between different tasks/datasets, and dataset selection for each skill. The problem of the proposed encoder-decoder model etc. Please check the weakness session for details. ", "strength_and_weaknesses": "[Strength]\n\n- The proposed benchmark involves four main functions, which are perceive, ground, reason, and act. Seems like a good combination of skills to test the unified model. \n\n- The author also proposes to use pre-trained representations to test the benchmark and evaluate two existing methods -- OFA and Unified-IO. \n\n[Weakness]\n\n- Creating a new benchmark is not simply aggregating existing datasets/skills. The more important for unified benchmark creation is what is the connection between each dataset, are there any shared concepts? what do we expect if the model performs well on one task and on others? Although the paper proposed interesting combinations of skills, there is no in-depth discussion on this. \n\n-  The reason for choosing the dataset in each skill is not clear. For example, In question answering, the authors choose GQA dataset for this benchmark. It is known that GQA is created by question templates and a lack of language diversities. VQA might be a better option. For an evaluation benchmark, the best option would be to use a mixed of existing benchmarks for the testing set. \n\n- The proposed framework uses frozen features and fine-tune the decoder on each task. The performance is far behind state-of-the-art results. Table 3 also doesn't have any STOA numbers on these tasks. My biggest concern is the trends shown in this table can not transfer or hint at other approaches. \n\n- In table 7, even with the fine-tuning encoder, the performance is still behind the state-of-the-art model. By freezing the encoder and having a separate decoder for each skill, the proposed framework is not a unified model anymore. \n\n- Table 4 shows two existing method results, OFA-HUGE and UIO-XL. These are interesting results, but many details are missing. For example, Is OFA-HUGE a single model or a model fine-tuned on each task? UIO-XL model didn't train on GQA datasets, this might be the main reason why the gap between OFA and UIO on VQA is huge. However, they may not reflect the general QA abilities since UIO also performed well on VQA datasets. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-MTlli1A-0", "reviewer": "Reviewer_U7v5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a unified collection of 11 vision (and vision-and-language) benchmarks, called G-VUE. One of the messages of this paper is that there are a lot of different evaluations of vision models these days, and they often cover very disjoint things. The paper suggests that vision models be evaluated on a variety of different tasks, exposing different capacities. Some of the tasks are a bit simplified (in terms of their format) to make evaluation easier (e.g. VCR is just the question answering part of that benchmark).\n\nThe paper evaluates ResNets and Vision Transformers, pretrained on both imagenet, web images, and egocentric videos, and with objectives spanning supervised and self-supervised learning. These are often treated as underlying encoders in the literature, and here it seems that the self-supervised approaches do best -- CLIP is better for grounding/reasoning, and MAE is better for raw perception tasks like depth estimation. The paper also evaluates recent large-scale vision-language models proposed in the last year (OFA and UnifiedIO) and finds significant headroom on many of of the proposed tasks.\n\nThe G-VUE benchmark will have an associated leaderboard for other researchers to use.", "summary_of_the_review": "To this reviewer, the introduction of a leaderboard that encourages evaluating on multiple tasks seems promising. However, it's not clear to this reviewer what the role of this leaderboard ought to be, and the analysis in the paper (i.e. a proxy of what we can learn from the leaderboard) could be improved by incorporating different classes of models.", "strength_and_weaknesses": "Strengths:\n* Benchmarks are important, and to this reviewer the overall message of the paper (models ought to be evaluated on a variety of vision tasks, not just low-level vision for instance), seems reasonable.\n* Making this leaderboard public will help other researchers compare to past work.\n* The analysis of low-level encoders (ResNet50 versus ViT-B); with different pretraining mechanisms / data sources is interesting. I'm not quite sure how much it extends to much larger models, though, or models that use language through other means (besides just a source of \"labels\" like CLIP).\n\nOne overarching question to this reviewer is the purpose of this benchmark collection -- are researchers supposed to finetune models on it, evaluate zero-shot, or do something else (like linear probing)? In hindsight, looking at what happened in the text domain like GLUE, to this reviewer it seems like the zero-shot evaluation paradigm won out as many of the GLUE subtasks were quite noisy and easy to overfit to if the right finetuning recipe was applied. I'd be curious as to the thoughts of the authors here.\n\nWeaknesses:\n* The benchmark could be improved -- and the results in this paper more insightful -- if more models were chosen. I'd like to see how well OFA / UIO do on a finetuned setting, if that's what the benchmark is about (since the underlying image encoders are finetuned). If the message is that e.g. OFA can't do depth perception out-of-the-box (because it wasn't trained on that) -- I'd be curious as to how it would do in a linear-probe setting. I'd also like to see other vision-language models included to see the benefits of training on other types of data -- like perhaps FLAVA (Singh et al 2021 https://arxiv.org/abs/2112.04482 ) which is another visual-bert like model trained on slightly different data versus OFA, or MERLOT Reserve (Zellers et al 2022 https://arxiv.org/abs/2201.02639) trained on videos. Finally, I'd be curious as to whether the results are different with size (e.g. is UIO-XL better than a smaller Unified-IO model?)\n* An issue with this collection of benchmarks, at least to this reviewer, is that they all have different formats (different inputs, outputs, metrics); different training distributions / validation sets, etc. For that reason, I'm not quite sure whether this benchmark collection will catch on. Though not as important for my overall score, I think it would be neat if some code was released that made it easy for different e.g. image encoders to be finetuned on the various tasks. However, I'm still not sure as to whether that code would be used -- as it would presume a model structure that might not be what's ideal for these various tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject