CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
Authors: Mohammadreza Salehi
Mehrdad Farajtabar
Maxwell Horton
Fartash Faghri
Hadi Pouransari
Raviteja Vemulapalli
Oncel Tuzel
Ali Farhadi
Mohammad Rastegari
Sachin Mehta
Venue: Accepted by TMLR ()

Abstract: Contrastive language image pretraining (CLIP) is a standard method for training vision-language models. While CLIP is scalable, promptable, and robust to distribution shifts on image classification tasks, it lacks object localization capabilities.  This paper studies the following question: Can we augment CLIP training with task-specific vision models from model zoos to improve its visual representations? Towards this end, we leverage open-source task-specific vision models to generate pseudo-labels for an uncurated and noisy image-text dataset. Subsequently, we train CLIP models on these pseudo-labels in addition to the contrastive training on image and text pairs. This simple setup shows substantial improvements of up to 16.3% across different vision tasks, including segmentation, detection, depth estimation, and surface normal estimation. Importantly, these enhancements are achieved without compromising CLIP's existing capabilities, including its proficiency in promptable zero-shot classification.

Published in Transactions on Machine Learning Research (06/2024)
CLIP meets Model Zoo Experts: Pseudo-Supervision for Vi- sual Enhancement
Mohammadreza Salehi'
University of Washington
Mehrdad Farajtabar
Maxwell Horton
Fartash Faghri
Hadi Pouransari
Raviteja Vemulapalli
Oncel Tuzel
Apple
Ali Farhadit
Allen Institute for Artificial Intelligence
Mohammad Rastegari
Meta
Sachin Mehta
Apple
Reviewed on OpenReview:
https: openreview. net/ forum? id= Y
Abstract
Contrastive language image pretraining (CLIP) is standard method for training vision- language models. While CLIP is scalable, promptable, and robust to distribution shifts on image classification tasks, lacks object localization capabilities. This paper studies the following question: Can we augment CLIP training with task-specific vision models from model 200s to improve its visual representations? Towards this end, we leverage open- source task-specific vision models to generale pseudo-labels for uncurated and noisy image-text dataset. Subsequently, to the contrastive training image and text pairs. This simple setup shows substantial improvements of up to 16.3% across different vision tasks, including segmentation, detection, depth estimation, and surface normal estimation. Importantly, these enhancements are achieved without compromising CLIP's existing capabilities, including its proficiency in promptable zero-shot classification.
train CLIP models these pseudo-labels in addition
Introduction
Foundation Models (FMs) are revolutionizing different domains of artificial intelligence and machine learning,
including computer vision [Radford et al. 2021 He et al. 2022 Kirillov et al. 2023 and natural language
processing (Devlin et al. 2019 Brown al. 2020 Touvron al. 2023).
FMs can be trained on web
"ook done during internship Apple.
Work done when employed Apple.

---

Published in Transactions on Machine Learning Research (06/2024)
Image encoder
Image encoder
Task heads
Reative Improvement (NN)
46
web-scale image-text data
40 16
Web-scale
Contrastive loss
Task losses
image-text data
Contrastive loss
Text encoder
Text encoder
Model (Task models)
Pseudo-labeling
X0090x (W6gg)
PAOCW
0000
(b)
(c)
Figure Training CLIP with pscudo-labels improves its visual representations. (a) shows the standard CLIP training. (b) shows CLIPTeX that trains CLIP with pseudo-labels from experts. Note that the main purpose of task heads to improve CLIP's image encoder with expert knowledge, and the heads can be discarded after training.
(c) shows the relative improvement that CLIPTeX obtains over CLIP-FT. Note that CLIP-FT is a stronger baseline than CLIP (see Section and Section 5 for details). Here, SSeg, OD, ISeg, SNE, and DE refer to semantic segmentation, object detection, instance segmentation, surface normal estimation, and depth estimation respectively.
crawled data without relying on crowd or expert annotations, and yet they demonstrate strong generalization capabilities (Jia al. 2021 Schuhmann et al. 2022
CLIP, one of the most prominent methods for FM training in vision, uses contrastive learning to align image and text representations (Radford et al. 2021 Jia et al., 2021). In addition to robustness to data distribution shifts, CLIP offers impressive zero-shot and cross modal retrieval capabilities on unseen datasets. Nevertheless, computer vision encompasses a broad range of tasks that require the ability to comprehend spatial relationships, semantic content, object localization, and 3D structures. In spite of CLIP's impressive zero-shot upen-vocabulary classification accuracy, it exhibits poor localization capabilities and often struggles in associating text with objects in an image (Thrush al. 2022 Ghiasi et al. 2022 Ranasinghe et al. 2023) Consoquently, in practice, many vision tasks (c.g., detection and segmentation), rely on CLIP through fine- tuning the entire model to compensate for these localization deficiencies.
In this work, we seek to answer the following question: Can we augment pretrained CLIP models with lask-specific vision models from model z0os to improve ils visual representations? That is, we seek to (1) use open-source task-specific vision models to generate hard pseudo-labels on web-scale noisy image-text dataset and, (2) train CLIP on image-text pairs along with pscudo-labels with multiple objectives. An overview of our approach, which we call CLIP Training with eXperts (CLIPTeX) is shown in Fig, [ We show that CLIPTeX enhances the visual representations of CLIP while retaining the pre-existing capabilities of CLIP.
We summarize our contributions as follows:
We introduce simple and effective method, CLIPTeX, to improve the visual representations of CLIP by leveraging experts specialized in object localization, depth estimation, and surface normal estimation. Through the generation of hard psoudo-labels on a noisy image-text dataset and the training of CLIP on these paired data points with multiple objectives, we achieve significant improvement in visual representations. Notably, our method yields up to 16.3% enhancement in probing accuracy across diverse set of vision tasks and datasets.
Our approach leads to positive transfer of representations down-stream tasks and preserves the inherent strengths of CLIP, including its ability perform zero-shot classification. This ensures that the model remains versatile and applicable across wide range of computer vision domains.
Experiments with multiple probes on variety of vision tasks and datasets (c.g., segmentation on PASCAL VOC and ADE20K, detection on COCO, depth estimation on NYU-v2, classification on ImageNet- 1k and Places-365, and surface normal estimation on NYU-v2) demonstrrate the effectiveness of CLIPTeX.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2
Related Work
Vision FMs. Vision FMs extended the concept of pre-training to vast datasets containing hundreds of
millions or even billions of images. This was in part driven by the introduction of ViTs Dosovitskiy et al., 2021j which demonstrated the scalability of training Transformers (Vaswani et al. 2017) to such large-scale datasots in the field of computer vision. Since then, numerous largo-scale pro- pre-training methods have emerged
in the domain of computer vision (e.g,, Radford et al.,
2021 Yu et al.
2022
Caron et. al. 2021 He et al.
2022 Arguably, one of the most prominent classes of vision FMs CLIP that specializes in aligning noisy
image-tett pairs from the web (Radford et al. 2021 Schuhmann et al. 2022;
Gadre et al.
2023 This
distinction is not only attributed to its scalability, but also to its prompting capabilities and robustness in handling dataset distribution shifts. Nevertheless, these models often face challenges in associating text with
individual objects and localizing them (Thrush et al. 2022: Ghiasi et al. 2022
Ranasinghe et al. 2023
This work focuses enhancing this capability through pseuudo-supervssin.
Pseudo-supervision with experts. The primary objective of pseudo-supervision (Lee et al., 2013 is facilitate model training by generating pseudo-labels for unlabeled data, typically leveraging experts trained on subset of the data containing ground truth labels. This methodology has also been applied to the training of foundation models (FMs). To the best of our knowledge, current approsches involve the acquisition of crowd labels for portion of the data on a single task, with the subsequent training of
experts on this labeled subset (e.g., Ghiasi et al. 2021 Zhang et al., 2022 Kirillov et al.) 2023: Liu et al. 2024). These trained experts are then utilized to create pscudo-labels for the remaining unlabeled data.
Essentially; these methods employ experts that have been trained on the same or similar data distribution as the unlabeled data, aiming to achieve positive transfer. For example, in GLIP (Li al. [002aa,, subset of web data is crowd-sourced to obtain localization labels, which is then used for expert training. Following expert training, these experts are employed to generate leaudo-labels for the remaining unlabeled web data. This combination of crowd labels and pscudo-labels is subsequently used to train the GLIP V2 (Zhang et al. 2022) model. SAM (Kirillov et al. 2023) also follows similar paradigm for creating large-scale segmentation dataset. Unlike previous approaches, our proposed method uses publicly accessible experts trained on diverse tasks with different data distributions and objectives.
Multi-task learning for FMs. Multi-tasking (Caruana 1997 Ruder 2017
standard method for
training on multiple tasks simultancously, is widely used in machine learning Liu et al.) 2019 Misra et al. 2016 Sun et al. 2020), including FMs (e.g., [Wang el al. 2022 Chen et al. 2022 Yang el al. 2022 Shukor et al. 2023; Zhang et al., 2023). Existing auulti-task FMs creates a unified multi-task datasets by either collecting a new labeled datasct (c.g., [Sun et al. 2022) or mixing existing labeled datascts (c.g., Lu et al. 2022), to facilitate positive transfer of knowledge to down-stream tasks, In contrast, CLIPTeX does not need any data collection and uses pseudo-superrisson for training.
3 CLIPTeX
CLIP, scalable FM, aligns image and text representations obtained from independent image and text encoders using contrastive loss. Although it offers zero-shot and cross-modal retrieval capabilities, it ex- hibits poor localization and dense prediction capabilities. This work, CLIPTeX, extends CLIP with pseudo- supervision from publicly available task experts specializing in localization, depth, and surface normal esti- mation. Our approach enhances CL.IP's representations without ony labeled data collection (Fig.
3.1 Modeling
Image and text encoders in CLIPTeX. To train on image-lext datasels, CLIPTeX uses two encoders, similar to CLIP: (1) an image encoder that takes RGB image as an input and produces an image embedding as an output and (2) a text encoder, that takes the text caption as an input and produces a text embedding as its output. Contrastive loss Lelip between image and text embeddings is one of the losses used train CL CLIPTeX, in CLIP.
3

---

Published in Transactions on Machine Learning Research (06/2024)
Task-specific heads. To train CLIPTeX with pseudo-labels, use task-specific heads that take the output of image encoder an input and generate predictions for the respective task. Previous work have shown that multi-scale representations provides significant benefit in tasks requiring localization and fine-grained visual understanding (Zhao et al., 2017 Lin et al. 2017). However, some image encoders (e.g., ViT) do not inherently possess these capabilities. To ensure CLIPTeX can learn better visual represenlations independent of the image backbone used, we include a single shared multi-scale module (Zhao al. 2017) between image encoder and task-specific heads. We feed the output of the image encoder through a multi-scale module (Zhao et al.) 2017), which in turn feeds into the lightweight task-specific classification regression heads. In our implementation, we use independent point-wise convolution as the head for each task.
Note that the spatial dimensions of the output from the task head may not be the same as the input image resolution, and certain tasks (e.g., segmentation, depth and surface normal estimation) may require them to be similar. In such instances, we perform nearest neighbour interpolation on head's output.
3.2 Training
To train CLIPTeX with pseudo-supervision on n tasks, we first generate hard pseudo-labels offline using publicly available task-specific experts. This is done on an uncurated web-scale dataset. We then train CLIPTeX with weighted sum of contrastive loss in CLIP Lelip and task-specific losses Ltask: as:
L = Actip Lelip + Σ Nessk Liesk:
(1)
where Ciask is the loss the t-th task. Here, Ntak and Aelip are the loss coefficients of t-th task and the standard CLIP loss, respoctively.
4 Experimental Setup
Probing is a standard method to study the representations learnt by neural networks (Alain & Bengio 2017 He et al. 2022: Radford et al., 2021). We probe CLIPTeX and other pre-trained models on different down- stream tasks and multiple datasets using clossifier regressor probes. This helps understand if training with hard pscudo-labels from experts improve the effectiveness of CLIP's image representations across different vision tasks. In the following sub-sections, expert models for pseudo-labeling (Section 4.1, baseline methods (Section 4.2) probes (Section 4.3 and
provide details of our experimental setup, including
down-stream tasks and datasets that are used for probing (Section 4.4
4.1 Task-specific Experts
Images provide valuable visual information about the appearance of objects in a scene, such as texture, color, and shape. However, they lack information about the spatial relationships between objects, specifically their relative distances and oricntation. To aid CLIP's image encoder in lcarning more comprchensive representations of visual content within an image, including object's 's orientation and location, we generate hard pseudo-labels from the following publicly available experts:
Semantic segmentation. We use Mask-RCNN (e al. 2017] with VIT backbone (Dosovitskiy al. 2021), trained on the COCO (Lin et al. 2014) with RangeAugment (Mehta el. al. 2022 to produce pseudo-labels for segmentation.
Monocular depth estimation: We use DPT (Rantl el. al., 2021), trained on MIX-6 datasel Ranlt. et al. 2021h, to generate monocular depth map lseudo-labels.
Surface normal estimation: We use NLL-AngMF Bae et 2021 as our surface normal expert. which is trained on ScanNet dataset (Dai al. 2017)
4

---

Published in Transactions on Machine Learning Research (06/2024)
4.2 CLIP Baselines
CLIP models pre-trained on billions of images have demonstrated impressive generalization properties. How- ever, training on such scale datasets with experts at high resolution (e.g., input image is 512 x 512 is com- putationally expensive. Therefore, to show the efficacy of our approach, we finctune pro-trained CLIP with and without pseudo-labels on CC3M (Sharma et al. 2018) for 30 epochs on 64 A100 GPUs. We compare with the following baselines to show the efficacy of pseudo-supervision:
CLIP. We use CLIP model by Mchta et al. (2022) pretrained on 1.2 billion images with a variable
resolution and batch sampler whose base input image's spatial resolution is 224 x 224. The model is robust
to input image's scale (likely due to multi-resolution training) and also, more performant compared to other open-source CLIP models trained on similar dataset size (e.g., OpenCLIP of Ilharco et al. (0021)
CLIP-FT. Many dense prediction tasks (c.g., segmentation) exhibit enhanced performance when pro- vided with high-resolution impul images. However, CLIP is initially pre-trained images with 224 224 spatial resolution. Therefore, to gain better insights into the benefits of pseudo-superrision, we finetune CLIP with contrastive loss on CC3M's image and text pairs. For this fine-tuning, we employ variable resolution and a batch sampler, whose base input image's spatial resolution 512 x 512. Importantly, this model serves as a fairer baseline for CLIPTeX compared to CLIP, as it has been adapted to CC3M data with high-resolution images. Therefore, any improvements observed over this baseline signify a pure transfer of knowledge from pseudo-sunerrvsion.
To show the generality of our approach, we conducted experiments with three image encoder backbones: VYT-B /1, ViT-H/16, 16, and ResNet-50. Also note that we finetune CLIPTeX on CC3M's image and text pairs along with pscudo-labels (Section 4.1 using the same settings as CLIP-FT. We use cross-entropy loss to train on segmentation pseudo-labels, and L1 loss to train on depth and surface normal pseudo-labels.
4.3 Classifier and Regressor Probes for Evaluation
To study the visual representations of different frozen pre-trained models (Section 4.2 our experimental setup involves both classification and regression tasks across different dalasets (Section 4.4) In this section, we provide details of task-wise probes.
Semantic segmentation. We use three probes with frozen image encoders: (1) linear head, point- wise convolutional layer that projects the features of the image encoder the number of classes in the semantic segmentation dataset. (2) DeepLabv3 head (Chen al., 2018), a standard non-linear head for dense prediction tasks. (3) PSPNet head (Zhao et al. 2017), another standard and widely used non-linear head for dense prediction tasks. During probing, we minimize the loss between predicted and ground-truth segmentation masks using a cross-entropy loss.
Object detection and instance segmentation. We use two probes with CLIP's frozen image encoder: (1) Mask R-CNN heads (He et al. 2017], a widely used non-lincar multi-task head for object detection and instance segmentation. (2) SSD head (Liu et al. 2016), another standard head for efficient object detection. During probing, we minimize with the same classification and localization losses as used in the original works.
Monocular depth estimation. Because it is a dense prediction task, we use the same probes as semantic segmentation task with frozen image encoder. We minimize the scale-shift invariant loss (SSI) Bhat et al. (2023) (in disparity space) between predicted and ground-truth disparity maps.
Surface normal estimation. We use the same probes as segmentation and minimize the the angular loss with learned allemuation (NLI-AngMP) between predicted and ground truth surface normal values (Bae et al. 2021).
Image classification. We use linear (i.e. fully-connected)) layer with the frozen image encoder. We minimize cross-entropy loss between predicted and ground-truth labels.
Many dense prediction tasks (e.g., segmentation and detection) require high resolution images.
5

---

Published in Transactions on Machine Learning Research (06/2024)
4.4
Evaluation Downstream Tasks and Datasets
We evaluate different models using classifier and regressor probes (Section
4.3 the following lasks:
Semantic segmentation. We use PASCAL VOC (Everingham et al. 2010 with 20 classes and ADE20K (Zhou et. al. 2019) with 150 classes for the task of semantic segmentation. Note that, the classes in the PASCAL VOC dataset are subset of the COCO classes on which the segmentation expert is trained (Section 4.1)) On the other hand, majority of the classes in ADE20k are not part of the segmentation expert's training corpora. Evaluating on these two types of datasets enables us to better understand the true gains of pseudo-supervision from experts. Following a standard convention, we report the accuracy on the validation sets of these datasets in terms of mean intersection over union (mloU).
Object detection and instance segmentation. We use the COCO dataset for detection and instance segmentation. Importantly, during training with pseudo-labels, we do not use the bounding boxes. In- stead, we convert instance masks into semantic segmentation pseudo-labels. This allows us to evaluate baselines on both instance segmentation and object detection, which are considered to be more challenging tasks than semantic segmentation. Following standard convention, evaluate the accuracy COCO's validation set in terms of mean average precision (mAP).
Monocular depth estimation. We use NYU-V2 (Silberman al. 2012 dataset our depth estima- tion benchmark. Note that DPT, the expert used for depth speudo-supervisoon is trained on different dataset, L.c., ScanNet. We use absolute relative error as metric for evaluation on the validation set.
Surface normal estimation. We use NYU-V2 for surface normal estimation. We follow Bae et al. (2021); ;; Qi et al. (2018) for training datasct, and evaluate the official test set of NYU-V2. Impor- lantly, it's worth noting that the surface normal expert (Section 4.1 has been trained on the ScanNet dataset. Therefore, CLIPTeX has not been exposed to images and labels from NYU-V2 during the pseudo- supervision process. This setup allows us to ascertain whether CLIPTeX can transfer to unseen datasets or not. Following Bac et al. [2021), , we use <<30 as the metric for evaluation.
Image classification. We evaluate on two standard image classification datasets, i..., ImageNet (Rus- sakovsky et al.) 2015) and Places365 Zhou et al. 2017). We use top-1 accuracy on the validation set as an evaluation metric.
Details about our implementation, including syper-parameters used for training on CC3M and probing tasks are given in Appendix A
5 Results
This section presents the probing results of CLIPTeX trained with pscudo-supervision from experts. Our results shows that CLIPTeX enhances the visual representations of the image encoder in CLIP, leading to significant improvements across a variety of tasks and datasets when probed using different probes.
5.1 Sreudo-supervistte improves Visual Representations
Semantic segmentation. Probing results for the task of semantic segmentation on PASCAL VOC and CLIPTeX shows consistent improvements over the baselines. Particularly note- worthy is the linear probing accuracy of CLIPTeX with VIT-B/16 and ViT-H/16 backbones on the PASCAL VOC dataset, which is about 10% and 16.3% better than CLIP-FT. These substantial improvements can be partially attributed to the fact that the semantic classes in PASCAL VOC are a subset of semantic classes in COCO, on which the segmentation expert is trained. We also observe improvements of up to 2.66% on the ADE20k dataset, despite the fact that the experts were not trained on the majority of its classes. These improvements underscores the enhancement in visual representations within the image encoder.
ADE20k are given Table [
It worth mentioning that while VIT backbones have generally exhibited superior performance compared to ResNets, observed that the gap in mloU between different models with ResNet image encoders

---

Published in Transactions on Machine Learning Research (06/2024)
Table Probing results for semantic segmentation. A higher value of mloU is better.
ADE20k
PascalVOC
Model
Linear
DeepLabV3
PSPNet
Linear
DeepLabV3
PSPNet
ViT-B/16
CLIP
6.78
16.15
17.32
18.66
43.75
45.53
CLIP-FT
26.60
37.11
38.80
62.47
77.67
78.22
CLIPTeX (Ours)
29.26
39.20
39.70
73.43
80.57
80.71
VIT-H/16
CLIP
24.18
33.39
34.86
56.18
73.12
75.37
CLIP-FT
32.20
43.05
14.24
62.95
81.73
82.94
CLIPTeX (Ours)
36.17
45.43
45.63
79.30
84.06
84.31
ResNet-50
CLIP
11.98
29.51
28.22
46.96
70.34
70.92
CLIP-FT
11.30
34.86
33.97
34.78
73.70
74.17
CLIPTeX (Ours)
12.93
35.45
34.80
40.31
75.82
75.58
is relatively small in comparison VIT backbones. This observation likely attributed to the inductive biases present in CNNs, which ViT-based models may lack. Note that CLIP's mIoU is notably lower in comparison to other models for VIT backbones. This discrepancy is likely attributed to the fact that the CLIP is pre-trained at a resolution of 224 x 224, whereas both CLIP-FT and CLIPTeX employ a higher resolution of 512 x 512.
Object detection and instance segmentation Table 2a shows probing results for the lask of object detection and instance segmentation. For Mask R-CNN head with ViT-B/ /66 as the frozen backbone, CLIPTeX delivers 13.69% and 1.68% better bounding box mAP over CLIP and CLIP-FT respectively. We observe similar gains when CLIPTeX is used with SSD. These results suggest that pseudo-supervision from experts improves CLIP's image representations for localization tasks.
Interestingly, ResNet-50-based CLIP models with different detection heads delivered better accuracy than ViT-based models. Our findings align with that of ViT-Det (Li et al.) (022b) which also indicates that ResNet-based models with less capacity delivers similar or better performance than ViT-based models on transfer learning for object detection.
Depth and surface normal estimation. CLIPTeX obtains lower error rate (Table 3a and higher value of a<300 (Table 3b) as compared CLIP and CLIP-FT across various probing heads on NYU-V2 for depth estimation and surface normal estimation respectively. These results indicate positive transfer of distance and surface normal knowledge to CLIPTeX 's image backbone, contributing to the improved performance.
Image classification. Unlike other dense prediction tasks discussed above, observe that CLIP achieves similar or slightly better accuracy compared to CLIP-FT and CLIPTeX (sce Table 2b This outcome can be attributed to the characteristics of image classification tasks, which typically involve assigning a single label to an entire image based on its 2D visual content. These tasks primarily focus on recognizing objects without requiring detailed information about object boundaries, spatial relationships, or the 3D structure of the scene.
5.2 Zero-shot Capabilities are Preserved in CLIPTeX
One of the important and powerful characteristics of CLIP is prompling, which enables zero-shot transfer to new datasets. Pseudo-supervisson with experts can potentially lead to catastrophic forgetting of previously Icarned knowledge, which may in turn affect the model's zcro-shot generalization capabilitics. Table 4 compares the zero-shot capabilities of CLIP-FT and CLIPTeX in classification on ImageNet- (Russakovsky et al. 2015), retrieval on Flickr-30k (Young et al. 2014). and 38 tasks used in OpenCT.IP (Tharco et al. 2021)

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Probing results for object detection, instance segmentation, and image classification. In (a), for Mask R-CNN, we report mAP (higher better) for bounding box and instance segmentation while for SSD, we report mAP only for bounding box on the COCO dataset. In (b) top-1 accuracy (higher is better) is reported.
(a) Detection and instance segmentation on COCO.
(b) Image classification.
Mask R-CNN
SSD
Model
ImageNet
Places365
Model
BBox
Instance
BBox
ViT-B/16
ViT-B/16
CLIP
80.24
55.52
CLIP
15.20
12.16
5.33
CLIP-FT
79.94
55.21
CLIP-FT
27.21
23.18
16.46
CLIPTeX (Ours)
79.64
55.36
CLIPTeX (Ours)
28.89
24.92
17.50
VIT-H/16
VIT-H/16
CLIP
84.85
56.96
CLIP
26.65
21.29
11.07
CLIP-FT
84.1
55.81
CLIP-FT
33.93
28.92
20.24
CLIPTeX (Ours)
83.2
55.96
CLIPTeX (Ours)
34.50
29.60
21.55
ResNet-50
ResNet-50
CLIP
29.49
25.61
20.32
CLIP
78.35
56.55
CLIP-FT
38.13
34.02
30.28
CLIP-FT
78.92
56.98
CLIPTeX (Ours)
38.23
34.04
28.62
CLIPTeX (Ours)
78.95
57.22
Table 3: Probing results for depth and surface normal estimation on NYU-V2 dataset. Following Ranftl et al. 2022), we report absolute relative error (lower is better) for depth estimation. For surface normal estimation, we report <<30 following Bae et al. (2021) (higher is better).
(a) Depth estimation.
(b) Surface normal estimation.
Model
Linear
DeepLabV3
PSPNet
Model
Linear
DeepLab PSPNet
VIT-B/16
VIT-B/16
CLIP
0.235
0.189
0.168
CLIP
28.49
45.17
47.29
CLIP-FT
0.215
0.145
0.139
CLIP-FT
29.06
47.74
47.91
CLIPTeX (Ours)
0.159
0.129
0.128
CLIPTeX (Ours)
39.96
50.95
50.80
VYT-H1/1
VTT-H/16
CLIP
0.212
0.151
0.132
CLIP
29.09
47.31
49.78
CLIP-FT
0.213
0.131
0.125
CLIP-FT
29.21
49.73
50.48
CLIPTeX (Ours)
0.138
0.118
0.117
CLIPTeX (Ours)
43.22
53.23
53.89
ResNet-50
ResNet-50
CLIP
0.212
0.156
0.147
CLIP
33.67
46.05
47. 47.28
CLIP-FT
0.239
0.160
0.155
CLIP-FT
28.72
46.99
48.66
CLIPTeX (Ours)
0.220
0.153
0.150
CLIPTeX (Ours)
31.56
47.92
49.44
respectively. CLIPTeX 's zero-shot performance on par with that of CLIP-FT, indicating that enhanced representations do not result catastrophic forgetting.
5.3 Positive Transfer of Representations from CLIPTeX to Downstream Tasks
The CC3M dataset is uncurated and noisy, and may have a skewed distribution towards specific object classes or scenes. Consequently, knowledge transfer from experts to CLIPTeX may also be skewed towards more frequent objects in the data. To explore this phenomenon, we quantified the frequency objects (bounding boxes or instances) in the pscudo-labels generated by the Mask R-CNN expert (Fig. 2a) on the CC3M dataset. Additionally, we examined class-wis. wise improvements in IoU of CLIPTeX with respect to CLIP. FT on the PASCAL VOC dataset (Fig. 2bb. CLIPTeX improves the loU for all classes in the PASCAL VOC

---

Published in Transactions on Machine Learning Research (06/2024)
Table 4: CLIP's zero-shot knowledge is preserved when trained with experts. Following CLIP
(Radford et al., 2021 we report zero-shot top-1 accuracy for ImageNet- 1K dataset and recall@1/5/10 for Flickr-30k dataset.
(b) 0-shot retrieval on Flickr-30k.
(a) 0-shot classification IeageNet.
Text Retrieval
Image Retrieval
Model
0-shot Top-1
Model
R:11
R@55
R@10
R@1
R@05
R@10
CLIP-FT
68.76 76
CLIPTeX (Ours)
68.25
CLIP-FT
85.90
96.70
98.60
71.66
91.00
94.94
CLIPTeX (Ours)
86.00
96.90
98.70
71.40
90.86
95.16
Average 0-shot accuracy 38 datasets in OpenCT.IP evaluation.
Model
0-shot avg. accuracy
CLIP-FT
59.44
CLIPTeX (Ours)
59.59
Class frequency
109
105
person
chair
car
boat
potted plant
bird
dining table
bottle
bicycle
sofa
horse
sheep
dog
cow
matorbike
aeropland
cat
train
(a) Bounding box frequency for PASCAL VOC classes in CC3M's pseudo-labels obtained with Mask R-CNN.
25
in
20
15
10
0
person
chair
car
boat
potted plant
bird
dining table
bottle
bicycle
sofa
horse
sheep
1v
dog
COW
motorbike
ampplann
cat
bus
train
(b) Class-wise IoU gap (in %) between CLIP-FT and CLIPTeX when linear probed the PASCAL VOC.
Figure 2: Positive transfer with CLIPTeX.
dataset and is not biased towards the most frequently occurring object classes. These findings, combined with insights in Section 5.1 suggests positive transfer of representations from CLIPTeX down-stream tasks.
5.4 Ablations
Role of pscudo-labels from experts in training CLIPTeX. Incorporating pscudo-supervision from task-specific experts, even from a single expert during training, results in substantial improvements in per- formance. These improvements observed when evaluating models various downstream tasks with

---

Published in Transactions on Machine Learning Research (06/2024)
Table 5: Role of pseudo-labels from experts in CLIPTeX training. Results with different probes for different dense prediction tasks are reported (see Section for details). For segmentation, we report the results on the PASCAL VOC dataset. We observe similar trends on ADE20k (see Appendix B
Row
Expert
Segueeniation (1)
Detection (1)
Depth (4)
Surface Noruall (1)
Segmentation
Dopth
Surface Normal
Linear
PSPNet
Mask R-CNN
SSD
Linear
PSPNet
Linear
PSPNet
R1
62:47
78.22
27.21
16.46
0:215
0.138
29.06
47.91
R2
R3
/
X
64.50
81.16
21.75
16.70
0.170
0.131
95.21
49.51
R5
/
/
x
73.06
81.40
28.83
17.57
0.162
0.130
37.05
49.60
R6
72.87
28.83
17.75
0.188
0.132
38.65
R7
/ /
64.20
81.17
27.90
17.00
0.765
0.120
39.50
51.01
R8
/
73.43
80.71
28.89
17.50
0.159
0.128
39.96
50.49
Table 6: Role of head complexity (light and heavy) when training with pscudo-labels on CC3m. #layers denote the number of convolutional layers used in the task head. Results with different probes for different dense prediction tasks are reported (see Section | for details). For segmentation, we report the results on the PASCAL VOC datasct. We observe similar trends ADE20k dataset.
Segmentation (1)
Detection (t)
Depth (4)
Surface Normal (t)
# layers
Linear
PSPNet
Mask R-CNN SSD
Linear
PSPNet.
Linear
PSPNet.
73.43
80.71
28.89
17.50
0.159
0.128
39.96
50.80
3
66. 70
80.24
28.64
17.43
0.155
0.127
40.55
51.72
Table 7: Joint multi-tasking with manual and pseudo-labels. For ADE20k segmentation, NYUv2 depth estimation, and NYUv2 surface normal, we use linear head while for detection COCO, use Mask R- R-CNN head.
Pseudo labels
Manual labels
Segmentation
Detection
Depth
Surface Normal
X
29.26
28.89
0.159
39.96
30.15
30.16
0.182
37.86
different probes (see R1 vs. rest; Table 5) Overall, our findings indicate that incorporating knowledge from all experts contributes to learning better visual representations. Therefore, use all experts for pseudo-supervision while training CLIPTeX.
Task-head complexity. As discussod in Scction 3 we use light-weight heads improve visual represen- (ations in CLIP's image encoder. We replace these heads with heavier counterparts (comprising of three standard convolutional layers) when training CLIPTeX with CC3M pseudo-labels. Table T shows that light- weight heads deliver similar performance to heavy-weight heads in most cases. Therefore, we use light-weight heads for pscudo-supervisie in our experiments to make the training more efficient.
Joint multi-tasking with manual and pseudo-labels. We trained variant of CLIPTeX with additional labeled data. Because there is no multi-task dataset available with annotations for all tasks, we instead extended CLIPTeX with ViT-B by feeding manual- and pscudo-labels in an itcrable fashion. For example. batch of segmentation dataset, batch of depth estimation, and batch 2 of pseudo-labels generated from experts on image-text dataset, and so on. We observe that tasks that has significant labeled data (e.g., COCO) improves the downstream tasks as expected (c.g., sogmentation and detection in Table 7 However, tasks with limited data (e.g., approx. about 1500 samples from NYU depth data) deliver worse performance compared to pseudo-labeling alone. These results suggests that pseudo-labeling is effective.
10

---

Published in Transactions on Machine Learning Research (06/2024)
6 Conclusion
As the field of machine learning research embraces openness, growing number of specialized expert models become publicly available. Our study showcased the potential of leveraging these publicly available expert models to enhance CLIP's visual representations, all without the necessity of collecting task-specific data. Our experiments revealed that CLIPTeX yields improvements across a wide range of tasks, highlighting its versatility and effectiveness.
References
Guillaume Alain and Yoshua Bengio. Undcrstanding intermediate layers using lincar classifier probos, 2017. URL http://pwwevieeeeee.eefrrrrr/d//
Gwangbin Bac, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the alcatoric uncertainty in surface normal estimation. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 13137-13146, 2021.
Shariq Farooq Bhat, Reiner Birkl, Diana Wolk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CV /EV/ international conference on computer vision, pp. 9650-9660, 2021.
Rich Caruana. Multitask learning. Machine learning, 28:41 75, 1997.
Giang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Con- ference on Computer Vision (ECCV), September 2018.
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Y Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreeviet http://parrrriiii..eeettrrrrriia net/forummid-
tJFaqsS22I3
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niessner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In 2017 IEEE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 2432 2443, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10.1109/CVPP PCV... 2017.261.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Min- neapolis, Minnesola, June 2019. Association for Computational Linguisties. doi: 10.18653 1011865/vv1NN19-11 URL htttp::/accaattholl.e http://ccaaaaooooooooooo//////// org/N19-112/
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL http:://ppppaeeeeeeeeeee//rrrnnp
Mark Everingham, Lue Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-33 338, 2010. URL hhttps://aaa sew..tissmmaticcolllarr

---

Published in Transactions on Machine Learning Research (06/2024)
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Gian- nis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mchdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Thirty-seventl Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreviiew http:/pwneweeiiia.teoooooorrrrta wottfoorunnidddaaaNNNY.
G. Ghiasi, B. Zoph, E. D. Cubuk, Q. V. Le, and T. Lin. Multi-task self-training for learning general representations. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 8836- 8845, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society. doi: 10.1109/ICCV48999
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Y Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In Computer Vision ECCV 2022: 17th European Conference, Tel Auin, Israel, October 23 27, 2022, Proceedings, Part XXXVI, pp. 540 557, Berlin, Heidelberg, 2022. Springer-Verlage ISBN 978-3-031-20058-8. doi:
31.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988, 2017. doi: 10.1109 10110/9CCVVV... 2011...2.........22.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000 16009, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi org/ 100.5221/zeeooo .01tmoddo.5177733.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana. Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- languag representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprinl orXiv:2304.0260 2023.
Dong- Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chumyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022a.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming Hc. Exploring plain vision transformer backbones for object detection. In European Conference on Compuler Vision, pp. 280 290-296.22 Springer, 2022b.
T. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 936 944, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10.1199/CVPR.2017.006 URL https://dor htt.ddttitteettmmtmorrrccceete
1199///PR./21177 106
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCT 2014: 19th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 750-755. Springer, 2014.
12

---

Published in Transactions on Machine Learning Research (06/2024)
S. Liu, E. Johns, and A. J. Davison. End-to-end multi-task learning with attention. In 2019 IEEE/CVE Conference Computer Vision and Pattern Recognétion (CVPR), pp. 1871-1880, Los Alamitos. CA, USA, jun 2019. IEEE Computer Society. doi: 10. 10.1099/CVPR.229.00005 URL https://doi.
iececonputersociety .aryy...r wtt//0..1///0000 1109/CVPR.. 2019.0019)
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Prismer: A
vision-language model with multi-task experts. Transactions on Machine Learning Research, 2024.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegody, Scott Reed, Cheng-Yang Fu, and Alexan- der C. Berg. SSD: Single Shot MultiBox Detector. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 21-37, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46448-..
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Represendations, 2022.
Sachin Mehta, Saeid Naderiparizi, Fartash Faghri, Maxwell Horton, Lailin Chen, Ali Farhadi, Oncel Tuzel, and Mohammad Rastegari. Rangcaugment: Efficient online augmentation with range lcarning. arXiv preprint ar Xix::2212.111 10558, 2022.
I. Misra, A. Shrivastava, A. Gupla, and M. Hebert. Cross-stitch networks for multi-task learning. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3994-4003, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. doi: 10.1109/CVPR.2216.433 URL https:
7/doi www.eompttoooocceito.....oo owg/oo..//.o 10. 1109/CVPR. 2016 2016. 433
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia, Geonet: Geometric neural network for joint depth and surface normal estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 283 291, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive lision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5571 5584, 2023.
R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datasel transfer. IEEE Transactions on Pallern Analysis and Machine Intelligence, 44(03):1623-1633 mar 2022. ISSN 1939-3539. doi: 2..1000/TAAM..200..000 3009997..
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Pro- ceedings of the IEEE/CVF international conference on compuler vision, pp. 12179 12188, 2021.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. 17X.v:1706.0500 05098. 2017.
arXiv preprint
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathoesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-255 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large- scale dataset for training next generation imago-text models. Advances in Neural Information Processing Systems, 35:25278 22622,,,2294, 2022.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hy- pernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, Mel- bourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ve / /P18 1238. URL http://accaattooooooorr////////
Mustafa Shukor, Corentin Dancelte, Alexandre Rame, and Matthieu Cord. UnIVAT: Unified model for image, video, audio and language tasks. Transactions Machine Learning Research, 2023, ISSN 2835- 8856. URL htttps://openreeieet http://porrooiio...eeeerrrrrrrra wttp.eeee/orrmmmmo0ddaa11a000000
Nathan Silberman, Derek Hoiem, Pushmeeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp.
Springer, 2012.
Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Luc Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu. Shift: a synthetic driving dataset for continuous multi-task domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21371-21382, 2022.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adasharr: learning what share for efficient deep multi-task learning. In Proceedings of the 34th International Conference Newral Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829544
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanprect Singh, Adina Williams, Douwe Kicla, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238 1888-524,, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint Xrx::23302 139711 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaisor, and Illia Polosukhin. Atlention is all you need. Advances in neural informalion processing systems, 30, 2017.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architecturos, tasks, and modalitics through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pp. 23318 23340. PMLR, 2022.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling, In European Conference on Computer Vision, pp. 521 539. Springer, 2022.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 78, 2014. doi: 10.1162/tacl a _0166. URL https
//aclanthology.o www.oolooyyorgg//4/./ org/Q14-1006
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text loundation models. Transuctions on Machine Learning Research, 2022. ISSN 2835-8856. URL httpp:///penreeveew. httpp////ppppriiiieeeeeeerrrraap wwt.:/eorrr///--//77733
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-langues under- standing. Advances in Newral Information Processing Systems, 35:36067 36080, 2022.
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multinodal learning. arxiv preprint arXiw:207. 10802, 2023.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya. Jia. Pyramid scene parsing network.
In Proceedings of the TEEE conference on computer vision and poltern vecognition, pp- 2881-2890 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. TEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
15

---

Published in Transactions on Machine Learning Research (06/2024)
A Hyperparameters
Hyper-paramen parameters used during training and probing CLIPTeX and other models are given in Table and Table respectively.
For selecting Aelip and Atak (where task = {depth, seg, surface normal) in Eq. (I linear search was used. The value of Aeli and Atakk was chosen as 0.1, 1.0, and 10 for cach of the tasks. We found that Actip A 1.0 worked well for all tasks except segmentation where we found A 0.1 delivered the best or close to the best performance. So, set these lyper-parameters in experiments.
B Additional results for pseudo-labeling on ADE20k
Table 10 shows the results for the task of semantic segmentation on ADE20k. Similar to observations in Table observe that incorporating knowledge from all experts contributes to learning better visual
representations.
Table 8: Hyper-parameters for training CLIPTeX on CC3M dataset.
Plyper-parameee
Value
Epochs
30
LR scheduler
cosipe
Warmup Steps
1000
Warmup Init LIt
Ic-06
Maximam
36-05
Minitum LR
1c-06
Batch size
32
Anptt
Amp
1.0
Aue
0.1
sormal
1.0
Table 9: Hyper-parameter: used for probing on different downstream tasks.
Segmennation
Detection
Depth
Surface Normal
Classifictton
Linoer
Desplab33
PSPNNs
Mask R-CNN
Duplain3
PSPNot
Linnar
Drsplabc3
PSPNot
Epochs
50
200
50
50
50
LR scholake
casine
cosine
casine
uailisstep
cosiue
conine
tosine
cceine
cssine
(06l:0
easine
Warmup
5000
500
250
5000
1300
1000
Warmup-
16<16
Masimun LR
de-15
de-10
3o45
3o-13
96-11
1c-04
Ic:04
Ic-04
164G
1c406
Minimum LR
3e-06
3e-06
36-36
NA
Io-06
I--6
1e-06
In-06
16:0
16436
1o-06
1646
Mikestomss
NA
NA
24]
NA
NA
LR Gamman
NA
NA
0.1
NA
NA
NA
Botch sine
32
32
32
16
16
16
16
16
125
Table 10: Role of pseudo-labels from experts in CLIPTeX training on ADE20k segmentation
dataset. For other tasks, see Table 5
Expert
Segmentation (1)
Segmentation Depth Surface Normal
Linear
PSPNet
26.00
38.80
x
X
31. 11.16
39.96
X
27.36
38.73
/
27.16
30.74
30.83
39.78
/
26.86
38.70
29.26
39.70
16


Reviews:
[{"review_id": "NQAmNWWS1d", "reviewer": "Reviewer_zknV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The proposed idea is intuitive and straightforward.\n2. The paper is overall easy to follow.\n3. Some of the results look promising.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors proposed a method called CLIPTeX, where the core idea is to leverage existing models of various vision tasks to provide additional supervision to the training process of CLIP. The authors demonstrated that the resulted representations provide both strong zero-shot classification ability like CLIP and a significant improvement in terms other downstream visual tasks compared to the vanilla CLIP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The major concern of the reviewer is that the reviewer doesn't see from the existing manuscript why such a representation is desired. In half of the downstream tasks, the proposed method doesn't significantly outperform the CLIP-FT baseline which simply enjoys a higher resolution finetuning. The reviewer would suggest the authors to further showcase that the marginal improvement on half of tasks could show a more significant impact on some important applications of the proposed representation to really demonstrate the importance and the necessity of the proposed method.\n\n2. The current experiment setting could use further comparison to gain more insights.\n\na. Despite of the use of teacher models relax the requirement of manual supervision, it is very interesting to compare with a variant that is trained additionally with the manually labeled data that is used to train the teachers.\n\nb. More analysis on how the knowledge distillation could be done is needed. Currently the results are based on tasks specific losses calculated with task-specific pseudo labels, but it is unclear whether other forms of supervision method could possibly work better. For example, unifying the loss formulation into contrastive loss by pushing the representation of the CLIP to be similar to the representation of task specific teachers.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2OCVZ9Uslq", "reviewer": "Reviewer_rFy4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well written and easy to follow.\nVisualizations and diagram are easily readable.\nThe introduced approach is sound.\n\nWeaknesses:\nThe validation could be strengthened (see comments below).\n\n\nThe introduced idea is a rather straight forward extension to the vanilla CLIP training objective. However, overall, I like the paper and the introduced idea. The presentation is very clear, and the reported results look solid. My main concerns are around validation:\n\n- First, the CLIP model is only trained using CC3M. How about CC12M or YFCC15M? Would we still see benefits of the approach if we were to train the CLIP model on LAION scale data? Adding larger datasets would make the comparison stronger and more convincing. \n\n- Second, CLIP models are oftentimes validated on large sets of datasets and not only on ImageNet (e.g. see tab 10 in https://arxiv.org/pdf/2103.00020.pdf). Adding more evaluation datasets would make the point about not affecting the original CLIP performance stronger.\n\n- Third, adding more CLIP implementations in comparisons would strengthen the paper, e.g. see OpenCLIP and MetaCLIP", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension to the vanilla CLIP training objective – it extends the training with additional objectives (e.g. segmentation, detection, etc). To obtain the pseudo labels needed to train the model, the paper proposes to leverage open-sourced foundation models. The reported results show that the new training objective leads to representations that obtain substantial improvements on tasks such as segmentation, detection, depth estimation without scarifying ImageNet linear probing results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Ablation of /lambda hyperparameters introduced in eq. 1 is missing. Please add an ablation.\n\n- Please add a citation to CLIP-FT model the first time it appears in the Intro and when describing the baselines.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HDQsKOREGk", "reviewer": "Reviewer_g5Ej", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nPaper largely well written and easy to follow.\nPresented approach leads to an improvement for downstream tasks compared to the baselines without multitask training (which seems expected given that CLIPTeX can benefit from task-specific models trained with labelled data)\nZero-shot capabilities are preserved (no catastrophic forgetting through finetuning)\n\n\n### Weaknesses\nPaper claims that the enhancement is obtained without any labelled data collection. However, the used experts are trained with supervised data, meaning that for the obtained results, labelled data was necessary. This should be clearer distinguished in the paper. \n\nExperiments are only performed wrt. to baselines but not in comparison to other multi-task training methods e.g., such as Unified-IO (Lu et al., 2022).\nFurther, why is it better to use pseudo-labels from a model which as been trained with supervision, instead directly using the original labelled dataset these models have been trained on? How far can we get with unsupervised trained experts? \n\nMissing details:\n- How long was the training on the CC3M dataset (on what kind of hardware)?\n- Results on ADE20k dataset for the ablations can be added to the appendix (instead of just stating that similar trends are observed).\n\nReferences need to be completed/updated:\n- Mulitple references are lacking the venue, e.g., \"Unified-io: A unified model for vision, language, and multi-modal tasks\", \"Cross-stitch networks for multi-task learning\", ...\n- Good practice to reference the conference/journal version of the paper instead of arxiv. E.g., \"Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer\" has been accepted to TPAMI, \"SSD: single shot multibox detector\" at ECCV, ...\n- There are duplicate references (\"Rangeaugment: Efficient online augmentation with range learning)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to improve the image encoder of CLIP for various downstream tasks beyond classification such as semantic segmentation, object detection, or depth estimation (among others). These tasks require improved localization capabilities which the original CLIP is lacking. The described approach suggest to finetune the visual representations of CLIP by extending it with different task heads whichare trained with pseudo-labels obtained by various expert models for different downstream tasks (final model called CLIPTeX). The experiments show that the finetuned visual representation perform better than the baseline (without multi-task training) for the different visual downstream tasks. Different from existing work, the proposed method uses multiple, diverse experts which are publicly accessible, instead of labelled datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The claim, that the presented method does not require collecting any task-specific data should be clarified given that it uses models trained with task-specific labelled data (such as COCO, ScanNet, ).\n\nThe experiment section could highly benefit from additinonal comparisons (see weaknesses above for examples, e.g., pseudo-labels vs. task-specific ground truth, unsupervised experts, existing other approaches)", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I accept the paper with minor revision, given my above assessment that the claim of the paper has been clearly supported by the experimental evidence.

One of the three reviewers leans toward acceptance. One of the three reviewers is on the fence. However, they did not have a chance to see the latest revision of the paper, which came in late and included the additional results incorporated in the pdf. The third reviewer's negative rating is, to my understanding, at least partially based on the concern of the predicted significance/impact, which is not part of the evaluation criteria for TLMR.

For this paper to be accepted, I expect the following changes:
- as g5Ej already noted in the original review, several references need fixing. Unfortunately, this was not fixed in the latest revision. I expect that *all* references are fixed as part of the minor revision, not just those explicitly mentioned by g5Ej
- Fix Formatting and minor errors: E.g., "Table 8: Hyper-parameters for training CLIPTeX on CC3M dataset..". This is bold, but the Table 9 caption is not. The Table 8 caption has to "..". Please check throughout the paper.
Decision: Accept (poster)