Enhancing Vision-Language Model with Unmasked Token Alignment
Authors: Jihao Liu
Jinliang Zheng
Boxiao Liu
Yu Liu
Hongsheng Li
Venue: Accepted by TMLR ()

Abstract: Contrastive pre-training on image-text pairs, exemplified by CLIP, becomes a standard technique for learning multi-modal visual-language representations. Although CLIP has demonstrated remarkable performance, training it from scratch on noisy web-scale datasets is computationally demanding. On the other hand, mask-then-predict pre-training approaches, like Masked Image Modeling (MIM), offer efficient self-supervised learning for single-modal representations. This paper introduces $\textbf{U}$nmasked $\textbf{T}$oken $\textbf{A}$lignment ($\textbf{UTA}$), a method that leverages existing CLIP models to further enhance its vision-language representations. UTA trains a Vision Transformer (ViT) by aligning unmasked visual tokens to the corresponding image tokens from a frozen CLIP vision encoder, which automatically aligns the ViT model with the CLIP text encoder. The pre-trained ViT can be directly applied for zero-shot evaluation even without training on image-text pairs. Compared to MIM approaches, UTA does not suffer from training-finetuning inconsistency and is much more training-efficient by avoiding using the extra $\mathrm{[MASK]}$ tokens. Extensive experimental results demonstrate that UTA can enhance CLIP models and outperform existing MIM methods on various uni- and multi-modal benchmarks.

Published in Transactions on Machine Learning Research (04/2024)
Enhancing Vision-Language Model with Unmasked Token Alignment
Jihao Liu CUHK MMLab
jihaoliu link. cu....ukk..u...k
Jinliang Zheng
2toinf(@ bupt. edu.cn
Institute for Al Industry Research (AIR) Tsinghua University
Boxiao Liu
liuboxiao Cerooaiaa@seesstttmee...mm
Sensetime Research
Liu Sensetime Research
liuyuisanai' @ gmail.com
Hongsheng
hsli ee. cuhkk edu
CUHK MMLab
CPI under ImoHHK
Shanghai Al Laboratory
Reviewed
Abstract
Contrastive pre-training on image-text pairs, exemplified by CLIP, becomes standard technique for learning multi-modal visual-language representations. Although CLIP has demonstrated remarkable performance, training it from scratch noisy web-scale datasets is computationally demanding. On the other hand, mask-then-predict pre-training approaches, like Masked Image Modeling (MIM), offer efficient self-supervised learning for single-modal representations. This paper introduces Unmasked Token Alignment (UTA), method that leverages existing CLIP models to further enhance its vision- language representations. UTA trains a Vision Transformer (ViT) by aligning unmasked visual tokens the corresponding image tokens from a frozen CL.IP vision encoder, which automatically aligns the ViT model with the CLIP text encoder. The pre-trained ViT can be directly applied for zero-shot evaluation even without training image-text pairs. Compared to MIM approaches, UTA does not suffer from training- loaning-iinetnnng inconsistency and is much more training-effician by avoiding using the extra [MASK] tokens. Extensive experimental results demonstrate that UTA can enhance CLIP models and outperform existing MIM methods on various uni- and multi-modal benchmarks.
1 Introduction
Contrastive pre-training, e.g., CLIP (Radford et al. 2021 with web-scale image-text pairs becoming the mainstream technique for learning multi-modal visual-language representations. The pre-trained CLIP model has unlocked the potential of various downstream applications. including zero-shot image classification and retrieval, and high-quality text-to-image generation (Rombach et al.) 2022 Ramesh et al.) 2022). Furthermore, the pre-trained visual and text encoders can be further used for multi-modal and even uni-modal tasks.
Unlike classical supervised learning on the trained on much noisier datasets found on the web such as LAION (Schuhmann et al. (022) and WIT (Radford et al. 202T), and require an extremely large batch size work well. Directly training on those datasets from scratch requires lot of computing resources, making not accessible most researchers. In contrast, the preeeic
classification dataset, CLIP and its variants are typically

---

Published in Transactions on Machine Learning Research (04/2024)
pre-training approaches, e.g., Masked Image Modeling (MIM) (He al. 2021 Xte al. 2021 and Masked Language Modeling (MLM) (Devlin et al., 2019), have been shown to be efficient and powerful way to learn single-modal (visual or language) representations in self-supervised manner and can achieve strong performance by fine-tuning the pre- vre-trained models on downstream tasks. The key design of those methods is to predict the masked tokens from the other visible and unmasked input tokens. We ask the question: can we take advantage of both types of methods and further enhance the vision-land -laguugge representations over CLIP? There are recent works, e-g.. EVA (Fang et al. 2023b), utilizing a pre-trained CLIP model for generating the prediction targets for MIM. The resulting vision models show stronger performance than the encoders pre- pre-trained using either only MIM only CLIP, demonstrating the effectiveness of combining MIM and CLIP for multi-modal feature learning. However, those methods are limited to learning single-modal representations, and extra contrastive fine-tuning is needed for multi-modal feature learning, as proposed in EVA-CLIP (Sun et al. 2023).
In this paper, we propose an efficient method, Unmasked Token Alignment (UTA), for enhancing the alignment between vision-language representations, which better utilizes existing pre-trained CLIP models. In particular. our method trains Vision Transformer (ViT) (Dosovitskiy et al. 2021) model from scratch by using the unmasked and sparse visual tokens to align with corresponding image tokens of a frozen CLIP model. For the model, we randomly mask portion of image tokens with a reversed masking strategy, where only the unmasked (i.e. kept) tokens (including the [CLS] token) are inputted into the ViT model and aligned with the output of the frozen CLIP visual model. We maximize the cosine similarity for token alignment, and therefore, the ViT model is automatically aligned with the CLIP text encoder in the normalized embedding space.
ViT
There are two major advantages of using the proposed unmasked token alignment strategy. 1) After pre-training the vision model, we can directly conduct zero-shot classification and retrieval using the normalized features of the trained VIT model and the CLIP text encoder. We illustrate the pre-training and fine-tuning pipeline of UTA in Fig. [ In contrast, the masked prediction objective used in existing MIM works (EVA (Fang et al. 2023b), BEiT-3 (Wang et al. )022bb) relies on the [MASK] tokens to predict the CLIP features while the unmasked tokens are not trained to align with the CLIP model as we do. They do not support zero-shot evaluation without contrastive fine-tuning as only the unmasked tokens are used for zero-shot evaluation. 2) MIM works suffer from the
inconsistency as a large portion of [MASK] tokens never appear during the fine-tuning. In contrast, our approach better maintains the finiiinnniietuiingggg. consistency by only inputting and aligning the unmasked tokens, which processed both in training and inference. We also empirically find that further adding the masked prediction objective our UTA results in much worse zero-shot performance.
Compared to the existing MIM approach that relies on the [MASK] tokens to predict the CLIP features with the masked prediction objective, our method is conceptually simple and computationally efficient by avoiding introducing the [MASK] tokens, which can reduce the training FL.OPs for up to 50%. But at the same time, our pre-trained models are also suitable for fine-tuning on downstream uni-modal or multi-modal tasks. In particular, our pre-trained ViT-L obtains 78.5% zero-shot accuracy on ImageNet without contrastive fine-tuning from image-text pairs. After fine-tuning with the DataComp- IB dataset (Gadre et al. 2023), we obtained 80.8% zero-shot accuracy on ImageNet, surpassing the DataComp baseline and EVV-002CLIIP by 1.6% and 1. 1.0%, respectively. On the more recent multi-modal benchmark, i.e.,
(Liu et al. 2023), we outperform CLIP and EVA-02 by 2.2% and 1.4%, respectively. We also fine-tune the pre-trained vision model on object detection and segmentation tasks and demonstrate better results than the competitive EVA-02 (Fang et al. 2023a) models on those tasks.
2 Method
In this section, we first review the widely used Masked Image Modeling (MIM) pre-training and its more advanced version equipped with a pre-trained CLIP model. We then introduce the Unmasked Token Alignment (UTA) approach and its implementation.
2.1
A Revisit of Masked Image Modeling with CLIP
MIM methods (Bao et al. [002 He et al.)
2021 Xie et al. 2021) typically use a Vision Transformer (ViT) (Dosovitskiy et al, (02T) for pre-training An input image first divided into non-overlapping image patches, which are converted into sequence of tokens with project layer and positional embedding. Then a portion of the tokens are randomly
2

---

Published in Transactions on Machine Learning Research (04/2024)
Masked Image Modeling (EVA)
Unmasked Token Alignment
Downstream Tasks
masked token
image
unmasked token
image
image
text
Vision Encoder
CLIP Vision Encoder
Vision Encoder
CLIP Vision Encoder
Vision Encoder
CLIP Text Encoder
prediction
alignment
uni-modal
multi-modal
Figure 1: Overview of Unmasked Token Alignment (UTA). During the pre-training of UTA, only the unmasked tokens are inputted into the vision encoder and aligned with the CLIP vision encoder. After pre-training, the pre-trained vision encoder is automatically aligned with the CLIP text encoder and can be directly applied for the zero-shot evaluation even without contrastive training on image-text pairs. The pre-trained vision encoder can be further fine-tuned for uni-modal or multi-modal downstream tasks.
sampled, where the masked tokens are filled with a special [MASK] token. The masked image is processed by the ViT to produce the latent representations, and lightweight head is utilized predict the original image based on the latent representations. After pre-training, the VIT is used for further fine-tuning downstream visual tasks.
Some recent papers (Peng et al. 2022 Fang et al. of a pre-trained CLIP model as the reconstruction targets and achieve much better performance than methods using the low-level pixels as the targets (He et al. 202I Xie et al. 202I). In particular, the unmasked image is fed into the visual encoder of the CLIP model for obtaining the full image's hidden feature map. The masked prediction objective to align the predicted feature with the CLIP's visual feature on the masked tokens.
2023b Hou et al. 2022 Xiao et al., 2022 utilize the hidden features
2.2 Unmasked Token Alignment
Using the masked prediction objective to align a train-from ftromssrrach ViT model with the pre-trained CLIP visual model still uses the problematic [MASK] tokens. causes training- finttuning inconsistency and makes the trained ViT unable to perform zero-shot classification without fine-tuning. To tackle the issue, we propose simple yet effective solution that does not utilize the extra [MASK] tokens. We align the feature maps of the two models with dense distillation objective, where the feature maps of the train-from-scrale scratch ViT model and CLIP vision encoder are obtained with a partial view and a full view, respectively. Specifically, given an input image, we use a random mask to mask a portion of image tokens. Unlike previous works that use the [MASK] tokens to fill in the masked patches, we directly drop the masked tokens and only input the rest tokens into the ViT encoder. For the pre-trained CLIP model, we input the original image and obtain full hidden feature map. Then select the corresponding unmasked (kept) tokens from the CLIP vision encoder's feature map, which are used as the targets for the teo---from-tattte VIT encoder.
The cosine similarity is maximized for the token alignment. After pre-training, the VIT encoder is aligned with the CLIP vision encoder in the normalized embedding space. Therefore, the ViT encoder is also aligned with the CLIP text coder as the CLIP's vision and text encoders share the same embedding space. As a result, we can directly conduct the zero-shot evaluation with the pre-trained ViT encoder and CLIP text encoder even without training on the image-text pairs. We show that we can already achieve decent zero-shot performance after the unmasked alignment.
Reversed biock-wise masking. Previous works (Bao et 2021) typically use block-wise masking to preserve the structure of input images. However, we note that such masking is spatially uncqualized, which tends to mask the center area of the images with much higher probability, and as result, the tokens in the border area are trained much more times than tokens in the center area. We introduce a reversed block-wise masking strategy, which first generates a mask with block-wise masking and then randomly reverses the mask with a probability of 0.5. Our masking strategy preserves the structure of the input images and also alleviates the spatial unequalization problem.
Pre-training efficiency analysis. As we do not need to process the extra [MASK] tokens during the pre-training, we can largely improve the masked training efficiency. In practice,
large mask ratio, e.g., 0.5, for pre-training.
3

---

Published in Transactions on Machine Learning Research (04/2024)
Thus, compared to EVA (Fang et al. 2023b) BEiT v2 (Peng et al. 2022) which require inputting extra [MASK] tokens, our UTA can reduce the training FL.OPs by 50%.
2.3 Implementation
Vision transformer architecture. We follow EVA-02 (Fang et al., 2023a) to introduce architectural modifications on vision transformer for improving the performance and training stability. In particular, we add extra relative positional embedding introduced by Suu et al. 2021) in the self-attention layer. We replace the original feedforward network (FFN) in vision transformer with the SwiGLU variant introduced by |Shazeer (2020). Moreover, we add an extra LayerNorm (Ba al., 2016) layer in the FFN stabilize the training as proposed by Wang et al.) (2022a). Note that those architectural modifications are not contributions of our method. We refer the readers to EVA-02 Fang al.
2023a for the detailed performance comparison of these modifications.
CLIP teacher model. Instead of using original CLIP models for pre-training, we follow Fang et al. (2023a) to use petter-performing CLIP model, i.e., giant- sized EVA-01-CLIP model (Sun et al. (023). for providing the alignment targets during pre-training. Our experiments show that the stronger CLIP model can bring large zero-shot accuracy improvements. Additionally, we find that our ViT-L model can surpass the giant-sized CLIP models (e.g., Open-CLIP and -V--II-CLIP) after contrastive fine-tuning as shown in Tab. [
3 Experimental Setup
To demonstrate the effectiveness of the proposed Unmasked Token Alignment (UTA), we conduct experiments to pre-train ViT to align with CLIP vision-language representation on large-scale dataset and apply the pre-trained models to downstream multi-modal and uni-modal tasks. The multi modti tasks include zero-shot classification, zero-shot retrieval, and the more recent ILL (Liu al., 2023). The uni-modal tasks include ImageNet classification (Deng al. 2009), object detection, and segmentation.
Pre-training. All VIT models are pre-trained on ImageNet-21KK 21 (Deng et al. 2009) dataset using 224x224 input resolution and patch size of 14. Unless otherwise specified, we pre-train for 150 epochs with batch size of 4096. We use Adam W (Loshchilov & Hutter 2017) optimizer with weight decay of 0.05. The learning rate is linearly increased to x,5x 3 with epoch of training and decays to 10-5 with cosine schedule (Loshchilov & Hutter 2016). By default, we use reversed block-wise masking with mask ratios of 0.4 and 0.5 for base and large models, respectively.
Contrastive fine-tuning, Although the pre-trained ViT model can already demonstrate excellent zero-shot capabilities even without contrastive fine-tuning, we also perform much shorter contrastive fine-tuning similar to other CLIP counterparts to further improve its zero-shot performance, especially for the out-of- -istrriiiint tasks. In particular, we initialize the vision and text encoders with the pre-trained ViT model and CLIP text encoder. Then we perform contrastive fine-tuning on the DataComp- 1B dataset (Gadre et al. 2023). The temperature parameter in the contrastive loss (Radford et al. (02T) fixed to 0.01 during our training as initially the vision encoder and text encoder are already aligned. To be comparable with other methods (Cherti al. 2023), we use patch size of 16 for fine-tuning UTA-B.
Fine-tuning, For evaluation on the LLaVA-Bench (Liu et al.
2023) and uni-modal tasks, only keep the pre-trained VIT. On LLaVA-Bench, we follow the default settings to first train a projection layer on CC-3M dataset (Sharma et al. for feature alignment and then fine-tune the project layer and Large Language Model (LLM) (Chiang et al. 2023 on ILL 150KK dataset (Liu et al. 2023). For object detection and instance segmentation tasks, we adopt the Cascade Mask R-CNN (He et al. 2017 Cai & Vasconcelos 2019) framework and separately fine-tune on the COCO (Lin al. 2014) and LVIS (Gupta et al.] (019) datasets. For semantic segmentation task, we adopt the
2018
UperNet (Xiao al. 2018) framework and fine-tune on the ADE20K (Zhou al., 2017) dataset. Please refer to the
appendix A. for more detailed configurations.
4 Main Results
In this section, we compare the proposed Unmasked Token Alignment (UTA) to prior arts on various benchmarks. We first conduct comparisons between UTA and previous zero-shot results in Sec. 4.1 We then compare UTA with other
4

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Zero-shot classification performance on ImageNet-1K (IN-IK), ImageNet-A (IN-A) (Hendrycks et al. 202 ImageNet-R (IN-R) (Hendrycks al., 2021a), ImageNet-Ve V2 (IN-V2) (Recht et al. 2019). ImageNet-Sketch (IN-
S) (Wang al., 2.19). and ObjectNet Barbu al., 2 We also report the average accuracy over the 6 datasets.
Following Sun al. 2023), the numbers of I-T pairs do not account for the I-T pairs used to train the teacher model.
Method
Model
# 1-T Pairs
IN-IK
IN-A
IN-R
IN-V2
IN-S
ObjectNet
Average
CLIP
B/16@224
13B
68.3
50.0
77.7
61.9
48.2
55.3
60.2
Open-CLIP
B/16@224
34B
70.2
38.2
80.6
62.3
56.1
56.0
60.6
EVA-02-CLIP
B/16@224
8B
74.7
54.1
82.5
67.0
57.7
62.3
66.4
UTA B/14@224
OB
76.0
54.2
76.7
68.1
52.5
63.6
65.2
UTA
B/16@222
2B
77.0
59.8
84.1
69.5
60.2
68.3
69.8
CLIP
L/14@224
13B
74.0
48.0
86.5
66.4
61.8
61.1
66.3
Open-CLIP
1/14@224
32B
75.5
70.8
87.8
69.9
59.6
69.0
72.1
DataComp
L/14@224
13B
79.2
69.6
90.8
72.1
68.0
74.3
75.7
EVA-02-CLIP
L/14@224
4B
79.8
76.1
92.7
72.9
68.1
75.3
77.5
UTA
L/14@224
OB
78.5
69.4
89.4
71.7
63.9
72.7
74.3
UTA
L/14@224
4B
80.8
79.1
92.3
73.7
68.4
77.6
78.6
CLIP
L/14@336
13B
76.6
77.5
89.0
70.9
61.0
72.0
74.5
EVA-02-CLIP
L/14@35
6B
80.4
82.9
93.2
73.8
68.9
78.4
79.6
UTA
L/14@ 336
4B
81.4
84.2
92.9
74.6
69.1
80.1
80.4
Open-CLIP
g/14@222
34B
78.5
60.8
90.2
71.7
67.5
69.2
73.0
EVA-OI-CLIP
g/14@224
11B
79.3
74.1
92.5
72.1
68.1
75.3
76.9
UTA g/14@224
OB
79.3
73.5
91.6
72.6
66.7
74.6
76.4
UTA
g/14@224
2B
81.5
81.9
93.5
74.8
69.6
79.7
80.2
UTA
g/14@336
3B
83.9
87.5
94.5
76.9
71.6
81.9
82.7
pre-training methods on LLaVA-Bench Sec. results on core vision tasks Sec.
4.2
To show the transferability of UTA, we present the transfer learning
4.1 Zero-Shot Results
We conduct zero-shot classification and retrieval and compare the results with other CLIP variants (Radford et al. 2021 Cherti et al. 2023 Sun et al. 2023). In Tab. we show that the pre-trained ViT-B model can obtain 76.0% zero-shot accuracy on ImageNet-1e 1K even without training on image-text pairs. After fine-tuning with only 2B image-text samples, our ViT-B obtains 77.0% zero-shot accuracy on ImageNet- 1K, surpassing Open-CLIP (Cherti al. 2023 and EVA-02-CLIP (Sun et al. 2023) by 2.3% and 1.0% respectively. On the challenging ObjectNet (Barbu al. 2019) dataset, outperform Open-CLIP CLIP and EVA-02-CLID by 11.3% and 6.0% points respectively. Our pre- pre-trained VIT-L model obtains 78.5% zero-shot accuracy on ImageNet- 1K. After fine-tuning with 4B samples, we achieve 80.8% accuracy, which outperforms Open-CLIP and EVA-02-CLIP by 5.3% and 1.0% respectively. Compared to strong EVA-02-CLIB we achieve an average 1.1% improvements over 6 evaluation datasets. We also fine-tune with x36x3336 input resolution using 200M samples, and we obtain an average of 1.8% points improvements on the evaluation datasets. We find that fine-tuning on the larger but noisier DataComp- 1B dataset (Gadre et al. 2023 can greatly boost the performance the ImageNet robust variants.
Table 2 presents the zero-shot retrieval results the Flickr30k (Young et al. 2014 and COCO (Lin et al. 2014 datasets. We find that the pre-trained model can already outperform other CLIP models on all evaluated metrics. In particular, the base model improves the Open-CLIP and EVA-02-CLIP by an average of 4% top-1 recall over the two datasets. For the large model, we improve the Open-CLIP and EVA-02-CLIP by an average of 3.4% and 1.8% top-1 recall, respectively. We also find that further fine-tuning on DataComp- IB dataset can improve the text retrieval performance but also degenerate the image retrieval performance, which may be related the filtering strategies used for DataComp-1B (aarre et al. 2023

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Zero-shot retrieval performance on Flickr30k (Young et al. 2014 and COCO (Lin al. 2014). R 1, R R and R @ 10 denote the recall performance among top-1, top-5, and top- 10, respectively, Following Sun numbers of I-T pairs do not account for the 1-T pairs used to train the teacher model.
2023 the
Text Retrieval
Image Retrieval
Method
Model
# I-T Pairs
Flickr30k R@5
COCO
Flickr30k R@5
COCO
R@1
R@
R@
R@5
R@
10
R@
R@ 10
R@
R@5
R@ 10
CLIP
13B
81.9
96.2
98.8
52.4
76.8
84.7
62.1
85.6
91.8
33.1
58.4
Open-CLIP
86.3
97.9
99.4
59.4
69.8
90.4
94.6
42.3
66.7
EVA-02-CLIP
85.7
96.7
98.9
58.7
88.2
71.2
91.0
94.7
42.4
66.9
76.3
UTA
88.4
98.5
99.5
63.4
83.9
90.0
75.5
93.1
96.4
46.8
71.5
80.8
UTA
2B
91.3
98.9
99.7
64.7
85.0
90.5
74.5
93.1
96.0
45,9
70.5
79.3
CLIP
13B
85.2
97.3
99.0
56.3
79.3
86.7
65.2
87.3
92.0
36.5
61.0
71.1
Open-CTII
34B
88.7
98.4
62.1
83.4
90.3
75.0
92.5
95.6
46.1
70.7
79.4
DataComp
89.5
98.6
99.7
63.3
90.4
73.4
91.7
95.4
45.7
70.0
EVA-02-CLIP
4B
89.7
98.6
99.2
63.7
84.3
90.4
77.3
93.6
96.8
47.5
71.2
79.7
UTA
OB
91.2
98.7
99.8
66.6
86.5
91.5
78.3
94.1
96.9
49.5
73.4
81.9
UTA
4B
93.0
99.0
99.7
66.5
86.9
92.2
77.4
93.8
96.6
48.7
72.3
80.9
Open-CLIP
34B
91.4
99.2
99.6
66.4
86.0
91.8
77.7
94.1
96.9
48.8
73.3
81.5
EVA-OI-CLIP
91.6
99.3
68.2
87.5
78.9
94.5
50.3
74.0
82.1
92.2
99.7
68.0
92.4
79.0
94.5
50.3
74.2
UTA
2B
93.2
99.4
99.8
68.2
87.6
93.0
78.2
94.4
96.7
48.7
72.9
81.1
Question: What is the position of the skateboard in the image?
EVA: The skateboard is on the ground, with the person standing top it.
UTA: The skateboard is positioned upright, with the wheels off the ground, and the deck facing upwards.
Question: What is the man sitting in the middle doing in the image?
EVA: The man in the image is sitting down, holding a glass of beer, and making a gesture or a sign with his hand.
UTA: The man in the image is sitting down, talking on his cell phone, and holding his hands up while doing so.
Figure 2: Qualitative examples generated by LLaVA models fine-tuned with EVA-02 and UTA.
4.2
Multi-Modal Results
The emergent multi-modal capabilities of GPT-4 (OpenAl 2023) have attracted widespread attention, and there are various re-implementations of such capabilities using open-sourced vision and large language models (Liu et al. 2023 Zhu et al.) 2023). We adopt the LLaVA framework Liu et al. (023) and evaluate pre-traine pre-trained models the VA- -A-BencBh Liu et al. 2023). The results are presented in Tab. 3 Note that all the results are obtained by fixing the vision encoders' parameters, which can directly reflect the representation quality of the pre-trained model. Notably, our model achieves the best results in the overall category. Compared to the original CLIP large model (Radford et al. 202T). we overall obtain an improvement 2.2% accuracy. Using the same pre-training dataset and iterations, we also outperform EVA-02 (Fang et 2023a) for 1. 1.4%. We compare the outputs generated by the two LLaVA models and highlight the difference Fig. We show that our approach can capture more fine- mnniina details produce better answers.

---

Published in Transactions on Machine Learning Research (04/2024)
Table 3: Results on LLaVA-Bent -ench (Liu et al. 2023). The results of CLIP and EVA-02 are obtained by our re- implementation with official checkpoints. Conversation, Detail, and Reasoning represent the performance of conversa- tion, detailed description, and complex reasoning, respectively. The overall result summary of the results of the three categories.
Method
Model
Conversation
Detail
Reasoning
Overall
CLIP
B/16
74.5
69.9
90.3
78.3
EVA-02
B/16
75.3
61.1
91.8
76.2
UTA
B/16
80.8
66.2
88.8
78.8
CLIP
L/14
78.7
70.4
90.0
79.8
EVA-02
L/14
80.4
71.6
91.1
80.6
UTA
L/14
81.4
72.2
91.8
82.0
EVA-01
g/14
79.9
72.2
91.0
80.8
UTA
g/14
84.1
71.3
93.5
83.1
Table 4: ImageNet classification and ADE20K segmentation results. 2S and FT denote the zero-shot and fine-tuning top-1 accuracy on ImageNet respectively. f denotes the model after contrastive fine-tuning. $ indicates the results equal to random choice without contrastive fine-tuning.
ImageNet
ADE20K
Method
Model
#Params
Input Size
ZS
FT
Input Size
mIoU
MAE
B
86M
224
83.6
512
48.1
BEiT v2
B
86M
224
85.5
512
53.1
CLIP
B
86M
224
68.3
85.7
EVA-02
B
86M
224
0.1
87.4
512
55.3
UTA
B
86M
224
76.0
87.5
512
55.6
UTA
B
86M
224
77.0
87.4
512
55.1
MAE
L
304M
224
85.9
512
53.6
BETT v2
L
304M
224
87.3
512
56.7
CLIP
L
304M
224
74.0
88.0
EVA-02
L
304M
224
0.13
89.0
512
58.3
UTA
L.
304M
224
78.5
89.2
512
58.8
EVA-01-CLIP
g
1011M
224
79.3
89.1
512
57.4
4.3 Core Vision Task Results
2021 demonstrate that the MIM pre-trained models have superior performance after fine-tuning to downstream tasks, including ImageNet classification, object detection, image segmentation, etc. There are some recent papers (Xie et al. 2023) that show the mask-then-peee then-predict objective is the key to such fine-tuning capabilities. our empirical evaluation, we show that our UTA pre-training also has such capabilities.
Prior arts (Bao et al. 2021 He et al.
We present the results of ImageNet classification in Tab. 4 Compared recent MIM works (e.g., BETT v2 (Peng al. ) which also utilize pre-trained CLIP model for pre-training, fine-tuning. We can also largely outperform the CLIP model for both the zero-shot and fine-tuning accuracy. Compared with EVA-02, although we slightly improve the fine-tuning accuracy, we can largely improve the zero-shot accuracy.
obtain improvement of ~2% points after
We show the results of performing object detection and instance segmentation on COCO and LVIS datasets in Tab. 5 Compared the MAE pre-training (He et al. (0221), we find our UTA can improve the Apbox for more than 1% mAP on COCO and 6% mAP on more challenging LVIS. Additionally, our approach also performs better than EVA-02, which demonstrates 2.0% and 0.6% mAP improvements LVIS for the base and large models respectively.
7

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: Object detection and instance segmentation results on COCO and LVIS datasets. i denotes the model after contrastive finentuning.
#Enc. Params
COCO
LVIS
Method
Model
Apbox
Apmask
Apboo
A A.maak
ViTDet
86M
54.0
46.7
43.0
38.9
EVA-02
B
86M
55.5
47.1
47.1
41.4
UTA
B
86M
55.8
47.7
49.1
43.1
UTA
86M
55.6
47.5
47.9
42.2
ViTDet
L
304M
57.6
50.0
49.2
44.5
EVA-02
304M
58.5
50.3
55.3
48.6
UTA
304M
58.7
50.5
55.9
49.5
EVA-OI-CLIP
1011M
58.7
50.4
53.8
47.7
Table The effect of pre-training objectives. FD denotes the re-implementation of the Feature Distillation method Wei ct 2022). ZS and FT denote the zero-shot and fine-tuned top-1 accuracy on ImageNet respectively.
ImageNet
COCO
LVIS
ADE20K
Config
FLOPs
ZS
FT
Aphox
A pmask
Aphox
Apmmsk
mloU
FD
1.0x
74.7
87.2
55.2
47.0
47.9
42.2
54.7
MIM
1.0x
86.9
54.7
46.6
46.6
41.1
54.3
UTA+MIM
1.0x
70.7
87.2
55.4
47.1
47.7
42.0
54.8
UTA
0.6x
75.0
87.3
55.7
47.4
48.9
43.1
55.4
5
Ablation Studies
In this section, we conduct ablation studies to evaluate the impact of different design choices of our proposed Unmasked Token Alignment (UTA). Unless otherwise specified, we the ViT-B backbone and pre-train it for 90 epochs the ImageNet-21K (Dng et al. 2009) dataset.
Pre-training objectives. We thoroughly explore the effect of pre-training objectives and show the results in Tab. 6 We also explore combining UTA and MIM by inputting masked and unmasked tokens simultaneously and conducting token alignment for unmasked tokens and feature prediction for masked tokens. find that UTA performs best on evaluated benchmarks while requiring the least computation cost. particular, we find the improvements on LVIS are most significant compared other approaches. Moreover, show that combining UTA and MIM can lead much worse zero-shot accuracy but similar fine-tuning accuracy ImageNet than using UTA alone. We suspect the inconsistency introduced by the extra [MASK] tokens is more significant when the backbone is fixed for evaluation.
Different pre-trained CLIP models. We study the impact of different pre-trained CLIP models on downstream performance. As shown in Tab. 7 we find that using stronger CLIP model can lead better downstream performance. Additionally, we observe that the performance gap was as significant as on COCO and ADE20K, probably because the classes of those datasets already be easily classified by CLIP-L/14.
UTA for pre-training the text encoder. While perform UTA to pre-train only the vision encoder by default, we also explore using it to pre-train a text encoder from scratch. We train smaller text encoder on DataComp- 1B for epoch. We use the text encoder of EVA-0I-CLIP as the teacher model and apply random masking during pre-training. The hyper--praeee parameters are the same as those used for pre-training vision model. Empirically, we only obtain 54.5% zero-shot accuracy after pre- pre-training, which is much lower than using the CLIP text encoder. We hypothesize that the text encoder is too small benefit from such pre- pre-training. Thus, we decide not perform UTA for pre-training the text encoder.
8

---

Published in Transactions on Machine Learning Research (04/2024)
Table 7: The effect of pre-trained CLIP model.
ImageNet ZS
ImageNet
COCO
ADE20K
CLIP Model
ZS
FT
Aphox
Apmusk
mloU
CLIP-L/14
74.0
67.7
86.6
55.6
47.3
53.7
EVA-OI-CLIP-gie
79.3
75.0
87.3
55.7
47.4
55.4
Table 8: The effect mask ratio (left) and mask type (right). Block-R denotes the reversed block-wise masking. We use mask ratio 0.5 for the mask type ablation.
ImageNet
COCO
ADE20K mloU
ImageNet
COCO Aphox
ADE20K
Ratio
FLOPs
Mask
ZS
FT
Aphox
Apmask
ZS
FT
Apmusk
mloU
0.0
1.0x
74.7
87.2
55.2
47.0
54.7
Block
74.2
87.2
55.3
46.6
54.1
0.4
0.6x
75.0
87.3
55.7
47.4
55.4
Random
74.7
87.2
55.1
46.4
54.6
0.5
0.5
74.8
87.3
55.3
46.8
55.0
Block-R
74.8
87.3
55.3
46.8
55.0
0.7
0,3x
74.0
87.0
55.0
46.6
54.8
Mask ratio and mask type. We examine the effect of the mask ratio and mask type on the final performance. As shown in Tab. (left,, we find that using a mask ratio of 0.4 achieves the best computation cumputation-pefformance performancce trade-off. Additionally, using block-reversed masking performs best on all evaluated datasets.
6 Related Works
Vision (-Language) Foundation Models. The Transformer architecture (Vaswani et al. 2017) has rapidly evolved to become a pivotal paradigm in both Computer Vision (CV) and Natural Language Processing (NLP). Models like BERT (Devlin et al. 2019) and the GPT (Floridi & Chiriatti 2020) series, built upon the Transformer architecture, have exhibited exceptional prowess across various language tasks. Simultaneously, in the field of CV, Vision Transformers (ViTs) (Dosoviskkyy et al. 2021) have emerged potent contenders, gradually displacing CNNs in various downstream vision tasks. Furthermore, the fusion of text and images in shared embedding space, exemplified by CLIP (Radford ct al., 2021). has rendered the Transformer an indispensable tool for versatile uni- and multi-modaa tasks. As training CLIP requires a large amount of computation resources, FLIP (Li et al. 2023b) proposes to mask the visual input tokens to accelerate the training process of CLIP. Recently, large-scale visual pre-training methods based on the Transformer architecture, such as BEiT-3 (Wang et al., 2022a) and EVA (Sun et al. (2023), have continuously pushed the performance boundaries of various downstream visual tasks. this work, we introduce simple yet effective large-scale pre-training method for enhancing the multi-moda representations and demonstrate competitive performance various uni- and multi-modal tasks.
Masked Image Modeling (MIM). MIM is popular pretext task where the vision model learns rich visual representa- tions by conducting reconstruction from corrupted images. Its initial introduction can be traced back to ViT (Dosovitskiy ct al. 202I) and iGPT (Chen et al., 2020). Subsequent ad udannceeenss in the field, exemplified by the notable con- tributions of BEiT (Bao al. (021). MAE (He et al.] (02T), and others (Wang et al. 20226 Liu et al., 2022 Xie et al. 202T), have consistently elevated the performance of the MIM method across diverse downstream tasks. Recent works (Fang al. 2023b: Peng et al. 2022 Hou et al.) 2022 Xiao et al.) 2022) have highlighted the utilization of carefully devised reconstruction targets, like the hidden features from pre-trained CLIP model, which has been shown to facilitate MIM in acquiring superior visual representations. However, these methods rely on the [MASK] tokens to predict the masked features/pixels which introduces the sinntingfinettningg. inconsistency. While UMT (Li et al.) 2023a) does not use the [MASK] tokens and only processes the unmasked tokens, it focuses on training vidco models and does not align with the CLIP text model without contrastive fine-tuning, In contrast, our UTA automatically aligns the main-from-scratee ViT model with CLIP text model and enables zero-shot evaluation even without training on image-text pairs.

---

Published in Transactions on Machine Learning Research (04/2024)
7 Conclusion
In this paper, we introduce the Unmasked Token Alignment (UTA) method, which enhances the alignment between vision and language representations by leveraging pre-trained CLIP models. UTA trains a Vision Transformer (ViT) by aligning the unmasked tokens with corresponding visual tokens of a frozen CLJP model. UTT does not suffer from inconsistency and is training-efficiett by avoiding using extra [MASK] tokens. The pre-tra pre-trained ViT model and CLIP text model can be directly applied for zero-shot evaluation even without contrastive training on image-text pairs. Experimental results demonstrate the effectiveness of UTA across various uni- and multi-modd downstream tasks, outperforming existing MIM and CLIP methods.
Limitations While the proposed UTA method presents promising results and advantages, it also has some limitations. Firstly, UTA relies on the availability of a pre-trained CLIP model, which may limit its applicability in scenarios where such models are not accessible or suitable. Additionally, although UTA achieves strong zero-shot performance without contrastive fine-tuning, it still benefits from further fine-tuning on large-scale image-text pairs, especially for robustness evaluation. While UTA shows great potential for enhancing multi- representations, further research is needed to address these limitations and improve its applicability in a wider range of applications.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmentation. IEEE transactions on pattern analysis and machine intelligence, 4383) 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- tanguage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 1818-200 2829, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* R chatgpt quality, March 2023. URL https: httpos://mmmmss.s.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. database. In CVPR, 2009.
Imagenet: A large-scale hierarchical image
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11 11331, 2023a.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19366 2023b.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30: 681 181-694, 2020.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14108, 2023.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEF/CVE conference computer vision and paltern recognition, pp. 5356-5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'au ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1 15262-15271, 2021b.
Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, and Sun- Yuan Kung. Milan: Masked image pretraining on language assisted representation. arXiv preprint arXiv:2208.0644 06049, 2022.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. arXiv preprint 1rXiv:2303.66058 2023a.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming Hc, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv:2111.11499 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre- training via masking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23390-23400, 2023b.
Tsung-Y Yi Lin. Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08488 2023.
Jihao Liu, Xin Huang, Yu Liu, and Hongsheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. arXiv preprint arXiv:2205. /3137, 2022.
Ilya Loshchilov and Frank Hutter.
Sgdr:
Stochastic gradient descent with warm restarts. arXiv preprint
arXiv:1608.03958 039883 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR,
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL CorpusID:227532815
org/
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector- quantized visual tokenizers. arXiv preprint arXiv:2208.06366 063666 2022.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06123, 1(2)3, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Robin Rombach, Andreas Blatmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
Christoph Schubmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 -222944, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-25655 2018.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.055202 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0980 2021.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Haohan Wang, Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506-10518 2019.
Hongyu Wang. Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj. Saksham Singhal, Alon Benhaim, et al. Foundation transformers. arXiv preprint arXiv:2210.064: 2022a.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442, 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling fine-tuning via feature distillation. arXiv:2205.1414.1 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv:2111.1142,, 2022.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu, Simmim: A simple framework for masked image modeling. In CVPR, 2021.
Zhenda Xie, Zigang Geng, Jingeheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14475-14485, 2023.
12

---

Published in Transactions on Machine Learning Research (04/2024)
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, arXiv preprint arXiv:1904.00962, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. CVPR, 2017.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
A Appendix
A.1 Training Details
Contrastive fine-tuning on DataComp-1B. We initialize the model with pre-trained ViT encoder and CLIP text encoder and fix the temperature value in CLIP loss 0.01. We use a total batch size of 49,152 for fine-tuning. Following Sun al. 2023), use LAMB You et al. for base and large models respectively. We use layer- learning rate for fine-tuning and set the decay rate to 0.75 and 0.85 for base and large models respectively. The weight decay is set to 0.05 for all models. We use cosine learning rate schedule and decay the learning rate to O. We use the prompt provided in CLIP (Radford et al. 2021 for zero-shot evaluation.
2019 optimizer with peak learning rate of 2x 10 and 4 10
Fine-tuning and evaluation with LLaVA. Following Liu al. (023). we use a two-stage procedure for LLaVA model training. Stage 1: Feature alignment. At this stage, train the linear projection layer between the frozen vision encoder and the Large Language Model (LLM) for epoch, utilizing a filtered dataset containing 585K image-text pairs from CC-3M (Sharma et al., 2018). We use Adam (Loshchilov & Hutter] (017) optimizer with a learning rate of x 10-3 The learning rate is linearly warmed up for the first 150 iterations and decayed to 0 with cosine schedule. We use a batch size of 128 and apply no weight decay. Stage 2: End-to-end fine-tuning. We fine-tune the LLaVA model using 158K unique language-image instruction-following dataset for 3 epochs while keeping the vision encoder weights frozen. We use the same optimizer and learning rate schedule as in the first stage except for changing the batch size 32 and setting the learning rate to 2 10-5, We do not apply weight decay during this stage either.
We evaluate the results on LLaVA-Bench (Liu et al. 2023, which measures the instruction following capabilities of multi-modal models over conversation, detailed description, and complex reasoning dimensions. We refer the readers to LLaVA (Liu et al. 2023) paper for the detailed description of LLaVA-Benchh
Object detection and segmentation. Following (Li al. 202 we adopt Cascade Mask R-CNN (He et al. 2017
Cai & Vasconcelos 2019) framework for fine-tuning on COCO (Lin et al. 2014) and LVIS (Gupta al. 2019). We
follow most of the Syper-parameter settings in EVA-02 (Fang al. 2023ab. On COCO,
batch size of 128
and fine-tune for 60k iterations. We use learning rate of 5 x 10 10-3/6 /6 x 10 10-3 drop path rate (Huang et al. 2016) of 0.1/0.4, layer-wise decay rate of 0.7/0.8 for base/large models. On LVIS, we use batch size of 64 and fine-tune for 100k iterations. The learning rate is set to 10 The drop path rate and layer-wise decay rate are the same as those used on COCO. We adopt the UperNet (Xiao al.) (2088) framework for semantic segmentation on ADE20K (Zhou al. 2017). In particular, we use batch size of and fine-tune for 60k iterations. We use learning rate of 6 10 10-%/4 x 10-b drop path rate of 0.15/0... layer-wise decay rate of 0.85/0.9 for base/large models.
13

---

Published in Transactions on Machine Learning Research (04/2024)
A.2 Analysis of Masking Strategies
We show the masking probabilities of different locations in Fig. 3 mask the center locations with much higher probability. In contrast, the reversed block-wise masking (Block-R) can guarantee spatial equalization.
We find that original block-wise masking tends to
Block
Block-R
0.8
2
0.8
0.6
0.6
0.4
8
0.4
10
14
10
15
5
10
15
Figure 3: Masking probabilities different locations. The probabilities are calculated by averaging over 5000 random
samples.
A.3 Loss Curve
In Fig. show the training and validation loss curves of UTA-B during the pre-training phase. For simplicity, we apply random masking with mask ratio of 0.5 for validation. Note that the loss is negative since minimize the negative cosine similarities during pre-training. We observe similar downward trend in both curves and do not observe over-fitting phenomenon over the pre-training process.
A.4 Comparison with the Teacher Model
To have more fair comparison with the teacher model, we conduct additional fine-tuning with the pre-trained teacher model to align the overall computation cost. Specifically, utilize the OpenAI CLIP-L (Radford al. 202 I) as the teacher model and pre-train a UTA--L. We also fine-tune the OpenAl CLIP-L on DataComp- 1B. For OpenAl CLIP-L, we fine-tune the model on DataComp-IB using about 2B samples. For UTA-L, we pre-train the model on ImageNet-21K for 90 epochs (about 1.2B samples), and then fine-tune it on DataComp-1B using about 0.8B samples. We obtain zero-shot accuracy scores of 76.5% for OpenAI CLIP-L and 77.8% for UTA-L. Since the model sizes are comparable, we use the number of samples as indicator to align the overall computation. Under such a setting, UTA-L still able outperform the teacher model. However, note that the 77.8% score for UTA-L lower than the result reported in benefit more from stronger teacher model, also shown in ablation study in Tab.
Tab. 1 indicating
Since our pre-training utilizes reversed block-wise masking to improve the training efficiency, we also fine-tune tune the OpenAl CLIP-L with such masking strategy. In particular, we fine-tune CLIP-L using about 4B samples with a mask ratio of 0.5. We keep the batch size the same to ensure a fair comparison. With this setup, we obtained a zero-shot accuracy score of 76.8% on ImageNet-Iee K,, which still lower than our UTA-L.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Training and validation loss
-0.5
Training
Validation
-0.60
Loss
-066
-0.70
0075
0.80
100k
200k
300k
400k
500k
Steps
Figure 4: Training and validation loss over the pre-training process.
15


Reviews:
[{"review_id": "XzItAmvD1c", "reviewer": "Reviewer_E4o2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros)\n+ This paper is well-written and easy to follow.\n+ Its core concept is both straightforward and impactful, especially the innovative approach of distilling a CLIP visual encoder by aligning partial tokens with the text encoder, eliminating the need for additional finetuning.\n+ The experimental results presented are remarkable and demonstrate significant performance enhancements.\n\n\nCons)\n- The paper lacks clear explanations about the effectiveness of its method:\n   - It does not sufficiently clarify why training on IN-21k and aligning tokens after dropping a few from the ViT backbone yields better results than the traditional approach of pre-training solely on IN-21k, followed by finetuning on IN-1k and additional contrastive finetuning with more images.\n   - There is no insight/intuition into why token dropping improves results compared to using all tokens for alignment.\n- The experiments conducted are not comprehensive enough. For instance, only the EVA-02 architecture was tested, leaving uncertainty about whether UTA could consistently enhance training across different baseline backbones. The paper would benefit from further experimentation with more ViTs from OpenAI-CLIP, OpenCLIP, or smaller/other CLIP variants.\n- The impact of the introduced Block-wise Reversed masking (Block-R) on ADE20k mIoU, as shown in Table 8, is significant. However, the paper does not provide insights into why this method is so effective.\n\nQuestions)\n- Understood that using IN-21k pre-trained ViT is to match EVA's setup, but the necessity of IN-21k pretraining for aligning with CLIP supervision is unclear (because doing that is costly). It would be beneficial for the authors to conduct additional experiments using ViTs pre-trained on IN-1k or with random initialization, to assess the impact of IN-21k pretraining on the effectiveness of their method.\n- In Table 1, EVA-01-CLIP outperforms UTA, despite UTA utilizing EVA-02, which is an improvement over EVA-01, and this is achieved without contrastive finetuning. An explanation from the authors on why this occurs, especially given the supposed advantages of UTA, would be helpful.\n- The difference between 'ZS' and 'ImageNet ZS' in Table 7 needs clarification from the authors. Additionally, the authors should explain the purpose of including Table 7 in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a distillation method for enhancing CLIP's visual encoder, named unmasked token alignment (UTA). UTA aligns the unmasked tokens with CLIP's supervisory tokens, inspired by the EVA and EVA-CLIP strategies, which utilize a pre-trained vision transformer to initialize the vision encoder alongside CLIP's text encoder. Unlike EVA, which relies on masked image modeling (MIM) and requires contrastive finetuning due to its training on ImageNet alone, UTA directly aligns with CLIP's visual encoder using only unmasked tokens. This approach differs significantly from MIM's reliance on masked tokens and avoids the need for additional contrastive finetuning while maintaining alignment with the text encoder. The authors support their methodology with experimental evidence and ablation studies to validate the effectiveness of UTA. The evaluations follow the evaluation protocol of CLIP to see the zero-shot capability on the ImageNet with ImageNet-variant datasets, LLaVA-bench for multi-modal capability, and further dense prediction metrics on ADE20k/COCO/LVIS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Reflect on the previously provided comments.\n- Include training and validation loss trends for the pre-training phase using token alignment to aid in understanding the proposed method.\n- Correct the typo in Table 1: Change \"B/14\" to \"B/16\".", "broader_impact_concerns": "The method itself may not directly raise ethical issues; however, employing datasets like DataComp could introduce concerns due to potential biases or problematic images within these datasets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jUkjUNNxdO", "reviewer": "Reviewer_agMp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[strengths]\n- Interesting results: The paper demonstrates high performance for different tasks without any finetuning with image and text pairs. \n- Well-Written: The paper is exceptionally crafted, presenting a clear and detailed overview of the method in section 2, enriched with insights from related works.\n- Innovative Approach: Introduces a simple yet effective method to enhance vision-language representations, building on pretrained CLIP models, using unmasked tokens, enhanced with a token alignment loss.\n- Addressing Existing Limitations: Skillfully highlights the limitations of current approaches that use masked tokens in the vision encoder.     \n- Effective Experiments: Conducts well-organized and thorough experiments that convincingly demonstrate the superiority of the proposed method over existing ones.\n\n[weaknesses]\n- some detailed analysis of the results is missing, e.g. page 5 last sentences: why finetuning UTA deacreases the performance for Image Retrieval tasks?\n- It is not clear for the reader if the improvements over related works comes from factors such as changes to the ViT architecture? or the dataset distibution?\n- What would you respond if I hypothesize the large gain after finetuning on datacomp is coming from the better distribution of dataset over CLIP models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method named Unmasked Token Alignment (UTA), aimed at enhancing the performance of vision foundation models through masked modeling-based distillation. The key contributions and findings of this paper can be summarized as: 1) porposing UTA incotrast to MIM 2) Extensive experiments and evaluation 2) superior performance over existing methods w and w/o finetuning", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Can you provide a fair comparisonbtween CLIP models (Open-CLIP or EVAL-02-CLIP) and UTA where both are using datacomp image and text pairs to see the effect of data distribution?", "broader_impact_concerns": "While not the main focus of this paper, there might be biases in the datasets used in large scale, i.e. datacomp", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "24M3Mp3095", "reviewer": "Reviewer_mDEF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- UTA does not use `[mask]` tokens in the encoder and unlike MAE also does not have `[mask]` tokens in the decoder. In fact, there is no need for a decoder as CLIP features are being distilled. Thus UTA avoids the train-test gap otherwise caused by `[mask]` tokens during pre-training.\n- Reversed block-wise masking (same as block-wise masking, but randomly inverting the mask to avoid center bias) is a simple and efficient masking method proposed in this manuscript. Ablation experiments in Table 8 (right) evaluate it on classification, detection and segmentation.\n\n--\n## Weaknesses\nThe manuscript restricts its scope to distilling CLIP, but does not ablate this position. The following experiments are needed:\n\n(a) If the FLOPS used to train UTA are instead used to fine-tune EVA-02-CLILP itself on the DataComp-1B dataset.\n\n(b) Same as (a), that is fine-tuning EVA-02-CLILP on DataComp-1B, but now also omitting vision tokens as UTA does.\nI think Table 1 or Table 6 might be a good place for these comparisons.\n\n--\n## Clarification questions\n- In page 4, line 1, \"ViT-L model can surpass the giant-sized CLIP model after contrastive fine-tuning\". Which table-rows should I compare for this claim?\n- In page 7, \"Compared with EVA-02, ..., we can largely improve zero-shot accuracy\". But in Table 4, EVA-02 does not have any zero-shot numbers to compare against.\n- In page 8, UTA for pre-training the text encoder, is unclear. Is a the CLIP text encoder being distilled into a text model? Is block-R masking still being used?\n- The delta on ADE20k between Random/Block and Block-R in Table 8, right, is massive. More analysis on what aspect of the models output has improved would be very helpful. Are the boundaries better? Is the classification better? etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This manuscript proposes Unmasked Token Alignment, UTA, a masked modeling based approach to distilling vision foundation models.\n- It distills a large CLIP vision encoder and demonstrates zero shot improvements when combined with the CLIP text encoder. \n- A wide range of experiments evaluate the model against baselines, including zero shot ImageNet classification, ImageNet robustness variants, zero shot retrieval, segmentation on ADE-20k, object detection on COCO and LVIS., etc.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The baselines suggested in the weaknesses section and the clarifications requested below it would be great.\n\nIn Table 1, I-T pairs are written as 0B for UTA because in the distillation process no Image-text (I-T) pairs are needed. But the original EVA-02-CLIP model being distilled used many image-text pairs. I believe that one should at least write 0B + 8B to clarify this (and similarly in other rows).", "broader_impact_concerns": "Pros: The manuscript shows that CLIP can be further improved without access to billions of image-text pairs and massive amounts of compute. With access to some Image-text pairs, further improvement is possible. This slightly furthers the democratization of these foundation models.\n\nCons: Detection and segmentation are sensitive research topics as there is potential for misuse. Although these problems are not the main focus of this manuscript, a broader impact statement in the appendix would be nice.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5tQKG26XuU", "reviewer": "Reviewer_6ntT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+The paper is clear and well written, the technical part is very easy to follow.\n+The presented method is simple yet effective. Overall I think the method is neat and sound.\n+The empirical evaluation seems to be quite thorough. Although some results over the previous method is marginal, some gains are pretty clear.\n\nWeaknesses\n\n-The motivation of this line of study is questionable. I understand the approach can attain similar performance to the teacher model with only distillation and much less I-T finetuning. However, the resulting model seems to have the same size as the original teacher model and the architecture is the same as well, with similar or slightly better results. If this is indeed the case, I don't quite understand the incentive of doing this -- why not just use the original teacher model?\n\nIn light of this, I am curious about the potential of this method of distilling into smaller models. Some results on this will be fantastic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper introduces a new visual pretraining approach dubbed UTA, which effectively extends the previous MIM + CLIP distillation methods to only distilling the unmasked tokens instead. Comprehensive evaluations including zero-shot image classification, multi-modal tasks (LLaVA-Bench) and other core vision tasks (detection and segmentation) are conducted. Also, several ablation studies shed some light on how this approach works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please add some results of applying your approach to smaller models (to distill from larger models).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ReojqHuvkC", "reviewer": "Reviewer_qzjH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n-\tThe paper proposes a simple and effective approach to learn enhanced vision-language representations based on pretrained CLIP models.\n-\tThe paper clearly points out the limitations of existing approaches which inputs masked tokens to the vision encoder and proposes a novel method which addresses such limitations by using the unmasked tokens as the only input while utilizing a token alignment loss. \n-\tOverall, well-organized experiments conducted in this study effectively demonstrates the effectiveness of the proposed method and its superiority over the previous methods. \n\n[Weaknesses]\n\n-\tThe paper lacks a detailed analysis of the experiment results. For example, in Table 2, further fine-tuning on the DataComp-1B dataset degenerates the image retrieval performance, as mentioned at the end of page 5. This phenomenon is observed in Tables 1 and 5 as well. Why would additional fine-tuning deteriorate performance? \n-\tAlso, in page 7, the last sentence of the first paragraph says “We can largely improve the zero-shot accuracy.” However, in Table 4, the zero-shot accuracy of EVA-02 is not reported.\n-\tWhile the paper slightly modifies (or improves) the backbone architecture as mentioned in Section 2.3, the paper does not investigate the impact of this modification on performance improvement.\n-\tAs a minor weakness, there is a lack of explanation about the conducted experiments. The paper does not provide a detailed explanation on the criteria reported in Table 3 (e.g., Conversation, Detail, Reasoning, and Overall).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel method called unmasked token alignment (UTA) which enhances representations of multi-modal vision-language models.\nDuring the training phase, UTA only inputs unmasked tokens to the vision encoder and trains these token representations to be aligned with the token representations extracted from the pretrained CLIP vision encoder.\nA key advantage of UTA is its applicability to zero-shot tasks eliminating the need for additional fine-tuning with masked tokens. This feature ensures a more consistent training and finetuning phase.\nThe experimental results demonstrate the superiority of UTA over existing methodologies in the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-\tDetailed explanation about the experiment settings and a deeper analysis of the experiment results are required.\n-\tIn addition, analysis of the impact of architectural modification on the performance improvement should also be included in the manuscript.", "broader_impact_concerns": "Although the model is trained with a large-scale dataset, it may have unintentionally learned spurious correlation between representations due to the inherent dataset bias within the training set. This may cause unwanted impact to certain minority groups when the model serves as a foundation model for various downstream tasks. This issue can be alleviated by encouraging the model to learn debiased features during the training or by utilizing post-processing strategies for debiasing. It is important to be aware of the potential issues regarding fairness.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims for enhancing representation power of vision-language models, and proposes a new method called unmasked token alignment (UTA), which, during the training phase, takes as inputs the unmasked tokens to the vision encoder and trains them to be aligned with the token representations extracted from the pretrained CLIP vision encoder.Thorough experiments demonstrate the superiority of UTA over existing methods.

All reviewers are satisfied with the novelty and contributions made from the paper. They initially made a few requests for revisions, including providing additional explanations and insights on effectiveness of the proposed method, additional experiments, and additional ablation studies. The authors made the corresponding revisions that reviewers have been satisfied with. Please the authors make sure all these revisions have been included in the final version.
Decision: Accept (poster)