Harmony: A Joint Self-Supervised and Weakly-Supervised Framework for Learning General Purpose Visual Representations
Authors: Mohammed Baharoon
Jonathan Klein
Dominik Michels
Venue: Accepted by TMLR ()

Abstract: Vision-language contrastive learning frameworks such as CLIP enable learning representations from natural language supervision and provide strong zero-shot classification capabilities. However, due to the nature of the supervisory signal in these paradigms, they lack the ability to learn localized features, leading to degraded performance on dense prediction tasks such as segmentation and detection. On the other hand, self-supervised learning methods have shown the ability to learn granular representations, complementing the high-level features in vision-language training. In this work, we present Harmony, a framework that combines vision-language training with discriminative and generative self-supervision to learn visual features that can be generalized across different downstream vision tasks. Our framework is specifically designed to work on web-scraped data by not relying on negative examples in the self-supervised learning path and addressing the one-to-one correspondence issue using soft CLIP targets generated by an EMA model. Moreover, Harmony optimizes for five different objectives simultaneously, efficiently utilizing the supervision in each data example, making it even more suited in data-constrained settings. We comprehensively evaluate Harmony across various vision downstream tasks and find that it significantly outperforms the baseline CLIP and outperforms the previously leading joint self- and weakly supervised methods, SLIP, MaskCLIP, and DetailCLIP. Specifically, when compared against these methods, Harmony shows superior performance in linear-probing, fine-tuning, and zero-shot classification on ImageNet-1k, semantic segmentation on ADE20K, and both object detection and instance segmentation on MS-COCO, when pre-training a ViT-B on CC3M. We also show that Harmony outperforms SILC on detection, linear and fine-tuning classification, and outperforms other self-supervised learning methods like iBOT and MAE across all tasks evaluated. Our code is publicly available at https://github.com/MohammedSB/Harmony.

Published in Transactions on Machine Learning Research (06/2025)
Harmony: A Joint Self-Supervised and Weakly-Supervised Framework for Learning General Purpose Visual Represen- tations
Mohammed Baharoon Harvard University
mohammmed bubaroon@@mss.arvrrt edu
KAUST
Jonathan Klein KAUST
eew.oousst.udd.... edu.sa
Dominik L. Michels KAUST
Reviewed on OpenReview: /ttps://opentie operreveew.reee...oeeeeee net/ forum? IcOBOuf
https: //
Abstract
Vision-language contrastive learning frameworks such as CLIP enable learning representa- tions from natural language supervision and provide strong zero-shot classification capabil- itios. However, due to the nature of the supervisory signal these paradigms, they lack the ability to learn localized features, leading to degraded performance on dense predic- tion tasks such as segmentation and detection. On the other hand, self-supervised learning methods have shown the ability learn granular representations, complementing the high- level features in vision-language training. In this work, we present Harmony, framework that combines vision-language training with discrininative and generative sell-supervision to learn visual features that can be generalized across different downstream vision tasks. Our framework is specifically designed to work on web-scraped data by not relying on negative examples in the self-supervised learning path and addressing the one-to-one correspondence issue using soft CT.IP targets generated by an FMA model. Moreover, Harmony optimizes for five different objectives simultaneously, efficiently utilizing the supervision in each data example, making it even more suited in rada-c settings. We comprehensively eval- uate Harmony across various vision downstream tasks and find that it significantly outper- forms the bascline CLIP and outperforms the previously lcading joint self- and weakly super- vised methods, SLIP, MaskCLIP, and DetailCLIP. Specifically, when compared against these methods, Harmony shows superior performance in linear aneer-probbing fine-tuning, and zero-shot classification on ImageNet-lk semantic segmentation on ADE20K, and both object detec- tion and instance segmentation on MS-COCO, when pre-training ViT-B on CC3M. We also show that Harmony oulperforms SILC on detection, linear and fine-tuning classification, and outperforms other self-supervised learning methods like iBOT and MAE across all tasks evaluated. Our code is publicly available at https:/ http://ggttbbbcom/MoamolsSS3
MohhammddBB ///r//aroyy
1 Introduction
Self-supervised and weakly-superviiee pre-training have recently shown remarkable success at learning visual
representations without direct supervision (Radford al. 2021 Oquab al.) 2024 Caron al. 2021b Chen
al. 2020a He et al. 2020 Grill et al. 2020 Zhou al. 2022 He et al. [021 Mu al.) 2021 Dong al.
2023
As vision training datasets continue to scale, it becomes progressively more difficult. and expensive to

---

Published in Transactions on Machine Learning Research (06/2025)
ViT Size
Models
80
Small
CLIP
Base
MaskCLIP
70
-3.3
+5.2
DetailCLIP
-13.0+11.5
SLIP
60
+2.5
SILC
Harmony
Performance
50
12.5
-7.0
+1.9
+5.4
+12.2
+7.0
+1.5
+2.0
40
+3.7
30
20
10
0
Image Classification
Semantic Segmentation Task
Object Detection
Figure Performance comparison across different lasks. Harmony significantly outperforms CLIP, SLIP, MaskCLIP, and DetailCLIP across all tasks, and generally has larger improvement in performance as we scale the ViT size, except for DetailCLIP which is low performing at ViT-S. Morcover, for classification and detection, the small version of Harmony outperforms the base version of the other methods. We used MS-COCO for object detection, ADE20K for semantic segmentation, and ImageNet for classification, using linear-probing. See Table [ for numerical values.
provide manual supervision in the form of labels, making the development of robust self-supervised learning (SSL.) and weakly-supervised learning (WSL) techniques more integral.
Weakly-supervised learning, specifically language-guided or language-sppervieed learning, was popularized by CLIP (Radford et al. 2021 and learns visual and textual representations using contrastive loss by
maximizing the similarity between image-captions pairs and minimizing the similarity of non-paired image- captions (Radford et al.) 2021 Cherti et al. 2022). Because this approach relics on semantic captions as a supervisory signal, language-supervised models are strong at high-level lasks like image classification, but significantly underperform on dense, low-level prediction tasks that require localized features (Radford et al. 2021 Wang et al. 2024 2022). In other words, these paradigms are good at lcarning what objects are present in visual input, but not where they are. One approach of introducing local information into WSL frameworks is to combine it with self-superiised learning (Dong et al.) 2023 Mu et al.] 2021 Yuan et al.
2021]. Unlike language-supervised learning that maps across modalities (e.g. image to text), SSL maps to the same visual modality, making it more granular and localized for visual tasks (Caron et al. 2021b He
et al. 2021 Assran et al. 2023).
Recent works self-supervised learning formulate the pre-training task as either discriminative or generative 2023 Doersch et al. 2016). For discriminative SSL methods, the model learns visual rep-
Ozbulak et al.
resentations from images by differentiating between positive images pairs, and optionally repelling negative
pairs CCen et al.
2020a Grill et al. 2020 Caron et al.) 2021b Oquab et al. 2024 He et al. 2020) Chen el al. 2020b). On the other hand, generalive approaches learn visual representations by masking certain parts an image and learning to reconstruct the missing parts, given the original image (He et al. 2021 Tong et al. 2022 Gupta et al. (003)). Bocause SSL methods do not require manually annotated labels, they can be used for training large neural networks on huge image datasets scraped from the web (Oquab al.
2024).
2

---

Published in Transactions on Machine Learning Research (06/2025)
Soé Conrattiv Lons
Cont Loss
Danilation
Costratire
Madked Imguuge modeling
Paol
Ditilation
Danllanio Lou
Masked maddling
Dv
Dr
Dv
DT
By
Ev
Eyy
Ev
Ey
Fr
ET
Ey
F
EMA
Ey
E
E
"A Cand
"A Canel [MASK in eupey grm
'A Canal [MASK] jn on expry gaus
[MASK] is ouere gass 6d2
Aug
Aug 2
Aug
Aug
"A Camel
Camel standing in an empty grass field'
"A Camel standing an empty grass field'
standing
empty grass
field'
SLIP
MaskCLIP
Harmony
Figure Harmony compared to previous methods. We show our Harmony approach next to previonsly leading joint methods MaskCLIP (Dong et al. 2023) and SLIP (Mu et al. 2021). Harmony optimizes five different objectives simultaneously, outperforming the other approaches across all vision downstream tasks.
Combining langnage-supervised and self-supervised learning has led to superior learning paradigms (Dong
et al. 2023 Mu et al. 2021 Yuan et al. 2021 Nacem et al. 2024 Monsefi et al. 2024). For example, SLIP Mu et al. 2021 combines CLIP with SimCLR, lcading to better zcro-shot capabilitics and overall more accurate models across multiple downstream lasks. MaskCLIP (Dong et al. 2023 combines CLIP with masked self-distillation, leading to further improvements in a variety downstream evaluations. More recently, STL.C (Nacem al.
2024) combined CLIP with DINO (Caron al. 2021b) to learn more localized image features for dense prediction tasks. DetailCLIP (Monseli et al. 2024) combines A-CLIP (Yang et al. 2023 with MAF (He al.
2021 and iBOT (Zhou et. al. 2022), outperforming all three methods in isolation.
Building on the aforementioned works, we present novel framework, which we will call Harmony, that combines discriminative and generative self-supervision with language-superisst learning in order to learn general purpose visual representations from image-captions pairs gathered from the web. Our approach is designed to learn semantic visual representations useful for high-level vision tasks and fineggraine,, low-level reprosentation for dense prediction tasks the same time.
Previous approaches like MaskCLIP (Dong et al. 2023) and SLIP Mu et al. 2021 rely one-to-one correspondences and hard negative examples which fail to model the inherent semantic relationship between non-paired samples (Andonian et al. 2022). This is because for a certain image, captions of other images in the batch could still describe this image with varying degrees, especially when the batch size is large, Morcover, particularly in an uncurated data setting, the caption paired with a certain image could simply incorrect, or the description could be only loosely associated with the image. To remedy these issues, many relationships by incorporating soft CLIP targets, and we designed our framework to utilize self-distillation methods like iBOT (Zhou al., (022) rather than not rely on negative examples by using SimCLR (Chen et al.] 2020a) like in SLIP (Mu al. 2021 We argue that this makes Harmony better designed for web-scraped data like CC3M (Sharma et al.] 2018
We summarize our main contributions in this paper as follows:
We present Harmony, a joint self-supervised and weakly-supervise, framework that learns global and local features, generalizing across vision tasks including classification, segmentation, and detection. Harmony optimizes for five different objectives simultancously, fully leveraging each data example, which makes method particularly suited for data-constrained settings.
3

---

Published in Transactions on Machine Learning Research (06/2025)
We introduce EMA-based method for generating soft-targets for CLIP and a lightweight text self- distillation objective for learning textual representation, improving CLIP's zero-shot capabilities.
Harmony built to avoid negalive examples in the sell-supervised learning path and reduce reliance on strict one-to-one correspondences in the text-guided contrastive path, achicving superior or com- petitive performance with leading methods when pre-trained on the web-scraped CC3M dataset.
We provide unified codebase for all methods that is composable, allowing users to combine any subset of our predefined set of methods, and with our own implementation of SILC and MaskCLIP, which were not previously open open-sourced.
2 Related Works
Discriminative self-supervision. Recent discriminative SSL approaches learn visual representations by discriminating between images that are positive pairs of each other, usually defined as different angmentations
of the same image, and optionally pushing away negative pairs. (Grill et al., 2020 Caron et al., 2021b He 2020 Chen ct al., 20) Methods like SimCLR (Chen et al.] (2020a) and MoCo (He al. 2020 Chen ct al. jet al. 2020b) use contrastive learning to maximize the embedding similarity between positive pairs while simultaneously reducing the similarity of negative pairs. These approaches ignore the inherent similarity between different images and treat the discriminative problem a one-to-one scheme, where an image is only similar to its positive pair. Other approaches like BYOL (Grill et al. 2020 and DINO (Caron et al., 2021b) use self-distillation to only maximize agreement between positive pairs, avoiding that. issue. These methods use a momentum encoder, usually refereed to as a target or teacher encoder, to generate embedding targets for an online or student encoder that is trainod simultancously with the teacher encoder. The lcarning lask in these paradigms to maximize the similarity between embeddings from both the teacher and student encoders given positive image pairs. Self-distillation methods regularly outperform contrastive methods, and
reach accuracies that are competitive with supervised learning (Caron et al. 2021b Zhou et al. 2022 Grill
ct al. 2020
Generative self-supervision. Rather than discriminating between images, generative SSL methods learn visual representations by masking certain parts of an images (or certain images in a video) and then learn to reconstruct the masked portions given the original signal (He et al. 2021 Tong et al.] 2022 Gupta et al., 2023). This reconstruction task can be viewed as a proxy for learning about visual features from images, improving representation ability and, therefore, performance on downstream tasks. A common approach in this paradigm is called Masked Autoencoder (MAE) (He et al., (022). which utilizes ViT's patching paradigm to mask a certain percentage of patches (usually 75%) and minimizes the L2 loss on predicted pixels from the masked patches. Generative SSL approaches are highly scalable due to their lightweight memory usage compared to discriminative approaches, and reach fine-Luning accuracies rivaling those methods (He et al. 2021).
Joint self-supervision methods. Many previous works have combined discriminative and generative self- supervised learning objectives, resulting in improved performance downstream classification and semantic segmentation tasks (Chen et al. 2023 Huang et al.] 2024 Mishra et al.] 2022). CMAE (Huang et al. 2024) and CAE (Chen et al. 2023) combine MAE's pixel reconstruction task with contrastive learning, outperforming either method in isolation. CAN (Mishra et al.
2022 further add a noise prediction task and masks both views in the contrastive learning objective. These works highlight the fact that SSL paradigms can be complementary to cach other.
Language-guided and self-supervised learning. Other works have combined SSL and language- guiddddd learning to learn more generalized representations (Mu et al. 2021 Dong al. 2023 Yuan et al. 2021 Yuan et al. (2021) combine vision-language training with contrastive SSL to learn visual representations from multimodal, image-text data. SLIP (Mu et al.) 2021) builds on this work further by showing that combining SimCLR (Chen et al., (220aa) with CLIP (Radford et al. 2021) objectives outperforms both mothods isolation by relatively large margins across multiple downstream tasks. SILC (Nacom et al., 2024) replaces SimCLR with DINO in SLIP, outperforming SLIP. In addition, MaskCLIP (Dong et al. 2023) presents masked self-distillation approach combined with contrastive vision-language pre-training,
4

---

Published in Transactions on Machine Learning Research (06/2025)
further outperforming CLIP and other joint WSL and SSL methods. More recently, DetailCLIP (Monsefi et al.j 2024g combines A-CLIP (Yang et al. 2023) with MAF and iBOT to learn more localized visual features for fine-grained tasks. We further build on these works by going an extra step of combining five different objectives, including self-distillation, pixel reconstruction, text-guided contrastive learning, and two additional lext-only losses: masked- languugg modeling and text self-distillation.
3 Harmony
We introduce Harmony, joint self-supervised and weakly-supervised framework for learning semantic and localized visual representations in the wild. The main components of our framework are shown in Figure 2 Namely we define vision student and teacher encoders, Ey and Ev, text student and teacher encoders, Er and Er, and vision and text decoders, Dy and Dr. Our teacher encoders are used generate self-distillation and soft targets, and are defined as being the exponential moving average (EMA) of the students. The vision and text decoders are used to map from embedding space to pixel and word token space, respectively. All components use the Transformer (Vaswani et al. 2023) architecture, and we the standard vision transformer (ViT) implementation for processing images (Dosovitskiy et al. 2021
Our framework optimizes five different objectives simultaneously with the goal of learning robust, general- purpose visual representations from web-scraped images-caption pairs. Our losses are (1) text-guided con- trastive learning that identical to CLIP (Radford et al. 2021] p but with added soft targets, (2) feature delf-disillaation following iBOT (Zhou et al. 2022], (3) pixel prediction following MAE (He et al. 2021)
(4) word prediction or MLM (Devlin et al.) 2019), and (5) text self-distillation, which is similar to iBOT's
patch-level objective but applied on word embeddings (Zhou et al. 2022). In the following sections, detail each objective in our framework.
3.1 Text- rext-guided Contrastive Learning with Soft Targets
Our first objective in Harmony image-text contrastive learning with soft targets. We begin by defining vision encoder Ey and text encoder Ey. We attach single layer projection head h to each encoder, resulting in h 9 h o f, where f is either Ey Er. Given batch of image-text pair collections [(11, (1), (12, 12),... [N)) where N is the batch size, we extract image and text embeddings v, gr (1:) and t, = (t+), where 9u and It are the student vision encoder Ev and student text encoder ET with the attached projection heads, respectively.
We maximize the similarity between paired embedding sets, V, and tú where i = and minimize the similarity between unpaired sets where i + j. More formally, we define the InfoNCE loss as our training objective van den Oord et al. (2019) following CLIP Radford et al. (2021), where LIntONCE = L, + L. and
N
N by log Po (v
1 N
C.
(1)
ky is an element in the identity matrix IN so it is set to one when i = j when the image-text embeddings are paired, and to zero otherwise. P. is the softmax function applied per image:
exp(cos(v1, (t)))/())
(.(,,ty:TT Ex exp(cos( Vi, (x)/T)
(2)
The function sim(vi, tj) is the cosine similarity, sim (vi, t2) T tj. and is learnable temperature param- eter. The loss L. and function Pt are defined in symmetrical way van den Oord et al. (2019). Since the above LIafONCE uses hard targets (Is and 0s), we will refer to it as LHard-
Notice that C, (and L, given the symmetry) is the cross-entropy function H(a, b) = - log applied across image caption pairs. Because of that, we can rewrite C. as H(IN, P(VTT: :T)) where V.T T E R xxd are matrices that contain batch of image and text embeddings, with d being the embedding size. P is the vectorized version of P. and P, in Equation 2 where the cosine similarities, cos(vi, tt), are calcualted through
5

---

Published in Transactions on Machine Learning Research (06/2025)
matrix multiplication. Therefore, Lard can be rewritten
Hard H(IN, P(VT"; (r)) + P(TV',
(3)
Soft targets. The loss function in the original CLIP implementation Radford et al. (2021) assumes that there is one-to-one correspondence between image and caption pairs. This is because the target in the cross entropy function in Equation 3 is the identity matrix IN. This means that, given an image embedding Vi and a set of textual embeddings {t1, t2..... tx², minimizing the objective can be viewed as an N-way multi-class classification problem. This assumption does not always accurately represent the relationship between imago-captions sets, especially in the case of uncurated data. Certain captions can describe many different images with varying degrees, regardless of their original pairing, and the larget should reflect some function of how semantically similar image-caption set is.
Recent works have tried to mitigate this issue by incorporating soft instead of hard targets (Andonian et al. 2022 Gao et al.] 2023 Scotti et al.) 2023). This is a non-trivial task since generating soft targets assumes
some pre-existing knowledge about how semantically similar image-caption pairs are. Gao et al. (2023 relies on a pre-trained object detection model to generate soft similarity targets, while Andonian et al. (2022) used
self-distillation approach dynamically generate soft targets without pre-training. In the latter approach the same model is used to both generate soft targets and make predictions for calculating the loss, by splitting the mini-batch into student and teacher targets. Instead, use an EMA self-distillation method for generating soft-targets, motivated by the fact that our framework already defines an EMA model for feature self-distillation in Section 3.2 so the addition of soft targets comes at little computational expense.
To generate soft largets, we define vision and text teacher models, Ey and ËT, respectively, and their corresponding In and gr. which are the encoders with the attached projection heads. We do not propagate gradients through the parameters of either Ev or ET and instead update their weights using the EMA of the students. Given the same image-text pair collections {(11, f1), (02, f2), t2),..., (uN, (N)) we generate embedding largels V. (1a) and i; (44). We can represent these largets as the matrices V and T like in Equalion with Av = P(VT": F) and AT = P(TV"; ;)). Our soft CLIP loss, Lsoft is then defined as
Lsoit P(VT" + P(TV
(4)
We set the teacher temperature 7 to 0.1. Our final contrastive loss is the sum of the two losses LHard and LsoR, where we progressively increase the influence of the soft loss Lsoft throughout training. In other words, the contrastive loss is defined as
Lc = a.CHard + (1 - cc.)
(5)
We start with a, = and progressively decrease it to a. = 0.2 using a cosine scheduler in the first 10 epochs of pre-training.
3.2 Feature Self-distillation
On top of the contrastive objective Lc, we add a self-supervised feature self-distillation loss following iBOT (Zhou al. 2022), which we find can significantly boost performance across all downstream tasks evaluated (see Table This loss consists of both global and local objectives that go hand-in-hand to learn generalized visual features. Conceptually, the goal of this loss in our framework is to learn visual features that might not be described in the caption of the contrastive loss, and learn more localized features by adding local objective that operates at the patch level.
Global objective. Following [Caron et al. 2021b Oquab et al. 2024 Grill et al. 2020 Zhou et al. 2022 we utilize a teacher encoder Ey that defined as the EMA of student encoder Ey, which trained with gradient optimization. The teacher and student models, Ey and Ey, are the same models used in the contrastive loss from Section 3.1. Just like in Caron et al. 2021b Zhou et al. 2022), we attach multi-layer
6

---

Published in Transactions on Machine Learning Research (06/2025)
perceptron (MLP) projection head h, such that g g models, respectively. Given input image x we generate two different augmentations, 21 and 22, defined as random crops or views of x, We feed these two augmentations to and and optimize the following loss function:
ho where f is either Ey Ev, resulting in g and
CELS M((((1)) + M((((22),
(6)
2
where H. is the cross entropy function and P is a softmax function, defined as
P(x) = explg(e)/T)
(7)
E exp(g(xx)/7)
T is a non-trainable temperature parameter that controls output distribution sharpness. P(x) is defined in the same way but with 9 instead of and F instead of T.
Local objective. Equation is applied on the class token (CT.S) of the vision transformer DDsovvisskiy et al. 2021), which is why we refer to as LCLS- Since CLS tokens aggregate information across different patches into single token, minimizing Lets learns more global features, sometimes disregarding granular details that are helpful for dense prediction tasks like semantic segmentation. On top of this global-level objective we aim to learn localized features by employing a patch-level loss using masked image modeling (MIM), following iBOT Zhoo et al. 2022). More precisely, given the sequence of image tokens x = {x1, X2...., xN} being processed by a ViT, we sample from the masking set m E [0,1) according to a ratio r. Tf m; = we replace the original x,, with a special mask token Xm, resulting in a masked view & formalized as & & {&, ((1 m,)x; + m;x,)) (Zhou et al., 2022). We subsoquently food x and & to I and g, respectively, optimizing the loss function over all patch tokens:
CMIM Σ my H(Px))
(8)
As shown in (Zhou et al. 2022) adding the patch-level loss LMIM on top of LCLS, which is originally from
DINO (Caron al. 2021b), improves downstream performance on both dense prediction and classification tasks.
Moreover, we also minimize the cross entropy loss between the embeddings of 21 and 22 extracted by the teacher model Ey and emboddings of smaller, local crops (21, Vr extracted by the student model Ey and generated using a multi-crop augmentation strategy (Caron et al. 2021a), which is shown to increase performance in (Caron et al.) 2021b; Zhou et al., 2022). This multi-crop optimization is added only to LCLS.
The final self-distillation loss LD is as the average of LELS and LMIM, i.c.,
LD = / (CoLs + Lxax)).
(9)
3.3 Pixel Reconstruction
The Ln loss is applied on the feature space. On top of this, we add another pixel level loss for learning more granular features, which we find slightly improve performance for segmentation tasks (see Table 4 Specifically, we follow the approach in MAE (He et al.] 2021) in that, given a sequence of patch token XN,,,,,..,,N,, we mask (remove) P number of the patch tokens at random (as opposed to replacing them with Fm like in Section 3.2). We feed the remaining (L P) + of tokens, where the 1 is the FçLs token and L is the total number of patches in the original images, to Ev, which is the same encoder used in the contrastive objective (Section 3.1p and feature sclf-distillation (Section 3.2). We end up with sequence of image embeddings e {ecI.s, e1, .., ew., where M = L P + 1. We then add P number of mask tokens Xm to e in the same position (or index) they were removed from to obtain the masked embedding è Subsoquently, è is passed to decoder Dy, which will up-sample the embeddings from dp to the H W, where dp is the embedding size of Dy, H and W are the height and width of the original patch size, respectively. We set. P P= I x 0.75, following He et al. 2021

---

Published in Transactions on Machine Learning Research (06/2025)
The loss is then calculated as the Mean Squared Error between the predicted pixels pt and the L2 normalized pixels of the original target in patch fú. In other words, the loss becomes:
CR
m (p: - -.)
(10)
3.4 MLM and Text Self-distillation
Even though the goal of this work is visual representation learning, improving textual representations can have a significant impact on zero-shot evaluations as supported by results from Dong et al. (2023) and our Table [ As a result, we added two textual objectives that the text. encoder Er will now optimize on top of the contrastive loss. These objectives are maskod language modcling and text self-distillation.
Masked language modeling. The first additional textual objective is masked language modeling, described in BERT (Devlin et al., 2019), which predicts masked words given the context of surrounding words. MLM can be viewed analogously as pixel prediction, in the sense that both optimize outputs that are in the same level as original inputs (tokenized words in the context of MLM and pixels in pixel prediction).
We feed a masked view i of a tokenized caption t to the student text encoder ET, where word tokens are randomly masked using Bernoulli distribution m Bernoullip)) with p 0.2, with m being the mask vector that is applied words with random words, nor keep some predicted words unchanged. Masked word tokens are replaced with a mask token im to generate f. Our final MLM loss is then formulated as
caption / to generate f. Unlike BERT (Devlin et al., 2019), don't replace
C CM Σ
m, m H(4, (Er(()))
(11)
where is the context length of the transformer model Ey and m, = only if f is masked.
Text self-distillation. On top of the MLM loss, we add self-distillation loss by utilizing the student and
teacher lext encoders, Er and Er, motivated by iBOT's patch-level objective [Zhou et al. 2022). Unlike the MLM loss, this objective will function at the embedding level, which offers softer targets for the student text encoder compared to MLM. We attach MLP projection heads to Er and Er, resulting in and respectively. We then optimize the equation
LTD = Σ m ((((()),,)()))
(12)
where and P are the softmax functions in Equation 7 but with the text instead of vision encoders.
3.5 Harmony's Objective
Our finalized framework, Harmony, optimizes the five described objectives simultaneously. In other words, our final loss is a linear combination of all five losses or the equation:
LH = Le + aCp + BCR + 7LM + SCTD.
(13)
The parameters a, B, 7, and 8 allow for weighting of the different losses. However, our experiments (sce Table G) choosing identical weights has usually been sufficient.
4 Experiments
Here, we experinentally evaluate Harmony against the bascline methods CLIP (Johnson et al. 2016), SigLIP (Zhai et al. 2023), iBOT (Zhou et al. 2022), and MAE (He et al. 2021). ) as well as previously leading joint SSL and WSL methods, SLIP (Mu et al. 2021), SILC (Naeem et al.] 2024), MaskCLIP (Dong et al. 2028) and DetailCLIP MMonsffi et al.) 2024). We start Soction 4.1 by describing our model architecture and training setup, then we present our results in Scction 4.2 and finally end with an ablation and hyper- parameter tuning study in Section 4.3
8

---

Published in Transactions on Machine Learning Research (06/2025)
26-
Harmany
Harmony
CLIP
CLIP
32.8
0-Shot ImogeNee-11
23.3
(P-Shok Imageteet-11
+6.2
44.0
22-
28.8
+4.3
20
23,3
+3.6
19.0
17.2
16
Small
Base Model Size
Large
CC3M
CC12Mpp
CC12M
Data Size
(a) Scaling ViT size. The gap in 0-shot performance between Harmony and CLIP increases as we increase model size, sug- gesting that Harmony scales better with respect to model size.
(b) Scaling data size. The performance gap between Harmony and CLIP narrows as dataset size increases. CC12MBM indi- cates using 6M images from CC12M. VII-B was used for all experiments.
Figure 3: Zero-shot performance comparison between Harmony and CLIP when scaling VIT size and data. (a) With increasing model size, Harmony's advantage grows. (b) With increasing dataset size, the performance gap narrows, though Harmony maintains an edge. Appendix B shows zero-shot results on CC12M across 8 other datascts.
4.1 Training Setup
Model architecture. Six transformer networks (Vaswani et al. 2023); three of which vision transformers (Dosovitskiy et al.) 2021), of different sizes and configurations, make up our final framework: (1, student and teacher vision encoders, Ey and Ev; (3, 4) student and teacher text encoders, Er and Er; (5) vision decoder Dv; and (6) text decoder Dr. For the vision encoders, Ev and Ev, we use ViT sizes small (S), base (B), and large (L). For the Lext. encoders, Er and Er, we use 12 layers, 512 embedding dimensions, and 8 heads, following CLIP (Radford et al.) 2021p. The number of text tokens is fixed to 77 with necessary truncations or paddings. Morcover, following He al. (2021), we define our vision docoder Dy as an layer, 512-embedding dimension, and 16-head VIT. Finally, our text decoder Dr, is made of up layers, 512 embedding dimension, and 8 heads.
Pre-training. We pre-train ViT-S, ViT-B, and ViT-L using Harmony on CC3M for 25 epochs. We use an AdamW optimizer [Loscchloov & Hutter mixed precision (Micikevicius et al. 2018). We set the number of global crops per given image to and the local crops to 8. We use a masking ratio of 75% for the pixel prediction task and pre-train up to 16 32GB V100 GPUs. Further information about our pre-training setup can be found in Appendix
2019) with an value of le to improve training stability using
Downstream. To asses the generalizability of learned features using our proposed method, evaluate our model variety of datasets, including ImageNet-Ik (Russakovsky et al. 2015) for fine-tuning, linear- probing, and zero-shot classifications, ADE20K (Zhou et al. 2017) for semantic segmentation, MS-COCO (Lin et al. 2015) for object detection and instance segmentation, as well as many other datasets for zero-shot evaluations (see Table 2) We fine-tune for 100 epochs on ImageNet-1k and for 160k itcrations for ADE20K, using UperNet. (Xiao al. 2018). For object detection and instance segmentation, we use a Cascade Mask R-CNN (Cai & Vasconcelos 2019 and fine-tune for 12 epochs. Further details on downstream settings can be found in Appendix
4.2 Results
We begin our evaluation of Harmony by analyzing its downstream performance for variely of visual tasks in different settings. We start Table ! by presenting our main result for classification, segmentation, and detection tasks. We show that Harmony significantly outperforms CLIP and SigLIP and outperforms iBOT [hou al.] (2022) and MAE (He et al. (2021) in all high-level and low-level tasks evaluated. We also show that. Harmony outperforms SLIP (Mu al.] 2021), MaskCI.IP (Dong al. 2023 and DetailCT.IP
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Comparing Harmony against baseline SSL and previous best methods. All methods are pre-trained on CC3M for 25 epochs. We used the same pre-training settings for MAF, iBOT, CLIP, SigLIP, SILC and Harmony. To reproduce SLIP, and DetailCLIP, we used their public code base (Mu et al. 2021 Monsefi al. 2024). We reproduced MaskCLIP and SLIC ourselves since their code was not publicly available (see Appendix for further detail MaskCLIP). FT: fine-tuning; LIN: linear-probing: SSEG: semantic
segmentation; DET: object detection; ISEG: instance segmentation.
Method
ViT
INet-1K
ADE20K
COCO
O-Shot
LIN
FT
SSEG
DET
ISEG
MAE
S
14.6
70.9
26.9
28.9
25.6
iOO
S
58.2
79.5
37.0
44.9
39.0
CLIP
S
17.2
57.0
76.7
35.3
41.0
35.7
SigLIP
S
16.0
57.0
76.8
36.2
40.8
35.6
MaskCLIP
S
17.7
56.1
77.0
34.5
39.4
34.5
DeI
S
18.2
50.2
77.7
28.7
42.5
37.1
SLIP
S
18.1
51.5
76.5
35.0
40.0
35.0
SILC
S
20.1
62.8
78.2
38.1
44.0
38.3
CLIP
B
19.0
61.4
79.8
40.7
42.5
37.0
SigLIP
B
18.0
61.8
79.9
40.7
42.5
37.0
MaskCLIP
B
19.1
58.6
79.5
38.2
40.2
35.1
DetailCLIP
B
21.1
63.2
81.6
40.9
45.5
39.5
SLIP
B
19.1
63.0
80.3
42.0
42.0
36.8
SILC
B
23.3
68.1
81.6
44.0
45.9
39.8
S
20.8
64.4
79.5
37.5
45.6
39.5
Harmony
B
23.3
69.6
82.4
44.5
48.1
41.6
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Zero-shot evaluations. We compare VIT-B Harmony against CLIP, SLIP, SILC, MaskCLIP, and DetailCLIP on general and natural image benchmarks for 0-shot classification. Harmony outperforms the other methods on average.
Average
Cal10
S Kin700
MNIST
CLEVR
INet- A
INet- C
LNet-I
CLIP
28.9
53.9
87.2
17.4
10.1
12.8
4.5
24.2
21.4
MaS
28.2
49.8
84.9
15.7
12.1
14.2
4.0
24.8
20.0
DetailCLIP
29.9
55.2
89.1
17.1
11.6
11.6
4.4
27.3
22.9
SLIP
28.3
50.5
85.2
16.2
10.8
13.3
4.9
23.7
22.2
SILC
32.7
58.6
91.5
19.8
15.1
12.4
6.9
29.2
28.4
Harmony
32.9
60.3
94.3
22.5
10.1
12.3
7.6
26.1
29.8
(Monsefi al. 2024), which are joint methods similar ours, We also show that Harmony is compettive with SILC (Nacem et al. 2024) on zero-shot classification, but outperforms SILC other tasks such Mlassfication.. Interestingly, MAE performs substantially worse than the
detection, linear and fine-tuning other methods, which could be due to the distribution of the web-scraped data in CC3M. Even then, using the pixel reconstruction objective in MAE still complements the features for Harmony (shown as an ablation in Table 4 Moreover, SLIP performs worse than expected on classification tasks, which could be due to the small batch size for SimCLR's contrastive loss (Chen et al. 2020a) (768 vs 4096 in the original SL.P (Mu) let al. 2021 ). This issue is less prominent in methods that use self-distillation such Harmony and SILC
[Dong et al. 2023 Nacem et al. 2024 because they do not rely on negative examples in the batch in their
self-supervised losses.
Scaling Harmony. In Figure we present the performance of Harmony and CLIP as we scale model sizes from small to base and large, and as we scale data size from CC3M to CC12M. As we scale the model, the performance gap increases at an increasing rate, which is a sign that Harmony scales better than CLIP with respect to ViT size. The same trend is also shown in Figure going from ViT-S to ViT-B for other tasks.
When scaling the dataset size, the performance gap between Harmony and CLIP narrows. This reduction is expected, improvements become harder to achieve at higher accuracy levels. However, it may also suggest that CLIP benefits more from larger datasets and begins to close the gap with Harmony as the data scale increases. Appendix B shows zero-shot results on CC12M across 8 other datasets.
Zero-shot evaluations across datasets. Moreover, we compare the O-shot capabilities of Harmony to CLIP, SLIP, SILC, MaskCLIP, and DetailCLIP across diverse datasets. We focus on general and natural datasets (as opposed domain specific like StanfordCars (Yang al. 2015) and FGVC-Aireraft (Maji al,, 2013)), and ImageNet robustness datasets like ImageNet-R. We present results in Table [ Harmony outperforms all other methods in 5 out of the 8 datasets, and performs better on average compared to all methods except SILC, where it is competitive. We show qualitative examples between Harmony's and CLIP's s zero-shot predictions in Appendix 4
Retrieval. We also evaluate Harmony for image-text zero-shot retrievals on MS-COCO (Lin et al. 2015 and Flickr30K (Yonng al. 20144. The results are presented Table 3 We observe that for top 5 (R@5) and top 10 retrievals, the gap between Harmony and the other methods grows rapidly, indicating that our method is more robust, since it still generates higher probabilities for the correct retrieval, even if it can't retrieve it within the first examples.
Li ct al
2022
Coates et al.
2011
Carreira a.
2022
Lecun el
098
Johuson al.
2016
Henddryks
b
Hendrycks
212Th
Hendrycks
2021
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Results of zero-shot image-lext retrieval Flickr30K and MS-COCO da lasels.
Flickr30K
MS-COCO
Image-to- Text
Text-to- Imaaee
Image-to- Text
Text-to-Image
R@1
R@5
R@10
R@1
R@5
R@110
R@1
R
R@10
R@5
R810
CLIP
4.9
13.8
20.2
4.6
12.6
18.1
15.4
36.2
48.0
13.8
32.9
43.9
MaskCLIP
4.8
14.0
20.5
12.5
17.9
16.1
36.3
48.1
13.4
32.0
42.8
DetailCLIP
6.4
16.8
24.2
5.8
14.8
20.7
17.9
39.4
51.3
14.8
33.7
44.8
SLIP
5.4
15.4
22.3
13.5
19.1
15.4
38.0
50.3
13.4
32.3
43.2
SILC
7.6
18.4
25.4
20.6
39.7
51.4
Harmony
9.8
23.7
32.6
8.6
20.7
28.1
22.1
46.9
58.7
19.1
42.1
53.8
Table Ablation study for Harmony. The upper section represents the addition of the main three objectives of framework. The feature prediction task (Cp) is described in Section 3.2 soft targets (Cson) in Section 3.1 and pixel reconstruction (Ln) in Section 3.3 The lower part shows the addition of the two text losses: Masked language modeling (Cwr and text self-distillation (LTD). We train VIT-S for 25 epochs and batch size 768 for all ablations. 'Text losses are compared to the original CLIP baseline.
Method
Compute
INct-1K
ADE20K
Mem
Time
O-Shot
FT
LIN
SSEG
CLIP
1.0x
1.0x
17.2
76.7
57.0
36.5
+ Lp
2.3x
3.0x
19.8
12.6
78.2
f1.5
61.7
14.7
37.9
f1.4
+ Lsoh
2.3x
3.2x
20.4
10.6
79.4
f1.2
64.3
12.6
37.0
10.9
+ LR
2.4x
3.9x
20.4
10.0
79.5
10.1
64.4
10.1
38.0
11.0
Text
+ LM
1.1x
1.3x
17.7
10.5
76.8
10.1
35.8
10.7
+ LTD
1.1x
1.4x
18.6
10.9
77.1
10.3
36.4
10.6
LH
2.5x
4.2x
20.8
13.6
79.5
12.8
64.4
17.4
37.5
11.0
12

---

Published in Transactions on Machine Learning Research (06/2025)
Table Equal compute comparison. Performance of CLIP, SLIP, and MaskCLIP when trained with the same hyperparameters and the same computational resources (light gray) as Harmony. We saved checkpoint every 10 epochs and reported results for the highest performance on zero-shot classification, because the models were overfitting due to the larger number of epochs. Total memory is calculated across all GPUs used, and GPU hour is calculated for a single GPU.
Method
GPU Hour
Total Memory (GB)
Batch Size
Epochs
0-Shot
FT
CLIP
90
76
768
25
17.2
76.7
CLIP
413
209
2400
120
17.9
77.7
SLIP
167
184
768
25
18.1
76.5
SLIP
426
240
1024
65
18.4
77.7
MaskCLIP
105
197
768
25
17.0
77.0
MaskCLIP
417
202
800
100
19.6
78.4
Harmony
414
219
768
25
20.8
79.5
4.3 Ablations
Building Harmony. To quantify the improvements from each objective in Harmony, we rebuild Harmony's architecture step-by-step, starting at CLIP (Radford et. al. 2021). The results for pre-training a CLIP on CC3M (Sharma et al.) 2018) are shown in Table 4 We continue by adding a visual feature prediction task in the form of self-distillation (Lp). This substantially boosts the accuracics across all tasks. Subsequently, we add soft targets (Csont by utilizing the same teacher network Ey used in the feature prediction, further increasing performance. However, we notice degradation in the SSEG performance. To remedy this, we add the pixel reconstruction loss (CR) from MAE (He et al. 2021), which operates in the pixel space, leading to more granular pixel-level features being Icarned, increasing SSEG closer its value after feature prediction.
Moreover, we investigate the effect of adding our two text losses to the original CLIP. Starting from the same baseline show Table 1 adding both MLM (EM) and text self-distillation (CTD) objectives increases the 0-shot classification by +1.4% compared CLIP. This is likely due boost in representation ability the text encoder ET. and the introduction of an EMA teacher text encoder Er. However, even in downstream tasks that do not utilize text encoders like FT, there is still slight increase in performance, likely due to the making the contrastive learning task more meaningful by enhancing textual representations, leading to more reflective similarity matrices in Equation 3
fyper-paramcter tuning. We tune the introduced a₂ parameter of Equation 5 that controls the weight of soft and hard CLIP targets, and the parameters Equation [3 that control the influence of each objective Harmony. Tn Appendix D we provide additional experiments such as the effect of (1) passing the two global crops compared
single image our pixel reconstruction objective, (2) using iBOT's data augmentation with CLIP, (3) using mask scheduler to change the MAE mask ratio throughout training, and (4) masking
the images passed to CL.P (Li et al. 2023 Yang et al. 2023
Influence of ae for soft targets. For the a. parameter in the contrastive loss, we compare two situations: A higher alpha value (a. = 0.2) reached early in the training (10 epochs) results a 0-shot accuracy of 20.5, while a lower alpha (a. = 0.05), reached later the training (15 epochs) results 0-shot accuracy of 16.6.
Loss weight. In Table F we adjust the weight parameter for each of the four loss functions added on top of the contrastive loss. We observe slight performance changes, with setting the weight to one generally performing better.
Batch size. We investigated the effect of changing the batch size for Harmony, using a VIT-S. Table shows the results. The performance for 0-shot classification gradually increases as we go from a batch size of 512 to 1024, but then decreases when we go to 2048.
Equal compute. As shown in Table 4 Harmony uses 2.5 times more compute and takes 4.2 times more time to train for the same number of epochs, compared to CLIP, SLIP and MaskCLIP. This makes a direct
13

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: ImageNet-I1 1k 0-shot accuracy for different Loss Weighting. We observe that setting the weight performs slightly better. We compare CLIP iBOT for a, CLIP MAE for B, CLIP + MLM for 7: and CLIP + MLM + TextDist for å with 7 set to 1.
Weight
8
7
0.5
0.5
0.1
0.2
0-Shot 19.1
19.2
16.8
17.2
17.1
17.3
17.8
18.0
comparison between the two methods, while controlling for the number of epochs is unintuitive. Instead, in Table we control for memory and time rather than the number of epochs. We increase the batch size to use more memory for the other methods. Harmony still significantly outperforms the other methods, which seem plateau after 25 epochs, increasing their accuracy only by 1% or less (except MaskCLIP). For each run, checkpoints were saved every 10 epochs, and only the best result is shown, because the models were overfitting due to the larger number of epochs.
Table 7: Influence of changing Harmony's batch size.
Batch Size
512
768
1024
2048
0-Shot
20.3
20.8
21.1
20.6
5 Conclusion
We present Harmony, joint self-supervised and veakly-supervised method for learning generalized visual features from web-scraped data, introducing a soft loss and a text self-distillation method. Harmony outper- forms or is competitive with leading methods and baselines across classification, segmentation, and detection tasks, highlighting how our multiple training objectives can complement each other learn stronger visual reprosentations.
6 Acknowledgement
We thank Paul Scotti from Stability AI for the discussion on soft targets for the contrastive loss. We also thank Yifan Yang and Weiquan Huan for sharing their retrieval evaluation code and for general discussion about attentive masking. The KAUST Supercomputing Lab has been gratefully acknowledged.
14

---

Published in Transactions on Machine Learning Research (06/2025)
References
Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progres- sive sclf-distillation, 2022.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embeddinng predictive archi- tecture, 2023.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers, 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmm: High quality object detection and instance segmentation, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments, 2021a.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021b.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note the kinetics 700 human action dataset, 2022.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmllb detection toolbox and benchmark, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations, 2020a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning, 2023.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive lcarning, 2020b.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning, 2022.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudik (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223. Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: //proceedings.mlr. 155 wttp://////atesaa...a1......... html
MMSSegmentatiion Contributors. MMSegmeentattinn: Openmmlab semantic segmentation toolbox and bench- mark.
2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A. Efrrss. Unsupervised visual representation learning by context prediction, 2016.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Maskelip: Masked sest-distillation advances contrastive language-inage pretraining, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu Enwei Zhang, Wei Liu, Jie Yang, Ke Li, and Xing Sun. Softclip: Softer mross-modal alignment makes clip stronger, 2023.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Talloc, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self- supervised learning, 2020.
Agrim Gupta, Jiajun Wu, Jia Deng, and Li Fei-Fei. Siamese masked autoencoders, 2023.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of dustr--distiibution generalization, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples, 2021b.
Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, and Jiashi Feng. Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1 13, 2024. ISSN 1939-3539. doi: URL http://dx..o...rrgg///// wtt.://////..11///P.....
33366525
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. doi: 0..1109/5.726799
Fei-Fei Li, Marco Andreeto, Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fino-grained visual classification of aircraft. Technical report, 2013.
Paulius Micikevieius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training, 2018.
Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishnan. simple, efficient and scalable contrastive masked autoencoder for learning visual representa- tions, 2022.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Amin Karimi Monsefi, Kishore Prakash Sailaja, Ali Alilooee, Ser-Nam Lim, and Rajiv Ramnath. Detailclip: Detail-oriented clip for fine-grained tasks. arXiv preprint arXin:249.00000 09.06899, 2024.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-
image pre-training, 2021.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Lue Van Gool, and Federico Tombari. silc. In European Conference on Computer Vision, pp. 38-55. Springer, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wee Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bo- janowski. Dinov2: Learning robust visual features without supervision, 2024.
Utku Ozbulak, Hyun Jung Lee, Beril Boga, Esla Timothy Anzaku, Homin Park, Arnout Van Messem, Wesley De Neve, and Joris Vankerschaver. Know your self-supervised learning: A survey on image-based generative and discriminative training, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015.
Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors, 2023.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Vidcomac: Masked autocncoders are data-cfficient learners for self-super video pre-training, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense languae inference, 2024.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation, 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
L. Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine-grained catego- rization and verification. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3973 3981, 2015. URL https:://app www.eeemmcooollaaaa......a sttp::://pp.eemmatttccccoooaa hrr.tooo.r....rrrroprpss:::21111
2510693
Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, Xinyang Jiang, Huiqiang Jiang, Fangyun Wei, Yin Wang, Han Hu, Lili Qiu, and Yuqing Yang. Attentive mask clip, 2023.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67 78, 2014. doi: 10.1162/tacl 00166. URL https:
//aclanthology.orgg org/Q14-1006
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning, 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEFE/CVI internalional conference compuler vision, pp. 11975-11986, 11986, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), pp. 5122 5130, 2017. doi: (0.1099/CVPR2201., 2VP5.2017.544
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer, 2022.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A Experimental Details
Pro-training.
We use the AdamW oplimizer (Loshchilov & Hutler 2019). and mixed precision with FP16, in all our pre- training experiments. Following DINO (Caron et al. (021b) and iBOT (Zhou et al. 2022) we scale the learning rate with the batch size through Ir = 5e x batchsize/250 The learning rate is ramped up lincarly from 0 to the value in the formula in the first 3 epochs, then decreases to 1e-6 using a cosine scheduler, by the end of training. For iBOT and DINO heads, we set the output dimension to 8192, and we share weights in both the iBOT and DINO heads for both the student and teacher. We also normalize the last layer for stability, and set the momentum parameter for the teacher to 0.996. For the masked image modcling in iBOT (Zhou et al. 2022), we follow the settings in their ViT-B ImageNet-22kk experiments. Specifically, the prediction ratio is selected randomly as either 0 and 0.3 plus an added variation of =0.2. Masking is performed block-wise. For the CLS objective in Equation G we set the temperature parameter 7 to 0.04. For the MLM objective shown in Equation S the temperature parameter is linearly scaled from 0.04 to 0.07 in 10 epochs. We use a weight decay that scaled with a cosine scheduler from 0.04 to 0.4, and use gradient clipping with a max gradient norm value of 3.0.
For data augmentation, likewise follow the settings in iBOT Zhou et al., 2022 For the global crops, we use cropping with a scale uniformly sampled from [0.32, 1]. In all our experiments, we use 8 local crops in the global objective, with cropping scale sampled from [0.05, 0.32). For the first global crop, we always apply a Gaussian blur, while we randomly apply it to the second crop with a probability of 0.1. We also randomly solarize the second crop with a probability of 0.2 and threshold of 128. We also apply a horizontal flipping to both crops with a probability of 0.5, color jitter with probability of 0.8, and grayscaling with probability 0.2. For the local crops, we take random crops of size 96 x 96, and apply the same horizontal flipping and color jitter. We also apply Gaussian blur with probability 0.5.
For the MAE pixel reconstruction objective, we randomly mask (remove) 75% of all patch tokens following He et al. (202T). We reconstruct both global crops from iBOT (see Table 12). The final pixel reconstruction loss is then calculated as the addition of the loss of both global crops. For the MLM, objective, each token is randomly masked with 20% probability.
For all VIT-S and VIT-B experiments, we use between and 16 V100 GPUS running for a day. For the VIT-L run, we 16 80GB A100 GPUs instcad, running for day.
Fine-tuning on ImageNet-lk. For fine-tuning evaluations, we use the pipeline in iBOT (Zhou et al. 2022), which follows the fine-tuning protocol from BEiT (Bao et al.] (022).. We fine-tune for 100 epochs in all experiments using the AdamW (Loshchilov & Hutter 2019) optimizer, with a batch size of 1024. Wo linearly warmup the learning rate to 4e- in 20 epochs then lower with a cosine scheulder to a final value of 66 We use a layer-wise decay of 0.65, and a weight decay of 0.05. We do not use layer scale. We use 8 32GB V100 GPUs for our fine-tuning experiments.
Linear-probing classification on ImageNet-lk. For our finear-probing experiments, we mostly follow DINO (Caron et al.] 2021b) and DINOv2 (Oquab et al. 2024). We train for 100 epochs and a batch size of 8192 in all our experiments. We use the CLS token only from the last layer of the ViT encoder. Following Oquab et al. 2004).. we instantiate multiple linear layers that all take the same encoder output, but are trained with differrntt learning rates for efficiency. This way, the larger backbone inference is done once per iteration, while multiple inferences are done on the much lighter lincar layers. We specifically span lcarning rale values of {le-s, 26-5. -5-- 5 1, 24-4, 5e -4, le- 2e 2e-3, 5e 5e- le le-2, 2e -2, 5e 5e- 2, 1e_,,, and we linearly scale them using the formula Ir x batchsize/256 We report the best performing value. We use stochastic gradient descent optimizer with the momentum set to 0.9 and no weight decay. We use 8 32GB V100 GPUs for our linear-probing experiments.
ADE20K semantic segmentation. In our semantic segmentation evaluations, we also use the iBOT (Zhou et al. 2022) pipcline, which follows BETT (Bao al. (2022). Specifically, we use UperNet (Xiao et al., 2018) from the implementation in mmsegmentaion (Contributors, 2020). We fine-tune for 160k iterations using a batch size of 16 and image size of 512 x 512. The AdamW optimizer is used with an initial learning
19

---

Published in Transactions on Machine Learning Research (06/2025)
rate of - that is linearly warmed in the first 1500 epochs, then decays to 0 throughout training. We use layer decay rale of 0.65 and weight decay of 0.05. We use Feature Pyramid Networks (FPNs) of four different scales to modify the feature map sizes generated by the ViT. Specifically, we upsample the output feature of the 4th block and 6th block by 4x and 2x, respectively, keep the output from the 8th block the same, and downsample the output feature of the 12th block by 2x. For data augmentation, we adopt the default settings in mmsegmentation, which includes random horizontal flipping, random re-scaling with a 2.0], and random photometric distortion. We use 32GB V100 GPUs for our semantic
ratio range of [005, segmentation experiments.
COCO object detection and instance segmentation. For object detection and instance segmentation, we use Cascade Mask R-CNN (Cai & Vasconcelos) 2019 He et al. 2018) implementation with mmdetection (Chen et al. 2019), which produces both bounding boxes and instance masks. We follow the settings in iBOT (Zhou et al., 2022) in that we use multi-scale training with the shorter size between 480 and 800 while the longer size is not larger than 1333. We use a learning rate of 1e_4, weight docay of 0.05, and layer
fine-tune for 12 epochs with the learning rate decayed with rate of 0.1 epochs 9 and 11. We decay rate of 0.75 and batch size of 16. We generate hierarchical feature maps by taking the outputs from layers 4, 6, 8, and 12, and passing them to two deconvolutions, one deconvolution, identity mapping, and max-pooling, respectively. We do not use multi-scale testing. We use 4 32GB V100 GPUs for our object detection experiments.
Zero-shot classification. For zero-shot classification, we follow the standard implementatiin [Raffrdd et al. 2021 Mu et al.] 2021) of encoding class labels in description, such as "a photo of {label}, and calualting the consine similarity between the text and image embeddings. We consider the highest generated similarity for each label pair to be the predicted class. For Harmony, we use the teacher text encoder.
20

---

Published in Transactions on Machine Learning Research (06/2025)
B CC12M Bero-shot classification performance
We evaluate zero-shot classification performance of Harmony and CLIP, both trained on the CC12M dataset, across diverse set of general vision benchmarks. As shown in Table S Harmony outperforms CLIP on average.
Table 8 Zero-shot evaluations on CC12M. We compare ViT-B Harmony against CLIP across a range of general and natural image benchmarks for zero-shot classification. Harmony continues to outperform CLIP average.
Average
Cal101
STL-10
Kin700
MNIST
CLEVR
INet-A
INet-O
INet-R
CLIP
36.6
72.8
94.8
26.6
10.7
16.2
7.9
39.5
44.1
Harmony
39.6
76.0
96.7
30.5
9.6
12.7
12.2
36.0
54.2
21

---

Published in Transactions on Machine Learning Research (06/2025)
C Reproducing MaskCLIP
As of writing this paper, the code for MaskCLIP's implementation is not yet public so we had to reproduce it ourselves. To do that, we followed the approach in their original paper (Dong et al. 2023). Namely, we define two vision encoders, student By and teacher Fy, and a text encoder Er. We optimize CLIPPs contrastive loss, masked self-distillation loss, and masked language modeling simultaneously.
C.1 Implementation of Masked Self-distillation
We pass a masked image to the student encoder, and the full image to the teacher encoder and optimized the the cross-entropy between masked student patches and unmasked patches from the teacher. More specifically, given an image x, we obtain masked embedding from the student. encoder by 6m ()F(xm))) where zm masked view of æ and D is transformer decoder with layer, 16 attention heads, and with the student encoder E. We also obtain target embedding from the
the same embedding dimension teacher encoder E by E(x). Given masked vector with indices corresponding to masked patches, the masked self-distillation loss function then becomes
MiskDiat
Σ m, (((()))P(em)))
(14)
m
P is the softmax function, and we used a masking ratio of 75%, removing all mask tokens, just like in MaskCLIP (Dong et al.) 2023). For the teacher encoder, linearly increased to 0.9999 throughout training, and we use minimal augmentation with cropping of scale (066, 1] and normalization only. We use the same learning rate parameters in MaskCTIP, and use loss weight of 0.05 for the masked self-distillation and MLM objectives.
momentum parameter of 0.999 that
C.2 Ablation of MaskCLIP
Additionally, we try out different parameters and architectures to further improve the performance of the framework. Namely, we try (1) adding the iBOT head (Zhou et al., 2022), which maps the ViT outputs to vectors of size 8192 using MLP layers, (2) adding the CLS-level objective from iBOT (Zhou et al. 2022) (Our Equation 6 and (3) adjusting the weight for the masked self-distillation and mlm losses. We show our results in Table 9 Having no CLS objective is almost identical to the original MaskCLIP implementation, depending on if centering and sharpening from DINO (Caron et al. 2021b) is done, which is not explicitly described in their paper. Removing the CLS objective and adding the iBOT head result in identical (-shot performances, both of which are only marginal improvements. We opt to use the iBOT head because it resulted in better generalization across values for our Table 2
Table 9: MaskCLIP ablations. The "default" settings uses both the MIM and CLS objectives and does not use the iBOT head. The first column after the default shows the effect of removing the CLS objective, the second column shows the effect of increasing the self-distillation and MLM loss weights from 0.05 to 1, and the third column shows the result of adding the iBOT head.
Default
No CLS obj.
Weighting to 1 iBOT head
iBOT head & no CLS
INct-1k 1kk 0-shot
17.3
17.7
16.7
17.7
17.4
22

---

Published in Transactions on Machine Learning Research (06/2025)
D Other Ablations
D.1 MAE Masking Scheduler
We tried the idea of increasing the masking ratio for pixel reconstruction throughout pre-training using linear scheduler. This idea motivated by the fact that having constant masking ratio is not optimal for scaling the number of itcrations, since the task of reconstruction does not become more difficult. This is unlike self-distillation where the teacher model produces better and better representation targets for the students to predict throughout training. As a result, we experimented with using a linear masking ratio scheduler that increases the ratio from 65% to 85% in the first 15 epochs of pre-training, In Table [O we compare our results to using a static mask ratio of 75%. Classification results mostly stay the same, but segmentation goes down by 1.2%. Further experimentation of the scaling behavior of using masking scheduler (especially scaling the number of iterations), and tuning of apper-parametes, like using lower end value or higher start value, might lead to different conclusion.
Table 10: Mask scheduler. Classification results stay the same but segmentation goes down by more than percent. The second row linearly scales the masking ratio from 65% to 85 % in the first 15 epochs of pre-training. Both experiments start with the same bascline of CLIP with soft targets + iBOT + MAE.
Masking Percentage
INET-IK
ADE20K
O-Shot
FT
LIN
SSEG
75%
20.8
79.5
64.4
36.2
65%
85%
20.6
79.6
64.5
35.0
D.2 CLIP and MAE Augmentations
We compare different data augmentation strategies on CLIP and MAF. For CLIP, we try using the standard augmentation of randomly cropping with scale in the range [0.4, 1] and random horizontal flipping 50% of the time. We compared this using the global crop augmentation from DINO and iBOT (Caron el al. 2021b;
Zhou al. 2022). which we described in Section A We show the comparison in Table 11 Both strategies produce comparable accuracios for classification, but iBOT's augmentation produces higher segmentation loU.
Furthermore, we explore the effects of using iBOT's global crops with MAE, Specifically, we compare two scenarios: using a standard augmentation as described above or foeding in both of iBOT's global crops, which is more computationally expensive. We showcase our results in Table 12 Reconstructing both crops substantially improves the fine-tuning performance, so continue using it in our final Harmony framework.
D.3 Random and Attentive Masking
We also tried adopting ideas from FLIP et et al. 2023 and Attentive Mask CLIP (Yang et al. 2023 in an atlempt to improve the efficiency of Harmony. In FLIP Li et al. 2023), they show that randomly masking
Table 11: CLIP augmentation. Here standard augmentation means random cropping and flipping, with no color distortions or Gaussian blurs. Both start from the same vanilla CLIP baseline.
Augmentation strategy
INET-1K
ADE20K
(-Shot
FT
SSEG
Standard
17.0
76.6
33.8
iBOT Global Crop
16.7
76.8
35.3
23

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: MAE augmentation. Here standard augmentation means random cropping and flipiig,, with no color distortions Gaussian blurs. For global crops, we feed both crops and calculate the final loss as the addition of the MSE from both crops.
Augmentation strategy
INET-IK (ACC)
ADE20K (loU)
FT
SSEG
Standard
64.2
25.2
iBOT Global Crops
70.9
26.9
Table 13: CLIP masking. Here standard augmentation means random cropping and flipping, with color distortions or Gaussian blurs. For global crops, we feed both crops and calculate the final loss as the addition of the MSE from both crops.
CLIP masking
INET-IK O-shot
No Mask
16.7
Attentive
Yang ct al. 2023
15.9
Random (Li el al. 2023
14.9
50% of the images being fed to CLIP's image encoder will retain its performance, while substantially im- proving efficiency. Attentive Mask CLIP builds this idea by utilizing ViT's attention matrix (generated using an MAE model, which is our teacher Ev) to mask only the patches with lowest relationship to the caption description. This is calculated using the CLS token of the vision encoder, which will encode semnantic information throughout training because of the nature of CLIP's objective. Both works used significantly larger training datasets than ours, so we wanted evaluate the idcas on a smaller scale.
We trained CLIP model with 50% random and attentive masking and present results in Table Attentive masking performs +1% higher than random masking, but is still worse than masking, We therefore continued with no masking for our vision-language contrastive learning objective.
24

---

Published in Transactions on Machine Learning Research (06/2025)
E
Qualitative Results
We show qualitative results for 0-shot classification of Harmony and CLIP in Figure Harmony has higher chance of identifying the correct guess. Note, that in the first image, Harmony selected a more specific category over the ground truth.
Harmony
CLIP
Maypole
Tripod
Pole
Hook, Claw
Pinwheel
Bow
Torch
Mosquito net
Water Tower
Dragonfly
Goldfinch
Duck
Toucan
Hen
Goose
Vulture
Hen
Toucan
Flamingo
Hummingbird
Sulphur
Cabbage
Monarch
Monarch
Lycaenid
Admiral
Cabbage
Sulphur
Ringlet
Lycaenid
Figure 4: Qualitative results of Harmony and CLIP. For 3 different input images, the top guesses of Harmony and CLIP are shown. Ground truth labels are marked in green.
25

---

Published in Transactions on Machine Learning Research (06/2025)
F
Pseudocode
We present a general pscudocode for our algorithm. More detailed implementation is shared on the GitHub repository. A backward step performed after each loss to free the computational graph, which saves memory.
Algorithm Harmony Pseudocode
Input:
Ev: Ev
// student and teacher vision network student and teacher text network // vision and text decoders // imagee and text self-distillation heads // network momentum // hard loss weight
Er, Er
Dy, Dr
hia lTa
1: for each batch in loader do
2:
i, I batch
// get image and text caption // global views // masked image views // mask text for mlm and self-distillation Us.Us + Ev) Ev(u) // extract image and text embeddings using student and teacher.
u. v f (u)ment(i), augment(()
û, t mask(u), mask(u)
i it mask(t)
7:
U + Ev(u). Ev(u)
8:
t.. + (t) (t)
lecle t- ET),
10:
11:
L +- (hi (Ua, Vs, Ues 12))
// distillation loss 11 performed at each step to save memory // soft and hard contrastive loss
12:
[.backward()
13:
L +- Hardd (Us (x) + (1 (1-a)) Lsoft (Uss u4, 4e)
14:
C.backward()
15:
L +- LR Dv Dvv(ua (uay)s)) v)).
// pixel reconstruction loss // masked language modeling loss
16:
C.backward()
17:
+- ENs (Dr (f,).t)
18:
C.backward()
19:
C + ( (f,), hr, (t:))
// text-distillation loss
20:
C.backward()
21:
22:
update(Ev)
// backpropagation based on the above losses
23: update (T)
24: Ey.params +- m Ev -pyramas + (1 (1-10 (1-m), -frrpammms
// EMA teacher update
25:
Ey.params + m Er. params (1 m), Params
26: end for
26

---

Published in Transactions on Machine Learning Research (06/2025)
G
Comparison with DINOv2
DINOv2 (Oquab et al. 2024 is an obvious comparison with Harmony, as it is an improved version of iBOT Zhoou et al. 2022). However, we do not include it in our results, because we observed poor performance when pre-training DINOv2 on CC3M (linear probing accuracy below 25%), using the publicly shared codebase. Given that the modifications introduced by DINOv2 beyond iBOT are relatively minor, with many being just hyperparameter changes (sce Table 1 in Oquab et al. (2024)), ) we would expect similar performance. We suspect bug may be present in the code, though be more fair to exclude this comparison.
unable to identify it. As result, felt. it would
27


Reviews:
[{"review_id": "Jifb7uYV0g", "reviewer": "Reviewer_S5KU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The motivation is clear and the writing is easy to follow.\n- The architecture is sophisticated and requires some engineering tuning.\n\nWeaknesses:\n\n- Pretraining on CC3M may not be scaled enough. It would be better if authors could include ablations on data scaling. In recent years, the pretraining scale has been ascended to scenarios where CC3M is considered the smallest pretraining scale [1]. Even for early work IBOT, they include pretraining on IN21K. Although it is not friendly to most labs, it may be an important issue for work in this direction. As it is witnessed that the zero-shot performance on the downstream datasets is quite low, the larger-scale dataset for vision-language pretraining is required.\n- Writing issue on related work: need to mention the differences of this work compared with prior works at each subsection to highlight technical differences/contributions to enhance readability.\n- Weakly supervised learning might be a broad term that can be narrowed down to language-supervised pretraining or something else.\n- Notation issue: the function $P$ in eq. 2 and 3 take different inputs.\n\n[1] No “Zero-Shot” Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a hybrid architecture for pretraining with self-supervision and language supervision to enhance the dense prediction performance on downstream datasets, which incorporates five loss functions: (1) vision student-teacher (ST) distillation loss; (2) language ST distillation loss; (3) visual masked modeling loss; (4) language masked modeling loss; and (5) vision-language contrastive ST distillation loss.\n\nExperimental results on IN1K, ADE20K, and COCO validate the effectiveness of the proposed architecture. Besides, better scaling results are demonstrated across the different model capacities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "If the author wants to draw solid conclusions for the pretraining community, more scaled experiments are required, e.g., CC12M.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5n9PEOpxS5", "reviewer": "Reviewer_XaFn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem is clearly introduced, and the paper transparently lays out the connection between prior work and how this work uses components from prior work to construct the Harmony model. The experimental details are clear and experiments are performed across a variety of datasets showing the gains of Harmony. Moreover, the ablations are meaningful and there are results which make clear the computational costs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes Harmony, a self-supervised and weakly supervised framework for training a visual encoder. Harmony builds on a line of work which have extended CLIP, which is here positioned as Weakly-supervised learning or text-guided learning, with additional self-supervised losses. Prior work SLIP adds the SimCLR loss to CLIP, and MaskCLIP adds self-distillation and masked image modelling losses.\n\nIn this paper this idea is extended even further, by (1) adapting the contrastive loss to use soft-targets for the text-guidance, (2) visual feature self-distillation (local and global), (3) MSE loss pixel reconstruction, (4) a masked language modelling loss, and (5) text self-distillation. \n\nExperimental results show that this approach leads to a stronger visual encoder across a variety of vision tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "W1: While the work as is seems fairly solid, leading to clear improvements (despite some computational cost gain during training) on vision tasks, it does (for the lack of a better word) seem a little dated. This area of research is very active, with many new works coming out, and yet most cited work is from before 2024 (with DINOv2 being the only reference from 2024). \n\nThis would not have to be an obstacle, but it is reinforced by the issue that the works which are compared against are all pre-2023, with newer versions of these models having being released since. For instance, SigLIP (for which the paper is never cited in this manuscript) in Table 1 has since been improved upon by SigLIP2 [A].\n\nWhilst SigLIP2 is very recent (February 2025), there are other works that are similar to the proposed paper that would be meaningful to compare against. Like SILC [B] and Brave [C]. Moreover, while DINOv2 is mentioned in the paper it is not compared against. Since the main contribution of this work is driven by performance it is necessary to compare against recent work.\n\n[A] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. Tschannen et al. 2025.\n[B] SILC: Improving Vision Language Pretraining with Self-Distillation. Naeem et al. ICLR 2024.\n[C] BRAVE: Broadening the visual encoding of vision-language models. Kar et al. ECCV 2024\n\nW2: This work builds on CLIP, which is a cross-modal approach that jointly learns a vision and a text encoder, and similarly many of the works that build on it similarly learn an encoder for both modalities, as is done for Harmony. Yet, the emphasis of the paper and the experiments is on vision tasks. Comparable works typically also evaluate on cross-modal tasks, such as image-to-text and text-to-image retrieval, and from the current paper it is not obvious how Harmony performs on cross-modal tasks. This may be by choice, but it would strengthen the paper if it was more clear why it is positioned and evaluated as a vision-first model, rather than a vision-language model.\n\nMoreover, it is not made explicit in the paper but presumably the 0-shot experiments are done in CLIP style, i.e., by encoding the class labels with the text encoder and then argmax over cosine distances between encoded class labels and visual features?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zZAPBwswwM", "reviewer": "Reviewer_oog6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1-The paper provides a thorough and well-structured review of relevant prior work.\n\n2-The problem addressed in this work is of critical importance, garnering substantial attention in computer vision research due to its broad practical applications in tasks such as cross-modal retrieval, image classification, and downstream vision systems.\n\n3-The authors conducted extensive experiments on multiple benchmark datasets, demonstrating that their approach outperforms existing methods\n\n**Weaknesses:**\n\n1-In contrastive learning, such as MoCo-v2 and SimCLR, the negative example plays a crucial role where positive pairs are attracted, and negative pairs are repelled in the embedding space. In this paper, particularly in the abstract and contribution, the authors state that their approach does not use negative examples. However, in different part of the paper the authors state that they are using soft contrastive learning. On page 5, paragraph 2, they describe that they are maximising the similarity between paired embedding sets, $v_i$ and $t_j$, where i = j and minimise the similarity between unpaired sets where $i \\neq j$.  which means that they are using negative examples to train the models. Also, Equations 2, 3, 4, and 5 which are used to align text and images in the shared space, demonstrating that they employ a contrastive approach, which relies on negative samples.  **Could the author clarify this confusion?**\n\n2- The notation $t_k$ in the denominator of equation 2 does not clearly refer to negative examples or what? **Could the author clarify what this notation refers to?**\n\n3-In Section 3.4, it is stated that the soft target is explained in Section 5 (Conclusion), where it is actually located in Section 3.\n\n4- The authors state in section 3.4: “Even though the goal of this work is visual representation learning, we hypothesise that improving the textual representations in our text encoder$ E_T$ can improve visual representation in the vision encoder $E_V$ both by making the soft learning targets described in Section 5 more meaningful and by improving the overall contrastive learning task”. **The statement requires further clarification on how the text encoder enhances visual representation in the vision encoder and how it improves the contrastive learning task.**\n\n5- The authors refer to Table 3 to support their claim about the text encoder.  In Table 3, they compare their approach with the baseline (i.e., CLIP), which does not have discriminative and generative parts (i.e., self-supervised learning) for training the image encoder. Thus, it is unclear whether this gain in performance is due to SSL or to the text encoder with MLM. **To make a fair comparison and prove that the addition part (i.e., EMA with MLM) caused this improvement, they should compare their approach with and without this part to see the effect of this part.** \n\n6- **We encourage the authors to incorporate pseudo-code for the algorithm in the paper, similar to the counterparts, to clarify any ambiguity in the approach.**", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Harmony proposes a framework for learning general-purpose visual representations by combining self-supervised and weakly-supervised learning. Instead of using hard targets for aligning text-image embeddings in a shared space, Harmony introduces soft targets to improve alignment. Additionally, it enhances visual representation learning by integrating both generative and discriminative self-supervised learning techniques into the image encoder.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness section.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews raised concerns about the scale of pretraining (especially given the low zero-shot performance), outdated comparison methods, and a need to clarify several aspects of the proposed work. 

These issues have been satisfactorily addressed in the rebuttal and the revised manuscript. For the final paper, the authors are required to add results on a larger dataset like YFCC15M as discussed in the rebuttal.
Decision: Accept (poster)