Data Efficient Language-Supervised Zero-Shot Recognition with Optimal Transport Distillation
Authors: Bichen Wu
Ruizhe Cheng
Peizhao Zhang
Tianren Gao
Joseph E. Gonzalez
Peter Vajda
Venue: ICLR 2022 Poster (2022)
Keywords: Zero shot learning
contrastive learning
optimal transport
vision and language

Abstract: Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be a broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use InfoNCE loss to train a model to predict the pairing between images and text captions. CLIP, however, is data hungry and requires more than 400M image-text pairs for training. The inefficiency can be \textit{partially} attributed to the fact that the image-text pairs are noisy. To address this, we propose OTTER (Optimal TransporT distillation for Efficient zero-shot Recognition), which uses online entropic optimal transport to find a soft image-text match as labels for contrastive learning. Based on pretrained image and text encoders, models trained with OTTER achieve strong performance with only 3M image text pairs. Compared with InfoNCE loss, label smoothing, and knowledge distillation, OTTER consistently outperforms these baselines in zero-shot evaluation on Google Open Images (19,958 classes) and multi-labeled ImageNet 10K (10032 classes) from Tencent ML-Images. Over 42 evaluations on 7 different dataset/architecture settings x 6 metrics, OTTER outperforms (32) or ties (2) all baselines in 34 of them. Our source code is open sourced at https://github.com/facebookresearch/OTTER.

Published as a conference paper at ICLR 2022
OTTER: DATA EFFICIENT SANGUAGE-SUPERVISSS ZERO-SHOT RECOGNITION WITH OPTIMAL TRANS- PORT DISTILLATION
Bichen Wu'; Ruizhe Cheng²; Peizhao Zhang', Peter Vajda', Josepp E. Gonzalez²
Meta Reality Labs, PC Berkeley
(wb,, stzpz, jajdap! ) fb. com, (chengruizhe, jegonzal beegoonaal/beeeeeeeey.ooo (@berkeley.edt
ABSTRACT
Traditional computer vision models are trained to predict a fixed set of predefined categories. Recently, natural language has been shown to be broader and richer source of supervision that provides finer descriptions to visual concepts than supervised "gold" labels. Previous works, such as CLIP, use InfoNCE loss to train a model to predict the pairing between images and text captions. CLIP, however, is data hungry and requires more than 400M image-text pairs for training. The inefficiency can be partially attributed to the fact that the image-text pairs are noisy. To address this, we propose OTTER (Optimal TransporT distillation for Efficient zero-shot Recognition), which uses online entropic optimal transport to find soft image-text match as labels for contrastive learning- Based on pretrained image and text encoders, models trained with OTTER achieve strong performance with only 3M image text pairs. Compared with InfoNCE loss, label smoothing, and knowledge distillation, OTTER consistently outperforms these baselines in zero-shot evaluation on Google Open Images (19,958 classes) and multi- ImageNet 10K (10032 classes) from Tencent ML-Images. Over 42 evaluations on 7 different dataset/archiiccctre settings x 6 metrics, OTTER outperforms (32) or ties (2) all baselines in 34 of them. Our source code is open sourced at https
INTRODUCTION
In real-world image recognition tasks, input images come from broad range of distributions, spanning tens of thousands of object categories unknown during training. It is thus important for large number of visual concepts that may or may be
computer vision models to generalize present in the training data. This problem is called zero-shot learning (ZSL), which aims to transfer knowledge from some known classes with training data to much larger number of unfamiliar classes.
Previous works on ZSL have explored using attributes (Romera-Pareee 2015 2013), class hierarchy (Wang al.) 2018 Kampffmeyer al. (019)) and pretrained word embeddings (Frome et al., |013] Norouzi et al.] (014)) to transfer knowledge from pretrained image representations recognize new classes. Recently, natural language has been used as a powerful source of supervision for visual representation learning. (Desai & Johnson] 2020)) Sariyildiz et al.
& Torr 2013 Akata et al.
2020 Zhang al.) 2020 Tia et al. 2020) demonstrate the effectiveness of pretraining
image-text data. Among them, CLIP (Radford et al. 2021) applies natural language supervision to zero-shot collects an enormous dataset with over 400M image caption pairs from the Internet, and trains an image encoder and text encoder jointly with a contrastive loss to maximize the cosine similarity of paired image and text embeddings. CLIP demonstrates good zero-shot classification results on a wide range of downstream image classification datasets. However, main constraint of CLIP is that requires over 400M image-text pairs for training. Collecting and training on such a huge dataset is very expensive. The inefficiency can be partially attributed to the fact that the training labels from image-text pairs are noisy. As shown in Figure [ in a typical image-texi dataset, observe that images and captions are loosely correlated. It is very common that one caption (image) can potentially match several other images (captions), and the ground-truth pairing is
image recognition.
"qqual contribution

---

Published as a conference paper at ICLR 2022
aeriat: waving The
dinemado perial shot the dramatic cosotinee at dlifs
view and forest
forast of sounted that contrast
landscape heyslack
newly built house to beach
pubilo traditional pub building comer
cottace
Original Captions
most dramatio mountain rangasi
picturesque
village
rock
seen
cinamatic
of
aerial:
landscape mown grarss
cottage picturesque
Permuted perial
woman waring ams
pine fonast
stunted
small the
Captions the dramatic
dramatio
traditionll
coantine
haystack
sharp
vilage
pub building corner
renges!
rook
seen
Figure I: Images and captions are only loosely correlated in many image-texi datasets. The ground- truth pairing is not the only sensible match between texts and images. In the example above, we can find permutations of text captions that can still match with original images.
not the only sensible match. Note that examples in Figure 1 are not hand-picked special cases. In fact, such noisy image-text matching is prevalent image-text datasets.
To quantitatively analyze this, we use a CLIP et al. 2021) VIT-B/32 pretrained on OpenAl's 400M dataset to estimate the matching probabilities between a batch of paired image-text samples. Specifically, we randomly sample 1000 batches from the CC3M (Sharma et al. 2018)) and YFCCI5M (subset of YFCC100M (Thomee et al. 2016) datasets, and use the pretrained CLIP model compute the image-to-text matching probabilities by taking the dot-product of the feature embeddings and taking a softmax along each row. For each batch, we compute three statistics (averaged across rows): default probability, non-default max probability, and non non-default average probability. Note in both datasets, the matching probability between paired samples are far smaller than 1.0, and the probability decreases with the batch size. This indicates that there exist image and text samples that are not paired, but have nontrivial matching probabilities. This is further confirmed by the max matching probabilities between unpaired samples. In the extreme cases (CC 3M, 2048 batch size), the average of max matching probability between unpaired image-text samples is very close to the average of probability of paired samples. Despite prevalent noisy matching between images and texts, CLIP uses the InfoNCE loss (Hadsell et al.) (2066) for training and uses the ground-trute pairings as hard labels. This ignores the many-to- to-many relationship within batch of images and text captions, leading to noisy training signals and lower data efficiency.
Table 1: Matching probabilities estimated by CLIP on Conceptual Captions and YFCC
Datasct
Batch Size
Paired
Unpaired Avg
Unpaired Max
512
0.565
0,001
0.215
CC 3M
1024
0.480
0.001
0.230
2048
0.398
0.000
0,238
512
0.628
0.001
0.197
YFCC 15M
1024
0.551
0.000
0.219
2048
0.469
0.000
0.239
To address this, we propose OTTER, Optimal TransporT distillation for Efficient zero-shot Recognition. We improve InfoNCE to consider the many-to-many relationship between unpaired images and texts. Specifically, given a batch of image and text tuples {(v., (=)NN we first use image/text encoders to estimate a similarity matrix whose elements denotes similarity from image v; text caption ty. Based on the similarity matrix, we use optimal transport find matching probability between each possible image-text combination. To model the many-to-many relationship. we add an entropic regularization to the optimal transport so that the match is softly assigned. Entropic-regularizzed optimal transport can be solved efficiently with the iterative Sinkhorn- Knopp algorithm (Cuturi 2013). Finally, we use the match as soft label to train the image and text encoders.
Based on pretrained image and text models, we use OTTER to train zero-shot models on the Conceptual Captions (CC) (Sharma al. 2018), (subset of) Wikipedia-be based Image Text (Srinivasan jet al., (02T), and YFCC 15M (Thomee et al.) (016) datasets, which contain 3M, 5M, and 15M image-caption pairs, respectively. We evaluate the image encoder's zero-shot recognition of common
visual concepts on Google Open Images (GOT) (Kuznetsova et al. 2020 (19,958 categories) and
2

---

Published as a conference paper at ICLR 2022
multi-labeled ImageNet 10K (10032 categories) from Tencent-ML-Images (Wu ct al. 2019a). Over 42 evaluations on different aatasst-architecturr settings X 6 metrics, OTTER outperforms (32) or ties (2) all baselines in 34 of them. We also propose quantitative vision-language compositionality benchmark and show comparable results to CLIP in Appendix D
2 RELATED WORKS
Zero-Shot Learning in Computer Vision: Zero-shot learning (ZSL) studies the generalization of knowledge to unseen classes. Previous methods for zero-shot recognition in computer vision mainly follow three paradigms. The first type, including DiVE (Frome et al., 2013) and ConSE (Norouzi) et a 2014), uses pretrained word embedding vectors to represent different categories and implicitly model their relationships. However, word embedding is a preliminary and limited representation
of class relationships, which hurts performance. The second paradigm, including GCNZ (Wang)
et al. 2018), DPGZ. (Kampfimeyer et al., 2019), and H7SL (Liu et al.) [220), explicitly models class relationships as a graph, and uses a graph convolutional network (GCN), or a predefined class hierarchy, such as WordNet (Feinerer & Hornik 2020). to learn the knowledge propagation between classes However, real-world class relationships are complicated and simple graph structures such as WordNet are too limited to model such relationships. Lastly, PRomera-Paredee & Torr 2013 Akata et al. 2015 2013) rely on human-labeled attributes to model semantics of classes. The scalability of these methods are limited by the need for attribute annotations. More recently, CLIP (Radford et al.) 2021) applies unguuag-ssupevision to ZSL by training on image caption pairs. Our work is based on CLIP and we generalize the InfoNCE loss to improve its data efficiency.
Vision and Language: Natural language has long been used as a source of supervision in fields like image-text retrieval Hironobu et al. (1999), object classification WWagg ct al. (2009). and video
understanding (Ramanathan et al. 2013). y. Socher et al. (2014); Karpathy et al. (2014); Li et al.
(2019); Chen al. (2021a) have proposed methods of learning visual and language representations
in a joint embedding space. More recently, (Lu et al.] 2019 Chen et al.] 2020c Qi et al., 2020)
propose using a cross-modal attention mechanism increase performance image-text matching.
In addition, (Joulin* et al. 2016: Li et al. 2017: Desai & Johnson 2020, Saaryildizz et al. 2020)
demonstrate that good visual representations can be learned by predicting image captions. To scale
up dision-languee joint training. CLIP (Radford et al. [021] and ALIGN (Jia et al. 2020) both collect their own image-text datasets with 400M and 1B image-caption pairs.
Optimal transport (OT) a theory that enables comparison of two probability distributions whose supports may not overlap. OT has been applied to many areas such as domain adaptation (Courty ct al. 2016), generative models (Salimans et al. 2018), and self-supervised vision representation learning (Caron et al. 2020 Asano al. 2019). In vision and language, (Chen et al. 2020a) uses OT to align objects in images and words in texts. The problem formulation of our work is similar to (Damodaran al. 2018), where OT is used mitigate the label noise in remote-sensing data under supervised learning. In our paper, we extend the method from supervised learning to contrastive learning, where OT is natural way to estimate pairings between images and texts. In another related work. (Chen et al. (2021b) adds an additional OT-based Wasserstein loss to contrastive representation distillation (Tian et al. 22019) The loss matches student representations to teacher representations in a batch. (Chen et al. (021b) is different from our method since it directly minimizes the Wassertein loss between two models' representations, while our method uses OT to estimate the pairing probability and use the probability for knowledge distillation. Directly minimizing Wasserstein loss between image/text embeddings in our case will lead to collapsed representations, where models generate constant output regardless of inputs.
Our work is also related areas including learning with noisy labels, contrastive learning, and knowledge distillation. Our method uses OT to estimate the matching probability of unpaired images and texts. This reminiscent to estimating label transition probability under noisy labels (Song et al., 2020). Our method is based on contrastive learning, which is commonly used in self-supervised visual representation learning (Chen al. 2020b He et al. 2020). For vision representation learning, (Robinson et al., 2020) argues that sampling hard negative pairs can improve learning efficiency. For language-supervise important to mitigate the noise of widely spread hard negative samples, since positive image-text pairs are usually only loosely correlated. Our method also an extension to knowledge distillation (KD) (Hinton et al. 2015). Typical KD directly relies teacher model to directly generate target distribution (Xie et al. 2020 Bagherinezhad et al. 2018 Caron et al. 202T). Our method is
Other related works:
representation learning, however, it is
3

---

Published as a conference paper at ICLR 2022
different since our target distribution is computed by OT based on the pairwise similarity estimated by a teacher model. Experiments show that this works better for image-text contrastive learning.
Image similarit
Text similarity
Image-text similarity
S
EMA update
Optimal Transport
Cross Entropy
Image encoder
encoder
Fly over
Dog
EMA update
Mount
Predicted distribution
Target distribution M
Figure 2: Architecture of OTTER. use image and text embeddings to compute similarity matrices S" (and S'), which is then used to solve for matching probabilities M (and M as targets.
METHODS
We introduce OTTER in this section. Let ((v;, t;)) be batch of paired image-text tuples sampled from data distribution p(v, t). Our model contains an image encoder (-) and a text encoder fc(-) that map image vy and text t; norrmraal embeddings z and respectively.
3.1 CONTRASTIVE LEARNING WITH INFONCE LOsS
CLIP (Radford et 2021 trains the image and text encoders with contrastive learning pull the paired image and text embeddings closer, and push the unpaired embeddings farther. This is achieved by miminizing the InfoNCE loss LIiONNC. L, + C. C, is the loss for matching images text captions, and L, is for text-to-image matching. L. is defined as
1 N
N
cxp((z)' (22/TT//
C.
Lg log (og,
Lgg log
(I)
N
Σ exp((z)) )/T)
where (z.T zT) is the cosine similarity between two fz-normalized embedding vectors. is (trainable) temperature parameter. [Ú is the element of an identity matrix In with 4a = 1, Vi and I₁ 0, Vi Symmetrically, we define L. and Pt in the same way as Equation (I except we normalize across z"
Note Pe is normalized across z for k l.... N in the denominator.
Equation (T) a rather redundant way of writing the InfoNCE loss, as lij are all zeros for unpaired image-text samples. However, this shows that InfoNCE is essentially the cross entropy between one-hot distribution It and the estimated probability Pe (z," z);T). One-hot distribution assumes that within batch of images and text captions, the only match for image V, is paired text caption f, However, shown in Figure [ and T this assumption is not true. Paired images and text captions are only loosely correlated. and vice versa. The ground-truth match provided by the dataset is not the only sensible match between images and texts. One-hot labels are therefore noisy, leading to degraded learning performance.
common that one image can match with several other texts
3.2 MODELING THE PROBABILITY UNPAIRED IMAGE- TEXT MATCHING
To better capture the many-to-man relationship in image-text datasets, we modify InfoNCE in Equation (1) to consider the matching probability of unpaired images and texts. For batch of N image-text pairs, we define Y; E (1,.... N) as random variable, and let (V,Y, = j|VI:N (1:)) be the probability that image v, should be matched with text caption t; the batch. We model this
=Y jiViNN ti.N) qo (Y; 4y (Y, # Mi aly (1 a)
(2)
4

---

Published as a conference paper at ICLR 2022
VI.N, 1,:N), M, 0 Vi is the conditional probability of image v being matched to t given that it is not matched to text tz. For simplicity, we write () = go (Y; = jlVi.N, t.:N). a E [0, 1] is the prior probability that image v, is matched with its paired text caption t.. a reflects the noise level in the dataset. In an ideal noiseless dataset, a = 1, so g)) = Iz.. This is the case where we should use the one-hot labels Liş for contrastive learning. However, in image-text datasets,, it is common for an unpaired text caption t to be better match for image vi, as
where Myj := go = j|Y; *
shown Table] T In this case, a < and using Jez as the target distribution is no longer accurate. So we generalize the InfoNCE loss in Equation I by replacing 1s with the more generic q.' (j) as
C. 1
JaTi + My) log log 2):T):
==
ali provides supervision on paired image-text samples and (1 - a Mi supervises unpaired samples. The question is how do we estimate My.. A simple estimation is to let My = (1 - L,)/ -1) ji.j be uniform distribution. This is equivalent to the label smoothing method proposed in (Szegedy et al. 2016). However, this completely ignores the contents of images V1:N and texts t1:N-
3.3
MODELING WITH OPTIMAL TRANSPORT
To design a better method of estimating My. we start from two intuitions: first, in a reasonable image-text dataset, there are no bad images or texts. We assume all the images and texts equally matchable so they should have equal matching probabilities. Second, the matching probability from image v; to caption tÚ should depend on their similarity estimation Sij. A relatively higher similarity Siy should lead to higher matching probability My,- An estimation for Mi, that satisfies the two intuitions can be obtained by solving the following entropic optimal transport problem (Cuturi 2013)
M arg max (M) (M).
(4)
MEM
S" RNNN is a similarity matrix whose elements Sü₃ denotes the similarity from v; to 6j. We discuss how to compute S in Section 3.4 (M, = Ex Sy is the Frobenius inner product between the similarity matrix S and the matching plan M. Maximizing this term ensures M is similar to S", i.e., larger Sy leads to larger Mij and vice versa. Meanwhile, we add an entropy regularization on M as H(M) = - Eij Mij log Mij. This ensures that M does not over concentrate on a few elements. We constrain the solution of Equation (4) to be a transportation polytope
1 1 IN, M
1
M = (ME MM
N 1x).
(5)
This constraint ensures that the solution M satisfies the first intuition - all images and texts are equally important and should be matched with equal probabilities. Moreover, as proven in (Cuturi) 2013). the solution to Equation ④ takes the form of a normalized exponential matrix
(6) where r, RV are and column normalization vectors and can be calculated through the iterative KunkhornKKnopp algorithm (Cuturi 2013). The Sinkhorn- Sinkhorn-Knopp algorithm can be efficiently implemented GPU and we provide pseudo-code implementation in Appendix B
M Diag(r): exp(S" Di)Dia)
From Equation 66 it clear that M satisfies our second intuition that a similarity S.y leads to higher matching probability since My ~ exp(S"), /A). The role of the entropic regularization is also clear. larger X or higher entropy regularization and leads "softer" distribution for Mg On the other hand, smaller À or lower entropy regularization leads to "harder" distribution for Mg.
3.4 COMPUTING THE SIMILARITY MATRIX
To compute the similarity from image v; to text ty, we can use a pair of teacher encoders f(())
compute fo-norrlldeedd embeddings 2j, 2/. Denoting 2", Ž RdxN matrcies whose columns are N ii respectively, we compute the similarity matrix as
S" 7. T Z" 7 Z' + T TTIN.
The first term ZT E R N compares the image similarities, as (Z" 2');; = 2 T cosine similarity between image embeddings. Intuitively, it assumes that for pair of similar images, it is likely that exchange their text captions. Similarly, ZT Z' compares the text similarities.
ž is the
5

---

Published as a conference paper at ICLR 2022
It assumes that if a pair of text captions are similar, it is more likely that one text caption match the other image. The term Zu Ze considers the similarity between the image and text embeddings. Finally, nyIn with + oo ensures the diagonal terms of S" are infinitely small. This effectively sets the diagonal terms of M to O, which is necessary since My is conditioned on Y, * i.
There are several options to instantiate fo(.) and fe(-). The simplest option is to use the original image and text encoder f(()) as fo(-), f.).. Alternatively, following recent works (He et al. 2020 20r0-Caron et al. 2021| Liu et al. 2021), (.)), can share the same model architecture with f.).. but their weights are updated as an exponential moving average as 0 + m8 + (1 (1- m 8,, where is the weight for fo()). fc(.). @ is the weight for fo f.).., fe(-), and m is a momentum parameter set to 0.999. Of course, we can also use trained image and text encoders such as CLIP for (+(-) and fe(-). We adopt the first two options in our paper, since we want to avoid using extra image-text pairs.
3.5 RELATIONSHIP WITH KNOWLEDGE DISTILLATION
OTTER is an extension of conventional knowledge distillation (KD) (Hinton et al. (013). Equation
computes the cross entropy H ( pp') between of (j) and pT( := P. Po(z (z) zy;: T), where 9. (j) is
the teacher distribution solved by OT and (j) is the student distribution with logits (z)" 2j /
computed by f())., f()t. A more conventional way to compute KD's teacher distribution is
cxp((2 25/T)
12. GliT))
(8)
Σ k= exp((2) Z0/T)*
where z,. 23 are computed by the teacher fo(.). f()). We can re- write Equation (8 in the matrix form as Q" Diag(r) exp( / /iag))i)a(()) c). where T; 1. and c, 1/ Ek=1 exp( (Z: TT) Note this teacher distribution has the same form as OTTER in Equation (6 but with two differences. First, OTTER's 's similarity matrix S" in Equation (7) have three more terms: 7, 2 Z" Z' Z', MT,NN- In comparison, KD ignores image-image, text-fext similarities and does not exclude diagonal terms. By 0, their similarity matrices are equivalent. Second, OTTER's normalization vectors r, in Equation (6) are solved with Sinkhorn- -nopp while for KD r, C are computed by a Softmax function. In fact, if we set the #iteration to 0 in Algorithm(2 (Appendix B), Sinkhorn Sinkhorn-Knopp equivalent to Softmax, as also noted by (Caron et al. 202T).
setting To- = "/2
4 EXPERIMENTS
In this section, we discuss our experiments validating the effectiveness of OTTER. We open-sourced our code at http:s::///// //g com/ follow CLIP (Radford et al. 2021) to train an image and a text encoder to predict the pairing of image and text samples using the infoNCE loss. Since the dataset used by CLIP is not released, we train on three publicly available datasets, Conceptual Captions 3M (CC) (Sharma et al. 2018). Wikipedia- Mikipedia-base Image-Texx Text Dataset (WTT), and YFCC 15M (Thomee al.) 2016). We only train on images with English captions in all 10 partitions of the WIT dataset, resulting in 5M image-text pairs in total. Since the datasets we use are small (~~10xx smaller than the one used by CLIP), we have to use pre-trat tre-trinnd models to initialize the image and text encoders. Also, due to the datasets' limited scale and concept coverage, models trained on CC or WIT do not perform well on domain-specift datasets such as Stanford Cars (Krause et al. 2013) and FGVC Aircraft (Maji et al. 2013). To test zero-shot recognition on common visual concepts, we evaluate our models on the test set of Google Open Image (GOI) (Kuznetsova et al. (020).. which contains 19,958 classes We also evaluate on the test set of labeled ImageNet 10K (10032 classes) dataset whose labels come from Tencent ML-Images (Wu et al. 2019a). Each image in ImageNet 10K is auto-labeled with highly-coreelated class labels from GOI, alleviating the single-label issue of ImageNet 21K and 1K. To compare with previous ZSL methods (Norouzi et al.) [0144] Frome et al. [0133] Wang et al. 2018 Liu et al. 2020). we report the ZSL performance of one of our models on N
/TTTER To setup baseline,
We adopt training recipe similar to BiT's finetuning strategy (Kolesnikov et al. 2020): We use SGD with an initial learning rate of 3e-3, a cosine annealing scheduler, momentum 0.9, and no weight decay. Input images are resized to 256x256 and randomly cropped to 224x224 while test images are resized to 256x256 and center-cropped to 224x224. We train on V100 GPUs using Pytorch (Paszke et al. 2019) distributed data parallel with a total batch size of 512 (64 per GPU) for
Training:

---

Published as a conference paper at ICLR 2022
10 epochs. While CLIP (Radford al, 2021) computes InfoNCE using sub-batches each GPU, we gather logits from all GPÜs for OTTER and baselines.
Inference:
For inference, we follow CLIP to compute the text embeddings for the target classes using the trained text encoder, and we use prompt template of "a photo of [label]" " augment the label texts. Next, we fit KNN using the text embeddings. Given an image, we find the top K nearest label embedding neighbors the image embedding based on cosine similarity.
Evaluation: GOI (Kuznetsova et al. 2020) and ImageNet 10K from Pencent-ML-Imagee 2019a) are multi-labeled. Following previous work on ZSL (Norouzi et al. 2014] Frome et al. 2013 Wang et al., 2018 Liu et al. 2020), we use flat hit k (FH@ K) for evaluation. FH@ K is the percentage of test images such that the top K predictions of the model overlap with true labels and is formally defined N E [vv[((v:) for the i-th image and L; is the set of true labels.
[Wu et al.
L; Ø)), where (J(v:)) k is the top K predictions
Table 2: FH@K on test sets of Google Open Images and ImageNet10K from Tencent-ML-Imaan
Image
Text
GOI @ K (%)
INIOK FH @ (%)
Data
Method
encoder
encoder
CLIP (400M)
ResNet50 ViT-B/32
CLIP Transformer
26.5
54.0
64.3
20.1
44.8
56.4
InfoNCE
27.5
55.3
22.5
49.1
60.7
Wide ResNet50x2
InfoNCE
28.6
58.6
11.0
29.9
40.6
InfoNCE
26.8
55.1
66.4
10.9
29.4 .4
40.5
LS
26.3
55.9
67.5
29.6
39.8
ResNet50
KD
26.7
55.3
67.1
10.0
27.5
38.5
OTTER
29.1
59.6
70.9
12.0
31.8
42.1
InfoNCE
22.8
50.0
61.5
7.9
23.7
33.0
LS
19.8
46.9
59.2
6.7
21.9
31.9
ResNet34
DeCLUTR -Sci-base
KD
21.1
47.9
59.8
7.3
23.0
32.5
OTTER
24.2
52.6
25.6
35.4
CC
(3M)
InfoNCE
27.2
57.0
69.0
10.0
27.9
38.5
LS
53.9
65.7
26.7
38.0
FBNeVV3-A
KD
26.9
56.7
68.4
10.7
28.9
39.7
OTTER
27.5
57.2
69.0
10.4
29.4
39.9
InfoNCE
25.7
54.3
66.1
8.7
25.8
35.8
LS
24.8
54.0
66.1
9.7
26.8
37.6
FBNe1V3-C
KD
26.6
55.8
67.6
10.5
28.2
38.9
OTTER
27.5
57.6
69.1
10.4
28.7
39.4
InfoNCE
25.5
52.2
62.8
9.5
26.1
35.9
Sentence -ERR--aae
LS
26.7
37.0
ResNet50
KD
25.6
52.3
62.4
9.8
26.2
36.0
OTTER
26.1
53.1
63.4
9.9
26.6
36.6
InfoNCE
13.5
34.0
44.8
6.3
19.2
27.8
WIT
DeCLUTR
LS
14.3
35.5
46.2
6.4
19.8
28.9
ResNet50
(5M)
-Si--ase
KD
14.4
35.0
45.9
6.2
19.3
28.0
OTTER
14.5
36.4
47.7
29.0
InfoNCE
18.8
42,9
53.6
8.9
26.3
36.9
YFCC (15M)
DeCLUTR
LS
19.6
44.9
55.7
28.2
38.8
ResNet50
-Sci-base
KD
19.5
43.5
54.2
8.9
26.0
36.7
OTTER
20.6
45.4
55.9
9.3
27.4
38.1
4.1
COMPARING OTTER WITH BASELINES
To compare with OTTER, we include three baselines: 1) InfoNCE with hard labels; 2) InfoNCE with label-smoothing (LS) SZzgedy et al. 2016), described in Section 3.2 3) InfoNCE with knowledge distillation (KD) Hinton et al. 2015). as described Section 3.5 In addition to the experimental setting described above, we the following OTTER hyper-parameters:
set the loss coefficient = 0.5, set y, = 7/ for the similarity matrix. We use the aaponentiallmovinng average (EMA) of the image/text encoders teachers and set the EMA decay 0.999. For Sinkhorn-Knopp
7

---

Published as a conference paper at ICLR 2022
Table 3: Flat hit @K on ImageNet 21K+1K
Flat Hit@k(%)
Dataset
Image Encoder
Text Encoder
Method
2
5
10
skip-gram skip-gram
DeViSE
0.3
0.9
2.2
3.6
INIK (1.2M)
ConSE
0.1
1.5
3.5
4,9
ResNet50
Glo
GCNZ
1.0
2.3
5.3
8.1
GloVe
HZSL
2.2
4.6
9.2
12.7
InfoNCE
3.2
4.8
8.8
12,9
CC
LS
3,4
5.1
9.4
13.7
FBNetV3-C
DeCLUTR- Sce-bbee
(3M)
KD
3.6
5.4
9.7
14.0
OTTER
3.7
5.5
9.9
14.3
CLIP (400M)
ResNer50 VIT-B/32
CLIP Transformer
13.5
19.7
30.5
39.4
CLIP
15.3
22.2
33.9
43.3
0.15 and the number of iterations to 5. For the KD baseline, we also use EMA teacher and set
0.5. For the label-smoothing baseline, we set ar 0.9, which yields better results than = 0.5.
On CC, we train the image-tex models based on four different pretrained image encoders: ResNet- (50, 34) (He et al. 3). F3 ( A, C) (Wu et al., 2019b Wan et al. 2020 Dai et al. 2020).
and two pretrained text encoders: DeCLUTR-Sei-bes (Lo et al. 2020) and Sentence BERT (Reimers & Gurevych 2019) pretrained on SNLI (Bowman
(Giorgi al. 2020) pretrained
S2ORC
(et 2015) and MultiNLI (Williams et al.] 2018). We also train ResNet50 + DeCLUTR-Sci--en on the (partial) WIT (Srinivasan et al. 12021) and the YFCCI5M (subset of YFCC 100M) (Thomee 10 on the test sets of both GOI and multi-labeled
al.) 2016) datasets. We report FH@K=1) ImageNet 10K (Wu et al., 2019a). As shown in Table 2 over the 42 evaluations on 7 different dataset-architecture settings x 6 metrics, OTTER outperforms (32) or ties (2) all other baselines in 34 of them. Compared with CLIP's performance on the GOI test set, a ResNer50 trained by OTTER outperforms CLIP-RN5O by 2.6 pts FH@ I and by 6.6 pts FH@ 10. To further illustrate the significance of the performance gain, we show that ResNet50 (25. 6M params) trained with OTTER outperforms a Wide ResNet50x2 (68.4M params) trained with InfoNCE under the same setting.
For reference, to put OTTER in the context of traditional ZSL methods, we present @H@ K. results on zero-shot transfer the ImageNet 21 K+IK (Deng et al. 2009 dataset, which contains 21,841 classes in total. The result is reported in Table 3 With 400M image-tett pairs, CLIP (Radford et al. 2021 vastly outperforms all other methods. ImageNet22K's classes contain many uncommon words, such as scientific names of animals or medical terms. While not directly comparable with traditional 7SL. methods due to differences datasets used and model architectures, OTTER is significantly better than previous ZSL methods, beating the previous SotA, HZSL (iu et al. 2020). by 68% relatively.
4.2 VISUALIZING OTTER
In order to check if the image/text matching found by OTTER is sensible, we provide visualizations of OTTER's ' matching results. In Figure B] we visualize the matching results on a small batch of image-text pairs. We set = 0.5 for paired image-text samples, shown in the diagonal elements in Figure 3 The off-diagonal elements are estimated by OTTER. Since the interpretation of the matching results are highly subjective, we leave the interpretation to readers.
OTTER to process larger batch of 512 image-text pairs. This is batch size for training. We pick the top-8 largest off-diagonal pairs from the optimal tranport result and show them in Figure As we can see, in large batch, we can easily find unpaired images and captions that turn out to be good matches. InfoNCE will simply regard these pairs as negative examples and push them away from each other while OTTER can better handle this by treating them semi- comi-positive pairs.
Next,
4.3 IMPORTANCE OF SIMILARITY MATRIX AND EMA
In Equation 7 design the similarity matrix S the composition of image, text, and image-text similarity matrices. In Table 4 we show experiments to validate the effectiveness of this composition and the necessity of using EMA. There are various levels of performance drop when we don't use the image or text similarities, or when EMA is turned off. Note that our baseline hyper-parameters so the accuracy also different. We compare different settings using FH TH@K=1 on the GOI test set. More in-depth ablation studies are shown in Appendix A
are different from Table 2
8

---

Published as a conference paper at ICLR 2022
0.5
0.003
0.0094
0.065
0.1
0.0044
0.32
1.1e-08
6.2e-07
0.0005
0.5
0.48
0.0086
0.00028
0.00057
0.0054
4.4e-08
86-06
0.003
0.0016
0.0082
0.011
0.0041
1.1e-07
8.1e-07
0.025
0.003
0.0001
0.5
0.12
0.35
0.0017
3.9e-00
1.3e-05
0.29
0.00039
0.0032
0.11
0.5
0.074
0.028
2.8e-07
6.7e-06
0.0074
0.0012
0.021
0.16
0.23
0.084
1.1e-05
3.1e-06
0.19
0.022
0.013
0.15
0.017
0.066
0.5
0.0039
0.031
1e-07
2.6e-07
1.2e-06
4.9e-06
1.2e-05
2e-05
0.023
0.5
0.48
4.2e-07
3.3e-05
4.1e-07
0.00049
1.4e-05
3.4e-06
0.0052
0.49
0.5
the
serial:
chemattc
ofa pine
forest of
rewly buit
public traditional pub old buildig
cottage the picturesque village
small house
dramatic
waeing arms on the rock
trees stand sharp contrast...
grass haystock
mountain
dramatic coastine at the ditls
sas
ranges
beach
have soon
conner
Figure 3: Visualization of OTTER's matching on batch of 9 image/text pairs.
ice hockey player the skratss against sports team
amartonn football player calebrates his touchdown run
Ice hockey player biocks a shot during the sacond period...
vector llsstration with a cup of coffee and hand drawn....
cule cit starring with sharp eyas
seamless sicetch of oup of hot coffee and steam
dose-up of callio cal playing with
profession cuts the hair the cliant with clipper
Figure 4: Visualization of top-8 image-text pairs matched by OTTER in batch of 512 samples. These image/text pairs are regarded as negative samples by InfoNCE.
Table 4: Validation of Similarity Matrix and EMA.
Yv
Yt
EMA
#iter
batch
FH @ K=1
baseline
0.5
1.0
1.0
0.1
512
31.0
0.5
0.0
1.0
/
0.1
512
28.8 (4 2.2)
similarity
0.5
1.0
0.0
/
0.1
512
27.8 () 3.2)
matrix
0.5
0.0
0.0
/
0.1
512
26.1 (} 4.9)
EMA
0.5
1.0
1.0
X
0.1
4
512
30.4 (L 0.6)
CONCLUSION
Image-text datasets collected from the Internet are noisy, and the InfoNCE loss used by previous works such as CLIP fails to recognize the potential matches between unpaired images and captions in batch. As solution, OTTER extends the InfoNCE loss consider the many-to-many relationship between unpaired images and texts by computing a pair-wise similarity matrix and using entropic optimal transport to solve for the off- off-diagonal matching probabilities. OTTER outperforms (32) ties (2) all other baselines on Google Open Images and ImageNet 10K in 34 out of 42 comparisons. In future research, want test the effectiveness of OTTER on larger datasets, such as 400M.

---

Published as a conference paper at ICLR 2022
REFERENCES
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for attribute-base classification. CVPR, 2013.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. CVPR, 2015.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous 2019.
clustering and representation learning. arXiv preprint
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving ima- genet classification through label progression. arXiv preprint arXiv: 180..800.44444 2018.
Samuel R. Bowman, Gabor Angeli. Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. 2015.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski. and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09888 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2/04. 14294, 2021.
Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. ICML, 2020a.
Liqun Chen, Zhe Gan, Dong Wang, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. CVPR, 2021b.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI., 2020b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020c.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. arXiv preprint arXiv:/507.00 00504,2, 2016.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. NeurIPS, 2013.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using neural acquisition function. arXiv preprint arXiv:2006.02049 2020.
Bharath Bhushan Damodaran, Rémi Flamary, Vivien Seguy, and Nicolas Courty. An entropic optimal transport loss for learning deep neural networks under label noise remote sensing images. arXiv preprint ar:iv:1880.000 163, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. pp. 248-255 2009.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. arXiv preprint arXiv:2006.0660
2020.
Ingo Feinerer and Kurt Horik. wordnet: WordNet Interface, 2020. URL https: //CRAN CRAN
R package version 0.1-15.
10

---

Published as a conference paper at ICLR 2022
Andrea AndreaFrome, Greg S. Jonathonn Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. NIPS, 2013.
John M Giorgi, Osvald Nitski, Gary D. Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006. 03669, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255: 2015.
Yasuhide Mori Hironobu, Hironobu Takahashi, and Ryuichi Oka. Image-to-word transformation based on dividing and vector quantizing images with words. In in Boltzmann machines", ", Neural Networks, pp. 405409, 1999.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint urXiv:2102.0591 2020.
Armand Joulin*, Laurens van der Maaten*, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, 2016.
Michael Kampffmmeye, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P. Xing. Rethinking knowledge graph propagation for zero-shot learning. CVPR, 2019.
Andrej Karpathy, Armand Joulin, and Li Fei-Fei. Deep fragment embeddings for bidirectional image sentence mapping. In Proceedings of the 27th International Conference Neural Information Processing Systems - Volume 2, NIPS' 14, pp. 1889-1897, Cambridge, MA, USA, 2014. MIT Press.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 19X2. 13700 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
Ang Li, Allan Jabri, Armand Joulin, and Laurens van der Maaten. Learning visual n-grams from web data. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4193-4202, 2017. doi: 10.1 N0.119//CV..017.450
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, 2019.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. CVPR, 2020.
Yen-Cheng Liu, Chih- Yao Ma. Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09480, 094800 2021.
11

---

Published as a conference paper at ICLR 2022
Kyle Lo. Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4969 4969-49833, Online, July 2020. Association for Computa- tional Linguistics. doi: 10. 10.18653/v1/2020..m aa0.0aam-l-maii4440 main. URL/ttps::/www.cclne org
anthology 2twplllggyyy22oooomm http:/2200...mmmaamam. acl-main. 447
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task lask-sgo visiolinguistic representations for vision-and-len -adn--nad-a tasks. arXiv preprint
2019.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. ICLR, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019, URL wttpap/pppp/pppper...eoooo...... pdf
library.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021.
Vignesh Ramanathan, Percy Liang, and Li Fei-Fei. Video event understanding using natural language descriptions. In 2013 IEEE International Conference on Computer Vision, pp. 905-912, 2013. doi:
10.1109/CCV.2013..11
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL /tattps:///rxii...om
10084
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Bernardino Romera-Paredes and Philip H. S. Torr. An embarrassingly simple approach to zero-shot learning. ICML, 2015.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris N. Metaxas. Improving gans using optimal transport. ICLR, 2018.
Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. arXiv preprint Ntt:22008.0199 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyeed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y.Ng. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Litiioomnniiiiiiiiiiiiiiiii Linguistics, 2:207 2014. doi: 10.1162/tacl a 00177. URL https:/ httpp//e.wwb...gg//attooooooooo/
12

---

Published as a conference paper at ICLR 2022
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac-Gil Lee. Learning from noisy labels with deep neural networks: A survey, arXiv preprint arXiv:2007.88199 2020.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. arXiv preprini arXiv:2103.01913, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 18262282, 2016.
Bart Thomee, David A. Shamma. Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfee100m The new data in multimedia research. Commun. ACM, 59(2):64 73, jan 2016. ISSN 0001-0782. 10. 2212522282222
doi: 10.1145/281280. URL https::/dbi.orrz
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1900. 10699, 2019.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 12965-12974 2020.
Josiah Wang, Katja Markert, and Mark Everingham. Learning models for object recognition from natural language descriptions. In Proceedings of the British Machine Vision Conference, 2009.
Xiaolong Wang, Yufei Ye. and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. CVPR, 2018.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report
California Institute of Technology, 2010.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational I. Linguisties: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. org/ anthology/NN8-
URL http:
Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, and Tong Zhang. Tencent ml-images: A large-scale multi-label image database for visual representation learning.
IEEE Access, 7, 2019a.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CVPR, 2019b.
Qizhe Xie, Eduard H. Hovy, Minh-Thang Luong, and Quoe V. Le. Self-training with noisy student improves imagenet classification. CVPR, 2020.
Yuhao Zhang, Hang Jiang. Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Con- trastive learning of medical visual representations from paired images and texts. arXiv preprint arXiv:2010.00747, 2020.
13

---

Published as a conference paper at ICLR 2022
A ABLATION STUDIES
In this section, we analyze the impact of hyper-parameters on the performance of OTTER in Table S Note that our baseline sypem--arameters are different from Table 2 so the accuracy also different. We compare different settings using P on the GOI test set.
Table 5: Ablation studies.
ResNet50 + DeCLUTR- -ciibbbaee evaluated on GOI test set.
Tv
7
EMA
X
#iter
batch
FH@K=1
baseline
0.5
1.0
1.0
0.1
4
512
31.0
0,1
1.0
1.0
0.1
4
512
29.9 (4 1.1)
a
0,9
1.0
1.0
0.1
512
28.4 (4 2.6)
0.5
0.0
1.0
0.1
4
512
28.8 (L 2.2)
similarity matrix
0.5
1.0
0.0
0.1
4
512
27.8 (.2
0.5
0.0
0.0
0.1
4
512
26.1 4.9)
EMA
0.5 1.0
1.0
X
0.1
4
512
30.4 (L 0.6)
0.5
1.0
1.0
0.05
4
512
29.0 2.0)
0.5
1.0
1.0
0.3
4
512
28.2 2.8)
Sinkhorn
0.5
1.0
1.0
0.1
0
512
29.1 (4 1.9)
0.5
1.0
1.0
0.1
2
512
29.3 1.7)
0.5
1.0
1.0
0.1
512
30.0 (1 1.0)
batch
0.5
1.0
1.0
0.1
4
256
25.6 (1 5.4)
size
0.5
1.0
1.0
0.1
4
768
28.1 (1 2.9)
Confidence in the image-text pairs: In Section 3.2 we define a = q (i) as the probability that the paired text caption is the correct match with the image. This reflects the confidence, or the noise level, in the ground truth pairs. We set a = 0.1, 0.5, 0.9 in our experiment, and found that both lack of confidence (0.1) or over-confidence (0.9) can hurt the performance. Relatively, a = 0.9 leads to worse performance, validating the necessity of mitigating label noise.
Image-to-image, text-to-text similarity: We included the image and text similarity when computing the pair-wise similarities for OT. The assumption is that samples with similar images text captions are likely to share labels. To test this, we set Yes "t to 0 in the experiments. We found that both image-to-image and text-to-text similarity are helpful. Relatively, text similarity seems to be more important than image similarity, as removing text similarity leads to larger performance drop.
Do we need EMA teacher? In the default setting, we used the exponential moving average of the image/text encoders to compute the similarity estimation. We test the alternative option of using the image/text encoders themselves, and found that this leads to small accuracy drop (0.6 points).
One key component of OTTER is to optimal transport match images with text captions within a batch. However, do we really need optimal transport? To compute the teacher distillation, a simple alternative is use a Sofimax function. As we discussed in Section 3.5] Softmax is equivalent to our Sinkhorn- Knopp implementation when we set the number of iterations So we validate the necessity of optimal transport by setting the #iteration to 0, 2, Experiments show that using Softmax (0 iteration of Sinkhorn) leads to the worst performance (-1.9). This validates the necessity of using optimal transport to ensure all images and texts within a batch are matchable. Besides, using 2 (fewer) and 6 (more) iterations also lead to accuracy drops (-1.7, -1.0). Using more iterations of Sinkhorn leads to a more converged solution to the optimal transport problem, but this does not seem to be positively correlated with better performance. We also explored the impact of entropy regularization controlled by À in Equation (6) Experiments show that the target distribution being too "hard" (X 0.05) too "soft" (A 0.30) can hurt the performance.
Impact of optimal transport:
Previous works on contrastive learning show that larger batch size raises the lower bound of mutual information (Hadsell et al.) 2006) and leads to better performance (Chen et al. 20206 He et al.) 2020). However, for noisy image-text pairs. larger batch sizes can potentially bring more unpaired matches. We study the impact of batch sizes by setting it to 256, 512, 768 in the experiments. We find that both smaller (256) and larger (768) batch sizes lead worse performance. We hypothesize that batch size needs to be co-adapted with prpra--prr settings of and X.
Batch size:
14

---

Published as a conference paper at ICLR 2022
estimates the noise level, which is positively correlated with the batch size. À yields different "softness" with different batch sizes. However, further investigation is to required to validate this.
B PSEUDOCODE FOR OTTER
Algorithm 1: PyTorch Pseudocode for OTTER
fs, ft: student and teacher model.
# tpi, tpd: learnable inverse temperature.
eta: large constant, e.g., 100.
alpha: loss coefficient.
I_N: NxN identity matrix.
xont: cross entropy function.
for img, in loader:
i Reqular InfoNCE loss
emb_v, emb_t (img, txt) nermalized embedd ngs.
logits = emb_v v @ emb_t.t prop Soitmax (logits tpi) normalize over t. prob Soitmax (logits. * tpi) normalize over v.
iinffNCE xent (prob_v, I_N) + xent (prob_t, I_N)
I Similarity estimation
emb_v_t, emb_t_t ft (img, txt) ()tachh() stop gradient, sim_vv, sim_tt emb_v_t 0 amb_v_t.d. emb_t_ t_t m_b_t_t.I sim_vt, sim_tv = emb_v. 8 emb_t_t.
8 emb_v_t."
S_v sim_vv sim_tt + sim_vt eta I_N s_t sim_t sim_v + sim_tv cta IN
# Optimal Transport Distillation
M_v sinkhorn ($_v)
M_t sinkhern ($_t)
l_d xent (prob_v, M_v) xent (prob_t, M_t)
Final loss
loss alpha L infoNCE + (1-alpha) L update (fs, ft, tpi, tpd)
Algorithm 2: PyTorch Pseudocode for Sinkhorn Sinkhor-Knoop
def sinkhorn (8, 1ambda-0. 15, ni niter-5):
T exp (S / Tambda)
T.sum
s.spape
iterative row/column normalization for in range (niter):
/= (T. sum (dim=1, keepdim=True)
row normalization
/= (T. (dim=0, keepdim=True) N) # column normalization
# Note if niter=0, this is equivalent to Softmax
return I // I. s. (dim=1, keepdim=True)
normalization
C VARIANCE ANALYSIS
our experiments, we noticed variance experimental results with identical settings. To study this, we repeat the experiments in Table 2 with a ResNet50 image encoder and DeCLUTR-Sci-base text encoder for times each using different random seed to analyze the variance of the experiments. We noticed higher variance GOI experiments. For example, for the FH@ 10 metric, the variance be up to 1.88 pts. Note that the mean accuracy's 's gap between OTTER and baselines are all significantly larger than the standard deviation, indicating that the performance improvement of OTTER is not
15

---

Published as a conference paper at ICLR 2022
result of randomness. However, such high variance is worth noting and requires future investigation on how to reduce
Table 6: Flat hit @K on test sets of Google Open Images and ImageNet10K from Tencent-ML. Images.
GOI F K (%)
IN1OK FH@K (%)
Data
Method
10
InfoNCE
0.23
56.1 0.92
66.4 1.05
10.9 1 0.38
29,4 0.75
40.5
0.76
CC
LS
26.7 + 1.00
55.9 1.31
0..51:311 1.31
10.1 + 0.67
29.6 = 0.81
39.8
1.03
(3M)
KD
26.7 1 0.81
55.3 + 1.67
67.1 + 1.71
10.0 = 0.75
27.5 = 1.42
38.5
1 1.14
OTTER
28.6 1.17
59.6 1.71
70.9 + 1.88
12.0 = 0.31
31.8 = 0.40
42.1 + 0.26
D QUANTITATIVE ANALYSIS ON THE IMAGE-TEXT COMPOSITIONALIIT ON CUB
ALIGN Jia et al. (2020) presents an interesting demonstration of the compositionality of image and text embeddings generated by language supervised vision models. On image retrieval task, query is formed by adding an image embedding vector to a text embedding vector. The returned image is expected to be similar to the image query and the text query. ALIGN demonstrates the compositionality by qualitatively showing several retrieval results, but does not provide any quantitative evaluations. In this paper, design preliminary benchmark based on the CUB dataset (Welinder et al. (0100) to evaluate the image-text compositionality.
Cosine Similarity
Image text queries
Query embeddings
Dataset image embedding
Image query emb
Image embedding
0.3
Query embedding m
Image embedding
0.9
yellow forehead color
Text query emb Text query emb
Image embedding
0.4
black eye color
Image embedding
white breast color
Text query emb
Figure Illustration of the image text -> image retrieval.
CUB (Welinder al. (2010) consists of 6033 bird images, and each image-i annotated with set of bird attributes, which we denote as A, In this dataset, there are in total 288 unique attributes. Given an image and several CUB bird attributes text, we generate image and text embeddings using our pretrained models and add the embeddings together to form query embedding vector. We use the query vector to match images in CUB, and choose the nearest neighbor, based on cosine similarity, as the retrieved image. This process is illustrated in Figure 5 To evaluate the retrieval quality, we compare the overlap between the retrieved image's sss attributes with the image-text query's attributes.
We now describe how generate text queries in addition text attributes to image queries may result in unmatchable queries. To ensure that the added text query is sensible and the combined image-text query can be matched to an image the dataset, we obtain a text query with the following approach: We first select pair of images, denoted as image-i and image-j. We use image-i as the image query, and let Q" A, the image query's attribute set. Then, to form the text query, we compare the differences of image-j and and let Q' A, - A; be the text query's attribute set. The combined image-text query should contain attributes Q = Q" UQ' We show an example of image query, text query, and combined query in Figure 6 Following this process, we generate 1,000,000 image-text queries from randomly sampled image pairs from CUB where each image pair shares least 10 common attributes. The image-text queries used in our experiment are provided the attached file image smage_text, rext_query, list. txt in the supplementary material.
image query. Randomly adding
16

---

Published as a conference paper at ICLR 2022
image-i
image-j
Image Query
Text Query
'grey wing color, 'wiit upperparts color', 'hite forehead color', white primary color'
'orange belly color', "orange underparts color,, "orange breast color', "blue nape color',
Image Attributes Q' = Ai
Text Attributes Q'= Aj- A;
A
Aj
Figure Image and text query example.
Table 7: Quantitative Vision-Language Compositionality Benchmark. OR represents Overlapping Rate, IOR represents Image Overlapping Rate, and TOR represents Text Overlapping Rate.
Model
Image Encoder
Text Encoder
Method
OR (%)
IOR (%)
TOR (%)
Baseline
27.7
31.3
23.2
CLIP
ResNet50
CLIP Transformer
InfoNCE
35.7
36.2
34.5
InfoNCE
34.5
33,6
36.8
DeCLUTR -Sci-base
LS
34.2
33.6
35.8
Ours
ResNet50
KD
33.2
32.9
34.4
OTTER
34.7
33.9
36.7
Our evaluation measures the overlap between the retrieved image's ' attribute set Rş with the image- text query set Qu, image query set Q text query set Q Specifically, we compute: 1) the attributes overlapping rate (OR) N
Σ IQunRal This measures the retrieval quality for the combined 1Qul
k=1
image-text query. 2) the average image attributes overlapping rate (IOR) 1 Σ
[QgnRRl This
[
evaluates if the returned image hits/misses attributes the image query. 3) the average text attributes overlapping (TOR) Y Σk=1 Q4
12(rRel This evaluates if the retrieved image hits/misses attributes
of text queries. An ideal match should achieve higher scores in OR, IOR, and TOR simultaneously.
We report our results comparing with CLIP and other baselines in Table First, we report the measurement random baseline. With this random baseline, for any image-text query, we return random image from the dataset. We can see that this gives non-trivial OR (27.7%), IOR (31.3%), and TOR (23.2%). This not surprising because images in CUB have many overalapping attributes. However, although both CLIP and our models are never directly trained on CUB nor trained to predict bird attributes, their overlapping rates are significantly higher than the random baseline. CLIP achieves comparable accuracy with our models, with 1% higher OR, 2.3% higher IOR, and -2.2% lower TOR. Among different training methods, OTTER outperforms all baselines OR and IOR, and achieves slightly worse (-0.1%) TOR than the InfoNCE baseline. Also note that models achieve similar IOR and TOR, which shows both image and text queries are considered. This demonstrates good compositionality.
In Figure 7 we show qualitative results our model trained by OTTER. For example, in the first row, we add yellow forehead color, black eye color, yellow upper-parts color and yellow breast
17

---

Published as a conference paper at ICLR 2022
yellow forehead color, black eye color', yellow upperparts color' while breast color'
'black wing color 'black leg color
"black forehead color',
'yellow color, 'yellow primary color
"striped wing pattern' 'black eye color'
"black throat color, 'white breast color', 'black wing color.. 'black under tail color'
'grey forehead color'
"black eye color' 'solid tail pattern'
Figure 7: Qualitative results of text + image retrieval. Blue lines indicate texts attributes, and red lines indicate image attributes. For the demonstration purpose, we only label parts the attributes that are easy to recognize.
color text attributes image of black bird. The retrieved bird image contains attributes from both image and text queries - has yellow body colors while maintains black leg and black wing colors. From both quantitative and qualitative results, model demonstrates good image-text compositionality.
18

---

Published as a conference paper at ICLR 2022
E IMAGE ATTRIBUTIONS
1. Figure
Paul Bica, Coast of Kauai, Hawaii, CC BY 2.0
2. Figure
James St. John, Columnar- jointed rhyolitic obsidian lava flow, CC BY 2.0
3. Figure]
Mordaka, QK9A 1397, CC BY-SA 4.0
4. Figure]
Alan Reid, Heathery moor the flank of Stone Saul, CC BY-SA 2.0
5. Figure
Jimmy Emmerson, The Tormented Valley, CC BY-NC-ND 2.0
6. Figure BY 2.0
Roman Boed from The Netherlands, Black Forest- Meadow (10561897306), CC
7. Figure
Daniel Clerc / CC-BY-SA-300, 2013 bois herpin 013, CC BY-SA 3.0
8. Figure
Dave Bevis, 22 and 24 High Street, Newcastle-under-Lynn CC BY-SA 2.0
9. Figure
Nilfanion, Thatched cottages in Coverack (8379), CC BY-SA
10. Figure
TheAHL, Chuck Kobasew (cropped), CC BY 2.0
11. Figure
Mark Mauno, Jasper Fitzi, CC BY 2.0
Figure
A Calico cat looking for food in a kitchen in the Farm section of Oak Hill, Fairfax County, Virginia, CC BY-SA 4.0
Famartin, 2020-04-27
Franklin
13. Figure
Beth, Haircut-4, CC BY 2.0
19


Reviews:
[{"review_id": "tW9Yiv5oyL0", "reviewer": "Reviewer_pk8X", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---\nStrengths:\n\n1. The work has a clear hypothesis and is well-motivated. Many-to-many relationships / soft correspondences are generally important for vision-and-language learning since there is never really one unique and perfect correspondence between an image and text. Additionally, improving data efficiency of pre-training vision-and-language models can be beneficial to the community as many recent works rely on enormous amounts of image-text data. Others would likely upon this work in a similar direction.\n\n2. The proposed method addresses the issue of many-to-many relationships in a technically valid way, and doesn't increase the complexity of the general approach significantly\n\n3. The authors present experiments with a good set of baselines (LS, KD, InfoNCE) as alternative methods and ablation studies on each component of their approach\n\n4. The experiments show that the proposed approach consistently outperforms alternative approaches (baselines)\n\n\n---\nWeaknesses:\n\n\n1. Although the proposed approach doesn't increase the complexity significantly, it does introduce a number of new hyperparameters – many of which the models seem highly sensitive to (Table 3).\nIntuitively, different datasets might have more or less similarities between samples, noise level, etc. - therefore the hyperparameters that work well on CC & WIT, which are of relatively high quality, might not be very transferable to more noisy datasets (like the one used in CLIP, for example).\nThe ablation studies indicate that, e.g., batch size has a very significant impact on the performance, and intuitively the best choice should depend on the particular dataset since, for a given batch size, the average number of similar samples might vary greatly among different datasets.\n\n2. The comparison with ZSL methods trained on ImageNet1K (Table 2) is misleading. These ZSL methods are trained on ImageNet1K and evaluated on ImageNet21K because of a relatively clear separation between seen and unseen (novel) classes. However, the proposed approach is trained on the CC dataset, which likely has a substantial overlap of concepts with the test set (ImageNet21K). Therefore, the comparison of the prior works and the proposed approach and claims of improving state-of-the-art are somewhat misleading because training on CC means that the model has effectively seen images + names (likely in captions) those classes that are supposed to be \"unseen\". Even though all works can be said to do zero-shot, the prior methods trained on ImageNet1K do zero-shot with respect to novel classes/concepts, while the proposed method & CLIP model do something more of zero-shot with respect to a dataset (but having possibly seen at least some of the concepts/classes already). Thus, making a side-by-side comparison without discussing this difference can be misleading.\n\n3. The problem of noise in datasets is discussed in the context of more than one possible matching for a given text/image. But what about the image-text pairs not being \"accurate\" - e.g. text not relevant, or not descriptive enough?\n\n\n---\nOther:\n\n- Question: Why do both $M^V$ and $M^T$ need to be computed (for the OTTER variant)? Shouldn't they be the same (at least for a larger number of Sinkhorn-Knopp iterations)?", "summary_of_the_paper": "The paper proposes an image-text training method similar to CLIP. However, in training, instead of using hard targets for image-text pairs, it relies on soft image-text correspondences based on pairwise similarities and optimal transport.\n\nAs image-text pairs can possibly have more than one positive match in the training batch, the authors motivate that the usage of soft targets is more appropriate because it better reflects the possible many-to-many relationship between the image and text samples.\n\nThe authors claim that such an approach leads to better data efficiency as it achieves higher performance on CC and WIT datasets, which are significantly smaller than the one used by CLIP.", "summary_of_the_review": "The work introduces a valid approach for considering many-to-many correspondences in image-text datasets which is an important problem for vision-and-language learning.\n\nAlthough the presented results show consistent improvements over alternative approaches, the approach does introduce a few new hyperparameters. Intuitively they might need to be tuned carefully for different datasets to be able to observe any improvement over simpler approaches. The ablation studies do suggest that within a given dataset the sensitivity to some of those hyperparameters is high.\n\nAdditionally, the work makes a side-by-side comparison with ZSL prior works, which, unlike this paper, do control the separation between seen and unseen classes/concepts in their training/test data. Such comparison and claims of exceeding ZSL their performance and reaching a new state-of-the-art might lead to misleading conclusions.\n\n**Update (after the authors' response):**\n\nRaised the recommendation to _6: marginally above the acceptance threshold_", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ggvj3HwWaak", "reviewer": "Reviewer_Qrbe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n+ The motivation of this paper is very clear.\n+ Overall, the paper is well written. In particular, the INTRODUCTION section has a nice flow. \n\nCons: \n\n +For the optimal transmission algorithm, in the METHODS part, the author should emphasize the difference from the existing algorithm.\n\n +More visualizations should be added to analyze the effectiveness of the proposed method (E.g Figure 2).\n\n +For IN10K FH@K, the proposed OTTER does not achieve satisfactory performance. The author should analyze the reason for this result.\n\n\n Minor comments: \n\n- In the 4th line in 4.2 ABLATION STUDIES, it should be \"define\" instead of \" defined \".", "summary_of_the_paper": "Review: This paper proposes a novel framework, OTTER, which considers the many-to-many relationship within a batch of images and text captions for data-efficient language-supervised zero-shot recognition. An improved InfoNCE is explored to consider the many-to-many relationship between unpaired images and texts. Then to further explore the relationship between images and texts, the optimal transmission method is introduced. Extensive experiments on six popular benchmark datasets demonstrate that OTTER well addresses this challenge, and it only needs fewer samples.\n", "summary_of_the_review": "Due to the limited novelty of the proposed method, I need to re-evaluate my judgments after the rebuttal.\n\nUPDATES: The authors have addressed all of my concerns, though experiments on 400M dataset are missed by uncontrollable reasons, as initial judgement, I tend to accept this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fKZI4Iou8R7", "reviewer": "Reviewer_qBzH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The authors showcase a fundamental problem of InfoNCE by providing interesting examples in Figure 1 and Figure 3. This is a valuable insight for the community, and may encourage future work to make more realistic assumptions about image-caption datasets.\n2. The proposed method is simple and elegantly formulated. It is versatile and can be applied to many vision-language tasks besides zero-shot recognition. There are hyperparameters alpha and lambda that control how much we trust the image-caption pairs, and how much we trust the similarity estimations made by the teacher model.\n3. The paper is well-written and easy to follow.\n4. The experiments show consistent improvements in various training and evaluation settings, on difficult and realistic classification tasks.\n\nWeaknesses:\n\n1. Although the improvements shown in Table 1 are consistent, they are not significant. Hence, using OTTER may not be worth the extra computational cost. \n2. The comparison to other ZSL methods in Table 2 is unfair, since the baselines are trained on a smaller dataset and with different pretrained image and text encoders.\n3. Although in Figures 1 and 3, the authors provide interesting evidence for the problem they identified, it is not clear how prevalent and important this problem is, especially considering the limited impact of the proposed method on the performance. Is it possible that only a handful of examples exist like those shown in the paper? Are there ways to quantify how common this problem is?\n4. Although zero-shot recognition is a relatively new problem, learning cross-modality embeddings using image-caption datasets has been studied for a long time, with applications ranging from text-to-image retrieval to visual grounding. It is not clear whether the problem of image-caption alignment noise has been identified and studied before, or the authors are the first to identify this. It is also not clear how effective this method would be for such vision-language tasks besides zero-shot recognition.\n5. In Table 1, it is strange that CLIP's performance using 400M images is close to the InfoNCE baselines trained on 3M images, on the GOI dataset. Is that only due to using pretrained image and text encoders? If so, it is important to elaborate the pretraining process, and compare the resources used for pretraining to CLIP's resources. It would also be interesting to show the performance without pretrained encoders, as a lower bound.\n", "summary_of_the_paper": "It is common to train cross-modality embedding models on image-caption datasets using contrastive objectives, assuming each caption correctly describes the corresponding image, and all other captions can be used as negative (incorrect) matches. This paper clearly illustrates that this common assumption is wrong, and it hurts the performance of the learned embeddings on down-stream tasks, namely zero-shot recognition. To address this problem, the authors propose a simple and elegant approach to refine the pairwise image-caption matching, which can be seen as a smarter variation of label smoothing. This simple trick improves the performance of the well-known CLIP model consistently across datasets and backbone architectures.", "summary_of_the_review": "The authors identify a new problem and propose a simple, elegant, and versatile solution to solve it, which has a consistent, incremental impact on the important task of zero-shot recognition. The insights and conclusions made by this paper are moderately significant, but not adequately supported, and not strong enough for acceptance.\n\nUPDATE: after reading the authors' response, I have decided to raise my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dg31U4v1kqG", "reviewer": "Reviewer_DJW4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper is very-well written and easy to follow. In particular, the approach section is clearly written and presented in a good pedagogical manner. \n+ Overall, IMO the proposed approach solves the missing match problem in language-vision contrastive pretraining in a sensible way.\n+ The results seem to suggest that the proposed approach is overall a good fix to existing language-vision pretraining methods, as it generally provides a gain (in certain cases bigger than others) across different target datasets and choice of encoder architectures. \n+ The authors have made good efforts to ablate different components of their approach.   \n+ The authors have promised to release their code for training and evaluation. \n\nSpecific things I like in the paper:\n+ I like Section 3.5 as it helps clearly distinguish the proposed method to related methods with term swapping.\n+ I like the visualization in Figure 3, which gives me a sense of what kind of missing matchings are mined. \n\nWeaknesses:\n- One issue I have for the approach is that, it's not clear to me why self-distillation is necessarily beneficial? Since the authors use the original encoder (or their momentum-updated counterparts) as teachers, what extra information does the distillation bring in? At a conceptual level, the only extra source of information is the introduction of text-text and visual-visual similarities in Eq. 7. But from the results (i.e., 'running SK for 0 iterations' vs the full OTTER method), it seems to suggest that's not the only reason for the improved accuracy. It would be good if the authors could have a summary on where they think all the gains come from. \n- The authors have motivated their approach by giving the example in Figure 1 to show that, there are many missed matches between images and texts. However, there lacks a quantitative study showing how prevalent this problem is. Although one can argue from the improved results, it would be more convincing to have a ballpark of how many missed matches there are in a batch. \n- The proposed method has many parameters (alpha gamma_v gamma_t, eta, lambda), there lacks a detailed sensitivity analysis for them (Table 3 only does coarse ablation that mostly turn on or off the term). \n- Page 5, \"we train on the two publicly available datasets, Conceptual Captions 3M (CC) (Sharma et al., 2018), and Wikipedia-base Image-Text Dataset (WIT)\", it would be helpful to train on a larger and more noisier dataset like Conceptual Captions 12M to see how the proposed method scales (my impression is that the benefit of the proposed method should be more significant due to higher noise level there). \n- In Table 1, for three image/text encoder pairs \"FBNetV3-A/DeCLUTR-Sci-base\", \"FBNetV3-C/DeCLUTR-Sci-base\" and \"ResNet50/Sentence-BERT-base\", OTTER only has very marginal gain over KD, especially on IN10k, any insights on why this is case?   \n- In Table 2, for OTTER, why not use ResNet50 as the image encoder to enable a more fair comparison? Again, for the \"FBNetV3-C/DeCLUTR-Sci-base\" setting, the gain is quite marginal between OTTER and KD. \n\nQuestions/confusions:\n- Page 3, \"Directly minimizing Wasserstein loss between image/text embeddings in our case will lead to collapsed representations\", any insight why this might be happening? \n- Page 3, \"tau is a (trainable) temperature parameter\", is there evidence showing the benefit of learning this parameter, instead handpicking one using validation? Also, what value does tau converge to? \n- Table 3, it would be good to have results for setting lambda to 0 (no entropy regularization). \n- Page 6, \"with a total batch size of 512 (64 per GPU) for 10 epochs\", how long does it take to train for 10 epochs? \n", "summary_of_the_paper": "This paper tackles the problem of zero-shot image recognition through language-supervised pretraining. The proposed method trains their visual-language encoder pair with a contrastive objective (similar to CLIP) on image-text pairs. The main problem the authors saw and aim to mitigate from previous approaches like CLIP is the existence of missing-matches between a batch of image-text pairs for training. The authors argue that, in a batch, one image can be matched to multiple texts and vice versa. They propose to use Optimal Transport optimization to solve for a soft image-text matching to replace the hard single-to-single matching and serve as the pseudo ground-truth for supervision. \n\nThen, the pretrained models visual and language encoders are taken to several target datasets (Google Open Images, ImageNet 10K, and ImageNet 21K+1K) for zero-shot recognition (text encoders are used to generate classifiers on the fly for labels in the target datasets). The proposed method demonstrated good results compared to several previous methods and also a set of variants for ablation studies. \n", "summary_of_the_review": "Overall, I like the idea proposed in the paper to use OT to mine missing matches for language-vision pretraining. The paper is clearly written and makes intuitive sense for most part -- I'm overall positive on accepting this paper but still like to hear the authors address my concerns listed above. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YdBRQRFk2NG", "reviewer": "Reviewer_ayPf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1)\tThe authors provide an interesting and solid idea based on CLIP.\n2)\tThe presentation is very good and easy to follow. The authors clearly present their ideas and describe the technical details.\n3)\tThe ablation study and visualization analysis of the experimental results are sufficient. The visualization of OTTER’s matching illustrates its effectiveness in handling many-to-many relationships.\n\nWeakness\nWhile the ablation study and visualization analysis are done, the key evaluations are missing. The details can be found as follows.\n\nThe experiment results in Tables 1 & 2 cannot plausibly prove OTTER is more effective than CLIP. The larger dataset may relate to more many-to-many relationships when training the model. For example, in the visualization in Figure 2, the maximum many-to-many relationship of a sample is 3. However, the dataset with 400M may contain too many many-to-many relationships like 7 or 8 (maybe more, the data scale is about 100 times the used datasets in this paper). In this case, OTTER may be degraded due to the multiple learning targets. In contrast, focusing on a solid target and ignoring the rest of potential targets, CLIP may still infer the generalized information provided by many-to-many relationships due to the wide range of data collection. Therefore, it is necessary to compare OTTER and CLIP on the same-scaled datasets. The authors provide OTTER using hard labels (InfoNCE) as a baseline, but ZSL methods are sensitive to hyper-parameters and training sets. I doubt that InfoNCE can represent the best performance of CLIP trained on CC (3M) and WIT(5M). Regarding that CLIP does not release the 400M dataset (mentioned in this paper by the authors), the authors may not be able to train OTTER on the 400M dataset. The provided experiments are more like baselines for self-evaluation other than state-of-the-art performance. Without the experimental results using the same training settings, I do not think the authors are able to prove the superiority of OTTER over CLIP fully.\n", "summary_of_the_paper": "This paper proposes an enhanced model Optimal TransporT distillation for Efficient zero-shot Recognition (OTTER) based on Contrastive Language–Image Pre-training (CLIP). The authors present the potential many-to-many visual to text relationships that CLIP cannot handle. The authors proposed to model the many-to-many relationships as optimal transfer, which portrays the probability relationship based on two intuitions: 1) all the images and texts are equally matchable and have equal matching probabilities; 2) the matching probability from image to caption should depend on their similarity estimation. The authors further use knowledge distillation to enhance the learning of similarity matrices. ", "summary_of_the_review": "According to my main review, I believe this paper still has room to improve", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the interesting  many-to-many assignement problem between a set of images and a set of text. Most reviewers, (and I agree with them) think that the  idea and its application worth being published although the performance improvement
is marginal. I request the authors to update the paper based on the discussions.
Decision: Accept (Poster)