What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs
Authors: Tal Shaharabany
Yoad Tewel
Lior Wolf
Venue: NeurIPS 2022 Accept (2022)
Keywords: weakly-supervised phrase-grounding
open-world
weakly-supervised localization
CLIP

Abstract: Given an input image, and nothing else, our method returns the bounding boxes of objects in the image and phrases that describe the objects. This is achieved within an open world paradigm, in which the objects in the input image may not have been encountered during the training of the localization mechanism. Moreover, training takes place in a weakly supervised setting, where no bounding boxes are provided. To achieve this, our method combines two pre-trained networks: the CLIP image-to-text matching score and the BLIP image captioning tool. Training takes place on COCO images and their captions and is based on CLIP. Then, during inference, BLIP is used to generate a hypothesis regarding various regions of the current image. Our work generalizes weakly supervised segmentation and phrase grounding and is shown empirically to outperform the state of the art in both domains. It also shows very convincing results in the novel task of weakly-supervised open-world purely visual phrase-grounding presented in our work.
For example, on the datasets used for benchmarking phrase-grounding, our method results in a very modest degradation in comparison to methods that employ human captions as an additional input.

What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs
Tal Shaharabany Yoad Tewel Lior Wolf
Tel-Aviv University
(shaharabany, yoadtaewl,,ade.el,.,o. wolf) tau ac
Abstract
Given input image, and nothing else, method returns the bounding boxes objects the image and phrases that describe the objects. This is achieved within an open world paradigm, in which the objects the input image may not have been encountered during the training of the localization mechanism. Moreover, training takes place in weakly supervised setting, where bound- ing boxes are provided. To achieve this, our method combines two pre-trained networks: the CLIP image-to-ter to-tett matching score and the BLIP image captioning tool. Training takes place on COCO images and their captions and is based on CLIP. Then, during inference, BLIP is used generate hypothesis regarding various regions of the current image. Our work generalizes weakly supervised segmentation and phrase grounding and shown empirically to outperform the state of the art in both domains. It also shows very convincing results in the novel task of weakly-supervised open-world purely visual hhrase-grounding pre- sented in our work. For example, on the datasets used for benchmarking phrase- grounding, our method results in very modest degradation in comparison to meth- ods that employ human captions as an additional input. Our code is available at h/tps::/gitan cwm./aassaaaaaabbnr/aaaaaaaa
where by-lkoking and a live demo can be found at https: wtt.e/tte.....aaaaaaeeeeeo wtm.ttaasaaaaaabbrraa.. whet-e--wweer-bbye by--eebb--oooonnng
Introduction
What does it mean, to see? The plain man's 's answer (and Aristotle's too) would be, to know what is where by looking. In other words, vision is the process of discovering from images what is present in the world, and where it is.
Marr 50
We address the task of detecting the objects that exist in a given image, without limiting ourselves to predefined list of objects, of the type that detection algorithms employ. This property is sometimes referred to open-word open-world vision. Our method assigns text phrase to each detected object. The similar task that employs an additional text input describing the image called phrase-grood grounding. We describe our approach as "purely visual" to denote that there is no text input. Since the task performed without the use of bounding boxes in any phase of training, the level of supervision employed corresponds to weakly supervised. As far as we can ascertain, the task of weakly weakly-supervised open-world purely visual grouudinggniiigg is entirely novel. For short, we refer this task, which illustrated in Fig as "what is where by looking" (WWbL).
In order tackle this task rely on recent pretrained vision-language models: CLIP 60 and
BLIP 41. These models are trained on large corpora of matching images and captions and can produce a matching score between an input image and an input text. Trained on millions of such pairs, CLIP has been extensively used in zero-shot recognition [94), image generation [2]] 45) image
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

tricks
mountains ciouds in the sky
with black shirt
graffiti the side of building
bike
Figure I: Given the image on the left, our method extracts multiple sentences that descrihe objects in the image, and generates foreground mask that localizes the object
Table 1: Comparison to different lines of previous work on image localization.
Name
References
Open-world
Weakly supervised
Purely visual
Object detection
Sarase-Grounding
42
Weakly supervised localization
66
Weakly supervised Phrase- Groundiio
V
What is where by looking (WWh1.)
This work
V
and other tasks [52]. BLIP, more recent addition, also incorporates into its training phase the ability to generate image captions. also relies on a bootstrapped captioning model to generate matching sentences and bootstrapped filtering model to eliminate false matches from its training set.
editing and manipulation [5
95], image and video captioning 53 72 [3]
While BLIP can produce matching score, we choose to use CLIP for producing such score and BLIP for generating plausible captions. This modular choice allows us to study the matching-basee components (such as generalizing weakly supervised localization methods) separately from the WWbL. It also emphasizes the modular nature of our method. Given time and resources, would be able to evaluate the method using only BLIP or replacing BLIP with CLIP-based image captioning methods, which are considerably slower than BLIP [73]
Our WWLL method comprises two phases. During training, an encoder-decoder g is applied to extract a foreground mask for an input image, given an image caption. This is the same level of supervision that required to train CLIP BLIP, albeit performed on much smaller dataset. Since no segmentation mask (or bounding box) is used, this is considered weakly supervised.
Four loss terms used. The first one requires that the CLIP matching score of the foreground region and the matching text be maximal. The second helps ensure that the background region is unrelated the text. The third term considers the explainability map of CLIP, given the input image and the caption [3] as a guide for the foreground mask. Lastly, in order to encourage compact foreground masks, a regularization term is added.
two-stage inference procedure employs the learned foreground segmentation network and a pretrained-BLII model. The first stage extracts object proposals using Selective Search algo- rithm (74). The second stage uses BLIP to generate captions for each proposal and find text clusters in CLIP's embedding space, in order avoid repetitive descriptions. Network is then used find the foreground mask for each sentence, given the entire image as input and the sentence.
Our results show that network g. weakly supervised localization network, outperforms the sate-of-thh-arts The same model also outperforms the state-of-the-at in phrase grounding. When the entire method is applied, with the two stages described above, the results obtained for WWbL are par with the results obtained using human captions for the task of phrase grounding. These results are considerably better than those obtained by the baselines.
Our main contributions are: (i) A new state-of-ther -hr---rtt solution for weakly-supervised localization tasks with text and image as input, and (ii) Tackling, for the first time, as far we can ascertain, the fundamental WWLL. task, which is an open- world detection task that does not rely on any localization information during training.

---

2
Related Work
Open-worlh computer vision is an emerging paradigm that stands in contrast to recognition systems that can only address the limited number of classes and types of examples that were observed during training. Specifically, open-set classification 86 38 sample is not from the observed classes.
requires the ability to understand when a new
Since large visual-language models are trained hundreds of millions of samples [60], many classes of objects are observed during training. It is impractical to enumerate all object classes. Moreover, these models can recognize phrases that were unseen during training, e-g., "a green dove". Therefore, the use of the term "open-world" for such models is not based on learned vs. unseen classes, but rather on testing the method without limiting the type of objects observed during the inference.
Open-world localization methods generalize more restricted methods, which rely on a relatively short list of detectable objects. Similarly, weakly supervised methods, which employ labeling the image level, generalize fully supervised methods that require bounding boxes during training. Finally, methods that do not receive input guiding information regarding the content of the image (we call these "purely visual") generalize methods that receive as input a list of objects to detect or text that describes the contents of the image.
Tab. D compares the task we tackle to other localization tasks. As can be seen, the task we tackle "what is where by looking" (WWbL) generalizes all other localization tasks. Unlike supervised object detection, is open-set and weakly supervised. Weakly supervised localization methods are not open-set. Lastly, phrase grounding methods employ textual input, while WWbL does not.
Object Detection One of the fundamental tasks of computer vision is object detection, where a closed set of object categories is localized based on training sets that contain costly bounding -bo annotation 4 20 Since the advent of deep learning, many fully supervised algorithms have been proposed for this task 66 26 44.. In contrast to these methods, our approach employs image-level annotation (global text description) and an open dictionary to describe the detected object.
Phrase Grounding is a weakly supervised phrase localization task, in which text phrases are associated with local image annotations and local numan-annotaie methods extract text embedding from a pretrained language model [66] 63 35), together with image representations, obtain common semantic space for image-text pairs. Our model is based on
signal 2 89 33]. Many
CLIP [60] as text-image relation space to extract the localization of objects. Li et al. [42] collected 27M image-text data points, of which 3M have local human annotation, and used a semi-supervised training method for chrase-groundie our work, since it uses additional supervision. Arbelle et al. 3 proposed a self-supervision method scheme for generating localization map.
which employs CLIP. It cannot be compared directly with
Weakly Supervised Localization (WSOL) Class Activation Map (CAM) explainability methods have been offered in recent years for solving WSOL tasks [92] 59 46) Most of these algorithms
train a classifier to distinguish between sub-categories of the main object (Birds, Cars, Dog etc). employing a localization loss term for the explainability map 80
28 55
Similarly to us, Shaharabany al. [66] employ external localization network in addition to the classifier. During training, the classifier supplies gradients to the localization network. In contrast to work: (1) they compare the classifier outputs with and without the foreground mask, while our algorithm compares the masked image with the text in the CLIP space, (2) there is no background loss term, and (3) explainability is not used as a signal.
Image Captioning Captioning is a fundamental lision-language task. Early methods applied
RNNs 49 37). Attention was added to identify relevant salient objects [83
64) Subsequently,
Transformers modeled interactions among all image elements with self-attention 75 56 22]. Recent
works have shown significant improvements in robustness and generalization by using large-scale
vision-language data sets 73 53
77). Our method employs BLIP [41]. which a vision-
language pre-training framework that trained on images and their caption, employing a method to filter out noisy data. The framework also provides an image captioning model and shows good performances in diverse image domains and for large variety of real-world objects.
Many methods generate a heatmap that indicates relevancy for CNNs, e-g., 65 47 67 Most relevant to our work is the use of GradCAM (65) relevancy maps as cue for weakly
Explainability
3

---

Z.
Joveried mapit
Figure 2: The architecture of our model and the optimization loss terms. The input image is encoded with the VGG16 encoder part of g. and test description is encoded with CLIP's text encoder. The resulting text embedding multiplics the spatial dimension of image embedding to find a similarity between regions in the image. The similarity map Z, weights the image embedding. The results are passed to the decoder part of to generate the output map. The loss term Lymap compares the output map the CLIP explainability map with L2 loss. Lore forces the output map focus the text-relevant pixels by increasing the similarity between O ((T and the text. Leack aims to guide IIg(I) to focus on the background by decreasing the cosine similarity between @ - (I) and the text. Lreg adds sparsity constraint on g(7)
supervised segmentation methods [54 79 including recent contributions that employ GradCAM heatmaps as pseudo labels for weakly supervised localization and segmentation algorithms 78 28 The literature on transformer explainability is sparse and is mostly limited to pure self-attention architectures [1 12]. A recent work by Chefer et al. [12] fully addresses bi-modal transformer networks, and is the method employed to obtain relevancy maps. The solution is based on Layer- Layer-wise Relevance Propagation [5], with gradient integration for the self-attention and co-attention layers.
3 Method
In this section, we describe our method for generating a localization map based on image and text input signals without local annotation. We use our method on three different tasks (i) Weakly supervised object localization (WSOL) (ii) Weakly supervised phrase-grounding (WSG) (iii) What is where by looking (WWbL).
At the core of our method lies segmentation network g. which is trained in weakly supervised manner. The network g has two inputs: an input image I R3xW* and an embedding of the input text Z For the simpler task of WSOL, network g receives only the first input I as input signal. The loss terms used in both cases are the same, and, for the sake of generality we write the equations with g that has two inputs.
The training of g does not involve any ground truth localization information. Instead, it relies two types of signals: (I) comparing the CLIP score of the extracted segment to that of the entire image, and (2) including the heatmap of the CLIP network, as obtained with the relevancy method of Chefer et al. [4], in the segmented region.
Four-loss terms used, using the input image I, the text t, and the relevancy meat-map of CLIP given and f, which we denote as H. The terms are (1) a foreground loss Lfrre (2) a background loss Loack (I, t). (3) relevancy heatmap loss Lymap (1, H), and (4) regularization loss Lyeg (I.
The foreground loss encourages the segmentation network g to output map that increases the similarity between the text and the masked image g(I,t) @ I, O being the Hadamard product.
Looe (1, CCLIP(((1, zT) 1,t).
(I)
where CLIP((J,)) denotes the CLIP matching score hetween an image and the text 8.
The background loss helps ensure that the complementary part to that returned by g(T, 2T) is irrelevant, the sense that its CLIP matching score with the text is low.
Lbock (1,t) -CLIP(1 9(1, 1,t),
(2)

---

The relevancy map H obtained by the method of Chefer et al. [4 on CLIP(I,t) provides us with another important cue. While is not entirely reliable in the sense that irrelevant regions are often marked and those regions often do not include the entire relevant part, we find that it often highlights some of the relevant parts. Since both H and g( z²) have the same range of values ((0,1), we employ the squared Euclidean norm of the difference.
I Temay (T, H) H g(,, )12
The background loss may encourage g , -T) to be as inclusive as possible, so that the complement is minimal. Additionally, the relevancy map loss Trmep may encourage the obtained foreground mask to be overly inclusive. To encourage to be more spatially limited, we add a regularization loss
Lreg (1, (1, I
The combined loss is defined as I, -= a1 oore Treg (T), where A A are fixed weighting parameters for all datasets, which were determined after a limited hyperparameters search on the CUB (76) validation set to be 1, 1, 4, 1 respectively.
( (1, + X2 T.back(
(T, () + À3 Trmma (T, H) X
3.1 Inference Tasks
Weakly supervised object localization (WSOL) The goal of the WSOL task is to localize single object the image I without local annotation. For each fine-grained dataset, there is a single type of object (birds, cars, dogs, etc), and we do not guide the segmentation process with any type of conditioning. The text is used only in the training time for the loss functions, and simply the name of the object ("birdss", "cars", "dogs").
For converting the obtained continuous segmentation map g(),, which a sigmoid output between 0 to I, to a binary mask, we follow the binarization used, e.g., by Qin et al. [59] 18 17 66). which is the standard in the benchmark. Namely, low-value pixels, below the threshold of 0. are zeroed out.
Then, the method of Suzuki et al. [70 is used to extract contours. From the largest contour found, the algorithm selects the bounding box of the object.
Weakly supervised chrase-grounding (WSG) The aim of weakly supervised groundigg is to generate localization map. given image and textual phrase t for specific object in the image. To extract the objects from the output map of M = g(I, zT),, we first zero low-value pixels, using threshold of 0.5. Next, we find contours with the method of Suzuki et al. [70]. For each contour, we extract the proper bounding box. The score for each box is calculated from the output map in the positions of the detected box. In the final stage, apply non-maximal suppression over the boxes with 0.3 loU, and filtering boxes with a low score (50% less) compared to the maximum score. See Appendix D for the visualization of these steps.
WWbL Inference The WWbL task is an open-world localization task, in which the only input is a given image. The task consists of localizing and describing all the elements composing the scene. To solve this task, we propose a two-stage algorithm based on a selective search [74] algorithm to extract object proposals to describe local regions in the image and incorporate the learned network g, the image captioning network BLIP, and the CLIP model.
Captioning models usually tend to describe the main element of the scene, neglecting other objects or the background. To address this issue, our proposed algorithm extracts local regions from the image with the Selective Search 774 procedure, which first over-segments the image into super-pixels, and then groups those in a bottom-up manner to propose candidate bounding boxes, see line 2 of Alg. Each region is passed through BLIP to generate local caption (line 3).
To avoid duplicate captions, the algorithm projects the captions to CLIP space (line 4) and applies the Community Detection (Cd) algorithm [9] to find caption clusters (line 5). The clustering algorithm uses cosine similarity in CLIP space between each caption such that a matrix of size M x M is obtained, where M is the number of captions. The algorithm employs two hyperparameters: (i) cosine similarity threshold for items in the same cluster and (ii) minimum size of a cluster such that, for each caption (row in the matrix), we find the number of captions above the threshold and check whether it above the minimum size for a cluster. The final stage of Cd filters out overlapping clusters, giving priority to larger clusters. We set the cosine similarity threshold to be 0.85, and the minimum size to
5

---

Algorithm 1: WWbL. inference method
Require: Input image
1: Load trained networks g, BLIP and Fie
2: [Pi]=
SelectiveSearch(L)
3: {Ti = BLIP((P))
4: {2T),, li-1 = Ee(
person ricing tractor
the sky with dlouds in the background
5: (T,, "==1== = C ( 2..55
6: D+
7:
for
1....m do
8:
M. g(1, Zi)
Generate map
9:
B, < (M; P Extract bounding box
10:
D DU
(B, (B,,
> Add object
11: return
Figure 3: Sample WWbL results for Filckr30K
be 2. This algorithm returns caption for each cluster and its embedding in CLIP space. For each cluster, generates a localization map. Next, the bounding boxes are extracted like in Sec 3.1
3.2 Architecture
Our network g is used in two types of tasks where (1) image captioning is used only for the loss function (WSOL) and g receives only the image (2) image captioning is used as input to g and the loss function (WSG and WWbL). In this section, describe each mode.
WSOL Architecture For WSOL, Network g is based on U-Net [62 architecture, with skip connections between the encoder and the decoder for signals with the same spatial resolution. The decoder of ! employs five upsampling layers, with each block containing two convolutional layers with batch normalization after the last convolutional layer before the activation function. The last activation function is Sigmoid, while the others are Relu.
Multi-Model Architecture For WSG and WWbL, Network is based on an encoder-decoder architecture adapted support text-based conditioning. This is illustrated Fig. 2
The encoding Ze for the input text is produced by the CLIP text encoder, where the size of the vector 7, is 512. The image encoder of g is VGG network [68], in which the receptive field is of size 16 16, i.e., the image is downscaled four times. The number of channels of the obtained map Zj is also 512. The VGG network is initialized using ImageNet pre-trained model.
We then consider the vector in 5212 associated with each spatial location of the tensor Zy. It is normalized to have a norm of one. The dot product with the vector Z, (which in CLIP is also normalized) is computed. Performing this over all spatial locations, we obtain map Ze with values between -1 and 1. Z, has single channel and the same spatial dimensions as Z₁.
Zg can be viewed as a text-conditioned importance map. All channels of Zı are multiplied by Z, and the result is passed to the decoder part of This decoder consists of three upsampling blocks, each with sampling factor of two. Each block contains two convolutional layers with a kernel size equal to 3 and zero padding equal to one. Batch normalization is used after the last convolution layer, before the activation function. The first layer's activation function is ReLU, while the last layer's activation function is a Sigmoid.
4 Experiments
We present our results for the three tasks: (i) weakly supervised object localization (WSOL), (ii) weakly supervised phrase grounding (WSG), with training either MSCOCO 2014 43] or the Visual Genome (VG) dataset [40], and (iii) the new task we present (WWBL). For the first task, employ three fine-grad tne-grained localization datasets, and for the other two, use the three datasets commonly used in WSG.
Datasets For the task of WSOL we evaluate our model on fine-grained datasets often used for this task. CUB-200-2011 (76) contains 200 birds species, with 11,788 images divided into 5994 training images and 5794 test images. Stanford Car [39] contains 196 categorics of cars, with 8144 samples

---

Table 2: results for the CUB dataset.
Table 3:
Localization accuracy (percentage) WSOL on the cars and dogs datasets.
Method
Backhone
Accuracy|
Method
Stanford-Can Cars
Stanford-Dogs
GAF
12
CLIP
68.01
CLIP 12
81.2
64.7
CAM 92
VGG16
55.10
CAM 92
65.2
66.0
ACoL 90
VGGI6
62.96
HaS 02
87
77.5
ADL
VGGI6
75.41
ADL 19
82.8
73.5
DANet
VGG16
67.70
RDAP 17
92.9
77.7
MEIL 48
VGG16
73.84
66
79.2
GCNe 46
VGGI6
81.10
Ours
98.9
86.4
POSL
VGG16
89.11
SPA
VGG16
77.29
Table Backhones comparison weakly super- vised segmentation, as evaluated on CUB
SLT 28
VGG16
87.60
FAM 5
VGG16
89.26
ORNet 82
VGG16
86.20
Backbone
Params[#]
Accuracyl %]
BAS 80
VGG16
91.07
Ours
VGG16
93,94
Resnet18
11.1M
94.85
Resnet34
21.3M
95.31
CAM 92
MobileNetVI
63.30
Resnet50)
23.5M
96.54
HaS 69
MobileNetV1
67.31
Resnet101
42.5M
96.64
FAM 51
MobileNetV1
85.71
RCAM
6
MobileNetV1
78.60
Hardnet39DS
3.5M
93.77
BAS 80
MobileNetV1
92.35
Hardnet68DS
4.2M
93.85
Ours
MobileNetVI
94.40
Hardnet68
17.6M
94.99
Hardnet85
36.7M
95.11
CAM 92
Resnet50
57.35
POSL 87
Resnet50
90.00
WTL
Resnet50
77.35
Table Ablation results weakly supervised localization on CU)
FAM
Resnet50
85.73
SPOL 78
Resnet50
96.46
R(g(())
Lemmap
Lfore
Lawak
Accuracy(S6 [%]
BAS 80
Resnet50
95.13
48.36
Ours
Resnet50
96.54
47,58
CAM 92
Inception V3
55.10
V
42.11
V
DANet 84
Inception V3
67.03
84.72
12C
Inception V3
72.60
90.02
GCNet
Inception
75.30
SPA 155
Inception V3
72.14
87.02
89.12
SLT 28
Inception V3
86.50
V
90.09
FAM 51
Inception V3
87.25
V
91.08
BAS
80
Inception V33
92.24
84.24
Ours
Inception V3
94.30
V
92.67
94.73
PSOL
88
DenseNet161
93.01
96.54
Ours
DenseNet161 93.71
in the training set and 8041 samples in the test set. Stanford dogs [36 consists 20,580 images, with split of 12,000 for training and 8580 for testing, where the data has 120 classes of dogs.
For the task of WSG and WWbL, our model trains on the split of MSCOCO and VG, and we evaluate it on the test splits of Flickr30k, Referlt, and VG. MSCOCO 2014 43 using the splits of Akbari et al. (2). consists of 82,783 training images and 40,504 validation images. Each image has five captions describing it. VG [0 contains 77.398 training, 5000 validation, and 5000 test images. Each image comes with a set of free-form text annotated bounding boxes.
Flickr30k 588 Entities, which is based on Flickr30k, contains 224K phrases describing objects in over 31K images, each described by 5 captions. For evaluation, we use the same images from the test split of Akbari et al. [2]. ReferIt (27 15) contains 20K images and 99,535 segmented images that also contain description for the entire image. These image regions were collected in a two-player game with approximately 130K isolated entity descriptions. We use the split of Akbari et al. 2
Algorithm receives an input image of size 224 x 224. During training, the image is first resized to 256x256 and then random crop of size 224x224 is extracted. During the evaluation, the image is resized to 224x224.
Implementation details In WSOL,
Network y for WSOL is based on an Imagenet pretrained visual encoder. We compare the per- formance obtained using VGG16 (68), MobileNetV1 [30], Resnet50 (29), Inception 77 and

---

Table 6: WSG results: "pointing game" on VG, Flickr30K, and Referlt.
Table 7: WWbL and WSG results: "pointing game" and
bounding box accuracy.
Method
Rackbonee
VG mained
MS-COCO trained
Model
Training
Test Accuracy
Bbox Accuracy
VG
Flicker Referlt
VG Flicker Refert
VG Flickr
Relerli
VG
Flickr
Relerht
Baseline
Random
11.15 27.24
24.00
11.15 27.24
24.50
MG MG GAE 12
COCO
32.91 50.12
36.34
11,48 23.75
13:31
Center
20.55 47.40
30.30
20.55
47,40
30.50
WWbL
19.48
38.06
12.23
GAB
13
CLIP
54.72
72.47
56.76
54.72
72.47
56.76
38.15
56.25
41.64
9.69
17.14
12.3]
RCVC
VGG
14.03 29.03
33.52
OuT
COCO
44.20
61.38
43.77
17.26 32.44
21.76
VGLS
R1
VGG
24.40
VG
43,91
58.59
44.89
17.77
51.16
18.89
TD
Incepion-2
VGG
30.03
19.31
42.40
39.98
31,97
MG 2
COCO
47.94
6
47.52
15.77
27.06
2
50.18
57.91
62.76
46.99
53.20
47.89
MG 2
VG
48.76
60.08
60.01
14.45 27.78
18.85
MG
8
VLM
48.76
60.08
60.01
47.94
61.66
47.52
GAH [P]
54.72
72.47
56.76
16.70 25.56
19. 19.10
GbS
k3
VGG
53.40
70.48
59.44
$2,00
72.60
$6.10
CDCO
59.09
75.43
61.03
27.22 35.75
30.08
CLIP-VGG
62.31
75.63
65.95
59,09
75.43
61,03
OUTS
VG
62.31 75.63
65.95
27.26
36.35
32.25
NenseNet161 31 is applied for CUB dataset to compare performances with the state-of- ofe-----tthrrrr baselines, while for the other datasets we used Resnet50, since that is what the literature baselines have published. A SGD optimizer with a batch size of 48 and an initial learning rate of 0.0003 for 100 epochs is used. The optimizer momentum of 0.9 and weight decay of 0.0001 are also used. During the training, a random horizontal flip with 0.5 probability is applied. All models are trained on single GeForce RTX 2080Ti Nvidia GPU. In WSG training (the same network is used for inference in both WSG and WWbL), following Akbari et al. (2). network g receives
input image of size 299 299 and generates a localization map of the same size. During training, the image is also resized to 224x224, which is the input size of CLIP. The relevancy map H is generated at this resolution and is resized to 299x x 299. Network 9 for WSG and WWLL is VGG16 [68] to ensure a fair comparison with the baselines. An SGD optimizer (batch size of 32 and initial learning rate of 0.0003) is used for 100 epochs. The optimizer momentum of 0.9 and weight decay of 0.0001 are also used. During training. a random horizontal flip is applied, with 0.5 probability. All models are trained on a double 2080Ti Nvidia GPU, while all experiments of Arbelle et al. 3 were conducted on four V100 GPUs.
Results For the WSOL task, the main accuracy metric measures whether the intersection over union (IoU) of the ground-trute truth bounding box and the detector's output are above 0.5. As can be seen in Tab. [ our method obtains oftc-of-theearr performance for the CUB 776 dataset, for all five different backbones. The method also outperforms GAE [1] with CLIP as a backbone. Our method improves the GAE map, which it uses during training, by more than 25% in the accuracy metric. The results listed in Tab. show that our method also achieves state-of-the-ee performance for the fine-grained localization datasets. Following previous work, Resnet-50 is used for these methods, except for GAE, which is applied with a CLIP backbone. Appendix presents sample results.
For the WSG task, the algorithm is evaluated with respect to the accuracy of the pointing game [8 which is calculated from the output map by finding the maximum- -vaue location for the given query and checking whether this point is located in the region of the object. Another metric report (B-Box accuracy) compares the extracted bounding boxes with the bounding-bos annotations in the same manner as the WSOL task above. For a fair comparison, splits as Akbari et al. (2) For Referlt, Visual Genome and Flickr30K, we treat each query as a single sentence. Tab. summmrisss the results for the Flickr30k. ReferIt and Visual Genome datasets for the WSG task. Evidently, our method is superior to all baselines, whether training takes place over VG or MS- MS-COCO. Tab. [7prseents bounding box accuracy results for the WSG task. Here, too, method outperforms the baseline methods. In appendix B we present sample output maps.
use the same training/valiie validation/ts
For the WWbl. task, we use the same metrics (pointing game and bounding- box accuracy). For each ground-truth pair of bounding box and caption, we select the closest caption in CLIP space from the output predictions and compare the output map to the bounding box using the pointing accuracy metric. In addition, bounding boxes are extracted for the heat-map, as described at Sec. 3.I and compared to the selected ground-truth bounding boxes with the same accuracy metric as above.
T presents the results for the WWLL task. In addition to our method (Alg. [ that utilizes network g to generate a heat-map, given an image and a text (line 8), we also test two alternative heatmap generation methods. The first one is the MG method of Akbari et al. 2 for the WSG task, and the second is the GAE explainability method [12] As can be seen in the results listed in Tab. 7 our proposed method obtains the best results among the three. For comparison, we also provide WSG results using Alg. and the ground truth text for the same three heatmap methods. Interestingly.
8

---

Linds tit
Figure 4: WWbL on few samples of web images.
00:30
Figure 5: WWLL on images from Week in pictures by BBC (April-May 2022).
the g that is trained on the VG dataset shows better performance for WSG, while in WWbL the results mixed. Fig Fig. show results for method from Flickr30K and web images samples respectively. Fig. [5 shows results for our method from BBC Week in pictures from the recent weeks (April-May 2022).
Ablation Study In this section, we compare the accuracy of g with different modes of the loss function and present the experiment used to determine the values of 2,,,,,4- In addition, also

---

compare the performance of two backbone families, Resnet (29 and HarDnet [10]. All experiments were conducted in the WSOL settings for the [ dataset. Tab. p the sensitivity of our network y to each loss term, as evaluated on the test set, using the Resnet50 backhone. The first part of the table compares the performance without ma,, using different combinations of the other loss terms. As can be seen, the best performance is achieved when we use the three terms fores Lbock: Lreg together. Each term improves the performance: Looe improves it by 21%, Loack by 5% and Lveg by 48%. The second part compares the performance with Lymap which improved by more than 6%. As can be seen, each the other terms improves the performance: fore improves by 4%, Lduck by 2%, and Lrey by 5.5%. Also, training only with Lymap decreases the performance by more than 9% compared to performance using all loss terms. The backbone ablation was conducted with two backbone families and is summarised in Tab. 4 For both families, performance improves as the number of parameters is increased.
Appendix C presents the sensitivity of the network to the A coefficient, which is the weight of Trmap and the only coefficient that is not set to in our experiments. The method seems to be largely insensitive to this parameter and it is stable for wide range of values. The ability of g to generalize well under different pre-trained CLIP models is examined in App. G Evidently, the variance in the performance of g is much lower than the variance in the zero-shot classification performance of the CLIP models used to train g.
Discussion and limitations
When biological systems see, the input is seldom limited to single images. Although our WWLL method was developed in the context of stills, it can be naturally extended to perform online localization of objects in video. For this, the detected regions and their matching concepts need to be tracked and must evolve over time, with the ability to correct previous hypotheses. We leave this effort for future research, since benchmarking this ability with the current datasets is challenging.
The common computer vision terminology makes the distinction between object localization and object detection. The former aims to locate the main object of a certain type in the input image, while the latter aims to find all the relevant objects and their individual boundaries. Our WWbL methoo aims to extract every existing object and can, therefore, detect all image objects.
The method employs the selective search method 774] This can be avoided by identifying one region at a time and iterating the process. Such a method is presented in Appendix A It uses y to recover the object map each stage. The results reported in this Appendix
somewhat lower than the method based on a selective search. Since the image datasets and pre-trained models our method employs may be biased towards western media concepts, it is culturally biased. For similar reasons, it may display discriminatory behavior.
Another limitation is that our weakly supervised learning scheme does not distinguish between multiple instances of the same object. While Algorithm can improved to somewhat mitigate this, by separating multiple objects that have the same caption, building such a solution robustly may be challenging without additional supervision. Fpp present visualization for this scenario.
Conclusions
Through the power of large pre-trained transformers and by integrating explainability cues, we build an effective weakly supervised thrase-grounding network g. By combining an off-the-shelf image captioning engine, we are able to identify and localize the objects within an input image. The task we solve an open-world one and the setting generalizes multiple existing tasks. It convincingly demonstrated that our weakly supervised solution is on par with the fully supervised alternatives that exist for the thrase-grounding task.
Acknowledgments
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974).
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928, 2020.
[2] Hassan Akbari, Svebor Karaman, Surabhi Bhargava, Brian Chen, Carl Vondrick, and Shih-Fu Chang. Multi-level multimodal common semantic space for image-phrass grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12476-12486, 2019.
[3] Assaf Arbelle, Sivan Doveh, Amit Alfassy, Joseph Shtok, Guy Lev, Eli Schwartz, Hilde Kuehne, Hila Barak Levi, Prasanna Sattigeri, Rameswar Panda, et al. Detector-free weakly supervised grounding by separation. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1801-1812, 2021.
[4] Sadbhavana Babar and Sukhendu Das. Where to look?: Mining complementary image regions for weakly supervised object localization. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 1010-1019, 2021.
[5] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, (0(7):e013014)) 2015.
[6] Wonho Bac, Junhyug Noh, and Gunhee Kim. Rethinking class activation mapping for weakly supervised object localization. ECCV, 2020.
[7] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1572-1572, 2016.
[8] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus- Robu-R Müller, and Woj- ciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pages 63-71. Springer, 2016.
[9] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P1000S 2008.
[10] Ping Chao, Chao- Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin. Hardnet: A low memory traffic network. In Proceedings of the IEFE//VV international conference on computer vision, pages 3552-3561, 2019.
[11] Hila Chefer, Sagie Benaim, Roni Paiss, and Lior Wolf. Image-based clip-guided essence transfer, 2021.
[12] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 397-406, October 2021.
[13] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Computer Vision and Pattern Recognition (CVPR), 2021.
[14] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 782-791, 2021.
[15] Kan Chen, Rama Kovvuri, and Ram Nevatia. Query-guided regression network with context policy for phrase grounding. In Proceedings of the IEEE International Conference Computer Vision, pages 822, 2017.
[16] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional Istm-enns. Transactions of the association for computational linguistics, 4:357-370, 2016.
[17] Junsuk Choe, Dongyoon Han, Sangdoo Yun, Jung-Woo Ha, Seong Joon Oh, and Hyunjung Shim. Region- dropout with attention prior for weakly supervised object localization. Pattern Recognition, 116: 10794, 2021.

---

[18] Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim. Evaluating weakly supervised object localization methods right. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3133-3142, 2020.
[19] Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised object localization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2219-2228, 2019.
[20] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016.
[21] Katherine Crowson. Vggan-clip, 2021.
[22]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. words: Transformers for image recognition at scale. ICLR, 2021.
An image is worth 16x16
[23] Hao Fang, Saurabh Gupta, Forrest landola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1482, 2015.
[24] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators.
[25] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint arXiv:2102.01645 2021.
[26] Ross B. Girshick. Fast R-CNN. CoRR,
2015.
[27] Michael Grubinger, Paul Clough, Henning Müller, and Thomas Deselaers. The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International workshop ontolmage, volume 2, 2006.
[28] Guangyu Guo, Junwei Han, Fang Wan, and Dingwen Zhang. Strengthen learning tolerance for weakly supervised object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7403-7412, 2021.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[30] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
[31] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[32] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLII, July 2021.
[33] Syed Ashar Javed, Shreyas Saxena, and Vineet Gandhi. Learning unsupervised visual grounding through semantic self-supervision. arXiv preprint arXiv: 065.06506, 2018.
[34] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models. arXiv preprint 10Xiv22110.0848.. 2021.
12

---

[35] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulate detection for end-to-end multi-modal understanding, Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[36] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine- -raiiGd Visual Categorization (FGVC), volume 2. Citeseer, 2011.
[37] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Fisher vectors derived from hybrid mixture models for image annotation. arXiv preprint 17Xiv:1411..3990 2014.
[38] Shu Kong and Deva Ramanan. Opengan: Open-set set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2021.
[39] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[40] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):3 2017.
[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- Languae understanding and generation. arXiv preprint 2201:22201.12086 2022.
[42] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chumyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. arXiv preprint arXiv:2 11..::2211..088.... 2021.
[43] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, volume 8693 of LNCS, pages 740-755, 2014.
[44] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yare Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, volume 9905 of LNCS, pages 21-37, 2016.
[45] Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fuse- dream: Training-free free text-to-image generation with improved clip+gan space optimization. ArXiv, abs/21 12.01173, 2021.
[46] Weizeng Lu, Xi Jia, Weicheng Xie, Linlin Shen, Yicong Zhou, and Jinming Duan. Geometry constrained weakly supervised object localization. ECCV, 2020.
[47] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[48] Jinjie Mai, Meng Yang, and Wenfeng Luo. Erasing integrated learning: A simple yet effective approach for weakly supervised object localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8766-8775, 2020.
[49] Mao, W. Xu, Y. Yang, Wang. and A. L. Yuille. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). CoRR,
2014.
[50] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[51] Meng Meng, Tianzhu Zhang, Qi Tian, Yongdong Zhang, and Feng Wu. Foreground activation maps for weakly supervised object localization. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 3385-3395, 2021.
13

---

[52] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. arXiv preprint arXiv:2112.05221, 2021.
[53] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:211 1..09334,, 2021.
[54] Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic. Is object localization for free?- weakly-supervised learning with convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 685-694, 2015.
[55] Xingjia Pan, Yingguo Gao, Zhiwen Lin, Fan Tang, Weiming Dong, Haolei Yuan, Feiyue Huang, and Changsheng Xu. Unveiling the potential of structure preserving for weakly supervised object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11642-11651, 2021.
[56] Yingwei Pan, Ting Yao, Ychao Li, and Tao Mei. X-linear attention networks for image captioning. In CVPR, 2020.
[57] Or Patashmik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2094, 2021.
[58] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phenn correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference computer vision, pages 2641-2649, 2015.
[59] Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking softmax with cross-entropy: Neural network classifier as mutual information estimator. arXiv preprint arXiv:/911.10663 2019.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 000000 2021.
[61] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.
[62] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[63] Justyna Sarzynska- Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304: 14135, 2021.
[64] Idan Schwartz, Alexander G Schwing, and Tamir Hazan. High-order attention models for visual question answering. NIPS, 2017.
[65] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-68 626,, 2017.
[66] Tal Shaharabany and Lior Wolf. Learning weight map for weakly-supervised localization. arXiv preprint 11Xiv:2111.14133 2021.
[67] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv: 1605:0160..0171. 2016.
[68] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
14

---

[69] Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek- Forcing a network to be meticulous for seuukly-supeviseee object and action localization. 2017 IEEE international conference on computer vision (ICCV), pages
IEEE, 2017.
[70] Satoshi Suzuki et al. Topological structural analysis of digitized binary images by border following. Computer vision, graphics, and image processing, 30(1):32-46, 1985.
[71] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2281 2826, 2016.
[72] Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li, ClipAcaption: Clip for video caption. In MM, 2021.
[73] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[74] J.R.R. Uijlings, K.E.A. van de Sande, T. Gevers, and A. A.M. Smeulders. Selective search for object recognition. International Journal of Computer Vision, 2013.
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[76] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[77] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint ar arXiv:2 10904, 2021.
[78] Jun Wei, Qin Wang, Zhen Li, Sheng Wang, S Kevin Zhou, and Shuguang Cui. Shallow feature matters for weakly supervised object localization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5993-6001, 2021.
[79] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568-1576, 2017.
[80] Pingyu Wu, Wei Zhai, and Yang Cao. Background activation suppression for weakly supervised object localization. arXiv preprini arXiv:21 12..0020 2021.
[81] Fanyi Xiao, Leonid Sigal, and Yong Jae Lee. Weakly-supervised visual grounding of phrases with linguistic structures. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition, pages 5945-59559999 59544, 2017.
[82] Jinheng Xie, Cheng Luo, Xiangping Zhu, Ziqi Jin, Weizeng Lu, and Linlin Shen. Online refinement of low-level feature based activation map for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 132 141, 2021.
[83] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057, 2015.
[84] Haolan Xue, Chang Liu, Fang Wan, Jianbin Jiao, Xiangyang Ji, and Qixiang Ye. Danet: Divergent activation for weakly supervised object localization. Proceedings of the IEEE/VE International Conference on Computer Vision pages 6589-6598 -6998, 2019.
[85] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vasion-language models. arXiv preprint 21X1722109.1179..0 2021.
15

---

[86] Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. ceasiffcation-reconsttuue learning for open-set recognition. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4016-4025, 2019.
[87] Chen-Lin Zhang, Yun-Hao Cao, and Jianxin Wu. Rethinking the route towards weakly super- vised object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 13460-13469 2020.
[88]
Chen-Lin Zhang, Yun-Hao Cao, and Jianxin Wu. Rethinking the route towards weakly super- vised object localization. In CVPR, pages 13460-13469, 2020.
[89] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1884-1102 1102, 2018.
[90] Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas S Huang. Adversarial complementary learning for weakly supervised object localization. In CVPR, pages 1325-1334, 2018.
[91] Xiaolin Zhang, Yunchao Wei, and Yi Yang. Inter-image communication for weakly supervised localization. arXiv preprint N0X.::000800000 2020.
[92] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.
[93] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2921-2929, 2016.
[94] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134 2021.
[95] Peihao Zhu, Rameen Abdal, John C. Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. ArXiv, abs/2110.08399. 2021.
16


Reviews:
[{"review_id": "xZ6v_UdSN_n", "reviewer": "Reviewer_4wQX", "summary": "The paper proposes the task of Weakly-Supervised Open-World Phrase-Grounding, which seeks to first generate captions for local regions and then ground the caption. Meanwhile, the paper proposes a way to train a mask generator given image and text with only weak supervision (based on CLIP). ", "questions": "See cons.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ The paper studies a new task and proposes a sensible baseline to this task.\n+ The paper contributes a way to train a mask generator with only weak supervision with decent performance on benchmark (e.g., Flickr30K), which is \"somewhat\" novel to my knowledge.\n\nCons:\n- The newly proposed task is not particularly novel or challenging. It seems like a direct combination of image-caption (region-based) + localization. I would have liked to see some discussion on the unique challenge this task poses and how solving it might benefit real-world needs or other tasks.\n- The method to the newly proposed task is also a simple combination of previously methods: selective search for proposing regions; BLIP for generating the captions given a region; a localization network.  \n- The paper also proposes a way to train a mask generator given image and text with only weak supervision (based on CLIP). However, the paper does not come with a discussion on the novelty of the proposed method and how it compares to previous weakly-supervised methods for localization. The idea of using a \"classification\" model's heatmap for localization is not particularly new so I would appreciate a detailed discussion.\n- The paper is a bit confusing on its core contribution: it lacks a full discussion on the uniqueness of the newly proposed task if the task if the core contribution; the method of training the mask generator lacks a discussion to prior work so it is also hard to judge the contribution of the method part.\n\nMinor points:\n\nProviding more backgrounds on BLIP would be helpful.\n\nMDETR should be referenced.\n\nGLIP [42] does not use CLIP initialization.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tutiBrOkaO", "reviewer": "Reviewer_aWBt", "summary": "In this paper, the authors explore using pre-trained CLIP for three tasks: (1) object localization, (2) phrase grounding and (3) WWbL: generating object masks and corresponding captions for an image. Their model encodes both image and text with VGG and CLIP respectively, and generates a heat map. They use two weakly supervised losses: (1) foreground loss which encourages the masked image with the heat map to match the text input, (2) background loss which encourages the remaining image to not match the text input, (3) relevancy heatmap which measures the difference between the generated heat map and the relevancy heatmap of CLIP, and (4) regularization loss for the generated heat map. Performing inference on (1) and (2) are trivial, whereas for (3) they designed an algorithm which first unsupervisedly propose region, and then get captions for them to convert the problem to a phrase grounding task. \n\nObject localization results on CUB, and phrase ground results on VG, Flickr and ReferIt show that their model is better than previous methods, and each of the four losses plays a role in improving the results. On their proposed new task, WWbL, the results are also in favor of their proposed model. Combining all of the results, this paper found that CLIP can be useful in various language related object segmentation tasks. \n", "questions": "Please refer to the weakness section above. ", "limitations": "It is nice of the authors to point out the limitation that their work only applies to single images. It would also be nice if the authors can also talk about the potential ethical impact of their paper, e.g. would the bias in the pre-trained models be reinforced or reduced in this method? ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nIt is natural to use a pre-trained vision-language model for object detection/localization. Straightforward and simple solutions like [1], have been proposed and could perform reasonably well in a zero-shot manner. This paper is also a neat method, based on the assumption that objects to localize have higher similarity to the text in CLIP embeddings, compared to background. With the four simple and intuitive losses, the authors show their model performs better than the previous methods.\nThe authors also contribute a new task, WWbL.\n\n[1] https://github.com/shonenkov/CLIP-ODS\n\nWeaknesses:\n\n1. The model may fail for ambiguous captions. For example, given an image of four apples, if the text is “an apple”, the desired output mask should be one of the apples for a deterministic model. Since the foreground loss and background loss maximizes the difference between the image representation along the text embedding dimension, one of the most likely results is that all of the apples are in the foreground (if CLIP “thinks” four apples to no apples is more different than one apple and three in terms of the caption “an apple”). This means that even for a perfect pre-trained model, such loss may still result in undesired output given ambiguous captions. \n\n2. Related to 1, this paper lacks a qualitative study of the effects of coefficient. It is shown in this paper that all of the losses are important for the metrics, but would different configurations have different tendencies? Can configurations be optimized for different applications? This might be out of scope, but if the authors had provided insights on this, this paper would have been much stronger. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_FTsFCbyMI", "reviewer": "Reviewer_J9Sc", "summary": "This paper proposes and studies a new task, which localizes image regions with masks and describes them with natural language. The paper studies the task in a weakly-supervised open-world setting. In order to tackle the task, two recently proposed pre-trained vision-language models, namely, CLIP and BLIP are leveraged to produce multi-modal matching score and generate candidate captions, respectively. Meanwhile, an encoder-decoder network is trained to generate foreground mask for grounding. Superior performances are shown in multiple benchmark datasets, for the three evaluation tasks.", "questions": "1. Discuss the novelty and elaborate the contributions (Concern #1).\n2. Discuss the empirical fairness (Concern #2).", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. A new perspective for vision-language community, and open-world applications. Comprehensive techniques are proposed with extensive supportive experiments.\n2. State-of-the-art quantitative results on multiple benchmark datasets for the three tasks (WSOL, WSG, WWbL).\n\nConcerns:\n\n1. Regarding novelty. The novelty might not reach the standard of NeurIPS conference. The proposed framework largely depends on CLIP-like pre-trained models and lacks original innovation. Moreover, the proposal claims to be \"purely visual\" (without textual input), but actually the textual information comes from pre-trained BLIP model at inference. I expect more discussion regarding the novelty.\n2. Regarding empirical results. I am not quite sure whether it is fair to compare with previous methods, considering the proposed method actually leverages the pre-trained information from CLIP and BLIP. And I am also not sure if the datasets (e.g., Flicker, ReferIt) used for comparison have overlapping (or similar) images with the large-scale datasets involved in the pre-trained models (i.e., CLIP and BLIP). The comparison fairness should be discussed.\n3. The writing quality and presentation should be improved. The writing and expression highly affect deeper understanding of this work. There are quite many expression issues and typos throughout the whole paper. Some of the typos are listed as follows.\n(1) L50, \"A two-stage inference time procedure\" shoud be \"A two-stage inference procedure\". (2) L69, \"during the inference test\" should be \"during the inference\". (3) L197, \"a encoder-decoder architecture\" should be \"an encoder-decoder architecture\". (4) L241, \"An SGD optimizer\" should be \"A SGD optimizer\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SdyZyw8-Xj", "reviewer": "Reviewer_MVej", "summary": "This paper proposes a new solution for weakly-supervised open-world phrase grounding. The method is built upon the off-the-shelf CLIP model and captioning model, only taking images as the input. The experiments are conducted on several datasets, showing the superior performance of the proposed solution. ", "questions": "1. Have the authors tried to change the CLIP model and BLIP to other models to see how the new alternative can perform? \n\n2. Have the authors compare the model with some pre-trained methods like CPT-Colorful Prompt Tuning, Frozen or Flamingo?", "limitations": "How can the proposed solution perform if there are no strong pre-trained models available? ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method is simple yet effective. By combining strong pretrained model and off-the-shelf model, it achieves good performance without text inputs. \n\n2. The empirical results are several benchmarks are promising. \n\nWeaknesses:\n1. The novelty of the paper is quite limited. Most of the components are not developed by the paper. \n\n2. It is not clear how this method can perform well in such a weakly-supervised setting. Its generalization ability may be contributed by the powerful pretrained model (CLIP). The author should performant some ablation study to change these models to other candidates to validate this. \n\n3. The paper ignored some work related to pretrained model for zero/few shot settings (with prompt, etc) [a,b]\na. A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models. ACL 2022.\nb. CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models\nc. SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. ICLR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept