Triple Alignment Strategies for Zero-shot Phrase Grounding under Weak Supervision

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 14:28:27

Abstract: Phrase Grounding, i.e., PG aims to locate objects referred by noun phrases. Recently, PG under weak supervision (i.e., grounding without region-level annotations) and zero-shot PG (i.e., grounding from seen categories to unseen ones) are proposed, respectively. However, for real-world applications these two approaches are limited due to slight annotations and numerable categories during training. In this paper, we propose a framework of zero-shot PG under weak supervision. Specifically, our PG framework is built on triple alignment strategies. Firstly, we propose a region-text alignment (RTA) strategy to build region-level attribute associations via CLIP. Secondly, we propose  a domain alignment (DomA) strategy by minimizing the difference between distributions of seen classes in the training and those of the pre-training. Thirdly, we propose a category alignment (CatA) strategy by considering both category semantics and region-category relations. Extensive experiment results show that our proposed PG framework outperforms previous zero-shot  methods and achieves competitive performance  compared with existing weakly-supervised methods. The code and data will be publicly available at GitHub after double-blind phase.

Main paper: 
 introduction: 
 Phrase Grounding (i.e., PG) [48] aims to locate objects referred by noun phrases. The PG task could be beneficial for various downstream works, such as image captioning [8, 31, 56] , vision navigation [3, 16, 24, 51, 52] , visual question answering [7, 10, 50, 54] , and other multi-modal researches [18, 19, 23, 28-30, 33, 44, 49] .Recently, some works on PG under weak supervision [11] have been proposed. The motivation is to alleviate the large annotation cost of bounding boxes(bbox). PG models under weak supervision are required to learn only from image-phrase pairs but without bounding boxes. To address this challenge, various approaches using visual detectors [9, 14, 32, 34] and constructing auxiliary tasks [1, 2, 12, 17, 41, 42, 55] are proposed. These works achieve significant performance. However, due to limited seen categories during training, they are difficult to apply in zero-shot settings. Very recently, some works propose the task of zero-shot PG. Various approaches using zero-shot learning [5, 40] and Vision-Language Pre-training (VLPs) models [4, 22, 43, 45, 58] are proposed. The former approaches assume that the entities of phrases in the training set have limited categories (i.e., seen categories) and differ from those (i.e., unseen categories) in the testing set. To bridge the gap, these works leverage semantic information shared across all categories, achieving good performance. However, these works still require collecting large-scale bounding box annotations for seen categories. The latter VLP-based methods ground objects on new data without being fine-tuned. However, these methods primarily showcase the generality of VLPs for various vision-language tasks, not focusing on PG. In addition, these VLP-based methods suffer from the collecting of enormous training data.To summarize, existing works have not addressed the challenges of PG in weak supervision and zero-shot settings, simultaneously. In effect, for real-world applications, the grounding models are required to fit images without bounding box annotations and to effectively generalize from limited seen categories to unseen ones. Motivated by this observation, we propose a framework of zero-shot PG under weak supervision.Here, three questions arise naturally for zero-shot PG under weak supervision by our observations. Firstly, how to associate attributes of seen categories with those of unseen categories? Take an example in Figure 1 a ). Suppose that two classes including "white police cat" and "black truck" have been seen before, and the model is required to ground the unseen class "black police car". To perform inference correctly, the semantic information of "black" and "police car" should be transferred to their corresponding visual regions. To this end, the model is required to design a region-text alignment strategy. Secondly, how to transfer knowledge learned from seen categories to unseen categories for the model's prediction? As shown in Figure 1 b ), during training, the model can only observe the data of limited classes, such as people, vehicles, and animals. We expect that the model could learn a domain-invariant representation space even being trained on seen categories. Thus, the model can take advantage of such invariance for grounding unseen categories, such as celebrities, animes, and artworks. To this end, we need to design a domain alignment strategy. Thirdly, how to measure the similarities and differences among seen categories and those of unseen ones? As shown in Figure 1 c ), for one thing, "yellow hound" is related to "yellow puppy", but not to "police car". Therefore, phrases can be used to categorize referred objects. For another, correct visual-textual relations can help in distinguishing categories. For "yellow puppy", we reduce its distance to the corresponding region and increase that to irrelevant regions. Then the model will identify the category and ground the region of a similar phrase such as "yellow hound". To this end, we need to design a category alignment strategy. To emphasize, the weakly supervised setting requires that the training datasets do not provide accurate location annotations in images. This makes it more difficult for PG models to generalize in the zero-shot setting.In this paper, we propose a novel PG framework (Figure 2 ) using triple alignment strategies. Firstly, we design a region-text alignment (RTA) strategy to build region-level attribute associations based on Contrastive Language-Image Pre-Training (CLIP). Specifically, we extract region-level visual semantics and gradient maps using CLIP [37] for given phases. Each region-level visual semantics corresponds to a certain text embedding. Then, we use a patch-level gradient map to refine the CLIP-based heatmap. Here, the CLIPbased heatmap is used as a pseudo-label. Secondly, we propose a domain alignment (DomA) strategy to transfer knowledge learned from seen classes. Specifically, we align the grounding-related features to those of the pre-trained model through learning a domaininvariant space. A domain alignment loss is designed to adaptively adjust the grounding-related features. Thirdly, we design a category alignment (CatA) strategy to distinguish grounding-region categories. Inspired by the class activation method [27] , we discriminate the categories based on the phrase embeddings. To construct accurate visual-textual relationships, we use CLIP to measure the similarity of image regions and phrases. Our strategy considers both category semantics and region-category relations.In summary, our main contributions are three-fold.We propose a novel PG framework with zero-shot under weak supervision. To the best of our knowledge, we are the first to study PG under the two settings, simultaneously.2) We propose triple alignment strategies for PG framework. First, we design RTA strategy to learn region-level attribute associations. Second, we propose DomA strategy to learn domain-invariant representation space. Third, we design CatA strategy to help the network distinguish categories.3) We conduct extensive experiments on benchmark datasets. The results consistently show that our approach significantly outperforms existing zero-shot methods, and achieves competitiveness compared with other weakly-supervised methods. 
related work 2.1 weakly-supervised pg: 
 Under weak supervision, PG models can solely learn from imagephrase pairs during the training. To address the challenge, detectorbased works use visual detectors and choose the correct proposals that are highly related to the corresponding phrases. For example, Datta et al. [9] align captions with caption-conditioned image representations. Lu et al. [34] combine diverse vision-and-language tasks to improve performance. Gupta et al. [14] associate image regions with caption words by maximizing mutual information. Liu et al. [32] design a relation-aware instance refinement module to construct the phrase-object relations. However, the performances of these methods heavily depend on the quality of visual detectors.Moreover, the other methods design auxiliary tasks for PG, such as intra-modal classifications and inter-modal alignments. Javed et al. [17] use an attention mechanism to share nouns in captions among images having a common visual region. Zhang et al. [55] propose contrastive attention for task-specific attention maps. Akbari et al. [1] designs a multi-level common semantic space by mapping visual and textual features for phrase grounding. Arbelle et al. [2] use source separation techniques to ground the referred entities in pixel-level. Shaharabany et al. [41] propose WWbl, which learns to create a phrase-related foreground with the CLIP explainability. Recently, Shaharabany and Wolf [42] employ a layer-wise relevance propagation method to integrate relevancy and gradient information. Gomel et al. [12] design a joint learning approach to train the grounding model and an object detector, simultaneously. Nevertheless, all previous weakly-supervised PG methods struggle on unseen categories due to the limited training data.Zero-shot PG aims to predict the bounding boxes for images of unseen categories but is trained only on those of limited seen categories. Currently, there exist two main approaches, namely zero-shot learning-based and VLP-based methods. In the former methods, Sadhu et al. [40] leverage the supervision of image captions, class names, and bounding boxes, and perform zero-shot predictions via mapping the visual representations and three types of annotations. Chen et al. [5] leverage object priors shared across all categories. The latter VLP-based methods ground phrases on new data without being fine-tuned. Recently, GAE [4] was proposed, advancing VLP-based method for PG. Although this method activates the most discriminative location, the grounding box cannot accurately cover the object. Li et al. [22] use super-pixels technique to generate high-resolution feature maps for phrase grounding. Zhou et al. [58] modify the image encoder of CLIP by transforming the value embedding layer to handle pixel-level predictions. Subramanian et al. [45] use the pre-trained detector to generate a set of proposals and then use CLIP to select the best association between the query and proposals. However, these works either Seen category Unseen category require collecting bounding box annotations for seen categories or enormous image-text pairs.× Tokens Fusion RTA Multi-Head Self-Attention ⊕ MLP ⊕ Q K V ×N a) d) b) c) e) 
methodology 3.1 problem formulation: 
 Given an image 𝐼 and a noun phrase 𝑇 , the task of PG requires the model to predict a bounding box 𝐵. For obtaining the box, a heatmap 𝐻 is also generated as an intermediate bridge. In zeroshot PG under weak supervision, the model grounds both seen and unseen classes during the inference, after only being trained on seen classes without bounding box annotations in images.The overview of our proposed framework is shown in Figure 2 .The framework comprises the CLIP-based module and the grounding module. The CLIP-based module (Figure 2 b ) is built on CLIP, which is pre-trained on 400 million image-phrase pairs [37] . It first extracts region-level image semantics by transferring the textual class token's semantics to visual patch tokens. Then, this module fuses these tokens, generating a CLIP-based heatmap 𝐴 * 𝐶 . The grounding module (Figure 2 c )) consists of an image encoder E 𝑖𝑚𝑔 (•), a text encoder E 𝑡𝑥𝑡 (•), and a grounding decoder D 𝑔𝑛𝑑 (•). Thus, the grounding module returns a heatmap 𝐻 as follows,EQUATIONwhere the image encoder uses the last layer of the pre-trained CNN in ImageNet as the visual embedding. The text encoder uses the text embedding branch of CLIP (VIT-B/32), which is frozen. Multimodal feature fusion calculates the similarity of text features and visual ones, 𝐴 𝑀 = E 𝐼𝑚𝑔 (𝐼 ) ⊗E 𝑇 𝑥𝑡 (𝑇 ). The attention is then given as 𝑅 𝑀 = E 𝑖𝑚𝑔 (𝐼 ) •𝐴 𝑀 , in which the symbol • means Hadamard product. The grounding decoder converts the high-dimensional fusion features into the grounding heatmaps 𝐻 . The decoder consists of two upsampling layers. Moreover, we propose triple alignment strategies, RTA, DomA, and CatA. RTA strategy helps align the grounding heatmaps 𝐻 with CLIP-based heatmaps 𝐴 * 𝐶 . DomA strategy aligns features of seen categories in the training and those in the pre-training. CatA strategy aligns phrases with the corresponding regions. Subsequently, we describe the three strategies in detail.To learn region-level attribute associations, we exploit the working process of VLP. In our strategy, we employ the parameter-fixed CLIP to generate region-level attribute associations. Our network is trained on a limited number of seen classes. Therefore, directly re-training CLIP tends to overfit the seen classes as the model parameters are optimized only for seen classes. Consequently, knowledge learned for entity concepts unseen from the training set might be ignored during re-training. Parameter-fixed CLIP could potentially alleviate this issue.Formally, for the CLIP image encoder, we denote the embeddings from the 𝑙-th transformer layer as {𝑐 𝑙 , 𝑃 𝑙 }, where 𝑐 𝑙 denotes the [CLS] token and 𝑃 𝑙 = {𝑝 1 𝑙 , 𝑝 tokens. Inspired by VIT [38] , the patch tokens of the last layer represent dense features. However, CLIP only focuses on the [CLS] token, and the patch tokens are unable to establish semantic associations with text embeddings [58] . A straightforward way is to transfer the [CLS] token's semantic information to the patch tokens in the last layer, discarding the preceding layers. Thus, each patch token only receives information from its corresponding semantic so that phrase-related visual entities are well grounded. In practice, we initialize the mask token from the pre-trained [CLS] token, i.e., 𝑚 1 = 𝑐 1 and append it to the above token sequence in the first layer of CLIP image encoder. Then the 𝑙-th transformer layer processes the mask tokens as follows,𝑚 𝑙+1 = 𝛼 𝑙 • Attn 𝑚 𝑙 , 𝑃 𝑙 + 𝛽 𝑙 • 𝑚 𝑙 , 𝑙 = {1, • • • , 𝐿 -1} (2)where 𝐴 denotes the attention weight. Hyperparameters 𝛼 and 𝛽 control the weights of the residual term. While the parameter 𝑙 < 𝐿 -1, we set 𝛼 and 𝛽 to avoid establishing strong semantic associations between the mask tokens and patch tokens in the shallow self-attention layers. These patch tokens tend to share holistic information, which is favorable in image-level tasks [47] .While the parameter 𝑙 = 𝐿 -1, we enhance the semantic transfer from previous layer's mask tokens to the last layer's patch tokens. In other words, in shallow layers the two parameters are set relatively small, while in the last layer they are set relatively large. CLIP-based heatmap generation. We obtain the final embedding vectors {𝑚 𝐿 𝑝 }, in which 𝑝 = 1, 2, • • • , 𝑍 for each patch token. We then compute the inner product between the text embedding and the final embedding vectors to fuse texts and image regions. Formally, we obtain the CLIP-based heatmap as follows,𝐴 𝐶 = exp E 𝑡𝑥𝑡 (𝑇 ) ⊗ 𝑚 𝐿 𝑝 𝛿 (3)where 𝛿 is the temperature scaling parameter in CLIP, E 𝑇 𝑥𝑡 denotes text encoder, and 𝐿 represents the last layer of image encoder.CLIP-based heatmap refinement. The gradient map is often used to ground specific objects [21, 45] , where only the gradient of patch tokens of the last layer is computed. In contrast, we focus on patch-to-patch attention of each multi-head self-attention (MHSA). It involves collecting grounding-related features. Thus, we extract the patch-level attentions 𝐴 𝑃 2𝑃 ∈ R 𝑍 ×𝑍 based on patch tokens, without considering the [CLS] token. The gradient map ▽𝐴 is computed as ▽𝐴 = 𝜕𝑄 𝜕𝐴 𝑃 2𝑃 , where 𝑄 is the model's output logit, i.e., the similarity score for the image-text pair. The refinement is formulated as:EQUATIONwhere 𝐴 * 𝐶 is the CLIP-based heatmap refined by the gradient map. Region-level attribute association learning. CLIP can associate attributes and entities. For example, CLIP can infer the image has "red elephant", though it has only seen the attribute "red" or the entity "elephant" during pretraining. Our CLIP-based heatmap 𝐴 𝐶 * inherits CLIP's property of attribute association. This is demonstrated in the image region (See Figure 3 ). To enable the grounding module to learn the similar property, smooth ℓ 1 loss [39] forces the grounding module to simulate the region-text alignment process applied to the CLIP-based heatmap. The ℓ 1 loss is given asEQUATION)in which, 𝑥 is the difference between the grounding heapmatp 𝐻 and the CLIP-based heatmap 𝐴 * 𝐶 .To transfer knowledge from CLIP, a direct method is to replace the image encoder in our grounding module with that of CLIP. Then we train the modified model on the image-text datasets containing phrases from a limited number of classes. However, such a choice might cause severe overfitting. This motivates us to seek a special kind of feature in the grounding module. Thus, we design DomA strategy: 1) Reconstruct groundingrelated feature in CLIP and our network. 2) Minimize the difference in feature distributions between the training and pretraining phases. Formally, our grounding module computes the matching attention between a phrase and an image byEQUATIONwhere 𝑘 and 𝑖 × 𝑗 donate the number of channels and spatial dimensions, respectively. The domain heatmap 𝐴 𝑀 , compared with the grounding heatmap 𝐻 , can avoid overfitting. We consider the domain heatmap 𝐴 𝑀 and CLIP-based heatmap 𝐴 * 𝐶 as features, respectively. Note that the weakly-supervised training does not change the feature distribution of the frozen pre-trained model. Domain Alignment Loss. To narrow the difference in feature distributions between the training and pre-training phases, We use contrastive learning [15] . We formulate an alignment process to learn to select the positive samples from a set of positive and negative matching attention maps. Specifically, we use the CLIP-based attention 𝐴 * 𝐶 as the criterion for determining positive and negative samples. We treat the matching attention from the same input as positive samples, while the matching attention from different inputs as negative samples. Given a pair of matching attention 𝐴 𝑀 and CLIP-based heatmap 𝐴 * 𝐶 , the alignment loss is given as               ℓ 𝑐𝑜𝑛 𝜉 (𝐴 𝑀 , 𝐴 * 𝜉 𝐶 ) = -log 𝛿 (1 -(𝐴 𝑀 • 𝐴 * 𝜉 𝐶 )), 𝜉 ∈ B, -log 𝛿 (𝐴 𝑀 • 𝐴 * 𝜉 𝐶 ), 𝜉 ∈ W, ℓ 𝑐𝑜𝑛 (𝐴 𝑀 , 𝐴 * 𝐶 ) = 1 |B ∪ W| ∑︁ 𝜉 ∈ B∪W ℓ 𝑐𝑜𝑛 𝜉 (7)where 𝐴 * 𝐶 is a given in Eq. ( 4). B and W denote sets of positive and negative samples, respectively. B ∪ W is the union of two sets, and 𝛿 is the sigmoid function. Thus, our domain alignment is relevant to grounding-related domain generalization.We treat the PG as a problem of phrase-region alignment. Inspired by the class activation method [27, 53] , we use the phrase embedding E 𝑡𝑥𝑡 (𝑇 ) to discriminate the category of grounding-related feature, and normalize it as a category label 𝑦. To construct regioncategory relations, we calculate CLIP matching scores based on phrase embeddings and grounding-region embeddings. To ensure each training class can be seen by pre-trained CLIP, we crop out the grounding regions along the bounding box.Object and Phrase matching loss. By exploiting the information from the phrase-related object by CLIP, the region of the object 𝐵(𝐻 ) is grounded by cropping the bounding box, reshaped as output 𝐻 size, and mapped to the image representation by CLIP image encoder, i.e., 𝑣 𝑂 = E 𝑖𝑚𝑔 (𝐵(𝐻 )). The cosine similarity between the object representation 𝑣 𝑂 and the phrase representation E 𝑡𝑥𝑡 (𝑇 ) is used for a loss,EQUATIONin which 𝑠 𝑛 is the cosine similarity. 𝑁 is the total number of generated heatmaps. Thus, the heatmap 𝐻 is gradually close to the phrase-related object under the supervision of ℓ 𝑂𝑃 . Random region and Phrase matching loss. To enlarge the distance between phrase-irrelevant regions and phrase representations, the region of non-object is grounded by cropping a randomly generated 𝐻 -size of the box 𝐵 𝑅 . The generated box should be less intersected with the bounding box 𝐵(𝐻 ). The box is represented by the CLIP image encoder, i.e., 𝑣 𝑅 = E 𝑖𝑚𝑔 (𝐵 𝑅 ). The cosine similarity between the box visual representation 𝑣 𝑅 and the phrase representation E 𝑡𝑥𝑡 (𝑇 ) is used for a loss,EQUATIONin which 𝑠 * 𝑛 denotes the cosine similarity. Thus, the heatmap 𝐻 retains less of the phrase-irrelevant region of the object.Regularization loss. To further exclude irrelevant background in the heatmap, we constrain the size of the region covered by the heatmap, i.e.,EQUATIONThus, the total loss of our model is given as follows,EQUATIONBased on the grounding module, we generate the bounding box as follows. First, we set zeroes for the low-value pixels with a threshold of 0.5. Then, we search for contours and extract suitable bounding boxes following [46] . We then calculate the scores of bounding boxes based on the area percentage of the heatmap 𝐻 . Finally, non-maximal suppression is applied with 𝐼𝑜𝑈 = 0.3, and the boxes 𝐵 with 50% less than the maximum score are filtered to complete the localization. The visualization of bounding box generation is shown in Figure 4 . 
experiment 4.1 datasets: 
 We evaluate our framework on zero-shot PG settings using Flickr-Split-S0, Flickr-Split-S1, VG-Split-S2, and VG-Split-S3 [40] . In addition, to compare with previous weakly-supervised grounding methods, we use the setting in MG [1] , which is adopted in various works using either MS-COCO [26] or VG [20] training splits, respectively. In both cases, the resulting models are evaluated on the testing splits of Flickr30K [35] , VG, and ReferIt [6, 13] .We compare our framework with various state-of-the-art PG baselines, which can be divided into the following three categories. 1) Supervised zero-shot baseline, i.e., ZSGNet [40] . 2) Zero-shot baselines, including detector + CLIP [45] , GAE [4] , Grad-CAM [45] , AdaptingCLIP [22] , and MaskCLIP [58] . 3) Weakly-supervised baselines, including MG [1] , GbS [2] , WWbl [41] , SMST [42] , BBR [12] , and VPT [25] . Three metrics, including "pointing game" accuracy [55] , bounding box accuracy [41] , and recognition accuracy are used for our evaluation. The "pointing game" accuracy measures the percentage of predicted maximum points of the heatmap that lie within the bounding box ground truth. The bounding box accuracy measures the percentage of heatmap bounding boxes that have an IoU greater than 0.5 for the testing set of "image-query" pairs. In addition, the recognition accuracy counts the rate of correct results over all test sets. The accuracy relies on human evaluation. We asked volunteers to judge whether the grounded region is cognitively appropriate.For a fair comparison, we use VGG-16 as the visual encoder in our framework. The model accepts an image size of 224 × 224 which is the input size of CLIP's visual branch VIT-B/32. It generates a heatmap 𝐻 of the same size. We trained 150 epochs using SGD optimizer (a batch size of 64 and an initial learning rate of 0.0003), where the optimizer momentum is 0.9 and the weight decay is 0.0001. In addition, the layer 𝐿 is set as 11. The parameters 𝛼 and 𝛽, they are set as 0.01 and 1 when 𝑙 < 11. They are set as 1 and 10 when 𝑙 = 𝐿. All methods were implemented on an NVIDIA RTX A6000. In all our experiments, the weights of our loss in Eq. ( 11) were set as follows, 𝜆 1 = 0.25, 𝜆 2 = 0.125, 𝜆 3 = 0.25, and 𝜆 4 = 1.Visual Encoder Flickr-Split-0 Flickr-Split- Zero-shot Evaluation on Unseen Phrase Classes. We report the zero-shot evaluation results on the test split of Flickr-Split and VG-Split in Table 1 . Unlike Flickr-Split, VG-Split's phrases contain a large amount of textual noise that does not describe the corresponding objects. Therefore, we set two IoU thresholds (0.3 and 0.5) in the evaluation. Specifically, our approach achieves superior results on IoU thresholds of 0.5 shown in column #6, #8, #10, and #12). This indicates that the model's grounding results cover unseen categories better than other methods. Although there still exists a gap compared to the supervised method, our method improves the performance significantly compared to the weakly-supervised methods. WWbl model masks the image with the heatmaps, followed by using external knowledge to measure the similarity between masked images and phrases as the loss function. However, the external discriminator rarely encounters mask-covered images, leading to an incorrect accumulation of category judgment. Compared to other datasets, Flickr-Split-1 is more stringent in defining the difference between training phrase categories and testing ones.However, our results are close to the supervised grounding SoTA on Flickr-Split-1. This result shows a strong generalization of our model on unseen phrase classes.Weakly Supervised Evaluation on Seen Classes. We report the performances of our PG framework compared with other weakly-supervised PG methods on Flickr30K, VG, and ReferIt in Table 2 . For a fair comparison, all trainable methods use VGG-16 as the image encoder to produce the final grounding output. The experimental results show that our method generates competitive results compared with other weakly-supervised PG methods. Our CLIP-based heatmap (CH) also surpasses the pseudo label (GAE) used by WWbl in terms of bbox accuracy and such information is not available in the training dataset. This explains the 9% increase in bbox accuracy over WWbl. Our method does not perform well on some metrics compared to BBR, and SMST. The main reason is that these methods provide more accurate supervision labels. SMST designs visual similarity maps, while BBR uses bounding boxes provided by the object detector as additional annotations. We did not consider using their labels because the zero-shot generalization of weakly-supervised PG is more important than the slight improvement in grounding accuracy. Although visual similarity maps and box annotations can improve the appearance of heatmaps, they are both text-agnostic. Simply using such annotations can disrupt the text-image alignment, which is crucial for zero-shot manner of weakly-supervised PG.Zero-shot Evaluation on Unseen Object Classes. We compare our framework with the supervised methods on unseen imageobject classes. The testing set was collected which contained 30 instances for each category from Google. These categories belong to novel concepts or fine-grained categories, which do not appear in training datasets. This dataset includes 10 categories: CElebrity names (CE), ANime names (AN), GAme character names (GA), ARtwork names (AR), RAre plant and animal phrases (RA), SMall object phrases (SM), EXclusive category phrases (EX), SEntence-level phrases (SE), REmote-sensing-related phrases (RE) and MEdicalrelated phrases (ME). Note that the supervised method uses box annotations during training, whereas our method does not require any form of additional annotations. Figure 5 shows the proportion of grounding results evaluated by human evaluation. The subjective results clearly show that our method outperforms the supervised method with large margins in novel concepts. Some fine-grained categories, including SM, EX, and SE with close semantics have been encountered during training. It is extremely advantageous to the supervised method because the box annotations build stronger associations among similar semantic entities.Effects of alignment Strategies. Firstly, Table 3 shows the effects of alignment strategies in weakly supervised settings. We use Flickr30K Entities as it contains clearer expressions without noise. Among all combinations of VLP and training stages, using all alignment strategies leads to the best performance. Secondly, we measure the impact of each loss on grounding performance. Table 4 shows quantitative comparisons among different combinations of loss functions. Our framework only obtains 16.85% when only ℓ 1 is used. An addition of ℓ 𝑐𝑜𝑛 improves IoU from 16.85% to 29.41% and the inclusion of ℓ 𝑂𝑃 and ℓ 𝑅𝑃 achieves absolute improvements 8.84% and 4.26%, respectively. ℓ 𝑅𝐸 can ensure the compactness of CLIP-based heatmaps and improve the IoU by 2.95% on the dataset. Thirdly, we compare the baseline (we replaced the image encoder of grounding module with CLIP (RN50)) with our domain alignment strategy. In our experiments, we evaluate seen classes on Flickr30K Entities, and unseen classes on Flickr S0 and Filckr S1. As shown in Figure 6 , our baseline has a large gap in bounding box accuracy on seen and unseen classes. Compared to the former, our domain alignment strategy achieves better domain adaptation between seen and unseen classes. Finally, we ablate each alignment strategy in zero-shot settings. In addition, Table 5 shows the quantitative comparisons among different combinations of alignment strategies. The results show that each strategy works in our PG framework.Effects of CLIP-based Heatmap. To show the impact of the pseudo-label effect on our grounding module, we show the performance of our CLIP-based heatmap in Table 2 and Table 6 , and evaluate the performance of our framework with different training labels (rows #5 and #6 in Table 6 ). Our CLIP-based heatmap has demonstrated competitive performance in seen classes. In addition, when our CLIP-based heatmap (CH) acts as a label, our network improves the grounding quality in most seen classes.We show several results of our alignment strategies in Figure 7 . We use the instances' ground-truth bounding boxes as proposals in Flickr30K. When using only RTA, the predicted CLIP-based heatmap tends to focus on the most discriminative region of the referred object. However, when equipped with DomA strategy, the predicted CLIP-based heatmap tends to capture the context of the phrase but may focus on different object categories. With triple alignments, our method successfully grounds the referred object.Furthermore, we compare our method with VLP-based and weaklysupervised PG methods, as shown in Figure 8 . We observe that our method typically grounds more complete object contents and less phrase-related background regions. Specifically, VLP-based methods and weakly-supervised PG methods may underestimate the region of blue jeans and the woman, or falsely ground the region of mountain bike and the subway station. In contrast, the regions grounded by our framework are more complete and compact.Finally, we show failure cases of our framework in Figure 9 . We categorize failure cases into two groups: similar dense objects and in-context entities-related objects. Our framework highlights connected regions rather than separate regions while it locates dense objects. The number of bounding boxes cannot be precisely determined. Furthermore, our framework extracts only noun phrases  
conclusion and future work: 
 In this paper, we propose a PG framework, which designs alignment strategies to address three key problems of zero-shot grounding with weak supervision. Our approach outperforms previous zeroshot methods and achieves competitive results on weakly supervised benchmarks.In the future, there are several interesting directions. For instance, we will consider interpretable solutions for several groundingrelated tasks, such as Grounded VQA and image captioning. In addition, we will consider how to use multi-modal large language models [36] to improve zero-shot PG under weak supervision. 
 
 Appendix:  


Decision: The AC goes through the paper, rebuttal and review comments. This paper got 3 borderline accept reviews. All reviewers acknowledge the motivation and novelty. Therefore, the AC thinks this paper can be accepted, but the authors should carefully improve the paper by leveraging the feedback.

Reviews: review: paper_summary: paper_summary: This paper proposes propose a framework of zero-shot phrase grounding under weak supervision, which is built on triple alignment strategies, including region-text alignment strategy, domain alignment strategy and category alignment strategy. Experimental results on zero-shot and weakly-supervised phrase grounding show the effectiveness of proposed method.
strength_weakness: strength_weakness: 1-It makes a research on phrase grounding task in weak supervision and zero-shot settings at same time, and proposes a zero-shot phrase grounding framework.

2-The proposed method designs region-text alignment strategy based on CLIP, and tries to minimize the difference between distributions of seen classes by narrow the difference in feature distributions between the training and pre-training phases.

3-It considers both category semantics and region-category relations at same time, which regards PG as a problem of phrase-region alignment.
limitations: limitations: 1-In line 101, it mentions that VLP-based methods ground objects on new data without being fine-tuned. But in lin 105, the paper says that these VLP-based methods suffer from the collecting of enormous training data. When using pretrained VLP model, why it suffers from the enormous training data.

2-It seems like Zero-shot Referring Expression Comprehension, whether the proposed method is also effective? 

3-In Table 2, compared with SMST and BBR, the proposed method seems limited. Please analyse the main reason. Whether can provide only in weak supervision setting or zero-shot setting.

4-In 3.3, the CLIP image encoder has been used (Fig-2-b), why in line 569 VGG-16 is used as the visual encoder (Fig-2-c). Whether exists multiple visual encoder? If the CLIP has been used in 3.3, why do not also use CLIP to replace VGG-16.

5-In fact, as shown in Fig-2-c, maybe the multimodal feature fusion is more effective if using CLIP visual encoder and CLIP text encoder. Please provide the related results in above settings, and analyse the reason.

6-For the BBox Generation & Crop (Fig-2-d), the feature map focuses on the motorcycle, thus the image is also corresponding it, but not both motorcycle and man, please refine this part for clearly understanding.
paper_summary: paper_summary: This paper presents a novel framework for Phrase Grounding (PG) that operates under weak supervision and zero-shot settings. The framework leverages three alignment strategies: region-text alignment (RTA), domain alignment (DomA), and category alignment (CatA). The proposed approach aims to bridge the gap between seen and unseen categories during training by aligning visual and textual information at different levels, ensuring robust performance in real-world applications without requiring extensive bounding box annotations.
strength_weakness: strength_weakness: 1. The exploration of zero-shot phrase grounding with weak supervision is meaningful.
2. The proposed DomA strategy can effectively avoid overfitting.
limitations: limitations: 1. Lack of comparison with other weakly-supervised models in the main table. As mentioned in 4.2, the author states that they compare their methods with several weakly-supervised methods. However, in Table 1, only WWbl is involved in the comparison. In Table 2, the performance of the proposed method is not satisfactory. Therefore, more demonstration is needed to justify the performance.
2. Comparisons with ZSGNet in Figure 5 are not persuasive enough. In this work, the author adopts CLIP to guide the model. CLIP is much more powerful in zero-shot prediction. Therefore, there is a need to have experiments compared to the VLP-based model to demonstrate the ability of the model on unseen objects.
3. The motivation of DomA is not clear enough and its effectiveness is not well demonstrated. In the introduction, the author states that DomA aims to transfer knowledge from seen classes. Meanwhile, in the method, they state that DomA is used to narrow the feature distribution of training and pre-training. It is a little bit confusing. On the other hand, the effectiveness of the DomA is not well demonstrated. It would be interesting to see the model accuracy on seen and unseen classes with and without DomA.
4. In Table 3, it is strange that the three independent strategies have poor results, but together they do perform well. The experiment results need more explanation.
paper_summary: paper_summary: This paper studies Phrase Grounding (PG) under both settings of weak supervision and zero-shot. It alleviates three issues: attribute association, knowledge transfer, and category sim & diff measurement, by triple alignment strategies, that is, region-text alignment (RTA), domain alignment (DomA), and category alignment (CatA). Experiments are carried out on FlickrSplit-S0, Flickr-Split-S1, VG-Split-S2, and VG-Split-S3 to show effectiveness.
strength_weakness: strength_weakness: [+] Zero-shot phrase grounding under weak supervision is more practical in reality, thus embarking on the right path.

[+] The paper is easy to follow and understand. 

[+] Many experiments are conducted to prove the effectiveness of core components, including loss functions, heatmap fusion, several strategy ablations.
limitations: limitations: [-] Sub-optimal designs. This paper uses heatmaps from frozen CLIP as guidelines without training, since limited training data makes it easy to overfit. However, fine-tuning CLIP using small amount of data has been widely-used recently, such as adapter, LoRA, and prompt tuning. Comparing to these fine-tuning operations, is triple alignment strategies (this paper) better? Please make comprehensive discussions and experiments.

[-] Sub-optimal performance. In Tab.1-2, this paper obtains moderate results, sometimes it is actually worse than VLP-based methods that does not require fine-tune CLIP. Also, in pursuit of fair comparison, please quantify training parameters, training/testing time cots in these tables.

[-] Special cases. In introduction section (Fig.1), this paper emphasizes attribute association and knowledge transfer, the corresponding visualization or statistical analysis should be reported in experimental results.